Attachment 'Minutes.txt'

Download

   1 		Cloud sprint in Seattle, 2nd to 4th November 2016
   2 
   3 Hosted by Zach at the Google offices - thanks!
   4 
   5 Present:
   6 ========
   7 
   8 (on-site)
   9  * James Bromberger (JEB)
  10  * Emmanuel Kasper (marcello^/manu)
  11  * Steve McIntyre (Sledge/93sam) 
  12  * Martin Zobel-Helas (zobel)
  13  * Bastian Blank (waldi)
  14  * Sam Hartman (hartmans)
  15  * Jimmy Kaplowitz (Hydroxide/jimmy)
  16  * Marcin Kulisz (kuLa)
  17  * Thomas Lange (Mrfai)
  18  * Manoj Srivastava (manoj/srivasta@{debian.org,google.com,ieee.org}) Affiliation: Debian/Google
  19  * Zach Marano - Google (zmarano)
  20  * David Duncan - Amazon (davdunc)
  21  * Tomasz Rybak (serpent)
  22  * Noah Meyerhans (noahm)
  23  * Stephen Zarkos - Microsoft (???)
  24 
  25 (irc/hangout)
  26  * liw
  27  * hug
  28  * damjan
  29 
  30 Agenda
  31 ======
  32 
  33 (Wed)
  34 == What does it mean to run in a cloud environment.
  35 === Priority of our users vs. technical priorities
  36 == In depth look at how Debian runs in major clouds (AWS, Azure, GCE, Oracle, on premise, ... etc)
  37 == Define an official Debian cloud image.
  38 === legal and trademark issues
  39 === test suite for images
  40 === official "Debian-based" images (for container platforms and other variations)
  41 === Versioning 'Cloud Images' and critical holes patching policy (example: dirty CoW, heartblead, etc)
  42 === SDKs and access for platforms (including release cycle mismatch vs cloud)
  43 
  44 (Thu)
  45 == Decide, if we want WGs for belows discussion items?
  46 == Look into different build processes of different image types (maybe short presentations)
  47 == Introspect the various image build tools and whittle the list down.
  48 == cloud-init maintenance
  49 == test suite
  50 == rebuilding and customisations
  51 == Current and future architectures
  52 == (Human) Language support
  53 
  54 (Fri)
  55 == supporting services (for platforms, mirrors, finding things)
  56 == Ideally, come to consensus on many of the open ended issues that have been talked about at DebConf and on the debian-cloud list. 
  57 == Better handling/publishing/advertising of cloud images by Debian
  58 === Getting updated packages into Debian - stable updates, -updates, backports
  59 === Website changes (better promote Debian Cloud images)
  60 == AOB
  61 == Going to the computer museum
  62 
  63 Meeting 2016-11-02
  64 ==================
  65 1. Everybody introduces themselves
  66 2. Go through planned agenda; it doubled so we prioritised
  67 
  68 Need for customizations of images.
  69 No vendor lock-in
  70 
  71 What does it mean to run in a cloud environment
  72 -----------------------------------------------
  73 Cloud - disposable computers. Many types of usage: long-term or really short (mere hours).
  74 Also - desktop in the cloud.
  75 We (Debian) have many architectures, many languages, many packages - can be useful.
  76 Instance has to be fast to boot.
  77 a) Official image -> customise it -> save as new image
  78 b) generate image from scratch (tooling)
  79 c) launch and customize image - without saving it for later (cloud init, etc.)
  80 
  81 For the cloud - all the users will be derivatives?
  82 
  83 Minimal image - for start
  84 Full-featured image for some users
  85 Look on Ubuntu and other Vendors, Ubuntu's image finder
  86 Advanced users - can help themselves, but need good starting point. Need solid base, not be be discouraged.
  87 Don't care about Rackspace, little information to go on here
  88 Openstack has multiple virtualization backends (LXC, KVM) and vari	ous ways of presenting images and network to the users
  89 
  90 Boot time - Xen tries to initialize non-existing devices; 30s latency in boot time
  91 Non-existing framebuffer. Blacklist driver?
  92 Enhanced monitoring of cloud stuff using agents from the platform providers. For
  93 instance google cloud agent is based on a fork of the collectd agent.
  94 Packaging theses agents is easy, but the problem is to get them to stable, as
  95 the cloud provider API they call change quickly. These agents are used for 
  96 instance, setting the initial root password. 
  97 
  98 Google won't provide a default username/password for base images, so for login there needs to be an agent installed. It will deal with things like ssh keys, also routes for multiple NICs etc.
  99 Agents are not required per se for running images on any of the common platforms, but they provide additional services. We need easy way to allow for users to opt-in to them. Otherwise they will use some other distribution/image.
 100 Azure has also its own agent for the same purposes, packaged in Debian. Stretch
 101 for Azure will maybe use cloud-init.
 102 
 103 Now common for people to want docker (etc.) with their cloud images too, often to be able to provide their workloads that way: Docker, Mesos, Kubernetes, ECS, ???
 104 
 105 Cloud-init in unstable - bugs
 106 
 107 Various services:
 108 a) monitoring
 109 b) hardware (mostly networking) management
 110 c) security related (SSH key injection, etc.)
 111 d) deployment (e.g. AWS DevOps)
 112 e) other (e.g. EC2 Run Command)
 113 
 114 Put fast-changing packages. Testing/unstable - too risky (breaking stuff, transitions). Backports? Volatile archive?
 115 
 116 Cloud-init. Is it required? GCE can use it but don't want to - it's very slow.
 117 Users don't know about it, it's not a great user experience.
 118 Preferred to have software installed in the image already, then just run apt update afterwards if needed
 119 for updates.
 120 
 121 We want a long-running daemon for GCE and Azure at least, to integrate with the full platform feature set.
 122 cloud-init only works at boot. Special daemon (agent) works all the time - allowing for creating new users, SSH keys, etc. So on GCE/Azure it is possible to change keys, create users, etc. all the time. AWS - only during boot.
 123 
 124 Common core + extensions
 125 Disk space vs. boot time and performance. But disk space for images, or for running instances?
 126 
 127 Secure Boot, UEFI are examples of places where different platforms will show differences
 128 
 129 Vagrant - VM image typically used for a reproducible environment for various purposes.
 130 Vagrant for cloud providers; good to move from developer machine to the cloud
 131 
 132 
 133 In depth look at how Debian runs in major clouds (AWS, Azure, GCE, Oracle, on premise, ... etc)
 134 ------------------------------------------------
 135 
 136 Demo of Debian on cloud providers; proposed on IRC
 137 
 138 AWS (James B.)
 139 ++++++++++++++
 140 Images are built on EC2 instances. You need to create snapshot of the volume - so it's easy to work from EC2 instace. There is volume import API - but I'm not sure that it's usable. James used it only once.
 141 All regions; China (behind Firewall). Gov Cloud (you need to be ITAR Certified for access)
 142 Packages mirroring: CDN (Cloud Front). Now with IPv6. Headers for expiring - depending on requested resources
 143 HTTPS: apt-transport-https package to be useful
 144 Statistics: about 1TB/day
 145 
 146 Image. A bit of mess with billing code (?) - no possibility of removing it, even though it is not useful for community images. Restrictions of usage - snapshot, clone volume, start. Can users do it?
 147 
 148 Rebuilding (of stable) Point releases, security issues, boot time improvements
 149 Building of testing: should be regularly
 150 Usage: 22k accounts subscribers to Debian images
 151 
 152 AMI upload to market place. Spreadsheet with detailed informations. It should be possible to automatise it.
 153 
 154 Ubuntu is shown in preferred AMIs, for Debian we need to look for it in Market place
 155 
 156 Instance creation: no encryption by default. Related to billing code?
 157 
 158 No direct support for key-refresh over time
 159 
 160 Vagrant (Emmanuel)
 161 ++++++++++++++++++
 162 Wheezy and Jessie images
 163 vagrant repository or custom one
 164 
 165 Virtual Box, synchronization of directories. By default on boot, ability to have during lifetime
 166 
 167 Packer to build image.
 168 Latest ISO, with checksum.
 169 JSON with description
 170 Makefile to call packer with appropriate parameters and metadata
 171 
 172 Uses Debian Installer in the background
 173 
 174 Test suite
 175 vagrant up
 176 install package
 177 
 178 No need for root during this process. But we need kvm or similar kernel module
 179 
 180 
 181 Azure (zobel, waldi and Steve Z)
 182 ++++++++++++++++++++++++++++++++
 183 Automated building in Jenkins
 184 Jessie and Wheezy daily, uploads to publishing account. They are then replicated to public Azure network (20something regions?)
 185 Modification of open-stack script. Bash-based. Problems with customization
 186 Mirror network in Azure. Updatetraffic completely in Azure network .2.5GB/s capacity
 187 Image 30GB
 188 Anybody can upload, only chosen ones can publish
 189 
 190 Manual release process.
 191 Daily images - not in marketplace, but through API. But users can use them.
 192 Removed after 14 days
 193 Manual publication to marketplace. And manual removal of those.
 194 Ordinary users should use published ones. Developers can use daily
 195 
 196 Classic UI. No good discoverabilty for Debian. It's in maintenance mode.
 197 New: Azure portal. Search for Debian
 198 Resource providers. Azure Resource Manager. Templating.
 199 No SSH key management in portal.
 200 Resource groups: ability to better clean resources
 201 Boot diagnostics and Guest OS diagnostics. Need for daemon for this
 202 Tools (CLI) in many languages: node.js, Python, Go
 203 Offer, skew, version (keyword: latest): URN for command line to identify image
 204 
 205 Agents: on github
 206 Account Azure, WALinuxAgent. SSH keys, diagnostics. Can live side by size with cloud-init.
 207 Chosed during image creation
 208 
 209 cloud-init: need the latest version, with many issues fixed
 210 systemd - need quite recent cloud-init
 211 Debian 7 - kernel from backports
 212 Drivers
 213 
 214 Testing: testing suite. 
 215 Images compatible with Azure Stack - Azure cloud on premise
 216 
 217 JSON to publish on marketplace
 218 
 219 
 220 GCE (Zack Marano)
 221 +++++++++++++++++
 222 uses boostrap-vz, code and manifest are in the upstream git repo
 223 3 daemons are running in the final created image, for clock synchro, credentials
 224 and ip forwarding(?)
 225 
 226 All tools are on github.
 227 No official packages - to many problems with various distributions. Custom tool in Ruby to build package.
 228 
 229 minimal version manifest, not currently used; just as basis if somebody needs something to start
 230 Usually run on VM in GCE; script using bootstrap-vz, starting VM, preparing everything
 231 
 232 
 233 Debian is default on GCE!
 234 
 235 GCE SDK is baked in on GCE images.
 236 
 237 Monthly publishing of images
 238 For security reasons it can be more frequently.
 239 One Debian image. Just last stable. Depreciating oldstable, LTS does not cover the needs.
 240 
 241 Testing of images. Including performance tests.
 242 Tests not yet open-sources (integrated with internal Google tools), but should be in some time.
 243 
 244 Image families.
 245 Global images - not per regions
 246 
 247 Ubuntu build their own images
 248 
 249 Propose to release managers to put cloud-init from backports to stable.
 250 It was proposed by zigo, but maybe we need more to convince them.
 251 For now cloud was not seen as important; maybe when we show them numbers of users, release team will be more eager to let neccesary packages uploaded
 252 
 253 
 254 Do we want to build images in VMs?
 255 Complex layouts, like LVM - might need that.
 256 Debugfs.
 257 
 258 a bit of systemd discussion
 259 
 260 GCE is happy with a single image, supporting only stable.
 261 
 262 ScaleWay
 263 ++++++++
 264 ARM64 cloud applience. ScaleWay
 265 
 266 Docker
 267 ++++++
 268 Example script showing how to create image. Deboostrap-based.
 269 Docker file - based around application you want to host. Docker images are usually used to host single applications. Minimal.
 270 Image namded "Debian" in DockerHub. We don't exactly know who is the publisher of this image. Supposedly some DD, but not member of debian-cloud team, and image not maintened for some time. ( possibly Tianon Gravi see http://joeyh.name/blog/entry/docker_run_debian/ )
 271 Can anyone publish something with name of "Debian"? Move to legal/trademark issues
 272 
 273 
 274 Official Debian cloud images
 275 ============================
 276 Discussion on mailing list started in November 2015.
 277 https://lists.debian.org/debian-cloud/2015/11/msg00005.html
 278 
 279 More/later info in:
 280 https://lists.debian.org/debian-cloud/2016/03/msg00042.html
 281 
 282 We're bit behind proposed schedule ;-)
 283 
 284 Non-controversial proposals
 285 +++++++++++++++++++++++++++
 286 Archive: main
 287 stable-updates
 288 Not necessarily stable-backports (maybe other images, not-so-official)?
 289 No extra archives
 290 
 291 It might be hard to publish non-backports.
 292 It'll be official (hopefully), but will need to be documented as with backports.
 293 
 294 Official: by DD, on Debian infrastructure. Scripts to build are in Debian.
 295 Published on Debian infrastructure, and also uploaded to appropriate cloud providers.
 296 Of course also publish checksums.
 297 
 298 Testing, with public test logs.
 299 Test suite should (must?) be also public.
 300 Possibly not test-suite for Strech, but requirement for Buster.
 301 
 302 Azure: root build on Debian hardware. Then upload to Azure infrastructure, and perform some last steps.
 303 For secure boot, do not store keys in cloud infrastructure.
 304 Keys never leave FTP master (HSM attached to FTP master).
 305 How to cooperate with cloud providers to allow for injecting custom images, and then sign them?
 306 How far will cloud providers want to go with signing.
 307 Sign grub, sign kernel, sign entire image?
 308 
 309 Signing should not be required before Stretch release.
 310 
 311 Signing of Debian by Microsoft?
 312 
 313 CD images are not yet signed using HSM.
 314 
 315 New architectures (for cloud) but not yet as stable?
 316 
 317 Stable - of course. But also monthly (weekly?) testing images.
 318 
 319 Tool chain for official images.
 320 One tool set to build on all clouds.
 321 Might be to late for Stretch.
 322 Test suite to catch discrepancies between different images/tools.
 323 Stable toolset - to be able to reproduce our image. Maybe use snapshots for older version of packages.
 324 Almost like reproducible builds - i.e. the same checksum. We might never be able to get there.
 325 Different UUID for FS on images.
 326 
 327 Maybe not possible to have just one toolset. Cloud providers might have different needs. 
 328 
 329 It is desirable for Stretch to have small number of tools. With customizability to suit different clouds and different use cases.
 330 Tests. Internal cloud providers' tests. Debian tests - based on policy how official image should look like.
 331 
 332 Cloud providers:
 333 Amazon does not care: Anybody can create and publish image. Anything above VM is customer responsibility; buyer beware.
 334 GCE - official image is Debian. But whether they will use Debian official image or their own - now known yet. They care that image runs.
 335 Quality, provenience.
 336 Azure: endorsed distributions. Suite of tests
 337 
 338 vendor data - ability to customize images. Is this useful, or more way of circumventing policy.
 339 
 340 Extra requirements
 341 ++++++++++++++++++
 342 
 343 Configuration
 344 
 345 Unattended upgrades?
 346 User experience from desktop - no unattended upgrades. But cloud is not desktop.
 347 Official GCE images: with unattended updates. Customers expect updates: ssl, kernel, etc?
 348 Option during running instance?
 349 Two images: minimal without unattended upgrads, and base with one.
 350 
 351 Official images - with provider-specific agents. Of course when such agens is in main.
 352 
 353 unattended-upgrades - should also be in Debian-Installer, with default answer "YES".
 354 ACTION: Sledge to push that on debian-devel
 355 
 356 But ability to disable - e.g. database servers, Tomcat, etc.
 357 Setting in cloud-init, with ability to set through user data
 358 Or unattended-upgrades has configuration file, by DebConf
 359 Long time to stop/start server during security upgrade, or glibc
 360 
 361 Epheremal ports. What's the issue here? - https://wiki.debian.org/Cloud/SystemsComparison
 362 The team creating the Amazon cloud images changed some kernel parameters
 363 about tcp ephemeral ports. 
 364 
 365 Performance tuning settings. Per cloud, but even per instance type
 366 
 367 Document what we're changing. It'll be documented in the tools to build image, but we'll need to document why we did the change - for performance, for users' needs, because of cloud provider technical needs, etc.
 368 
 369 Consistency has its value. Both between cloud and desktop, and between cloud providers. We should stay consistent except where there is a strong reason to do otherwise.
 370 
 371 Disabling IPv6. Introduces delay when not available on platform. May lead to confusion of users.
 372 
 373 Platform specific tweaks are accepted as long as they are documented (like
 374 the IPv6 example above)
 375 
 376 SSH installed by default?
 377 
 378 legal and trademark issues
 379 --------------------------
 380 Risks: unofficial images with trojans or other malware. Or even not-well tuned images with various problems, diluting Debian name. Trademark needs to be protected.
 381 Official Debian and related to Debian, or Debian-based.
 382 
 383 "Debian provided by XXX" or "Debian provided by Debian". Let's avoid "Official Debian".
 384 
 385 Policy proposal. If any image called "Debian" differs from official images, we (or all users?) should see the list of changes.
 386 
 387 Test suites (https://wiki.debian.org/Testing%20Debian%20Cloud%20Images)
 388 
 389 We don't have one. We need one. We had some for CD, (from GSoC) but it rotted now.
 390 Test should be based on policy. Or does policy is defined by available tests?
 391 
 392 See gobby.debian.org /Sprints/CloudSprint2016/TestIdeas for test ideas
 393 
 394 Docker image
 395 ------------
 396 Should be built also in the debian infrastucture. We should contact the current
 397 author of the Docker Image (Tianon Gravi). 
 398 
 399 
 400 Versioning 'Cloud Images' and critical holes patching policy (example: dirty CoW, heartblead, etc)
 401 ==================================================================================================
 402 Because of the latest kernel issue, steve did a 8.6.1 release of OpenStack
 403 image. Steve has a cron job to monitor the updates of packages (security updates)
 404 which are included in the openstack image.
 405 
 406 Up to now we used the last digit in case of errors in the build process.
 407 We could also use time stamps for that (YYYYMMDD or similar)
 408 The securiy team should inform of us in case.
 409 
 410 ACTION: Sledge to switch the version of the openstack image to include timestamp and look at adding changelog
 411 ACTION: Sledge to start organising HSM for the CD build machine
 412 CVE fixed should be included in the changelog of the image.
 413 
 414 Discussion of labelling in AWS and GCE. 
 415 Codenames are sometimes difficult to follow for users.
 416 
 417 Debian-cd does not use version number in testing CD on purpose (to make easy
 418 for people to distinguish from stable images)
 419 
 420 Names should be searchable but don't need to be typed.
 421 
 422 What should be in Critical holes patching:
 423 * everything.
 424 We should rebuild images after each package security updates.
 425 This could happen once a day.
 426 Rebuilding each night is not a problem, the problem is the signing of the images.
 427 A human has to do that.
 428 
 429 Daily builds, even when nothing security-related changed, can help with discovering problems with building. We don't have publish them.
 430 
 431 We need automated testing though - manual checking for few weeks will lead to boredom and problems with catching problems after longer time.
 432 
 433 Signing by humans or automatically?
 434 
 435 
 436 SDKs and access to platforms
 437 ----------------------------
 438 Not sure of a consistent policy for volatile updates.
 439 Currently a version small number of packages go through proposed-updates.
 440 
 441 
 442 jessie-updates -> proposed-updates -> Jessie next point release
 443 
 444 Delay in getting new packages
 445 
 446 Many different languages times cloud providers - we can grow volatile manyfold.
 447 
 448 Just package, or also plan to keep it current?
 449 Should Cloud team take more active role in maintaining packages?
 450 
 451 Packaging requires human touch - check accordance with Debian Policy, etc.
 452 No automated solution will fix all problems. Automates can help with initial packaging though, and then DD can just fix few small issues.
 453 
 454 But then it takes time to maintain package - updates, reaction to issues, etc.
 455 
 456 FPM?
 457 Hack, but there is nothing better than that for now
 458 
 459 Cross-platform
 460 Cross-distribution - from the point of view of cloud provider. They want to serve different distributions
 461 
 462 For Debian source package is source. Many tools rely on content of debian/ directory. BTS, etc. debian/control is basic when deciding what is the package. debian/changelog for version and closed bugs, etc.
 463 
 464 Debian is about integration. We take care that all (or at least most) of our packages to work well together, not to conflict with each other, and to know what is required (and provided) by packages.
 465 Debian policy is market differentiator. Tools are for enforcing policy (helping with that).
 466 
 467 New solutions - like snap - to solve (or work around) packaging problems. But they do not solve problem but just bundle everything. Those do not make family of packages.
 468 
 469 Tool must conform to Debian policy. But it can cooperate (or at least not work against) other distributions.
 470 
 471 Google packages. They are in Ubuntu, but are they in accordance with Debian Policy?
 472 
 473 We have 1 month to get daemons and SDKs to Stretch.
 474 
 475 What do we do later, with updates?
 476 Additional apt source => not official image. Also problems when e.g. Google decides to put new version of glibc.
 477 
 478 Backports. Might be problematic with library transition.
 479 Nonetheless we need to put new version into testing so package has some usage.
 480 Updates might be better than backports.
 481 
 482 Commitment from release team?
 483 Discussion with Adam in Cambridge during BTS next week - Send email to them.
 484 Sledge to work through that
 485 
 486 We'd like to have daemons in Stretch
 487 Google - everything needs to be done.
 488 Better not to have such package in stable than have stale version.
 489 But still upload, and maybe keep in unstable (with RC bug).
 490 Example: cloud-init is broken in stable
 491 
 492 Build official images with stable + updates
 493 
 494 AWS - basic done (cloud-init only needed)
 495 Rest is optioal but nice to have
 496 
 497 Azure?
 498 
 499 SDKs
 500 Mostly convenience, but really nice to have
 501 GCE - release cadence every week
 502 But compatibility is preserved
 503 
 504 Old SDKs do not break, but no new features. e.g. new regions
 505 
 506 Solving daemons should help solving SDKs.
 507 Agents do not depends on SDKs
 508 Feature-dependent, but not code-dependent
 509 
 510 Azure - some SDK is packaged (Python?). No details
 511 
 512 Cloud providers are encouraged to provide SDKs or help with packaging.
 513 
 514 Meeting 2016-11-03
 515 ==================
 516 Almost all people are interested in both build tools and testing tools, so no splitting into working groups.
 517 
 518 So we start with presentations of build tools.
 519 
 520 bootstrap-vz, presented by James.
 521 Attach volume, install Debian in chroot, create snapshot, register it.
 522 
 523 We all seem to be doing bootstrapping.
 524 But some tools use D-I.
 525 
 526 Sam: taksonomy
 527 1. use or not D-I
 528 2. whether tool uses VM (creates VM)
 529 3. whether tool has customization built-in, or is just script you need to hack
 530 bootstrap-vz you can customize, open stack is script you need to hack
 531 4. whether we're bootstraping into mounted FS, or create tar which you can import into cloud
 532 Jimmy pointed that for GCE we have tar of disk image, so this might not be so distinct division
 533 
 534 Demo is nice, but more important it's to see details
 535 
 536 Adding plugin is easy - but Sam does not fully agree.
 537 Manifest knows about subtasks
 538 Just adding subtask is not enough - you need to add it to code building manifests.
 539 It doesn't have plugin factories.
 540 Jimmy remembers that it is not to hard to add plugins to GCE.
 541 
 542 Plugins: phases, and dependencies.
 543 
 544 Some strange dependencies.
 545 It's fixable, but not to good for not-cloud-team-member person who wants to customize image.
 546 
 547 Commands plugin - to run command in chroot
 548 Copy files plugin
 549 
 550 Not good enough documentation for end user
 551 So we need proper documentation for users, not only for developers
 552 
 553 Tool to build images needs to do it correctly (?)
 554 Legal, needs to work, needs to be correct
 555 Legal - for "Official" Debian images
 556 
 557 If tool needs something from outside of the archive, it might prevent us from calling "official"
 558 
 559 ANI on EC2 - Advanced Networking. It's not in archive. So Debian is not on par with other operating systems. So we're telling users to use other OS.
 560 
 561 Discussion about whether we want something perfect, but taking long time, or just working and then build on that.
 562 
 563 Tools can be in main if they can be used to build DFSG-free image. If users want to use it to build non-free images, that's their choice.
 564 
 565 Going back to tools.
 566 Martin - bootstrap-vz does not give proper error messages, but Python backtrace.
 567 
 568 Manoj: error handling and documentation as two really important areas of improvement.
 569 man page is incoherent. Code is better documentation that documentation.
 570 Marcin: there is package with documentation
 571 
 572 Thomas: tool is in Python. Do our target audience need to know Python? 
 573 Mixture of Python and shell commands (to apt)
 574 Manifests: config data
 575 But some configuration is in the Python code (e.g. ntp servers). Also lists of packages to install is hardcoded. And usage of pip to install something in plugin.
 576 Repeated things: user name, etc.
 577 No templates or inheritance of manifests
 578 It has tests.
 579 OO-overhead
 580 Jimmy: it was rewritten from shell to Python to allow for extensability
 581 Plugins need to be in Python
 582 
 583 Let's try to understand reason behind critisism.
 584 bootstrap-vz is hard to audit because of this mixture of code and configuration. Changing cloud provider changes list of installed packages. And those packages are not in the manifest but in code, sometimes in some hidden plugin.
 585 
 586 Providers make some assumptions, e.g. EC2 assumes that we are building on EC2 instances.
 587 
 588 We need to make sure that all above comments are upheld by any tool.
 589 
 590 
 591 Tool we choose should be team-maintaned. Not a single person.
 592 Anders is not active upstream (formally is) and maybe we'll (cloud team) need to become upstream.
 593 
 594 Currently bootstrap-vz is used by AWS, GCE, Oracle
 595 
 596 Good thing (Sam): it has plugins.
 597 It undestands that image creation is complex and needs to be extensible.
 598 Simple tool is atractive, but may be limiting in the future.
 599 
 600 Manoj. End user perspective. Debian builds. Build machines must behave the same no matter which backend is behind them. They do not need to be bit-by-bit identical but should be as similar as possible.
 601 Lack of root manifest. Might not be so bad - in theory it would be the only file we need to look at for auditing.
 602 
 603 Martin: bootstrap-vz is lot of magic. James - more modularity might be solution to that.
 604 Martin: chroot is more familiar. shell scripts are more familiar for admins
 605 shell scripts (based on open stack) was fastest way to build images on Azure
 606 
 607 Shell scripts are not really extensible.
 608 
 609 Shell scripts need to be hacked to have different images, providers, etc.
 610 zigo's purpose (according to Sam) is to require hacking for extensibility.
 611 Tool needs to provide hooks to change behaviour.
 612 
 613 It's easy to audit. But might not be good for customization.
 614 
 615 Image should be useful for the ordinary user.
 616 Reasonalble manifest file.
 617 Full log.
 618 Do we want to have log of generated images?
 619 Let's publish log along the link to the image.
 620 We want to have disk image at "images.debian.org" and log near this.
 621 User can download such an image and use it. At the same time we can also upload this to appropriate cloud provider.
 622 
 623 Discussion regarding managing credentials. Credentials on Debian machines? Long-term credentials are risky.
 624 Building of images - on Debian infrastructure.
 625 Testing - as well. Preferably automated.
 626 Upload - by human, after verifying tests results.
 627 
 628 Plugin can do anything - they are code.
 629 You need to attach them in appropriate time-slot.
 630 But there is no uniformity
 631 
 632 bootstrap-vz is state machine. It builds graph of plugins and travels along it.
 633 
 634 FAI - is it appropriate tool?
 635 http://fai-project.org/	
 636 presentation of FAI by Thomas
 637 Config spaces and classes.
 638 Class - distribution (Debian, CentOS).
 639 Priority of classes.
 640 Solution for inheritance and extensability for "manifests"
 641 FAIBASE - all machines (images) will inherit this information
 642 
 643 Task - define classes.
 644 
 645 Ability to have human-generated cache (basefile)
 646 Pre-build image, serving as basis for many images
 647 Sometimes it uses all matching classes, sometimes one with the highest priority
 648 
 649 Customization scripts.
 650 
 651 fai-diskimage -v -u X -S size -c CLASS,CLASS /disk/file.raw
 652 fai-kvm -Vu 5 /disk/file.raw
 653 
 654 Advance partitioning
 655 Does not use kpartx - there is no need for now.
 656 
 657 It runs as root.
 658 
 659 DebConf preseeding
 660 
 661 Config files, dropped into target images? More like used by various scripts to steer them.
 662 
 663 Mature code, with good documentation.
 664 
 665 Packages installation - via apt, aptitude, yum - configurable
 666 Ability to use shell variables there
 667 Logical joining of classes - class might be dependent on other class
 668 
 669 Some of the configuration depends on used tools - e.g. for installing or removing packages.
 670 
 671 Hooks directory.
 672 Priority of the classes.
 673 Configuration from command line, database, other sources. Priorities of classes
 674 
 675 ainsl instead of echo >>
 676 Templates for files.
 677 Tree to be injected into target images
 678 
 679 FAI - shell
 680 partitioning and packages maintenance - in Perl
 681 
 682 Replacing of tasks by some other scripts
 683 
 684 2 abstraction levels.
 685 Separation of code and configuration.
 686 
 687 Classes created by advanced users, used by ordinary users
 688 
 689 FAI can do more than disk images.
 690 Config files instead of API (?)
 691 
 692 You need to grok config space - takes about 1h.
 693 Then you can use FAI.
 694 
 695 Update to new Debian release takes few hours. You just need to make few builds to test all the changes.
 696 
 697 Decision regarding number of created and/or uses classes.
 698 
 699 Do you need to read the code to grasp what it's doing. Hard to start.
 700 Risk of debugging shell script
 701 
 702 Test suite - task called test. Not much - grepping though the logs.
 703 No regression tests.
 704 Manual testing of images.
 705 
 706 FAI - just classes for cloud. User can also create their own classes.
 707 Volume resize?
 708 
 709 Not changing anything for Jessie. Potential new tool - only for Stretch
 710 
 711 Sam should be able to provide deeper feedback by the end of next week.
 712 
 713 Variables for shell scripts used in scripts
 714 
 715 No dry-run. Also configuration is split among many files, and usully we use many classes. So we need to run FAI with appropriate options and check if it runs sucessfully.
 716 
 717 UEFI and GPT - was not tested for now.
 718 
 719 ------------------------------------------
 720 vmdebootstrap. Great test suite.
 721 Command line - a bit similar to pbuilder(distribution, mirror, architecture)
 722 Shell script.
 723 Only one shell script. It does not use apt as package resolver, but debootstrap. A bit problematic with more sophisticated dependency graphs.
 724 
 725 It cannot be used alone as the tool for building images. Might be used as a basis, with conjuntion with other scripts.
 726 Sam wrote Python3 library to help with that.
 727 Need more that one tool. More code than configuration.
 728 No correct API of debootstrap.
 729 
 730 Customizations is neccessary for image builder. debootstrap provides no customization.
 731 
 732 Test (yarn). Can we extract it? yarns have meta language. 
 733 
 734 Everything needs to be provided as command line parameters. Config file - is just shell script wrapping it.
 735 
 736 Tools for foreign architectures.
 737 vmdebootrap is good for that. --foreign; uses qemu
 738 We'll probably use VM. Not fast, but doable.
 739 
 740 vmdebootstrap authors might not want to extend it to suit our needs - they want to keep it simple. We might then need to have wrapper - but this will mean that we need to write much code, and some working around limitations.
 741 
 742 vmdebootstrap is ill-suited, but its test suite is great.
 743 
 744 Auditability is non-existent in bootstrap.
 745 
 746 FAI - categories for users
 747 
 748 Do we want to inspect what was installed, or inspect what would be installed? We'll need the list of what was installed (dependencies and alternatives), but sometimes we might want to know what will happen.
 749 
 750 People are opened to evaluating FAI.
 751 
 752 ... keysigning ...
 753 
 754 cloud-init
 755 ==========
 756 
 757 0.7.8-1 uploaded, currently in unstable and testing. Fixed a few bugs, including fragile test suite
 758 upstream are Ubuntu folks, major issue with CLA :-(
 759 Written in python
 760 
 761 Google don't like cloud-init as it's slow (adds 5s to boot time)
 762 there's a forked rewrite in Go which is much faster
 763 there was talk acout a re-architecture/rewrite, but no visible progress
 764 
 765 ~20k loc of python
 766 It's object oriented, which might add overhead
 767 
 768 It's good for uniformity between distributions - at least in theory.
 769 But there are different versions and forks in distributions, so it's not always true.
 770 Quite a lot of patches from various folks, not pushed upstream (possibly because of CLA)
 771 
 772 GCE is not eager to add cloud-init, at least until there is uniformity.
 773 Upstream does not really care.
 774 
 775 Time to fork? 
 776 Should we use the Go version? Maintained by CoreOS. Could be a problem :-(
 777 Do we want to have Go-based program in base? Compiled but statically linked. But there is possibility to dynamically linked - only for AMD64.
 778 Its upstream accepts patches.
 779 Might be dying upstream, they're pushing for "ignition" instead of cloud-init probably
 780 
 781 It's for configuration, but has much functionality. Any replacement should offer all functionality.
 782 
 783 ACTION: Bastian, Marcin and Jimmy volunteer to be a new upstream team for cloud-init in case it comes to that.
 784 ACTION: Various cloud platform folks will talk to Scott about moving to this new upstream to work around the CLA that's blocking people.
 785 
 786 Open bug for release asking why we need cloud-init in the next point release.
 787 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=789214
 788 
 789 test-suites
 790 ===========
 791 
 792 What do we want to test? Check that things match our policies. What are they?
 793 
 794 (see lists of simple tests further up)
 795 
 796 How do we manage tests? We build images and do we wait for cloud providers to run their tests?
 797 AWS: 2 sets of images. Our images (in Debian account) and ones in marketplace. Marketplace ones might have more tests.
 798 
 799 We should get information when cloud provider's tests fails - and then add such case to our test suite. We might not get exact source code of test, but we should get at least description to be able to reproduce it.
 800 
 801 Stretch - do we rely on cloud providers for running tests?
 802 
 803 2 things: tests and framework to run them.
 804 
 805 Framework will depend on cloud provider.
 806 Tests should test image in different variants - on different machine types, with various disk options, etc.
 807 
 808 More details in the second document.
 809 
 810 For test suite we'll need ability to programatically register image, start instance, etc.
 811 For this we'll need SDK or API access from chosen language.
 812 For AWS we have libraries, e.g. Boto.
 813 For Google we don't have SDK in Debian. There is Apache libcloud, or Google provided projects on github: https://github.com/GoogleCloudPlatform
 814 Google SDK, for which we have RFP, is command-line tools, not API access library: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759578
 815 
 816 Discussion about automatic updates.
 817 Non-https may leave users vulnerable to analysis which packages they have installed (might be problematic in some jurisdictions).
 818 
 819 Mostly positive responses to proposal on debian-devel@. Problems with upgrading services. We need to work on better restarting them.
 820 
 821 Rebuilding and customization
 822 ----------------------------
 823 We want to allow for it.
 824 
 825 Versions and updates (again)
 826 ============================
 827 We'll build images for stable and point releases. After important fixes we should rebuild - but this might be problematic. On AWS we need to give AMI ID so it will be problematic. Not only security updates, but also feature updates for agents and/or SDKs. So we don't need to keep them in lockstep, but then we need to provide detailed changelog - to let users know why the difference in timestamps.
 828 
 829 We want to update kernels. Feature releases by updates make more probable for things to break. Also - having dozens of instances just spinning updating may mean measurable costs for users.
 830 
 831 We do once a weekly builds for testing. Weekly and monthly retention, to avoid having to many images.
 832 Should we publish them? Different providers might have different meaning of what it means to have image published.
 833 When we'll have better testing and building automation, it'll make easier to users to build images.
 834 
 835 Tools for tranforming images to format acceptable by cloud provider. Qemu should be able to deal with most of them.
 836 
 837 Current and future architectures
 838 ================================
 839 
 840 amd64. There are some ARM clouds. IBM has Power cloud. Linaro has ARM64 images for developers. Steve promises to have Open Stack ARM64 cloud before Stretch.
 841 
 842 Currently there is not many architectures, but there might be in the future. But as long as we have generic tooling (and Debian has many architectures) we should be able to deal with it.
 843 
 844 ARM64 - UEFI with grub-uefi should do.
 845 
 846 Do we want to have multi-arch enabled by default? IT should be customizable, i.e. users should be able to generate such an image. Users are not asking providers for 32-bit support.
 847 
 848 Maybe we might mirror 32-bit repositories.
 849 
 850 We might want to have some ? policy? about sunsetting some of the architectures. 32-bit are getting less relevant.
 851 
 852 Human language support and localised images
 853 -------------------------------------------
 854 
 855 By default we're using "C". How do we allow for users to switch language e.g. through cloud-init. We have quite good language support, but do we need to have it in the cloud.
 856 
 857 Do current images have all locales installed?
 858 
 859 At least some of the cloud providers provide non-English UI (web frontend).
 860 
 861 Amazon Linux has localizations and ability to change language through cloud-init. It's just one image with many different locales. Ubuntu is English-only.
 862 
 863 ACTION: James to ask on the mailing list(s) if anybody wants more languages for the cloud image(s).
 864 
 865 Other things to consider - which mirror to use? Needed for users in China, for example.
 866 
 867 Sam proposal: ask few questions during first login. It'll break automation, but might be useful for some (really specific) needs. Not by default! Is it worth engineering effort? (Probably not)
 868 
 869 Meeting 2016-11-04
 870 ==================
 871 
 872 Supporting services (for platforms, mirrors, finding things)
 873 ============================================================
 874 
 875 Mirrors
 876 -------
 877 
 878 the httpredirect mirror httpredir.debian.org is depricated.
 879 When somebody has 1000s of instances and we have autoupdate, we'd like to avoid killing any mirror
 880 Zach: external mirrors are often faster.
 881 But some of the cloud providers want to keep as much traffic inside their network as possible.
 882 Azure has internal mirror network. 25TB of storage; pushed from official mirror
 883 CDN for Amazon; headers with different expiry to deal with different refreshment needs (e.g. packages themselves (*.debs), Packages files) Ordinary mirror and security mirror
 884 
 885 Our solution will have lowest maintenance possible. There is not enough manpower.
 886 
 887 James tried to S3 - it was really slow to populate this (6h). That's why he moved into CDN.
 888 It used to require instances to rewrite headers but now Cloud Front can set TTL (I'm not sure I understood this part correctly)
 889 
 890 Google Cloud CDN can serve content from inside GCE. But we could have instance with redirect.
 891 
 892 Disk space is not concern. Low maintenance and monitoring are priorities.
 893 Tracefile of the mirror should be monitored to see whether we're in sync.
 894 Official mirror script updates it last so it can serve as a canary
 895 
 896 Official mirrors - 4 pushes a day
 897 
 898 Signature cannot be older than 10 days - so maximum frequency is 7 days
 899 
 900 deb.debian.org - backed by 2 commercial CDNs (Fastly and cloudfront/Amazon). Stretch apt can directly use CDN behind it, without need to redirect (as needed by older apt)
 901 Fastly has peering connection to GCE
 902 Bastian can pass on documentation he has.
 903 
 904 Martin shows traffic to ftp.debian.org; it has 10G connection on university network
 905 200Mbps on average
 906 
 907 James shares his setup of Apache for redirects and expiry times
 908 AWS - 500 requests per minute to the interception header host; 1TB per day
 909 We have details which files are requested the most - so we could get something from those statistics
 910 
 911 CDN with HTTPS enabled. security-cdn certificate
 912 
 913 After Google has set CDN, it'll be integrated into our network
 914 
 915 mismatch Hashsum; we need small TTLs to avoid clients getting stale index files
 916 It'll dissapear with Stretch apt.
 917 
 918 Finding things
 919 --------------
 920 Ubuntu image finder: cloud-images.ubuntu.com
 921 Home page with links to all images of releases. They also have manifests
 922 For various architectures: amd64, IBM Z, arm32
 923 
 924 Link to AWS goes directly to launch wizard - quite useful.
 925 On Oracle it goes to market place - but it might be because we were not logged in to Oracle cloud
 926 
 927 We should also provide JSON with images so users can automatise work on our images.
 928 
 929 Martin: Ask Colin Watson if the code for Ubuntu image locator (cloud-images) is available. or Martin will write it!
 930 
 931 What users expect from image finder page. List of cloud providers? Distributions versions? Problem - filters are on the bottom while they should be at the top.
 932 Separate page with stable and daily images.
 933 Logos on the top to make it easier for users to see what do we offer.
 934 
 935 Support Debian in Juju?
 936 
 937 Should we provide base files, pregenerated when we build images? It'll make life easier for users not on Debian systems.
 938 
 939 == Better handling/publishing/advertising of cloud images by Debian
 940 
 941 Some nice! web pages for showing our images. ( example for parsing json files: https://msdn.microsoft.com/library/cc836466(v=vs.94).aspx )
 942 
 943 What's more than image finder?
 944 
 945 James sends signed emails with AMI IDs. But anybody can edit wiki and change IDs to something malicious. Should we lock wiki page?
 946 Image finder should help here.
 947 
 948 Register cloud.debian.org. Maybe debian.cloud? (i.e. .cloud TLD). It's reserved, we'd need to speak with SPI for trademark issues.
 949 
 950 === Getting updated packages into Debian - stable updates, -updates, backports
 951 
 952 How does clamav get on?
 953 In volatile times it was pushed through without much problems.
 954 
 955 Many paths. jessie, jessie-proposed-updates, something else?
 956 stable-updates
 957 proposed-updates
 958 
 959 Should we (team) salvage some of relevant packages, like python-boto, boto3?
 960 
 961 there is python-libcloud, and it seems to be quite recent
 962 
 963 There is Azure CLI in NEW --> contact maintainer to join the debian cloud team and set maintainer to the c, including related python-azure, python-azure-storage
 964 
 965 Action item (serpent): contact *boto* maintainers asking about update before freeze, backporting, maybe salvaging it.
 966 
 967 === Website changes (better promote Debian Cloud images)
 968 
 969 Needs to happen :-)
 970 James will try to do something with it after getting access
 971 Manoj offers to help too
 972 
 973 == AOB
 974 
 975 We go through cloud-related bugs in BTS.
 976 Add locales-all to the list of packages we install by default
 977 
 978 ACTION: Sledge to write up policy for official images and post it on the website (#706052)
 979 
 980 What do we do about things like non-free GPU drivers and other drivers that won't go upstream?
 981 We *could* add non-free unofficial images too, but definitely make it easier for people
 982 to build their own images.
 983 
 984 Resolved - do the extra non-free images, and make sure people can find them:
 985  * appropriate warnings that non-free is bad
 986  * *NOT* directly in the same image search area, etc., but maybe a second one which is linked
 987 
 988 == Going to the computer museum
 989 
 990 Yay!

Attached Files

To refer to attachments on a page, use attachment:filename, as shown below in the list of files. Do NOT use the URL of the [get] link, since this is subject to change and can break easily.
  • [get | view] (2016-11-08 00:05:57, 2.8 KB) [[attachment:CloudFrontProxyInterceptionConfig.txt]]
  • [get | view] (2016-11-08 00:05:00, 41.4 KB) [[attachment:Minutes.txt]]
  • [get | view] (2016-11-08 00:05:32, 2.0 KB) [[attachment:TestIdeas.txt]]
  • [get | view] (2016-11-30 15:11:10, 272.0 KB) [[attachment:group_photo.jpg]]
 All files | Selected Files: delete move to page copy to page

You are not allowed to attach a file to this page.