Attachment 'Minutes.txt'
Download 1 Cloud sprint in Seattle, 2nd to 4th November 2016
2
3 Hosted by Zach at the Google offices - thanks!
4
5 Present:
6 ========
7
8 (on-site)
9 * James Bromberger (JEB)
10 * Emmanuel Kasper (marcello^/manu)
11 * Steve McIntyre (Sledge/93sam)
12 * Martin Zobel-Helas (zobel)
13 * Bastian Blank (waldi)
14 * Sam Hartman (hartmans)
15 * Jimmy Kaplowitz (Hydroxide/jimmy)
16 * Marcin Kulisz (kuLa)
17 * Thomas Lange (Mrfai)
18 * Manoj Srivastava (manoj/srivasta@{debian.org,google.com,ieee.org}) Affiliation: Debian/Google
19 * Zach Marano - Google (zmarano)
20 * David Duncan - Amazon (davdunc)
21 * Tomasz Rybak (serpent)
22 * Noah Meyerhans (noahm)
23 * Stephen Zarkos - Microsoft (???)
24
25 (irc/hangout)
26 * liw
27 * hug
28 * damjan
29
30 Agenda
31 ======
32
33 (Wed)
34 == What does it mean to run in a cloud environment.
35 === Priority of our users vs. technical priorities
36 == In depth look at how Debian runs in major clouds (AWS, Azure, GCE, Oracle, on premise, ... etc)
37 == Define an official Debian cloud image.
38 === legal and trademark issues
39 === test suite for images
40 === official "Debian-based" images (for container platforms and other variations)
41 === Versioning 'Cloud Images' and critical holes patching policy (example: dirty CoW, heartblead, etc)
42 === SDKs and access for platforms (including release cycle mismatch vs cloud)
43
44 (Thu)
45 == Decide, if we want WGs for belows discussion items?
46 == Look into different build processes of different image types (maybe short presentations)
47 == Introspect the various image build tools and whittle the list down.
48 == cloud-init maintenance
49 == test suite
50 == rebuilding and customisations
51 == Current and future architectures
52 == (Human) Language support
53
54 (Fri)
55 == supporting services (for platforms, mirrors, finding things)
56 == Ideally, come to consensus on many of the open ended issues that have been talked about at DebConf and on the debian-cloud list.
57 == Better handling/publishing/advertising of cloud images by Debian
58 === Getting updated packages into Debian - stable updates, -updates, backports
59 === Website changes (better promote Debian Cloud images)
60 == AOB
61 == Going to the computer museum
62
63 Meeting 2016-11-02
64 ==================
65 1. Everybody introduces themselves
66 2. Go through planned agenda; it doubled so we prioritised
67
68 Need for customizations of images.
69 No vendor lock-in
70
71 What does it mean to run in a cloud environment
72 -----------------------------------------------
73 Cloud - disposable computers. Many types of usage: long-term or really short (mere hours).
74 Also - desktop in the cloud.
75 We (Debian) have many architectures, many languages, many packages - can be useful.
76 Instance has to be fast to boot.
77 a) Official image -> customise it -> save as new image
78 b) generate image from scratch (tooling)
79 c) launch and customize image - without saving it for later (cloud init, etc.)
80
81 For the cloud - all the users will be derivatives?
82
83 Minimal image - for start
84 Full-featured image for some users
85 Look on Ubuntu and other Vendors, Ubuntu's image finder
86 Advanced users - can help themselves, but need good starting point. Need solid base, not be be discouraged.
87 Don't care about Rackspace, little information to go on here
88 Openstack has multiple virtualization backends (LXC, KVM) and vari ous ways of presenting images and network to the users
89
90 Boot time - Xen tries to initialize non-existing devices; 30s latency in boot time
91 Non-existing framebuffer. Blacklist driver?
92 Enhanced monitoring of cloud stuff using agents from the platform providers. For
93 instance google cloud agent is based on a fork of the collectd agent.
94 Packaging theses agents is easy, but the problem is to get them to stable, as
95 the cloud provider API they call change quickly. These agents are used for
96 instance, setting the initial root password.
97
98 Google won't provide a default username/password for base images, so for login there needs to be an agent installed. It will deal with things like ssh keys, also routes for multiple NICs etc.
99 Agents are not required per se for running images on any of the common platforms, but they provide additional services. We need easy way to allow for users to opt-in to them. Otherwise they will use some other distribution/image.
100 Azure has also its own agent for the same purposes, packaged in Debian. Stretch
101 for Azure will maybe use cloud-init.
102
103 Now common for people to want docker (etc.) with their cloud images too, often to be able to provide their workloads that way: Docker, Mesos, Kubernetes, ECS, ???
104
105 Cloud-init in unstable - bugs
106
107 Various services:
108 a) monitoring
109 b) hardware (mostly networking) management
110 c) security related (SSH key injection, etc.)
111 d) deployment (e.g. AWS DevOps)
112 e) other (e.g. EC2 Run Command)
113
114 Put fast-changing packages. Testing/unstable - too risky (breaking stuff, transitions). Backports? Volatile archive?
115
116 Cloud-init. Is it required? GCE can use it but don't want to - it's very slow.
117 Users don't know about it, it's not a great user experience.
118 Preferred to have software installed in the image already, then just run apt update afterwards if needed
119 for updates.
120
121 We want a long-running daemon for GCE and Azure at least, to integrate with the full platform feature set.
122 cloud-init only works at boot. Special daemon (agent) works all the time - allowing for creating new users, SSH keys, etc. So on GCE/Azure it is possible to change keys, create users, etc. all the time. AWS - only during boot.
123
124 Common core + extensions
125 Disk space vs. boot time and performance. But disk space for images, or for running instances?
126
127 Secure Boot, UEFI are examples of places where different platforms will show differences
128
129 Vagrant - VM image typically used for a reproducible environment for various purposes.
130 Vagrant for cloud providers; good to move from developer machine to the cloud
131
132
133 In depth look at how Debian runs in major clouds (AWS, Azure, GCE, Oracle, on premise, ... etc)
134 ------------------------------------------------
135
136 Demo of Debian on cloud providers; proposed on IRC
137
138 AWS (James B.)
139 ++++++++++++++
140 Images are built on EC2 instances. You need to create snapshot of the volume - so it's easy to work from EC2 instace. There is volume import API - but I'm not sure that it's usable. James used it only once.
141 All regions; China (behind Firewall). Gov Cloud (you need to be ITAR Certified for access)
142 Packages mirroring: CDN (Cloud Front). Now with IPv6. Headers for expiring - depending on requested resources
143 HTTPS: apt-transport-https package to be useful
144 Statistics: about 1TB/day
145
146 Image. A bit of mess with billing code (?) - no possibility of removing it, even though it is not useful for community images. Restrictions of usage - snapshot, clone volume, start. Can users do it?
147
148 Rebuilding (of stable) Point releases, security issues, boot time improvements
149 Building of testing: should be regularly
150 Usage: 22k accounts subscribers to Debian images
151
152 AMI upload to market place. Spreadsheet with detailed informations. It should be possible to automatise it.
153
154 Ubuntu is shown in preferred AMIs, for Debian we need to look for it in Market place
155
156 Instance creation: no encryption by default. Related to billing code?
157
158 No direct support for key-refresh over time
159
160 Vagrant (Emmanuel)
161 ++++++++++++++++++
162 Wheezy and Jessie images
163 vagrant repository or custom one
164
165 Virtual Box, synchronization of directories. By default on boot, ability to have during lifetime
166
167 Packer to build image.
168 Latest ISO, with checksum.
169 JSON with description
170 Makefile to call packer with appropriate parameters and metadata
171
172 Uses Debian Installer in the background
173
174 Test suite
175 vagrant up
176 install package
177
178 No need for root during this process. But we need kvm or similar kernel module
179
180
181 Azure (zobel, waldi and Steve Z)
182 ++++++++++++++++++++++++++++++++
183 Automated building in Jenkins
184 Jessie and Wheezy daily, uploads to publishing account. They are then replicated to public Azure network (20something regions?)
185 Modification of open-stack script. Bash-based. Problems with customization
186 Mirror network in Azure. Updatetraffic completely in Azure network .2.5GB/s capacity
187 Image 30GB
188 Anybody can upload, only chosen ones can publish
189
190 Manual release process.
191 Daily images - not in marketplace, but through API. But users can use them.
192 Removed after 14 days
193 Manual publication to marketplace. And manual removal of those.
194 Ordinary users should use published ones. Developers can use daily
195
196 Classic UI. No good discoverabilty for Debian. It's in maintenance mode.
197 New: Azure portal. Search for Debian
198 Resource providers. Azure Resource Manager. Templating.
199 No SSH key management in portal.
200 Resource groups: ability to better clean resources
201 Boot diagnostics and Guest OS diagnostics. Need for daemon for this
202 Tools (CLI) in many languages: node.js, Python, Go
203 Offer, skew, version (keyword: latest): URN for command line to identify image
204
205 Agents: on github
206 Account Azure, WALinuxAgent. SSH keys, diagnostics. Can live side by size with cloud-init.
207 Chosed during image creation
208
209 cloud-init: need the latest version, with many issues fixed
210 systemd - need quite recent cloud-init
211 Debian 7 - kernel from backports
212 Drivers
213
214 Testing: testing suite.
215 Images compatible with Azure Stack - Azure cloud on premise
216
217 JSON to publish on marketplace
218
219
220 GCE (Zack Marano)
221 +++++++++++++++++
222 uses boostrap-vz, code and manifest are in the upstream git repo
223 3 daemons are running in the final created image, for clock synchro, credentials
224 and ip forwarding(?)
225
226 All tools are on github.
227 No official packages - to many problems with various distributions. Custom tool in Ruby to build package.
228
229 minimal version manifest, not currently used; just as basis if somebody needs something to start
230 Usually run on VM in GCE; script using bootstrap-vz, starting VM, preparing everything
231
232
233 Debian is default on GCE!
234
235 GCE SDK is baked in on GCE images.
236
237 Monthly publishing of images
238 For security reasons it can be more frequently.
239 One Debian image. Just last stable. Depreciating oldstable, LTS does not cover the needs.
240
241 Testing of images. Including performance tests.
242 Tests not yet open-sources (integrated with internal Google tools), but should be in some time.
243
244 Image families.
245 Global images - not per regions
246
247 Ubuntu build their own images
248
249 Propose to release managers to put cloud-init from backports to stable.
250 It was proposed by zigo, but maybe we need more to convince them.
251 For now cloud was not seen as important; maybe when we show them numbers of users, release team will be more eager to let neccesary packages uploaded
252
253
254 Do we want to build images in VMs?
255 Complex layouts, like LVM - might need that.
256 Debugfs.
257
258 a bit of systemd discussion
259
260 GCE is happy with a single image, supporting only stable.
261
262 ScaleWay
263 ++++++++
264 ARM64 cloud applience. ScaleWay
265
266 Docker
267 ++++++
268 Example script showing how to create image. Deboostrap-based.
269 Docker file - based around application you want to host. Docker images are usually used to host single applications. Minimal.
270 Image namded "Debian" in DockerHub. We don't exactly know who is the publisher of this image. Supposedly some DD, but not member of debian-cloud team, and image not maintened for some time. ( possibly Tianon Gravi see http://joeyh.name/blog/entry/docker_run_debian/ )
271 Can anyone publish something with name of "Debian"? Move to legal/trademark issues
272
273
274 Official Debian cloud images
275 ============================
276 Discussion on mailing list started in November 2015.
277 https://lists.debian.org/debian-cloud/2015/11/msg00005.html
278
279 More/later info in:
280 https://lists.debian.org/debian-cloud/2016/03/msg00042.html
281
282 We're bit behind proposed schedule ;-)
283
284 Non-controversial proposals
285 +++++++++++++++++++++++++++
286 Archive: main
287 stable-updates
288 Not necessarily stable-backports (maybe other images, not-so-official)?
289 No extra archives
290
291 It might be hard to publish non-backports.
292 It'll be official (hopefully), but will need to be documented as with backports.
293
294 Official: by DD, on Debian infrastructure. Scripts to build are in Debian.
295 Published on Debian infrastructure, and also uploaded to appropriate cloud providers.
296 Of course also publish checksums.
297
298 Testing, with public test logs.
299 Test suite should (must?) be also public.
300 Possibly not test-suite for Strech, but requirement for Buster.
301
302 Azure: root build on Debian hardware. Then upload to Azure infrastructure, and perform some last steps.
303 For secure boot, do not store keys in cloud infrastructure.
304 Keys never leave FTP master (HSM attached to FTP master).
305 How to cooperate with cloud providers to allow for injecting custom images, and then sign them?
306 How far will cloud providers want to go with signing.
307 Sign grub, sign kernel, sign entire image?
308
309 Signing should not be required before Stretch release.
310
311 Signing of Debian by Microsoft?
312
313 CD images are not yet signed using HSM.
314
315 New architectures (for cloud) but not yet as stable?
316
317 Stable - of course. But also monthly (weekly?) testing images.
318
319 Tool chain for official images.
320 One tool set to build on all clouds.
321 Might be to late for Stretch.
322 Test suite to catch discrepancies between different images/tools.
323 Stable toolset - to be able to reproduce our image. Maybe use snapshots for older version of packages.
324 Almost like reproducible builds - i.e. the same checksum. We might never be able to get there.
325 Different UUID for FS on images.
326
327 Maybe not possible to have just one toolset. Cloud providers might have different needs.
328
329 It is desirable for Stretch to have small number of tools. With customizability to suit different clouds and different use cases.
330 Tests. Internal cloud providers' tests. Debian tests - based on policy how official image should look like.
331
332 Cloud providers:
333 Amazon does not care: Anybody can create and publish image. Anything above VM is customer responsibility; buyer beware.
334 GCE - official image is Debian. But whether they will use Debian official image or their own - now known yet. They care that image runs.
335 Quality, provenience.
336 Azure: endorsed distributions. Suite of tests
337
338 vendor data - ability to customize images. Is this useful, or more way of circumventing policy.
339
340 Extra requirements
341 ++++++++++++++++++
342
343 Configuration
344
345 Unattended upgrades?
346 User experience from desktop - no unattended upgrades. But cloud is not desktop.
347 Official GCE images: with unattended updates. Customers expect updates: ssl, kernel, etc?
348 Option during running instance?
349 Two images: minimal without unattended upgrads, and base with one.
350
351 Official images - with provider-specific agents. Of course when such agens is in main.
352
353 unattended-upgrades - should also be in Debian-Installer, with default answer "YES".
354 ACTION: Sledge to push that on debian-devel
355
356 But ability to disable - e.g. database servers, Tomcat, etc.
357 Setting in cloud-init, with ability to set through user data
358 Or unattended-upgrades has configuration file, by DebConf
359 Long time to stop/start server during security upgrade, or glibc
360
361 Epheremal ports. What's the issue here? - https://wiki.debian.org/Cloud/SystemsComparison
362 The team creating the Amazon cloud images changed some kernel parameters
363 about tcp ephemeral ports.
364
365 Performance tuning settings. Per cloud, but even per instance type
366
367 Document what we're changing. It'll be documented in the tools to build image, but we'll need to document why we did the change - for performance, for users' needs, because of cloud provider technical needs, etc.
368
369 Consistency has its value. Both between cloud and desktop, and between cloud providers. We should stay consistent except where there is a strong reason to do otherwise.
370
371 Disabling IPv6. Introduces delay when not available on platform. May lead to confusion of users.
372
373 Platform specific tweaks are accepted as long as they are documented (like
374 the IPv6 example above)
375
376 SSH installed by default?
377
378 legal and trademark issues
379 --------------------------
380 Risks: unofficial images with trojans or other malware. Or even not-well tuned images with various problems, diluting Debian name. Trademark needs to be protected.
381 Official Debian and related to Debian, or Debian-based.
382
383 "Debian provided by XXX" or "Debian provided by Debian". Let's avoid "Official Debian".
384
385 Policy proposal. If any image called "Debian" differs from official images, we (or all users?) should see the list of changes.
386
387 Test suites (https://wiki.debian.org/Testing%20Debian%20Cloud%20Images)
388
389 We don't have one. We need one. We had some for CD, (from GSoC) but it rotted now.
390 Test should be based on policy. Or does policy is defined by available tests?
391
392 See gobby.debian.org /Sprints/CloudSprint2016/TestIdeas for test ideas
393
394 Docker image
395 ------------
396 Should be built also in the debian infrastucture. We should contact the current
397 author of the Docker Image (Tianon Gravi).
398
399
400 Versioning 'Cloud Images' and critical holes patching policy (example: dirty CoW, heartblead, etc)
401 ==================================================================================================
402 Because of the latest kernel issue, steve did a 8.6.1 release of OpenStack
403 image. Steve has a cron job to monitor the updates of packages (security updates)
404 which are included in the openstack image.
405
406 Up to now we used the last digit in case of errors in the build process.
407 We could also use time stamps for that (YYYYMMDD or similar)
408 The securiy team should inform of us in case.
409
410 ACTION: Sledge to switch the version of the openstack image to include timestamp and look at adding changelog
411 ACTION: Sledge to start organising HSM for the CD build machine
412 CVE fixed should be included in the changelog of the image.
413
414 Discussion of labelling in AWS and GCE.
415 Codenames are sometimes difficult to follow for users.
416
417 Debian-cd does not use version number in testing CD on purpose (to make easy
418 for people to distinguish from stable images)
419
420 Names should be searchable but don't need to be typed.
421
422 What should be in Critical holes patching:
423 * everything.
424 We should rebuild images after each package security updates.
425 This could happen once a day.
426 Rebuilding each night is not a problem, the problem is the signing of the images.
427 A human has to do that.
428
429 Daily builds, even when nothing security-related changed, can help with discovering problems with building. We don't have publish them.
430
431 We need automated testing though - manual checking for few weeks will lead to boredom and problems with catching problems after longer time.
432
433 Signing by humans or automatically?
434
435
436 SDKs and access to platforms
437 ----------------------------
438 Not sure of a consistent policy for volatile updates.
439 Currently a version small number of packages go through proposed-updates.
440
441
442 jessie-updates -> proposed-updates -> Jessie next point release
443
444 Delay in getting new packages
445
446 Many different languages times cloud providers - we can grow volatile manyfold.
447
448 Just package, or also plan to keep it current?
449 Should Cloud team take more active role in maintaining packages?
450
451 Packaging requires human touch - check accordance with Debian Policy, etc.
452 No automated solution will fix all problems. Automates can help with initial packaging though, and then DD can just fix few small issues.
453
454 But then it takes time to maintain package - updates, reaction to issues, etc.
455
456 FPM?
457 Hack, but there is nothing better than that for now
458
459 Cross-platform
460 Cross-distribution - from the point of view of cloud provider. They want to serve different distributions
461
462 For Debian source package is source. Many tools rely on content of debian/ directory. BTS, etc. debian/control is basic when deciding what is the package. debian/changelog for version and closed bugs, etc.
463
464 Debian is about integration. We take care that all (or at least most) of our packages to work well together, not to conflict with each other, and to know what is required (and provided) by packages.
465 Debian policy is market differentiator. Tools are for enforcing policy (helping with that).
466
467 New solutions - like snap - to solve (or work around) packaging problems. But they do not solve problem but just bundle everything. Those do not make family of packages.
468
469 Tool must conform to Debian policy. But it can cooperate (or at least not work against) other distributions.
470
471 Google packages. They are in Ubuntu, but are they in accordance with Debian Policy?
472
473 We have 1 month to get daemons and SDKs to Stretch.
474
475 What do we do later, with updates?
476 Additional apt source => not official image. Also problems when e.g. Google decides to put new version of glibc.
477
478 Backports. Might be problematic with library transition.
479 Nonetheless we need to put new version into testing so package has some usage.
480 Updates might be better than backports.
481
482 Commitment from release team?
483 Discussion with Adam in Cambridge during BTS next week - Send email to them.
484 Sledge to work through that
485
486 We'd like to have daemons in Stretch
487 Google - everything needs to be done.
488 Better not to have such package in stable than have stale version.
489 But still upload, and maybe keep in unstable (with RC bug).
490 Example: cloud-init is broken in stable
491
492 Build official images with stable + updates
493
494 AWS - basic done (cloud-init only needed)
495 Rest is optioal but nice to have
496
497 Azure?
498
499 SDKs
500 Mostly convenience, but really nice to have
501 GCE - release cadence every week
502 But compatibility is preserved
503
504 Old SDKs do not break, but no new features. e.g. new regions
505
506 Solving daemons should help solving SDKs.
507 Agents do not depends on SDKs
508 Feature-dependent, but not code-dependent
509
510 Azure - some SDK is packaged (Python?). No details
511
512 Cloud providers are encouraged to provide SDKs or help with packaging.
513
514 Meeting 2016-11-03
515 ==================
516 Almost all people are interested in both build tools and testing tools, so no splitting into working groups.
517
518 So we start with presentations of build tools.
519
520 bootstrap-vz, presented by James.
521 Attach volume, install Debian in chroot, create snapshot, register it.
522
523 We all seem to be doing bootstrapping.
524 But some tools use D-I.
525
526 Sam: taksonomy
527 1. use or not D-I
528 2. whether tool uses VM (creates VM)
529 3. whether tool has customization built-in, or is just script you need to hack
530 bootstrap-vz you can customize, open stack is script you need to hack
531 4. whether we're bootstraping into mounted FS, or create tar which you can import into cloud
532 Jimmy pointed that for GCE we have tar of disk image, so this might not be so distinct division
533
534 Demo is nice, but more important it's to see details
535
536 Adding plugin is easy - but Sam does not fully agree.
537 Manifest knows about subtasks
538 Just adding subtask is not enough - you need to add it to code building manifests.
539 It doesn't have plugin factories.
540 Jimmy remembers that it is not to hard to add plugins to GCE.
541
542 Plugins: phases, and dependencies.
543
544 Some strange dependencies.
545 It's fixable, but not to good for not-cloud-team-member person who wants to customize image.
546
547 Commands plugin - to run command in chroot
548 Copy files plugin
549
550 Not good enough documentation for end user
551 So we need proper documentation for users, not only for developers
552
553 Tool to build images needs to do it correctly (?)
554 Legal, needs to work, needs to be correct
555 Legal - for "Official" Debian images
556
557 If tool needs something from outside of the archive, it might prevent us from calling "official"
558
559 ANI on EC2 - Advanced Networking. It's not in archive. So Debian is not on par with other operating systems. So we're telling users to use other OS.
560
561 Discussion about whether we want something perfect, but taking long time, or just working and then build on that.
562
563 Tools can be in main if they can be used to build DFSG-free image. If users want to use it to build non-free images, that's their choice.
564
565 Going back to tools.
566 Martin - bootstrap-vz does not give proper error messages, but Python backtrace.
567
568 Manoj: error handling and documentation as two really important areas of improvement.
569 man page is incoherent. Code is better documentation that documentation.
570 Marcin: there is package with documentation
571
572 Thomas: tool is in Python. Do our target audience need to know Python?
573 Mixture of Python and shell commands (to apt)
574 Manifests: config data
575 But some configuration is in the Python code (e.g. ntp servers). Also lists of packages to install is hardcoded. And usage of pip to install something in plugin.
576 Repeated things: user name, etc.
577 No templates or inheritance of manifests
578 It has tests.
579 OO-overhead
580 Jimmy: it was rewritten from shell to Python to allow for extensability
581 Plugins need to be in Python
582
583 Let's try to understand reason behind critisism.
584 bootstrap-vz is hard to audit because of this mixture of code and configuration. Changing cloud provider changes list of installed packages. And those packages are not in the manifest but in code, sometimes in some hidden plugin.
585
586 Providers make some assumptions, e.g. EC2 assumes that we are building on EC2 instances.
587
588 We need to make sure that all above comments are upheld by any tool.
589
590
591 Tool we choose should be team-maintaned. Not a single person.
592 Anders is not active upstream (formally is) and maybe we'll (cloud team) need to become upstream.
593
594 Currently bootstrap-vz is used by AWS, GCE, Oracle
595
596 Good thing (Sam): it has plugins.
597 It undestands that image creation is complex and needs to be extensible.
598 Simple tool is atractive, but may be limiting in the future.
599
600 Manoj. End user perspective. Debian builds. Build machines must behave the same no matter which backend is behind them. They do not need to be bit-by-bit identical but should be as similar as possible.
601 Lack of root manifest. Might not be so bad - in theory it would be the only file we need to look at for auditing.
602
603 Martin: bootstrap-vz is lot of magic. James - more modularity might be solution to that.
604 Martin: chroot is more familiar. shell scripts are more familiar for admins
605 shell scripts (based on open stack) was fastest way to build images on Azure
606
607 Shell scripts are not really extensible.
608
609 Shell scripts need to be hacked to have different images, providers, etc.
610 zigo's purpose (according to Sam) is to require hacking for extensibility.
611 Tool needs to provide hooks to change behaviour.
612
613 It's easy to audit. But might not be good for customization.
614
615 Image should be useful for the ordinary user.
616 Reasonalble manifest file.
617 Full log.
618 Do we want to have log of generated images?
619 Let's publish log along the link to the image.
620 We want to have disk image at "images.debian.org" and log near this.
621 User can download such an image and use it. At the same time we can also upload this to appropriate cloud provider.
622
623 Discussion regarding managing credentials. Credentials on Debian machines? Long-term credentials are risky.
624 Building of images - on Debian infrastructure.
625 Testing - as well. Preferably automated.
626 Upload - by human, after verifying tests results.
627
628 Plugin can do anything - they are code.
629 You need to attach them in appropriate time-slot.
630 But there is no uniformity
631
632 bootstrap-vz is state machine. It builds graph of plugins and travels along it.
633
634 FAI - is it appropriate tool?
635 http://fai-project.org/
636 presentation of FAI by Thomas
637 Config spaces and classes.
638 Class - distribution (Debian, CentOS).
639 Priority of classes.
640 Solution for inheritance and extensability for "manifests"
641 FAIBASE - all machines (images) will inherit this information
642
643 Task - define classes.
644
645 Ability to have human-generated cache (basefile)
646 Pre-build image, serving as basis for many images
647 Sometimes it uses all matching classes, sometimes one with the highest priority
648
649 Customization scripts.
650
651 fai-diskimage -v -u X -S size -c CLASS,CLASS /disk/file.raw
652 fai-kvm -Vu 5 /disk/file.raw
653
654 Advance partitioning
655 Does not use kpartx - there is no need for now.
656
657 It runs as root.
658
659 DebConf preseeding
660
661 Config files, dropped into target images? More like used by various scripts to steer them.
662
663 Mature code, with good documentation.
664
665 Packages installation - via apt, aptitude, yum - configurable
666 Ability to use shell variables there
667 Logical joining of classes - class might be dependent on other class
668
669 Some of the configuration depends on used tools - e.g. for installing or removing packages.
670
671 Hooks directory.
672 Priority of the classes.
673 Configuration from command line, database, other sources. Priorities of classes
674
675 ainsl instead of echo >>
676 Templates for files.
677 Tree to be injected into target images
678
679 FAI - shell
680 partitioning and packages maintenance - in Perl
681
682 Replacing of tasks by some other scripts
683
684 2 abstraction levels.
685 Separation of code and configuration.
686
687 Classes created by advanced users, used by ordinary users
688
689 FAI can do more than disk images.
690 Config files instead of API (?)
691
692 You need to grok config space - takes about 1h.
693 Then you can use FAI.
694
695 Update to new Debian release takes few hours. You just need to make few builds to test all the changes.
696
697 Decision regarding number of created and/or uses classes.
698
699 Do you need to read the code to grasp what it's doing. Hard to start.
700 Risk of debugging shell script
701
702 Test suite - task called test. Not much - grepping though the logs.
703 No regression tests.
704 Manual testing of images.
705
706 FAI - just classes for cloud. User can also create their own classes.
707 Volume resize?
708
709 Not changing anything for Jessie. Potential new tool - only for Stretch
710
711 Sam should be able to provide deeper feedback by the end of next week.
712
713 Variables for shell scripts used in scripts
714
715 No dry-run. Also configuration is split among many files, and usully we use many classes. So we need to run FAI with appropriate options and check if it runs sucessfully.
716
717 UEFI and GPT - was not tested for now.
718
719 ------------------------------------------
720 vmdebootstrap. Great test suite.
721 Command line - a bit similar to pbuilder(distribution, mirror, architecture)
722 Shell script.
723 Only one shell script. It does not use apt as package resolver, but debootstrap. A bit problematic with more sophisticated dependency graphs.
724
725 It cannot be used alone as the tool for building images. Might be used as a basis, with conjuntion with other scripts.
726 Sam wrote Python3 library to help with that.
727 Need more that one tool. More code than configuration.
728 No correct API of debootstrap.
729
730 Customizations is neccessary for image builder. debootstrap provides no customization.
731
732 Test (yarn). Can we extract it? yarns have meta language.
733
734 Everything needs to be provided as command line parameters. Config file - is just shell script wrapping it.
735
736 Tools for foreign architectures.
737 vmdebootrap is good for that. --foreign; uses qemu
738 We'll probably use VM. Not fast, but doable.
739
740 vmdebootstrap authors might not want to extend it to suit our needs - they want to keep it simple. We might then need to have wrapper - but this will mean that we need to write much code, and some working around limitations.
741
742 vmdebootstrap is ill-suited, but its test suite is great.
743
744 Auditability is non-existent in bootstrap.
745
746 FAI - categories for users
747
748 Do we want to inspect what was installed, or inspect what would be installed? We'll need the list of what was installed (dependencies and alternatives), but sometimes we might want to know what will happen.
749
750 People are opened to evaluating FAI.
751
752 ... keysigning ...
753
754 cloud-init
755 ==========
756
757 0.7.8-1 uploaded, currently in unstable and testing. Fixed a few bugs, including fragile test suite
758 upstream are Ubuntu folks, major issue with CLA :-(
759 Written in python
760
761 Google don't like cloud-init as it's slow (adds 5s to boot time)
762 there's a forked rewrite in Go which is much faster
763 there was talk acout a re-architecture/rewrite, but no visible progress
764
765 ~20k loc of python
766 It's object oriented, which might add overhead
767
768 It's good for uniformity between distributions - at least in theory.
769 But there are different versions and forks in distributions, so it's not always true.
770 Quite a lot of patches from various folks, not pushed upstream (possibly because of CLA)
771
772 GCE is not eager to add cloud-init, at least until there is uniformity.
773 Upstream does not really care.
774
775 Time to fork?
776 Should we use the Go version? Maintained by CoreOS. Could be a problem :-(
777 Do we want to have Go-based program in base? Compiled but statically linked. But there is possibility to dynamically linked - only for AMD64.
778 Its upstream accepts patches.
779 Might be dying upstream, they're pushing for "ignition" instead of cloud-init probably
780
781 It's for configuration, but has much functionality. Any replacement should offer all functionality.
782
783 ACTION: Bastian, Marcin and Jimmy volunteer to be a new upstream team for cloud-init in case it comes to that.
784 ACTION: Various cloud platform folks will talk to Scott about moving to this new upstream to work around the CLA that's blocking people.
785
786 Open bug for release asking why we need cloud-init in the next point release.
787 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=789214
788
789 test-suites
790 ===========
791
792 What do we want to test? Check that things match our policies. What are they?
793
794 (see lists of simple tests further up)
795
796 How do we manage tests? We build images and do we wait for cloud providers to run their tests?
797 AWS: 2 sets of images. Our images (in Debian account) and ones in marketplace. Marketplace ones might have more tests.
798
799 We should get information when cloud provider's tests fails - and then add such case to our test suite. We might not get exact source code of test, but we should get at least description to be able to reproduce it.
800
801 Stretch - do we rely on cloud providers for running tests?
802
803 2 things: tests and framework to run them.
804
805 Framework will depend on cloud provider.
806 Tests should test image in different variants - on different machine types, with various disk options, etc.
807
808 More details in the second document.
809
810 For test suite we'll need ability to programatically register image, start instance, etc.
811 For this we'll need SDK or API access from chosen language.
812 For AWS we have libraries, e.g. Boto.
813 For Google we don't have SDK in Debian. There is Apache libcloud, or Google provided projects on github: https://github.com/GoogleCloudPlatform
814 Google SDK, for which we have RFP, is command-line tools, not API access library: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759578
815
816 Discussion about automatic updates.
817 Non-https may leave users vulnerable to analysis which packages they have installed (might be problematic in some jurisdictions).
818
819 Mostly positive responses to proposal on debian-devel@. Problems with upgrading services. We need to work on better restarting them.
820
821 Rebuilding and customization
822 ----------------------------
823 We want to allow for it.
824
825 Versions and updates (again)
826 ============================
827 We'll build images for stable and point releases. After important fixes we should rebuild - but this might be problematic. On AWS we need to give AMI ID so it will be problematic. Not only security updates, but also feature updates for agents and/or SDKs. So we don't need to keep them in lockstep, but then we need to provide detailed changelog - to let users know why the difference in timestamps.
828
829 We want to update kernels. Feature releases by updates make more probable for things to break. Also - having dozens of instances just spinning updating may mean measurable costs for users.
830
831 We do once a weekly builds for testing. Weekly and monthly retention, to avoid having to many images.
832 Should we publish them? Different providers might have different meaning of what it means to have image published.
833 When we'll have better testing and building automation, it'll make easier to users to build images.
834
835 Tools for tranforming images to format acceptable by cloud provider. Qemu should be able to deal with most of them.
836
837 Current and future architectures
838 ================================
839
840 amd64. There are some ARM clouds. IBM has Power cloud. Linaro has ARM64 images for developers. Steve promises to have Open Stack ARM64 cloud before Stretch.
841
842 Currently there is not many architectures, but there might be in the future. But as long as we have generic tooling (and Debian has many architectures) we should be able to deal with it.
843
844 ARM64 - UEFI with grub-uefi should do.
845
846 Do we want to have multi-arch enabled by default? IT should be customizable, i.e. users should be able to generate such an image. Users are not asking providers for 32-bit support.
847
848 Maybe we might mirror 32-bit repositories.
849
850 We might want to have some ? policy? about sunsetting some of the architectures. 32-bit are getting less relevant.
851
852 Human language support and localised images
853 -------------------------------------------
854
855 By default we're using "C". How do we allow for users to switch language e.g. through cloud-init. We have quite good language support, but do we need to have it in the cloud.
856
857 Do current images have all locales installed?
858
859 At least some of the cloud providers provide non-English UI (web frontend).
860
861 Amazon Linux has localizations and ability to change language through cloud-init. It's just one image with many different locales. Ubuntu is English-only.
862
863 ACTION: James to ask on the mailing list(s) if anybody wants more languages for the cloud image(s).
864
865 Other things to consider - which mirror to use? Needed for users in China, for example.
866
867 Sam proposal: ask few questions during first login. It'll break automation, but might be useful for some (really specific) needs. Not by default! Is it worth engineering effort? (Probably not)
868
869 Meeting 2016-11-04
870 ==================
871
872 Supporting services (for platforms, mirrors, finding things)
873 ============================================================
874
875 Mirrors
876 -------
877
878 the httpredirect mirror httpredir.debian.org is depricated.
879 When somebody has 1000s of instances and we have autoupdate, we'd like to avoid killing any mirror
880 Zach: external mirrors are often faster.
881 But some of the cloud providers want to keep as much traffic inside their network as possible.
882 Azure has internal mirror network. 25TB of storage; pushed from official mirror
883 CDN for Amazon; headers with different expiry to deal with different refreshment needs (e.g. packages themselves (*.debs), Packages files) Ordinary mirror and security mirror
884
885 Our solution will have lowest maintenance possible. There is not enough manpower.
886
887 James tried to S3 - it was really slow to populate this (6h). That's why he moved into CDN.
888 It used to require instances to rewrite headers but now Cloud Front can set TTL (I'm not sure I understood this part correctly)
889
890 Google Cloud CDN can serve content from inside GCE. But we could have instance with redirect.
891
892 Disk space is not concern. Low maintenance and monitoring are priorities.
893 Tracefile of the mirror should be monitored to see whether we're in sync.
894 Official mirror script updates it last so it can serve as a canary
895
896 Official mirrors - 4 pushes a day
897
898 Signature cannot be older than 10 days - so maximum frequency is 7 days
899
900 deb.debian.org - backed by 2 commercial CDNs (Fastly and cloudfront/Amazon). Stretch apt can directly use CDN behind it, without need to redirect (as needed by older apt)
901 Fastly has peering connection to GCE
902 Bastian can pass on documentation he has.
903
904 Martin shows traffic to ftp.debian.org; it has 10G connection on university network
905 200Mbps on average
906
907 James shares his setup of Apache for redirects and expiry times
908 AWS - 500 requests per minute to the interception header host; 1TB per day
909 We have details which files are requested the most - so we could get something from those statistics
910
911 CDN with HTTPS enabled. security-cdn certificate
912
913 After Google has set CDN, it'll be integrated into our network
914
915 mismatch Hashsum; we need small TTLs to avoid clients getting stale index files
916 It'll dissapear with Stretch apt.
917
918 Finding things
919 --------------
920 Ubuntu image finder: cloud-images.ubuntu.com
921 Home page with links to all images of releases. They also have manifests
922 For various architectures: amd64, IBM Z, arm32
923
924 Link to AWS goes directly to launch wizard - quite useful.
925 On Oracle it goes to market place - but it might be because we were not logged in to Oracle cloud
926
927 We should also provide JSON with images so users can automatise work on our images.
928
929 Martin: Ask Colin Watson if the code for Ubuntu image locator (cloud-images) is available. or Martin will write it!
930
931 What users expect from image finder page. List of cloud providers? Distributions versions? Problem - filters are on the bottom while they should be at the top.
932 Separate page with stable and daily images.
933 Logos on the top to make it easier for users to see what do we offer.
934
935 Support Debian in Juju?
936
937 Should we provide base files, pregenerated when we build images? It'll make life easier for users not on Debian systems.
938
939 == Better handling/publishing/advertising of cloud images by Debian
940
941 Some nice! web pages for showing our images. ( example for parsing json files: https://msdn.microsoft.com/library/cc836466(v=vs.94).aspx )
942
943 What's more than image finder?
944
945 James sends signed emails with AMI IDs. But anybody can edit wiki and change IDs to something malicious. Should we lock wiki page?
946 Image finder should help here.
947
948 Register cloud.debian.org. Maybe debian.cloud? (i.e. .cloud TLD). It's reserved, we'd need to speak with SPI for trademark issues.
949
950 === Getting updated packages into Debian - stable updates, -updates, backports
951
952 How does clamav get on?
953 In volatile times it was pushed through without much problems.
954
955 Many paths. jessie, jessie-proposed-updates, something else?
956 stable-updates
957 proposed-updates
958
959 Should we (team) salvage some of relevant packages, like python-boto, boto3?
960
961 there is python-libcloud, and it seems to be quite recent
962
963 There is Azure CLI in NEW --> contact maintainer to join the debian cloud team and set maintainer to the c, including related python-azure, python-azure-storage
964
965 Action item (serpent): contact *boto* maintainers asking about update before freeze, backporting, maybe salvaging it.
966
967 === Website changes (better promote Debian Cloud images)
968
969 Needs to happen :-)
970 James will try to do something with it after getting access
971 Manoj offers to help too
972
973 == AOB
974
975 We go through cloud-related bugs in BTS.
976 Add locales-all to the list of packages we install by default
977
978 ACTION: Sledge to write up policy for official images and post it on the website (#706052)
979
980 What do we do about things like non-free GPU drivers and other drivers that won't go upstream?
981 We *could* add non-free unofficial images too, but definitely make it easier for people
982 to build their own images.
983
984 Resolved - do the extra non-free images, and make sure people can find them:
985 * appropriate warnings that non-free is bad
986 * *NOT* directly in the same image search area, etc., but maybe a second one which is linked
987
988 == Going to the computer museum
989
990 Yay!
Attached Files
To refer to attachments on a page, use attachment:filename, as shown below in the list of files. Do NOT use the URL of the [get] link, since this is subject to change and can break easily.You are not allowed to attach a file to this page.