Debian images - build setup
This is a working document to describe how we build and publish a variety of official (and not-so-official!) images on Debian servers. While it might help as inspiration for other people, if you want to use this setup you'll need to be a Debian Developer and ask for access to one of two groups: debian-cd or cloud-build. This document describes the specifics of the debian-cd setup.
For $reasons, we have two machines in common use for Debian images:
casulana.debian.org is the main build box. It's large, with lots of CPU, memory and fast storage. It's hosted for us at Bytemark in the UK. It's a great machine with good performance for building images, but is not intended to be used for publishing them due to network bandwidth setup.
pettersson.debian.org is the machine that we use for publishing the images. It used to be our big build machine, but casulana is much newer and faster. Pettersson is hosted at Umeå University in Sweden, alongside a group of other machines that are set up for publishing images via HTTP, HTTPS and Bittorrent over their very fast network connection.
Our process is therefore split up into steps:
- Build images on casulana
- Sync the images to pettersson
- Publish from pettersson (and nearby machines)
Building images - the easy way
Making installation images with the debian-cd software is comparatively easy - they do not need any elevated privileges. We build things directly as the debian-cd user. To work on this, you need to log in to casulana and use sudo to become the debian-cd user:
$ sudo -u debian-cd -i bash
NOTE: for this to work, you will need to be in the debian-cd group to have access to the machines, and also you will need to have set up a sudo password for your Debian account - see the web interface at https://db.debian.org/ .
To work on the official debian-cd images, change directory into /home/debian-cd/build.$RELEASE. In this directory you'll find scripts and config files that debian-cd uses, and also a checkout of the debian-cd code to match the release. The scripts and config here may be found in the setup git repo.
TODO: add more information about running debian-cd
To run a weekly build of all our normal images and publish them, simply run the cronjob.weekly script here:
What the script does can be tweaked by setting flags in the environment, e.g.:
debian-cd@casulana:~/build.bullseye$ NOLIVE=1 ./cronjob.weekly
will run the normal set of installer image builds, but will not build live images.
Running a debian-cd release build (major releases, point releases, d-i alpha/RC releases) is triggered by adding RELEASE_BUILD=$release_name here, e.g. for the last buster point release:
debian-cd@casulana:~/build.buster$ RELEASE_BUILD=buster_release ./cronjob.weekly
This RELEASE_BUILD setting changes behaviour in a few key ways - look at the script for more information ehere.
There is a similar cronjob.daily script that also runs regularly, just more frequently and with a smaller set of images.
Each script does its job (too much to go into in detail here) and pushes all the output (images, log files, etc.) into a subdirectory under /home/debian-cd/publish.
Building images - the hard(er) way
Some build processes (like live builds) can't be easily run without privileges. The normal reason for this is simple - you need to be root to run mknod to create device nodes for a working root filesystem. This is why (for example) debootstrap needs root access.
So, how can we make those things possible on a shared-access machine where the admins (sensibly!) do not want to give out admin privileges? The chosen method on casulana is to run these things inside a VM. For Debian's live images and OpenStack images, we have existing VM images set up and configured for the debian-cd user to run. We have extra scripts that drive those VMs to build things. They're not pretty, but they work OK - see the live-setup repo if you want to know more.
On casulana, this repo is checked out in /home/debian-cd/live , and cronjob.weekly-live will use these scripts to start up VMs, run things in them then shut them down.
TODO: add more information about running things in the VM
Again, scripts push all the output (images, log files, etc.) into a subdirectory under /home/debian-cd/publish.
Syncing to pettersson
/home/debian-cd/build.$release/common.sh contains quite a few handy shell functions that are used by the various build scripts. Of particular interest here are the last two:
rsync_to_pettersson uses rsync to sync a specified directory from casulana to an incoming area on pettersson (the publishing machine).
publish_on_pettersson tells pettersson to do any needed processing on a specified incoming directory, then publish the output. (see the publish_from_casulana script for the receiving end).
Why the incoming directory? We don't want to have half-finished syncs visible. Specifically for installer images, we can use jigdo to transfer only jigdo files across from casulana to pettersson, then recreate ISO images on the far end. For other types of image, we can't do that so we simply rsync the files directly.
Publishing on pettersson
The final location for the output files on pettersson is /mnt/nfs-cdimage. This is directly visible on the web as the content for various sites: https://cdimage.debian.org/cdimage , https://get.debian.org/images/ , etc.
The publishing setup in Umeå can see the contents of that NFS mount directly, and that's how things are served for HTTP. Torrents are picked up separately - see the trigger_seeders script for our interface to that.
Normal daily and weekly builds will be signed with the debian-cd testing key, then pushed into the right place on pettersson automatically via our scripts such that they will be visible. They are signed with the testing key as a basic security measure only. There is no guarantee that they actually work in any way - no human testing has happened.
Conversely, release builds are not published automatically this way. Instead, they are unsigned at this point. They will accumulate in hidden directories as they appear from the sync process. Manual testing is then done, and only once the images team are happy with quality are they then signed with the proper Debian CD release key and published by hand.
We've used something like this setup for a number of years for:
- d-i images built using debian-cd (normal Debian, and later Debian-Edu)
- live images built using live-build, then live-wrapper from Stretch onwards
OpenStack images built using build-openstack-debian-image
OpenStack images post-Buster should now be migrated to the more generic setup of the cloud team. They also use casulana and pettersson like the images team, but with very different configuration, permissions etc.
If you'd like to build another type of Debian image, we can accommodate in one setup or another. Consider how often new images should be published. If you're building small images that are likely to change a lot, then maybe we could build them daily. If you're building larger images, it might be more appropriate to add them to the weekly cycle instead.
We also have rules on what can be in an official Debian image that we publish. First and foremost, it obviously must be totally DFSG-free. Secondly, it should be supported and supportable by the appropriate team in Debian. The images team will help with setting up builds, but do not want to be left supporting everything!
The Debian ports team use our setup to make semi-regular images for various of the architectures in debian-ports.
We can (and do!) also publish some non-free image variants regularly, but they are explicitly tagged as non-free and unofficial, e.g. https://cdimage.debian.org/cdimage/unofficial/non-free/images-including-firmware/ . We consider support to be best-effort only.