Contents
Debian Buildd Setup
This page describes setting up a new buildd. This can either be as part of the official Debian buildd network, with the machine run by DSA (Debian System Administrator), or as part of the debian-ports 'unofficial' buildd network for new/old/not-released ports, where the admin is not by DSA.
Mostly because DSA has particular admin requirements some of the setup is different. See the relevant section.
This documentation supersedes https://buildd.debian.org/docs/buildd-setup.txt
This example uses arm64 because that's what I was doing. A few of the details are affected by the way that the buildds were behind a firewall so didn't have direct SSH or email access. Adjust as needed for your circumstances.
Overview
The buildd connects over SSH to a wanna-build instance (normally buildd.debian.org (official)).
The buildd is started by a cron job every half hour (which does nothing if the buildd is already running).
Having connected and got a list of things to build it builds one, then emails logs to buildd.debian.org, and duploads the binaries.
Emails can arrive during this time to say that a build has been superceded and should be aborted.
You can manually SSH to the wanna-build machine and run commands to affect the database. Any DD can connect to buildd.debian.org and use wanna-build command to examine the database. She can only make changes if she is in the right group.
Considerations
It is much easier to run a buildd using stable. The tools are built for that, external admin people (DSA) expect it, and stability is good. Even if you have a new architecture it is a good idea to run stable on the base machine, and only use unstable with the recently-bootstrapped stuff in the build chroot if you can. Not all architectures have this option of course, and you may have to run unstable <new-arch> on the bare machine too. The problem here is that you may not have everything you need (at least sbuild, buildd, schroot, MTA, gnupg, dupload) built yet, and changes over time (general unstable breakage, new breakage in your new arch, ABI changes etc) will cause whoever is administering the machine some hassle. It will work like this though so ultimately the choice is yours. Once your architecture is accepted into testing it is best to use the current codename for the testing suite so that the machine starts using stable after a release.
Admin
To be adopted by DSA a machine must have these things
<list>
If you plan to become an official buildd then bear these in mind.
To be used as a buildd the machine needs:
Minimum:
- To be able to SSH out (to the wanna-build machine for instructions)
- To send and receive email
- Ability for buildd admin to SSH in and get root access
Preferable:
- Remote serial access and power control
- If the machine should serve as a porterbox, incoming SSH connections are needed.
Setup
debian.org Buildd setup
Much is now done by puppet. You will need to do the following before handing over to DSA:
<fill in here>
DSA's Puppet configuration can be found in https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/tree/master/modules/buildd/manifests.
Debian-ports Buildd Setup
- (as root) Create buildd user, and add to sudo group
adduser buildd adduser buildd sudo
- (as buildd) Generate a SSH public/private key pair of type RSA in the default location without a passphrase. As the buildd will use SSH to talk to wanna-build, the public part needs to be passed on to the wanna-build administrators.
echo | ssh-keygen -N ''
- (as root) Point at special buildd.debian.org versions of buildd/sbuild tools
apt-get install apt-transport-https ca-certificates debian-archive-keyring wget -O - https://buildd.debian.org/apt/archive-key.asc | apt-key add - echo "deb https://buildd.debian.org/apt/ wheezy main" > /etc/apt/sources.list.d/buildd.list
For debian-ports you need the debian-ports archive key:
apt-get update apt-get install debian-ports-archive-keyring
- (as buildd) set up buildd directories
install --directory --mode=2770 --owner=buildd --group=buildd build logs old-logs upload-security install --directory --mode=2775 --owner=buildd --group=buildd stats stats/graphs upload
- (as root) install tools
apt-get install sbuild buildd
- (as buildd) .forward must be filled with "|/usr/bin/buildd-mail" to set up the buildd to handle log reply mails.
echo '|/usr/bin/buildd-mail' > ~/.forward
- (as root) Add buildd user to sbuild group
sbuild-adduser buildd
The above will configure a mail daemon if you haven't already installed one.
- Remember to log in again as buildd at this point to gain new group
- (as buildd) generate sbuild keys
sbuild-update --keygen
Then to have a cup of tea as this tends to take ages on servers
- If you have lvm available then set up LVM snapshots:
(as root)
YOUR_VG="vg0" lvcreate --size 15G --name buildd-trees $YOUR_VG install --directory --mode=2700 --owner=buildd --group=buildd ~buildd/build-trees mkfs.ext3 /dev/$YOUR_VG/build-trees echo "/dev/$YOUR_VG/build-trees /home/buildd/build-trees ext3 rw 0 2" >> /etc/fstab mount ~buildd/build-trees
(as buildd)
sbuild-createchroot --arch=arm64 --keyring=/etc/apt/trusted.gpg unstable $YOUR_VG <url of bootstrap repo>
(for debian-ports, you need to use the debian-ports-archive-keyring.gpg keyring, add --include=debian-ports-archive-keyring, and possibly add --extra-repository="deb http://deb.debian.org/debian-ports unreleased main" if the buildd needs packages from unreleased )
The buildd-sid-$arch device containing the chroot is LVM snapshotted and used to install build-deps during builds The source and actual build files go in the build-trees device (bind-mounted on ~buildd)
Change /etc/schroot/buildd/fstab to mount /home/buildd/build-trees as /build in the build chroot:
# Mount a large scratch space for the build, so we don't use up # space on an LVM snapshot of the chroot itself. /home/buildd/build-trees/ /build none rw,bind 0 0
(If you choose not to use this separate build scratch space you will need to create much larger chroot snapshot above (15G recommended as currently all builds will fit in that space) so that there is room for both build-deps and source+build in the snapshot filesystem)
If you have a source of entropy start using it now. (e.g install ekeyd-egd-linux and an ekey, or arrange to use another one over the net).
(As buildd) Create the OpenPGP key: Use this script file ('keygen') to get it right (set your own hostname and arch):
GPG_OPTS='--cert-digest-algo SHA256' gpg --batch --gen-key ${GPG_OPTS} --status-fd 3 3>keygen.log <<EOT %echo Generating key for ${host} ... %no-protection Key-Type: RSA Key-Usage: sign Key-Length: 4096 Name-Real: buildd autosigning key ${host} Name-Email: buildd_${arch}-${host}@buildd.debian.org Expire-Date: 365d %commit EOT
Generate the key with (setting your own hostname and arch): host=turfan arch=arm64 ./keygen
In chroot: For debian-ports buildd use standard buildd package, or delete /etc/schroot/setup.d/99builddsourceslist and instead put in static config:
deb [arch=amd64] http://ftp.debian.org/debian/ unstable main contrib deb-src http://ftp.debian.org/debian/ unstable main contrib deb <your bootstrap repo binary deb line>
Configure apt (don't install recommends, no pdiffs). For speed allow unsafe io (fine on snapshot chroots), and favour .gz over .xz(unless your machine is much faster than your network)
APT::Install-Recommends 0; Acquire::PDiffs "false"; Acquire::Languages "none"; DPkg::Options {"--force-unsafe-io";}; Acquire::CompressionTypes::Order { "gz"; "bz2"; }
For Debian buildd /etc/schroot/setup.d/99builddsourceslist will set up your sources for different suites, including experimental, backports, security etc. On debian-ports buildds this file does not help, so remove it.
Set the default mirror(and disable incoming if you are not authorised to use that) in /etc/schroot/conf.buildd:
debian_mirror="http://ftp.cn.debian.org/debian/" debian_incoming=no
Set sbuild config (which arch(es) to build, chroot arch, log mailto/from addresses: (changes from defaults)
$build_arch = 'arm64'; $host_arch = 'arm64'; $mailfrom = 'buildd_arm64-turfan@turfan.debian.net'; $mailto = 'logs@buildd.debian.org';
Notes:
- $build_arch is the chroot arch
- $host_arch is the arch packages are built for
- $mailfrom needs to be a deliverable address with valid mx as the receiving end will check
It's important to change the maintainer name on buildd uploads otherwise the real maintainer gets a message every time you do an upload, which they generally don't want. Set this (to match the name/mail used in the in signing key) /etc/sbuild/sbuild.conf:
$maintainer_name = 'buildd autosigning key turfan <buildd_arm64-turfan@buildd.debian.org>';
Install archive key package:
- For debian-ports: debian-ports-archive-keyring
- For debian: debian-archive-keyring (debootstrap will have done this)
- Install archive key for your bootstrap repo (if signed), or set apt to allow unauthenticated packages.
Set up email: Install an MTA (nullmailer, ssmtp if there is a smarthost to deliver through, otherwise exim, or another you prefer).
Ensure the machine can deliver mail, and has valid DNS (at least an mx record)
(as root) Configure dupload (in /etc/dupload.conf)
If you are uploading to debian proper or using dupload version 2.9.3 or newer (present in "buster") there is nothing to do. If you are uploading to debian-ports and using older versions of dupload, it needs adding a stanza for debian-ports:
$cfg{'debian-ports'} = { fqdn => "ports-master.debian.org", incoming => "/incoming/", dinstall_runs => 1, };
Set up buildd config (in /etc/buildd/buildd.conf). Confusingly some of these options look a lot like the sbuild.conf ones. (changes from defaults, for a debian-ports buildd)
$build_arch = 'arm64'; $host_arch = 'arm64'; $distributions = [ { dist_name => ["unstable"], built_architecture => "arm64", wanna_build_ssh_host => "buildd.debian.org" wanna_build_ssh_user => "wb-buildd", wanna_build_db_name => "", wanna_build_db_user => "buildd_arm64-turfan", logs_mailed_to => 'logs@buildd.debian.org', sign_with => '0123456789ABCDEF0123456789ABCDEF01234567', } ]; $upload_queues = [ { dupload_local_queue_dir => "upload", dupload_archive_name => "debian-ports", } ];
- dupload_archive_name should match stanza 'cfg' label in dupload.conf
- sign_with is the generated buildd OpenPGP key to be used for signing uploads
- wanna_build_db_user should match user part of email used in the OpenPGP key
setting up debian.net entry
The buildd machine can be in any domain, but if you don't have a better one putting it in debian.net makes sense. This is easy to do. The process is documented at DebianDotNet.
For example to set an A record for the smarthost/net-visible machine (arm64.debian.net) and an mx record for the buildd behind it (turfan.debian.net) execute this:
gpg --clearsign <<EOF | mail changes@db.debian.org arm64 in a 10.11.12.13 turfan in mx 10 arm64.debian.net. EOF
Note that you cannot set up subdomains in debian.net.
- If you still have a /var/lib/sbuild/build line in /etc/schroot/buildd/fstab you need to change that to /srv/build-trees/ as otherwise it will (quietly!) prevent buildd from doing anything other than quitting.
Registering with buildd admin
Once you know the SSH key, OpenPGP archive-signing key, machine IP and the email to use to mail the machine, you can register the buildd with the wanna-build admins. They will also have to set up the architecture database if it's new.
For Debian ports, send a signed mail to ftpmaster@ports-master.debian.org
Please add this buildd for architecture $arch. OpenPGP key: 56A29A85 OPenPGP uid: buildd autosigning key ${machinename} <buildd_${arch}-${machinename}@buildd.debian.org> Address range SSH logins will come from: email address to mail the buildd: SSH key and OpenPGP key attached
(and don't forget to attach the keys and sign the mail
Get the OpenPGP key ID from gpg --list-secret-keys (as buildd)
Once they have installed this info check the SSH connection works (You don't get a shell, only the ability to run the 'wanna-build' command). You need to do this manually at least once in order to confirm that the machine is OK to SSH to and have it added to 'known hosts'.
ssh ${wanna_build_ssh_user}@${wanna_build_ssh_host} wanna-build -A ${arch} --list bd-uninstallable
Find the correct values in /etc/buildd/buildd.conf. So mine was:
ssh wb-buildd@buildd.debian.org wanna-build -A arm64 --list bd-uninstallable
If this gets you a very long list of packages then you are read to roll.
Running the buildd
buildd is the daemon. It's not started by a normal sysvinit/upstart/systemd script - but by a cron job which runs buildd-watcher every 15 mins to check that buildd is running and if not to restart it.
You can start the daemon manually with 'buildd-watcher'. Better is to run buildd manually in 'no-detach' mode to see what it is doing. If you haven't noticed the bit about "removing /var/lib/sbuild/build line in /etc/schroot/buildd/fstab" above then it will just be complaining about this.
It logs progress in ~buildd/daemon.log
buildd SSHs into the wanna-build machine and runs 'wanna-build <job-status> to get a lits of jobs in the specified status (note: this fails if 'debug' is enabled). Then it works through the list of jobs, uploading the ones that succeeded, and mailing logs. when it runs out of things to build it logs on again for more.
You can stop the daemon cleanly by touching EXIT-DAEMON-PLEASE in the ~buildd directory. Touching NO-DAEMON-PLEASE stops it restarting again.
example files for debian-ports buildd
example .sbuildrc (or /etc/sbuild/sbuild.conf) file:
# THIS IS MANDATORY FOR DEBIAN BUILDDS, OTHERWISE THE LOG SUBJECT IS BROKEN. $arch = 'sparc64'; $mailfrom = 'buildd on sompek <buildd@sompek.debian.org>'; ## ## DPKG-BUILDPACKAGE OPTIONS ## $maintainer_name='sparc/sparc64 Build Daemon (sompek) <buildd_sparc64-sompek@buildd.debian.org>'; $purge_build_directory="always"; $sbuild_mode = "buildd"; $mime_build_log_mails = 1; $compress_build_log_mails = 1;
example .builddrc (or /etc/buildd/buildd.conf) file:
# nice level to run sbuild. Dedicated build daemons should not be # niced. #$nice_level = 10; # time to sleep when idle (between wanna-build --list=needs-build calls) #$idle_sleep_time = 5*60; # Should buildd send "Should I build" messages? $should_build_msgs = 0; # tell the packages build system to run in parallel # check the number of available CPUs first $ENV{'DEB_BUILD_OPTIONS'} = 'parallel=8'; # mail addr of admin $admin_mail = 'aurelien@aurel32.net'; # mail address where to send statistics summaries $statistics_mail = $admin_mail; # log success messages from upload queue daemon? $log_queued_messages = 1; # list of distributions that buildd should take packages from @distributions = ( { dist_name => ["unstable"], built_architecture => 'sparc64', wanna_build_ssh_host => "buildd.debian.org", wanna_build_ssh_user => "wb-buildd", wanna_build_ssh_socket => 'buildd.debian.org.ssh', wanna_build_ssh_options => [], wanna_build_db_user => "buildd_sparc64-sompek", dupload_local_queue_dir => "upload-debian-ports", logs_mailed_to => 'logs@buildd.debian.org', sign_with => '0123456789ABCDEF0123456789ABCDEF01234567', }, ); @upload_queues = ( { dupload_local_queue_dir => "upload-debian-ports", dupload_archive_name => "debian-ports", }, );
Debugging
If nothing much is happening check these things: you can SSH as buildd (you may well get a 'no PTS' error because you may not be able to run a shell, only the wanna-build command). But check it connects OK with no password.
See what ~buildd/deamon.log says
run build on its own. (mine said "should get rid of /var/lib/sbuild/build in /etc/schroot/buildd/fstab (at least for d.o-systems) at /usr/bin/buildd line 41") This did indeed need doing.
Note that if you set $debug = 1; in /etc/buildd/buildd.conf then you will get some output about what buildd is doing, but it will also break the SSH connections to wanna-build so that it always gets 'invalid data format'. So turn $debug off.