Differences between revisions 21 and 22
Revision 21 as of 2014-04-17 08:51:17
Size: 16930
Editor: wookey
Comment:
Revision 22 as of 2014-04-17 10:03:30
Size: 17160
Editor: wookey
Comment:
Deletions are marked like this. Additions are marked like this.
Line 279: Line 279:
Once you know the ssh key, gpg archive-signing key, machine IP and the email to use to mail the machine, you can register the buildd with the wanna-build admins. They will also have to set up the architecture database if they haven't done that already. Once you know the ssh key, gpg archive-signing key, machine IP and the email to use to mail the machine, you can register the buildd with the wanna-build admins. They will also have to set up the architecture database if it's new.
Line 285: Line 285:
ssh key:
gpg key:
gpg key: 56A29A85
gpg uid: buildd autosigning key ${machinename} <buildd_${arch}-${machinename}@buildd.debian-ports.org>
Line 288: Line 288:
email address to mail the buildd:
}}}
email address to mail the buildd:

ssh key and gpg key attached
}}}
(and don't forget to attach the keys and sign the mail :-)

Get the gpg ID from gpg --list-secret-keys (as buildd)

Debian Buildd Setup

This page describes setting up a new buildd. This can either be as part of the official Debian buildd network, with the machine run by DSA, or as part of the debian-ports 'unoffical' buildd network for new/old/not-released ports, where the admin is not by DSA.

Mostly because DSA has particular admin requirements some of the setup is different. See the relevant section.

This documentation supercedes https://buildd.debian.org/docs/buildd-setup.txt

This example uses arm64 because that's what I was doing. A few of the details are affected by the way that the buildds were behind a firewall so didn't have direct ssh or email access. Adjust as needed for your circumstances.

Overview

The buildd connects over ssh to a wanna-build instance (normally buildd.debian.org (official) or buildd.debian-ports.org (unofficial).

The buildd is started by a cron job every half hour (which does nothing ifd the buildd is already running).

Having connected and got a list of things to build it builds one, then emails logs to buildd.debian.org, and duploads the binaries.

Emails can arrive during this time to say that a build has been superceded and should be aborted.

You can manually ssh to the wanna-build machine and run commands to affect the database. Any DD can connect to buildd.debian.org and use wanna-build command to examine the database. She can only make changes if she is in the right group.

Considerations

It is much easier to run a buildd using stable. The tools are built for that, external admin people (DSA) expect it, and stability is good. Even if you have a new architecture it is a good idea to run stable on the base machine, and only use unstable with the recently-bootstrapped stuff in the build chroot if you can. Not all architectures have this option of course, and you may have to run unstable <new-arch> on the bare machine too. The problem here is that you may not have everything you need (at least sbuild, buildd, schroot, MTA, gnupg, dupload) built yet, and changes over time (general unstable breakage, new breakage in your new arch, ABI changes etc) will cause whoever is administering the machine some hassle. It will work like this though so ultimately the choice is yours. Once your architecture is accepted into testing it is best to use the current codename for the testing suite so that the machine starts using stable after a release.

Admin

To be adopted by DSA a machine must have these things

  • <list>

If you plan to become an official buildd then bear these in mind.

To be used as a buildd the machine needs:

Minimum:

  • To be able to ssh out (to the wanna-build machine for instructions)
  • To send and receive email
  • Ability for buildd admin to ssh in and get root access

Preferable:

  • Remote serial access and power control

Setup

debian.org Buildd setup

Much is now done by puppet. You will need to do the following before handing over to DSA:

<fill in here>

Debian-ports Buildd Setup

  • (as root) Create buildd user, and add to sudo group

adduser buildd
adduser buildd sudo
  • (as buildd) Generate a SSH public/private key pair of type RSA in the default location without a passphrase. As the buildd will use SSH to talk to wanna-build, the public part needs to be passed on to the wanna-build administrators.

echo | ssh-keygen -N ''
  • (as root) Point at special buildd.debian.org versions of buildd/sbuild tools

apt-get install apt-transport-https ca-certificates debian-archive-keyring
wget -O - https://buildd.debian.org/apt/archive-key.asc | apt-key add -
echo "deb https://buildd.debian.org/apt/ wheezy main" > /etc/apt/sources.list.d/buildd.list

For debian-ports you need the debian-ports archive key:

wget -O - http://ftp.debian-ports.org/archive/archive_2014.key | apt-key add -
  • (as buildd) set up buildd directories

install --directory --mode=2770 --owner=buildd --group=buildd build logs old-logs upload-security
install --directory --mode=2775 --owner=buildd --group=buildd stats stats/graphs upload
  • (as root) install tools

apt-get install sbuild buildd 
  • (as buildd) .forward must be filled with "|/usr/bin/buildd-mail" to set up the buildd to handle log reply mails.

echo '|/usr/bin/buildd-mail' > ~/.forward 
  • (as root) Add buildd user to sbuild group

sbuild-adduser buildd

The above will configure a mail daemon if you haven't already installed one.

  • Remember to log in again as buildd at this point to gain new group
  • (as buildd) generate sbuild keys

sbuild-update --keygen

Then to have a cup of tea as this tends to take ages on servers

  • If you have lvm available then set up LVM snapshots:

(as root)

YOUR_VG="vg0"
lvcreate --size 20G --name buildd-trees $YOUR_VG
install --directory --mode=2700 --owner=buildd --group=buildd ~buildd/build-trees
mkfs.ext3 /dev/$YOUR_VG/build-trees
echo "/dev/$YOUR_VG/build-trees /home/buildd/build-trees ext3
 rw 0 2" >> /etc/fstab
mount ~buildd/build-trees

(as buildd)

/usr/share/sbuild/create-chroot <url of bootstrap repo> --arch=arm64 unstable $YOUR_VG 5G

(I had to hack this script a bit to add --keyring=/etc/apt/trusted.gpg to the debootstrap internal invocation, and nobble the lvm creation on all attempts after the first one). Note that if you don't put in a URL above it gets its mirror config from /etc/schroot/conf.buildd. For debian-ports you want to disable debian_incoming, and set a suitable mirror (or point it at your bootstrap repo to get initial new-arch packages from). Mine is:

debian_mirror="http://debian.nctu.edu.tw/debian-ports/"
debian_incoming=no

The buildd-sid-$arch device containing the chroot is LVM snapshotted and used to install build-deps during builds The source and actual build files go in the build-trees device (bind-mounted on ~buildd)

Change /etc/schroot/buildd/fstab to mount /home/buildd/build-trees as /build in the build chroot:

# Mount a large scratch space for the build, so we don't use up                                                 
# space on an LVM snapshot of the chroot itself.                                                                
/home/buildd/build-trees/       /build  none    rw,bind         0       0 

(If you choose not to use this separate build scratch space you will need to create much larger chroot snapshot above (20G recommended) so that there is room for both build-deps and source+build in the snapshot filesystem)

If you have a source of entropy start using it now. (e.g install ekeyd-egd-linux and an ekey, or arrange to use another one over the net).

(As buildd) Create gpg key: Use this script file ('keygen') to get it right (set your own hostname and arch):

GPG_OPTS='--cert-digest-algo SHA256'
gpg --batch --gen-key ${GPG_OPTS} --status-fd 3 3>keygen.log <<EOT
%echo Generating key for ${host} ...
Key-Type: RSA
Key-Usage: sign
Key-Length: 4096
Name-Real: buildd autosigning key ${host}
Name-Email: buildd_${arch}-${host}@buildd.debian-ports.org
Expire-Date: 365d
%commit
EOT

Generate the key with (setting your own hostname and arch): host=turfan arch=arm64 ./keygen

In chroot: For debian-ports buildd use standard buildd package, or delete /etc/schroot/setup.d/99builddsourceslist and instead put in static config:

deb [arch=amd64] http://ftp.debian.org/debian/ unstable main contrib
deb-src http://ftp.debian.org/debian/ unstable main contrib
deb <your bootstrap repo binary deb line>

Configure apt (don't install recommends, no pdiffs). For speed allow unsafe io (fine on snapshot chroots), and favour .gz over .xz(unless your machine is much faster than your network)

APT::Install-Recommends 0;
Acquire::PDiffs "false";
Acquire::Languages "none"; 
DPkg::Options {"--force-unsafe-io";};
Acquire::CompressionTypes::Order { "gz"; "bz2"; }

For Debian buildd /etc/schroot/setup.d/99builddsourceslist will set up your sources for different suites, including experimental, backports, security etc. On debian-ports buildds this file does not help, so remove it.

Set the default mirror(and disable incoming if you are not authorised to use that) in /etc/schroot/conf.buildd:

debian_mirror="http://ftp.cn.debian.org/debian/"
debian_incoming=no

Set sbuild config (which arch(es) to build, chroot arch, log mailto/from addresses: (changes from defaults)

$build_arch = 'arm64';  
$host_arch = 'arm64';
$mailfrom = 'buildd_arm64-turfan@turfan.debian.net';
$mailto = 'logs@buildd.debian-ports.org';

Notes:

  • $build_arch is the chroot arch
  • $host_arch is the arch packages are built for
  • $mailfrom needs to be a deliverable address with valid mx as the receiving end will check

It's important to change the maintainer name on buildd uploads otherwise the real maintainer gets a message every time you do an upload, which they generally don't want. Set this (to match the name/mail used in the in signing key) /etc/sbuild/sbuild.conf:

$maintainer_name = 'buildd autosigning key turfan <buildd_arm64-turfan@buildd.debian-ports.org>';

Install archive key package:

  • For debian-ports: debian-ports-archive-keyring
  • For debian: debian-archive-keyring (debootstrap will have done this)
  • Install archive key for your bootstrap repo (if signed), or set apt to allow unauthenticated packages.

Set up email: Install an MTA (nullmailer, ssmtp if there is a smarthost to deliver through, otherwise exim, or another you prefer).

Ensure the machine can deliver mail, and has valid DNS (at least an mx record)

(as root) Configure dupload (in /etc/dupload.cf)

If you are uploading to debian proper there is nothing to do. If you are uploading to debian-ports adding a stanza for debian-ports:

$cfg{'debian-ports'} = {
        fqdn => "ftp.debian-ports.org",
        incoming => "/incoming/",
        dinstall_runs => 1,
};

Set up buildd config (in /etc/buildd/buildd.conf). Confusingly some of these options look a lot like the sbuild.conf ones. (changes from defaults, for a debian-ports buildd)

$build_arch = 'arm64';
$host_arch = 'arm64';
$distributions = [
{
               dist_name => ["unstable"],
               built_architecture => "arm64",
               wanna_build_ssh_host => "buildd.debian-ports.org"
               wanna_build_ssh_user => "buildd_arm64",
               wanna_build_db_name => "",
               wanna_build_db_user => "buildd_arm64-turfan",
               logs_mailed_to => 'logs@buildd.debian-ports.org',
               sign_with => 'E567D7FA',
}
];
   $upload_queues = [
        {
                dupload_local_queue_dir => "upload",
                dupload_archive_name => "debian-ports",
        }
];
  • dupload_archive_name should match stanza 'cfg' label in dupload.conf
  • sign_with is the generated buildd gpg key to be used for signing uploads
  • wanna_build_db_user should match user part of email used in gpg key

setting up debian.net entry

The buildd machine can be in any domain, but if you don't have a better one putting it in debian.net makes sense. This is easy to do. The process is documented at DebianDotNet.

For example to set an A record for the smarthost/net-visible machine (arm64.debian.net) and an mx record for the buildd behind it (turfan.debian.net) execute this:

gpg --clearsign <<EOF | mail changes@db.debian.org
arm64 in a 10.11.12.13
turfan in mx 10 arm64.debian.net.
EOF

Note that you cannot set up subdomains in debian.net.

  • If you still have a /var/lib/sbuild/build line in /etc/schroot/buildd/fstab you need to change that to /srv/build-trees/ as otherwise it will (quietly!) prevent buildd from doing anything other than quitting.

Registering with buildd admin

Once you know the ssh key, gpg archive-signing key, machine IP and the email to use to mail the machine, you can register the buildd with the wanna-build admins. They will also have to set up the architecture database if it's new.

For Debian ports, send a signed mail to admin@debian-ports.org

Please add this buildd for architecture $arch.
gpg key: 56A29A85
gpg uid: buildd autosigning key ${machinename} <buildd_${arch}-${machinename}@buildd.debian-ports.org>
Address range ssh logins will come from:
email address to mail the buildd:

ssh key and gpg key attached

(and don't forget to attach the keys and sign the mail :-)

Get the gpg ID from gpg --list-secret-keys (as buildd)

Once they have installed this info check the ssh connection works (You don't get a shell, only the ability to run the 'wanna-build' command)

ssh ${wanna_build_ssh_user}@${wanna_build_ssh_host} wanna-build -A ${arch} --list  bd-uninstallable

Find the correct values in /etc/buildd/buildd.conf. So mine was:

ssh buildd_arm64@buildd.debian-ports.org wanna-build -A arm64 --list bd-uninstallable

If this gets you a very long list of packages then you are read to roll.

Running the buildd

buildd is the daemon. It's not started by a normal sysvinit/upstart/systemd script - but by a cron job which runs buildd-watcher every 15 mins to check that buildd is running and if not to restart it.

You can start the daemon manually with 'buildd-watcher'. Better is to run buildd manually in 'no-detach' mode to see what it is doing. If you haven't noticed the bit about "removing /var/lib/sbuild/build line in /etc/schroot/buildd/fstab" above then it will just be complaining about this.

It logs progress in ~buildd/daemon.log

buildd sshs into the wanna-build machine and runs 'wanna-build <job-status> to get a lits of jobs in the specified status (note: this fails if 'debug' is enabled). Then it works through the list of jobs, uploading the ones that succeeded, and mailing logs. when it runs out of things to build it logs on again for more.

You can stop the daemon cleanly by touching EXIT-DAEMON-PLEASE in the ~buildd directory. Touching NO-DAEMON-PLEASE stops it restarting again.

example files for debian-ports buildd

example .sbuildrc (or /etc/sbuild/sbuild.conf) file:

# THIS IS MANDATORY FOR DEBIAN BUILDDS, OTHERWISE THE LOG SUBJECT IS BROKEN.
$arch = 'sparc64';

$mailfrom = 'buildd on sompek <buildd@sompek.debian.org>';

##
## DPKG-BUILDPACKAGE OPTIONS
##

$maintainer_name='sparc/sparc64 Build Daemon (sompek) <buildd_sparc64-sompek@buildd.debian-ports.org>';
$purge_build_directory="always";
$sbuild_mode = "buildd";

$mime_build_log_mails = 1;
$compress_build_log_mails = 1;

example .builddrc (or /etc/buildd/buildd.conf) file:

# nice level to run sbuild.  Dedicated build daemons should not be
# niced.
#$nice_level = 10;

# time to sleep when idle (between wanna-build --list=needs-build calls)
#$idle_sleep_time = 5*60;

# Should buildd send "Should I build" messages?
$should_build_msgs = 0;

# tell the packages build system to run in parallel
# check the number of available CPUs first
$ENV{'DEB_BUILD_OPTIONS'} = 'parallel=8';

# mail addr of admin
$admin_mail = 'aurelien@aurel32.net';

# mail address where to send statistics summaries
$statistics_mail = $admin_mail;

# log success messages from upload queue daemon?
$log_queued_messages = 1;

# list of distributions that buildd should take packages from
@distributions = (
        {
                dist_name => ["unstable"],
                built_architecture => 'sparc64',

                wanna_build_ssh_host => "buildd.debian-ports.org",
                wanna_build_ssh_user => "buildd_sparc64",
                wanna_build_ssh_socket => 'buildd.debian-ports.org.ssh',
                wanna_build_ssh_options => [],
                wanna_build_db_user => "buildd_sparc64-sompek",

                dupload_local_queue_dir => "upload-debian-ports",

                logs_mailed_to => 'logs@buildd.debian-ports.org',
                sign_with => 'E567D7FA',
        },
);

@upload_queues = (
        {
                dupload_local_queue_dir => "upload-debian-ports",
                dupload_archive_name => "debian-ports",
        },
);

Debugging

If nothing much is happening check these things: you can ssh as buildd (you may well get a 'no PTS' error because you may not be able to run a shell, only the wanna-build command). But check it connects OK with no password.

See what ~buildd/deamon.log says

run build on its own. (mine said "should get rid of /var/lib/sbuild/build in /etc/schroot/buildd/fstab (at least for d.o-systems) at /usr/bin/buildd line 41") This did indeed need doing.

Note that if you set $debug = 1; in /etc/buildd/buildd.conf then you will get some output about what buildd is doing, but it will also break the ssh connections to wanna-build so that it always gets 'invalid data format'. So turn $debug off.