Differences between revisions 8 and 9
Revision 8 as of 2014-03-06 18:58:26
Size: 10707
Editor: wookey
Comment:
Revision 9 as of 2014-03-06 19:08:50
Size: 10720
Editor: wookey
Comment:
Deletions are marked like this. Additions are marked like this.
Line 56: Line 56:
* (as root) Create buildd user  * (as root) Create buildd user
Line 61: Line 61:
* (as buildd) Generate a SSH public/private key pair of type RSA in the default location without a passphrase. As the buildd will use SSH to talk to wanna-build, the public part needs to be passed on to the wanna-build administrators.  * (as buildd) Generate a SSH public/private key pair of type RSA in the default location without a passphrase. As the buildd will use SSH to talk to wanna-build, the public part needs to be passed on to the wanna-build administrators.
Line 66: Line 66:
* (as root) Point at special buildd.debian.org versions of buildd/sbuild tools  * (as root) Point at special buildd.debian.org versions of buildd/sbuild tools
Line 73: Line 73:
* (as buildd) set up buildd directories  * (as buildd) set up buildd directories
Line 79: Line 79:
* (as root) install tools  * (as root) install tools
Line 84: Line 84:
* (as buildd) .forward must be filled with "|/usr/bin/buildd-mail" to set up the buildd to handle log reply mails.  * (as buildd) .forward must be filled with "|/usr/bin/buildd-mail" to set up the buildd to handle log reply mails.
Line 89: Line 89:
* (as root) Add buildd user to sbuild group  * (as root) Add buildd user to sbuild group
Line 95: Line 95:
* Remember to log in again as buildd at this point to gain new group

* (as buildd) generate sbuild keys
 * Remember to log in again as buildd at this point to gain new group

 * (as buildd) generate sbuild keys
Line 102: Line 102:
* If you have lvm available then set up LVM snapshots:  * If you have lvm available then set up LVM snapshots:
Line 218: Line 218:
* dupload_archive_name should match stanza in dupload.conf
* sign_with is the generated buildd gpg key to be used for signing uploads
* wanna_build_db_user should match user part of email used in gpg key
 * dupload_archive_name should match stanza in dupload.conf
 * sign_with is the generated buildd gpg key to be used for signing uploads
 * wanna_build_db_user should match user part of email used in gpg key

Debian Buildd Setup

This page describes setting up a new buildd. This can either be as part of the official Debian buildd network, with the machine run by DSA, or as part of the debian-ports 'unoffical' buildd network for new/old/not-released ports, where the admin is not by DSA.

Mostly because DSA has particular admin requirements some of the setup is different. See the relevant section.

This documentation supercedes https://buildd.debian.org/docs/buildd-setup.txt

Overview

The buildd connects over ssh to a wanna-build instance (normally buildd.debian.org (official) or buildd.debian-ports.org (unofficial).

The buildd is started by a cron job every half hour (which does nothing ifd the buildd is already running).

Having connected and got a list of things to build it builds one, then emails logs to buildd.debian.org, and duploads the binaries.

Emails can arrive during this time to say that a build has been superceded and should be aborted.

You can manually ssh to the wanna-build machine and run commands to affect the database. Any DD can connect to buildd.debian.org and use wanna-build command to examine the database. She can only make changes if she is in the right group.

Considerations

It is much easier to run a buildd using stable. The tools are built for that, external admin people (DSA) expect it, and stability is good. Even if you have a new architecture it is a good idea to run stable on the base machine, and only use unstable with the recently-bootstrapped stuff in the build chroot if you can. Not all architectures have this option of course, and you may have to run unstable <new-arch> on the bare machine too. The problem here is that you may not have everything you need (at least sbuild, buildd, schroot, MTA, gnupg, dupload) built yet, and changes over time (general unstable breakage, new breakage in your new arch, ABI changes etc) will cause whoever is administering the machine some hassle. It will work like this though so ultimately the choice is yours. Once your architecture is accepted into testing it is best to use the current codename for the testing suite so that the machine starts using stable after a release.

Admin

To be adopted by DSA a machine must have these things

  • <list>

To be used as a buildd the machine needs:

Minimum:

  • To be able to ssh out
  • To send and receive email
  • Ability for buildd admin to ssh in and get root access

Preferable:

  • Remote serial access and power control
  • Ability to ssh in for remote access

Setup

debian.org Buildd setup

Much is now done by puppet. You will need to do the following before handing over to DSA:

Debian-ports Buildd Setup

  • (as root) Create buildd user

adduser buildd
  • (as buildd) Generate a SSH public/private key pair of type RSA in the default location without a passphrase. As the buildd will use SSH to talk to wanna-build, the public part needs to be passed on to the wanna-build administrators.

echo | ssh-keygen -N ''
  • (as root) Point at special buildd.debian.org versions of buildd/sbuild tools

apt-get install apt-transport-https ca-certificates debian-archive-keyring
wget https://buildd.debian.org/apt/archive-key.asc
apt-key add archive-key.asc
  • (as buildd) set up buildd directories

install --directory --mode=2770 --owner=buildd --group=buildd build logs old-logs upload-security
install --directory --mode=2775 --owner=buildd --group=buildd stats stats/graphs upload
  • (as root) install tools

apt-get install sbuild buildd 
  • (as buildd) .forward must be filled with "|/usr/bin/buildd-mail" to set up the buildd to handle log reply mails.

echo '|/usr/bin/buildd-mail' > ~/.forward 
  • (as root) Add buildd user to sbuild group

sbuild-adduser buildd

The above will configure a mail daemon if you haven't already installed one.

  • Remember to log in again as buildd at this point to gain new group
  • (as buildd) generate sbuild keys

sbuild-update --keygen
  • If you have lvm available then set up LVM snapshots:

YOUR_VG="vg0"
lvcreate --size 20G --name buildd-trees $YOUR_VG
install --directory --mode=2700 --owner=buildd --group=buildd ~buildd/build-trees
mkfs.ext3 /dev/$YOUR_VG/buildd-trees
echo "/dev/$YOUR_VG/buildd-trees /home/buildd/build-trees ext3
 rw 0 2" >> /etc/fstab
mount ~buildd/build-trees
/usr/share/sbuild/create-chroot http://ftp.debian.org/debian --arch=arm64 unstable $YOUR_VG 5G

If you have a source of entropy start using it now. (e.g install ekeyd-egd-linux and an ekey, or arrange to use another one over the net).

(As buildd) Create gpg key: Use this script file ('keygen') to get it right (set your own hostname and arch):

GPG_OPTS='--cert-digest-algo SHA256'
gpg --batch --gen-key ${GPG_OPTS} --status-fd 3 3>keygen.log <<EOT
%echo Generating key for ${host} ...
Key-Type: RSA
Key-Usage: sign
Key-Length: 4096
Name-Real: buildd autosigning key ${host}
Name-Email: buildd_${arch}-${host}@buildd.debian-ports.org
Expire-Date: 365d
%commit
EOT

Generate the key with (setting your own hostname and arch): host=turfan arch=arm64 ./keygen

In chroot: For debian-ports buildd use standard buildd package, or delete /etc/schroot/setup.d/99builddsourceslist and instead put in static config:

deb [arch=amd64] http://ftp.debian.org/debian/ unstable main contrib
deb-src http://ftp.debian.org/debian/ unstable main contrib
deb <your bootstrap repo binary deb line>

Configure apt (install-recommends=0, no pdiffs). For speed allow unsafe io (fine on snapshot chroots), and favour .gz over .xz(unless your machine is much faster than your network)

For Debian buildd /etc/schroot/setup.d/99builddsourceslist will set up your sources for different suites, including experimental, backports, security etc.

Set the default mirror(and disable incoming if you are not authorised to use that) in /etc/schroot/conf.buildd:

debian_mirror="http://ftp.cn.debian.org/debian/"
#debian_incoming=no

Set sbuild config (which arch(es) to build, chroot arch, log mailto/from addresses: (changes from defaults)

$build_arch = 'arm64';  (This is chroot arch)
$host_arch = 'arm64'; (this is arch pakcgaes are build for)
$mailfrom = 'buildd_arm64-turfan@turfan.arm64.debian.net';
$mailto = 'logs@buildd.debianports.org';

Should we have this?:

$maintainer_name = 'arm64 build daemon (turfan)';

Install archive key package:

  • For debian-ports: debian-ports-archive-keyring
  • For debian: debian-archive-keyring (debootstrap will have done this)
  • Install archive key for your bootstrap repo (if signed), or set apt to allow unauthenticated packages.

Set up email: Install an MTA (nullmailer, ssmtp if there is a smarthost to deliver through, otherwise exim, or another you prefer).

Ensure the machine can deliver mail

Send .ssh key and gpg key (in a signed mail) to the admins at debian-ports.net.

Once installed check you can log in.

(as root) Configure dupload (in /etc/dupload.cf)

If you are uploading to debian proper there is nothing to do. If you are uploading to debian-ports adding a stanza for debian-ports:

$cfg{'debian-ports'} = {
        fqdn => "ftp.debian-ports.org",
        incoming => "/incoming/",
        dinstall_runs => 1,
};

Set up buildd config (in /etc/buildd/buildd.conf). Confusingly some of these options look a lot like the sbuild.conf ones. (changes from defaults, for a debian-ports buildd)

$build_arch = 'arm64';
$host_arch = 'arm64';
$distributions = [
{
               dist_name => ["unstable"],
               built_architecture => "arm64",
               wanna_build_ssh_host => "buildd.debian-ports.org"
               wanna_build_ssh_user => "buildd_arm64",
               wanna_build_db_name => "",
               wanna_build_db_user => "buildd_arm64-turfan",
               logs_mailed_to => 'logs@buildd.debian-ports.org',
               sign_with => 'E567D7FA',
}
];
   $upload_queues = [
        {
                dupload_local_queue_dir => "upload",
                dupload_archive_name => "debian-ports",
        }
];
  • dupload_archive_name should match stanza in dupload.conf
  • sign_with is the generated buildd gpg key to be used for signing uploads
  • wanna_build_db_user should match user part of email used in gpg key

exmaple files for debian-ports buildd

example .sbuildrc (or /etc/sbuild/sbuild.conf) file:

# THIS IS MANDATORY FOR DEBIAN BUILDDS, OTHERWISE THE LOG SUBJECT IS BROKEN.
$arch = 'sparc64';

$mailfrom = 'buildd on sompek <buildd@sompek.debian.org>';

##
## DPKG-BUILDPACKAGE OPTIONS
##

$maintainer_name='sparc/sparc64 Build Daemon (sompek) <buildd_sparc64-sompek@buildd.debian-ports.org>';
$purge_build_directory="always";
$sbuild_mode = "buildd";

$mime_build_log_mails = 1;
$compress_build_log_mails = 1;

example .builddrc (or /etc/buildd/buildd.conf) file:

# nice level to run sbuild.  Dedicated build daemons should not be
# niced.
#$nice_level = 10;

# time to sleep when idle (between wanna-build --list=needs-build calls)
#$idle_sleep_time = 5*60;

# Should buildd send "Should I build" messages?
$should_build_msgs = 0;

# tell the packages build system to run in parallel
# check the number of available CPUs first
$ENV{'DEB_BUILD_OPTIONS'} = 'parallel=8';

# mail addr of admin
$admin_mail = 'aurelien@aurel32.net';

# mail address where to send statistics summaries
$statistics_mail = $admin_mail;

# log success messages from upload queue daemon?
$log_queued_messages = 1;

# list of distributions that buildd should take packages from
@distributions = (
        {
                dist_name => ["unstable"],
                built_architecture => 'sparc64',

                wanna_build_ssh_host => "buildd.debian-ports.org",
                wanna_build_ssh_user => "buildd_sparc64",
                wanna_build_ssh_socket => 'buildd.debian-ports.org.ssh',
                wanna_build_ssh_options => [],
                wanna_build_db_user => "buildd_sparc64-sompek",

                dupload_local_queue_dir => "upload-debian-ports",

                logs_mailed_to => 'logs@buildd.debian-ports.org',
                sign_with => 'E567D7FA',
        },
);

@upload_queues = (
        {
                dupload_local_queue_dir => "upload-debian-ports",
                dupload_archive_name => "debian-ports",
        },
);

}}}