Differences between revisions 16 and 66 (spanning 50 versions)
Revision 16 as of 2012-07-18 11:58:11
Size: 4439
Editor: ?Sylvestre Ledru
Comment: buildd
Revision 66 as of 2021-04-14 09:24:55
Size: 6720
Editor: GuillemJover
Comment: Use OpenPGP instead of GPG when referring to keys or the standard
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
This is a step by step tutorial about Build services setup for Wanna-build. This is a step by step tutorial about Build services setup for Wanna-build. Here I assume that you have already gone through the steps of that tutorial.
Line 9: Line 9:

== Setup Buildd and Sbuild ==

=== Install of buildd ===

0. Installing sbuild
In order to build packages, buildd uses sbuild so the first thing we're going to do is install sbuild and its dependencies.
{{{
apt-get install sbuild schroot debootstrap buildd
}}}
sbuild uses schroot to build packages. We'll create a directory for it and configure sbuild to use it.
{{{
mkdir -p /srv/chroot/<distribution>-<architecture>-sbuild
debootstrap wheezy /srv/chroot/<distribution>-<architecture>-sbuild
chown -R buildd:buildd /srv/chroot/<distribution>
}}}
Lets configure the chroot:
{{{
vim /etc/schroot/schroot.conf
}}}
{{{
[wheezy]
description=Debian wheezy (testing)
directory=/srv/chroot/wheezy-amd64-sbuild
users=buildd
groups=buildd
root-groups=buildd
aliases=testing,default
}}}
Configuring sbuild is explained here: http://wiki.debian.org/sbuild
{{{
vim /etc/sbuild/sbuild.conf
sbuild-update --keygen
sbuild-createchroot --keyring=/etc/apt/trusted.gpg --arch=$arch $dist /srv/chroot/$dist-$arch-sbuild $mirror

vim /etc/fstab
mount -a
}}}
We'll now configure the schroot environment. We'll create two users and edit the sources.list file (change <distribution> to the name of the distribution you're going to build for).
{{{
usersfile=/srv/chroot/<distribution>/root/users
echo "#!/bin/sh" > "$usersfile"
echo "groupadd -g $(id -g buildd) buildd" >> "$usersfile"
echo "useradd -u $(id -u buildd) -g $(id -g buildd) buildd" >> "$usersfile"
echo "groupadd -g $(id -g sbuild) sbuild" >> "$usersfile"
echo "useradd -u $(id -u sbuild) -g $(id -g sbuild) sbuild" >> "$usersfile"
chmod a+x "$usersfile"

chroot /srv/chroot/<distribution>
/root/users
vim /etc/apt/sources.list
}}}
1. Disabling sbuild environment filters
{{{
vim /usr/share/perl5/Sbuild/Chroot.pm
}}}
2. Install buildd and sudo
{{{
apt-get install buildd sudo
}}}

3. Patching buildd
# patch buildd.patch

This file:
{{{
/var/lib/buildd/build/SBUILD-GIVEN-BACK
}}}
contains packages that will be skipped because the've already failed. There are cases in which you may want to edit or delete it.

'''Entropy tip'''
If while generating gpg-key there's not enough entropy, here's a command that can be useful for generating entropy:

{{{
find /usr /var /tmp /opt -type f -print0 | xargs -0 cat > /dev/null
}}}

--------
This tutorial follows the installation of [[DebianWannaBuildInfrastructureOnOneServer|Wanna Build]].
Line 91: Line 13:
0. Installing reprepro === Install reprepro ===
Line 95: Line 17:
1. Creating directories for the repository
{{{
mkdir -p /org/ftp.debian.org/ftp/apt/
cd /org/ftp.debian.org/ftp/apt/

===
Create directories for the repository ===
{{{
mkdir -p /srv/ftp.debian.org/ftp/apt/
cd /srv/ftp.debian.org/ftp/apt/
Line 101: Line 24:
2. Configuring reprepro
{{{
cd /org/ftp.debian.org/ftp/apt/conf/

===
Configuring reprepro ===
{{{
cd /srv/ftp.debian.org/ftp/apt/conf/
Line 109: Line 33:
Codename: wheezy Codename: sid
Line 124: Line 48:
Allow: wheezy wheezy-backports Allow: sid
Line 149: Line 73:
touch override.wheezy.main
touch override.wheezy.main.debian-installer
touch override.wheezy.main.src
touch override.sid.main
touch override.sid.main.debian-installer
touch override.sid.main.src
Line 155: Line 79:
3. Generating gpg key === Generate an OpenPGP key ===
'''Tip'''
If while generating an OpenPGP key there's not enough entropy, here's a command that can be useful for generating entropy:

{{{
find /usr /var /tmp /opt -type f -print0 | xargs -0 cat > /dev/null
}}}

Line 160: Line 92:
4. Export the repository
{{{
reprepro -vb /org/ftp.debian.org/ftp/apt export
}}}

5. Adding packages to the repository
=== Export the repository ===
{{{
reprepro -vb /srv/ftp.debian.org/ftp/apt export
}}}

=== Adding packages to the repository ===
Line 168: Line 100:
reprepro includedeb wheezy <deb filename>
reprepro includedsc wheezy <dsc filename>
reprepro includedeb sid <deb filename>
reprepro includedsc sid <dsc filename>
Line 174: Line 106:
reprepro includedeb wheezy openssh-client_5.9p1-5_amd64.deb
reprepro includedsc wheezy eglibc_2.13-32.dsc
}}}
reprepro includedeb sid openssh-client_5.9p1-5_amd64.deb
reprepro includedsc sid eglibc_2.13-32.dsc
}}}


--------

== Setup Buildd and Sbuild ==

=== Install sbuild and buildd ===

We're going to install sbuild, buildd and its dependencies.
{{{
apt-get install sbuild buildd sudo schroot debootstrap
}}}

=== Configure sbuild and schroot ===
Configure sbuild and schroot as described on the [[sbuild]] page.

=== Configure buildd ===
User buildd has to be in the sbuild group for buildd to work properly:

{{{
usermod -a -G sbuild buildd
}}}

Edit the buildd configuration file

{{{
vim /etc/buildd/buildd.conf
}}}

Here is an example configuration file: (note: the bracket types and other punctuation in this example differs from that in the example shipped in the package, this version is right, the example shipped in the package is wrong --plugwash)

{{{
$build_arch = 'amd64';
$distribution = 'sid';
@distributions = (
        {
                dist_name => ['sid'],
                built_architecture => 'amd64',
                wanna_build_ssh_host => "127.0.0.1",
                wanna_build_ssh_user => "buildd",
                wanna_build_db_user => 'wbadm',
                dupload_local_queue_dir => "upload",
                logs_mailed_to => $admin_mail,
        }
);
$host_arch = 'amd64';
$daemon_log_file = '/var/log/buildd.log';
$idle_sleep_time = 300;
$debug = 16;
$verbose = 16;
$admin_mail = 'root';
$log_queued_messages = 1;
$apt_get = 'apt-get';
$mailprog = '/usr/sbin/sendmail';
$ssh = 'ssh';
$sudo = 'sudo';
$upload_queues = [
        {
                dupload_local_queue_dir => "upload",
                dupload_archive_name => "anonymous-ftp-master",
        }
];
$statistics_mail = 'root';
$wanna_build_built_architecture = 'amd64';
$wanna_build_db_user = 'wbadm';
$wanna_build_ssh_host = 'localhost';
$wanna_build_ssh_socket = '/var/buildd.sock';
$wanna_build_ssh_user = 'buildd';
1;
}}}

You have to create a log file for buildd and change its permissions so that the buildd user can write to it.

{{{
touch /var/log/buildd.log
chown buildd:buildd /var/log/buildd.log
}}}

You need to let buildd access the wanna-build database. The simplest way to do this is to generate ssh keys (buildd uses ssh to connect to wanna-build).

{{{
su - buildd
ssh-keygen
}}}

You will be asked some questions. You should enter an empty passphrase (so that buildd can use the key without human intervention). You can leave everything else to the default value. If you used the default values, your key should be in /var/lib/buildd/.ssh/

{{{
cd /var/lib/buildd/.ssh/
cat id_rsa.pub > authorized_keys
ssh 127.0.0.1
}}}

When you enter the last command you'll be asked to accept the key. This is all you have to do here - now buildd can automatically connect to wanna-build. The IP address 127.0.0.1 is the location of wanna-build - I'm assuming it's located on the same machine.

=== Patch sbuild and buildd ===
The latest version of sbuild has a feature to clean up the environment before starting the building process. However there are some bugs associated with it and the simplest way to go around them is to disable the filters from the script. There is a patch that solves this problem and some more. The binary packages sbuild and buildd are located in the same source package. After you have installed the binary packages, you can patch them together using [[attachment:sbuild.patch]]. Apply the patch (assuming that you have it in /root/sbuild.patch):

{{{
cd /usr/share/perl5/
patch -p2 < /root/sbuild.patch
}}}

== Run and test buildd ==
Now, when everything is configured, we have to import the packages from our repository to wanna-build's database:
{{{
su - wbadm
cd /srv/wanna-build/triggers/
./trigger.local
exit
}}}

Start the buildd daemon:
{{{
/etc/init.d/buildd start
}}}

The log file of buildd is /var/log/buildd.log
All the files that buildd creates are stored in the /var/lib/buildd/build/ directory.

'''Tip'''
This file:
{{{
/var/lib/buildd/build/SBUILD-GIVEN-BACK
}}}
contains packages that will be skipped because they've already failed. There are cases in which you may want to edit or delete it.

This is a step by step tutorial about Build services setup for Wanna-build. Here I assume that you have already gone through the steps of that tutorial.

Translation(s): none

(!) ?Discussion


This tutorial follows the installation of Wanna Build.

Setup reprepro for local repository

Install reprepro

apt-get install reprepro

Create directories for the repository

mkdir -p /srv/ftp.debian.org/ftp/apt/
cd /srv/ftp.debian.org/ftp/apt/
mkdir conf dists incoming indices logs pool project

Configuring reprepro

cd /srv/ftp.debian.org/ftp/apt/conf/
vim distributions

Origin: Alexander Pashaliyski
Label: Alexander Pashaliyski
Codename: sid
Architectures: i386 amd64 source
Components: main
Description: Alexander Pashaliyski APT Repository
SignWith: yes

Create and edit incoming conf file:

vim incoming

Name: default
IncomingDir: incoming
TempDir: /tmp
Allow: sid
Cleanup: on_deny on_error

Create and edit options conf file:

vim options

verbose
ask-passphrase
basedir .

Create and edit uploaders conf file:

vim uploaders

allow * by unsigned

Then create some other files

cd ../indices
touch override.sid.main
touch override.sid.main.debian-installer
touch override.sid.main.src
cd ..

Generate an OpenPGP key

Tip If while generating an OpenPGP key there's not enough entropy, here's a command that can be useful for generating entropy:

find /usr /var /tmp /opt -type f -print0 | xargs -0 cat > /dev/null

gpg --gen-key

Export the repository

reprepro -vb /srv/ftp.debian.org/ftp/apt export

Adding packages to the repository

To add a package to the repository, use one of these commands (replace <...> with the name of your package):

reprepro includedeb sid <deb filename>
reprepro includedsc sid <dsc filename>

Example:

reprepro includedeb sid openssh-client_5.9p1-5_amd64.deb
reprepro includedsc sid eglibc_2.13-32.dsc


Setup Buildd and Sbuild

Install sbuild and buildd

We're going to install sbuild, buildd and its dependencies.

apt-get install sbuild buildd sudo schroot debootstrap

Configure sbuild and schroot

Configure sbuild and schroot as described on the sbuild page.

Configure buildd

User buildd has to be in the sbuild group for buildd to work properly:

usermod -a -G sbuild buildd

Edit the buildd configuration file

vim /etc/buildd/buildd.conf

Here is an example configuration file: (note: the bracket types and other punctuation in this example differs from that in the example shipped in the package, this version is right, the example shipped in the package is wrong --plugwash)

$build_arch = 'amd64';
$distribution = 'sid';
@distributions = (
        {
                dist_name => ['sid'],
                built_architecture => 'amd64',
                wanna_build_ssh_host => "127.0.0.1",
                wanna_build_ssh_user => "buildd",
                wanna_build_db_user => 'wbadm',
                dupload_local_queue_dir => "upload",
                logs_mailed_to => $admin_mail,
        }
);
$host_arch = 'amd64';
$daemon_log_file = '/var/log/buildd.log';
$idle_sleep_time = 300;
$debug = 16;
$verbose = 16;
$admin_mail = 'root';
$log_queued_messages = 1;
$apt_get = 'apt-get';
$mailprog = '/usr/sbin/sendmail';
$ssh = 'ssh';
$sudo = 'sudo';
$upload_queues = [
        {
                dupload_local_queue_dir => "upload",
                dupload_archive_name => "anonymous-ftp-master",
        }
];
$statistics_mail = 'root';
$wanna_build_built_architecture = 'amd64';
$wanna_build_db_user = 'wbadm';
$wanna_build_ssh_host = 'localhost';
$wanna_build_ssh_socket = '/var/buildd.sock';
$wanna_build_ssh_user = 'buildd';
1;

You have to create a log file for buildd and change its permissions so that the buildd user can write to it.

touch /var/log/buildd.log
chown buildd:buildd /var/log/buildd.log

You need to let buildd access the wanna-build database. The simplest way to do this is to generate ssh keys (buildd uses ssh to connect to wanna-build).

su - buildd
ssh-keygen

You will be asked some questions. You should enter an empty passphrase (so that buildd can use the key without human intervention). You can leave everything else to the default value. If you used the default values, your key should be in /var/lib/buildd/.ssh/

cd /var/lib/buildd/.ssh/
cat id_rsa.pub > authorized_keys
ssh 127.0.0.1

When you enter the last command you'll be asked to accept the key. This is all you have to do here - now buildd can automatically connect to wanna-build. The IP address 127.0.0.1 is the location of wanna-build - I'm assuming it's located on the same machine.

Patch sbuild and buildd

The latest version of sbuild has a feature to clean up the environment before starting the building process. However there are some bugs associated with it and the simplest way to go around them is to disable the filters from the script. There is a patch that solves this problem and some more. The binary packages sbuild and buildd are located in the same source package. After you have installed the binary packages, you can patch them together using sbuild.patch. Apply the patch (assuming that you have it in /root/sbuild.patch):

cd /usr/share/perl5/
patch -p2 < /root/sbuild.patch

Run and test buildd

Now, when everything is configured, we have to import the packages from our repository to wanna-build's database:

su - wbadm
cd /srv/wanna-build/triggers/
./trigger.local
exit

Start the buildd daemon:

/etc/init.d/buildd start

The log file of buildd is /var/log/buildd.log All the files that buildd creates are stored in the /var/lib/buildd/build/ directory.

Tip This file:

/var/lib/buildd/build/SBUILD-GIVEN-BACK

contains packages that will be skipped because they've already failed. There are cases in which you may want to edit or delete it.