Differences between revisions 52 and 53
Revision 52 as of 2013-09-05 10:48:00
Size: 17390
Editor: ?EmanueleAina
Comment: Change 'unstable' to 'sid' in the explanatory text too
Revision 53 as of 2013-09-08 16:35:06
Size: 18236
Editor: ?IanCampbell
Comment:
Deletions are marked like this. Additions are marked like this.
Line 328: Line 328:
'''Note by ijc''': Indeed... I think it should be possible to do by using the command-prefix chroot option to run a script within the chroot which sets the necessary variables and then exec's the remainder of its command line. There are a couple of other problems, first `dpkg-architecture -qDEB_HOST_ARCH` is evaluated at setup time and not when the chroot is constructed, so it doesn't work for cross arch (even biarch, e.g. i386 on amd64) chroots, second the cache directory is owned by root but the builds do not run as root. I can't see how to inject either $USER or $HOME into the fstab dynamically.
Line 348: Line 350:

'''Note by ijc''': I expect this doesn't work for similar reasons to why the ccache example is bogus, the chroot-setup-commands are not run as a prefix to the build commands. I expect this would be easyily solved with command-prefix.

Translation(s): none


sbuild

sbuild is used on the official buildd network to build binary packages for all supported architectures. It can also be used by individuals to prepare packages. Alternatives to sbuild are cowbuilder and pbuilder.

This page is intended as a short guide to sbuild. It documents how to set up sbuild and build packages with it, but intentionally does not document all possible options to keep it short and simple.

Setup

To get started so you may build packages for Debian sid, run the following.

   1 sudo apt-get install sbuild
   2 sudo sbuild-update --keygen
   3 sudo sbuild-adduser $LOGNAME
   4 newgrp -
   5 sudo sbuild-createchroot --make-sbuild-tarball=/var/lib/sbuild/sid-amd64.tar.gz sid `mktemp -d` http://ftp.debian.org/debian

Now for a brief explanation on what these commands do.

The first line installs sbuild onto the system.

The second line generates apt keys used internally by sbuild. (If you get an error about lack of entropy, do something else on the system, like browsing the web or running find /usr or so. Or skip down to creating the chroot, and come back to this step.) This only needs to be done once.

The third line will add your username so that it may use the sbuild command. Additional users may be added by running sudo sbuild-adduser USER1 USER2 ....

sbuild-adduser will prompt you to copy the template sbuild configuration in /usr/share/doc/sbuild/examples/example.sbuildrc to each user's ~/.sbuildrc, to be used as their user sbuild configuration. You can customize sbuild settings here, but you usually won't need to customize anything. This should be done once per user.

The fourth line uses newgrp to update the active user group set to include sbuild without the re-login.

The fifth line uses sbuild-createchroot to create a chroot used by sbuild meant for building packages targeting Debian sid main. The chroot is saved as a tarball in /var/lib/sbuild/sid-amd64.tar.gz. The apt repository used is http://ftp.debian.org/debian. This can be changed to use a URL for a local mirror of the Debian archive. You can run this command once per distribution you want, and pass --arch=i386 to create a chroot for a different architecture (the default is your host architecture).

Configuration

If you're setting up sbuild for personal use, instead of as part of a build server, you might want to use the following options in your ~/.sbuildrc. These can also be set on the command line when running sbuild.

   1 $build_arch_all = 1;
   2 $build_source = 1;
   3 $distribution = 'sid';

The $build_arch_all variable will enable building of architecture independent packages by default. Since official build servers are used to build an existing package for a different architecture (e.g., the uploader builds for i386, uploads arch-i386 and arch-all binary packages, and a build server builds arch-amd64 packages), this is off by default. You can also enable this per build by passing -A to sbuild.

The $build_source variable will enable building of source packages by default. Again, on official build servers, this isn't wanted, but for personal use it's generally useful. You can enable this per build with -s.

The $distribution variable will set the distribution to build for as 'sid'. You can set the distribution per build by passing it to the -d option. Be careful not to use -d just to select a specific chroot (use -c for that, see below), as it will override the distribution set in debian/changelog and may lead you to upload a package to a distribution it was not intended for.

Updating

The chroot should be up-to-date before building packages. Use the sbuild-update to perform updates.

First, note the name of the sbuild chroot to be updated. All sbuild chroots built with sbuild-createchroot will have a suffix of '-sbuild' thus to find the names of all sbuild chroots, run the following.

   1 schroot -l | grep sbuild

If you followed the setup instructions above, there should be one chroot named source:sid-$arch-sbuild where $arch is the architecture installed on your machine.

After noting the name of your sbuild chroot, run the following.

   1 sudo sbuild-update -udcar sid-$arch-sbuild

The arguments '-udcar' will tell sbuild-update to run an apt-get update, dist-upgrade, clean, autoclean, and autoremove in the chroot.

You can also pass --apt-update --apt-distupgrade to the individual sbuild invocation to update the temporary copy of the build chroot, but this won't cause any changes to happen in the persistent copy of the chroot (in the .tar.gz file). So if you are building more than once, you should run sbuild-update instead of relying on this.

Building packages

To build a package from the source directory of a debianized package, simply run the following.

   1 sbuild

Alternatively, you may pass in the '.dsc' file of a package generated by dpkg-buildpackage, git buildpackage, and so forth so that it may be built with sbuild. For example, to build sbuild from its '.dsc' file, do the following.

   1 sbuild sbuild_*.dsc

To build packages available from the apt repositories used in the sbuild chroot, pass in a package to sbuild in the format $package_$version. For example, to build sbuild version 0.63.2-1.1, do the following.

   1 sbuild sbuild_0.63.2-1.1

Everything needed to build sbuild version 0.63.2-1.1 will be downloaded from the repositories and will be saved in your current directory after building of the packages is finished.

To build packages that are not yet in the repositories but which have their source package available online, the URL to the source package's '.dsc' can be passed into sbuild. This requires that the system have libwww-perl installed (so that it has the Perl module LWP::UserAgent).

First ensure that libwww-perl is installed.

   1 sudo apt-get install libwww-perl

Now you can build any package from its '.dsc' file, whether it is stored locally on the system, or it is available online. For example, to build sbuild using the '.dsc' file from the Debian archive, do the following.

   1 sbuild http://ftp.debian.org/debian/pool/main/s/sbuild/sbuild_0.63.2-1.1.dsc

Using lintian

The use of lintian with sbuild can greatly aid with increasing the quality of Debian packages. To use lintian with sbuild, first install lintian.

   1 sudo apt-get install lintian

Next, edit your ~/.sbuildrc configuration file. Open ~/.sbuildrc with your favorite text editor and edit the lines with the following variables.

   1 $run_lintian = 1;
   2 $lintian_opts = ['-i', '-I'];

The $run_lintian variable will enable running of lintian after a successful build with sbuild.

The $lintian_opts variable is an array of options to pass to lintian. Here the options are '-i' which will output information about lintian warnings and errors, and '-I' which will output lintian "info" messages.

Using piuparts

The use of piuparts with sbuild is another feature meant to enhance the quality of Debian packages built with sbuild. To use piuparts, first install piuparts.

   1 sudo apt-get install piuparts

Next, edit your ~/.sbuildrc configuration file. Open ~/.sbuildrc with your favorite text editor and edit the lines with the following variables.

   1 $run_piuparts = 1;
   2 $piuparts_opts = ['-b', '/var/lib/sbuild/sid-amd64.tar.gz'];

The $run_piuparts variable will enable running piuparts after a successful build with sbuild.

The $piuparts_opts variable is an array of options to pass to piuparts. Here the options are the "-b <path/to/tarball>" option in piuparts. This will instruct piuparts to use sbuild chroot '/var/lib/sbuild/sid-amd64.tar.gz' as the chroot used in its testing of packages.

Bind mounts

Directories on the local filesystem can be bind mounted and made available in the sbuild chroots. To do this, add an entry in /etc/schroot/sbuild/fstab.

The following example shows how to bind mount the apt archive cache inside an sbuild chroot. This is useful when not using a local mirror of the Debian repository.

   1 sudo echo /var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0 >>/etc/schroot/sbuild/fstab

Customizations of sbuild chroots

Sometimes it is desirable to further customize your sbuild chroot environment. Typical customizations done are installing more packages inside the chroot, modifying /etc/apt/sources.list, and installing custom scripts to be run inside the chroot.

To modify a chroot, start a session for the chroot with the prefix 'source:'. For example, to modify the sid-$arch-sbuild chroot, start a session for the chroot as follows.

   1 sudo sbuild-shell source:sid-$arch-sbuild

This will start a session inside the sid-$arch-sbuild chroot. Any modifications done inside the chroot will be saved upon exiting. Inside this chroot, you may run apt-get commands to install or remove packages as desired. For example, to install ccache, do the following in the chroot session.

   1 apt-get install ccache

Note that you are already root inside a chroot session. Also, sudo would typically not be installed in the chroot anyway.

Making other modifications such as editing /etc/apt/sources.list or adding scripts inside the chroot is best done from outside the chroot session. In order to do this, leave the current chroot session open and start another terminal session. In the other terminal session, find the path used for the existing chroot session as follows.

   1 schroot --info --all-sessions | grep Path

This should output exactly one line and should specify the path of the chroot session. It should look something like this.

  Path                   /var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c

Doing an ls on this directory should resemble a standard listing of the '/' directory.

With the above path, the sources.list file will be in the following path.

/var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

Open the sources.list file at this path with root privileges using your favorite editor. For example, using nano, do the following in your secondary terminal session.

   1 sudo nano /var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

For the graphically inclined, here is an example using kate, a KDE based text editor.

   1 kdesudo kate /var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

Here's another example using gedit

   1 gksudo gedit /var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

Proceed to edit the sources.list file as you see fit, then save.

To add scripts inside the chroot, simply place the scripts in the following directory.

/var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/usr/local/bin

Be sure to make the scripts executable.

   1 sudo chmod a+x /var/lib/schroot/mount/sid-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/usr/local/bin/*

Other modifications may be done to the chroot, either by running commands available within the chroot session, or by running commands with root privileges via the secondary terminal session. Once you are done making modifications to the chroot, simply exit the session. Inside the chroot session, do the following.

   1 exit

After exiting, your modifications will be saved and made available for every new chroot session created afterwards.

External Commands

sbuild supports running external commands at various stages of the build process. This is useful for cases such as running a script inside the chroot after it has been setup. As an example, to run a script in /usr/local/bin/myscript, edit the $external_commands in the sbuild configuration file ~/.sbuildrc as follows.

   1 $external_commands = {
   2                         'post-build-commands' => [],
   3                         'chroot-setup-commands' => ['/usr/local/bin/myscript'],
   4                         'chroot-cleanup-commands' => [],
   5                         'pre-build-commands' => []
   6                       };

sbuild can also translate certain percent escaped keywords for external commands during certain portions of a build. For example, in post build commands, %SBUILD_CHANGES is changed to the path of the '.changes' file for a successfully built package.

Here is an example of adding a post build command to run /usr/local/bin/postbuildscript with %SBUILD_CHANGES as an argument.

   1 $external_commands = {
   2                         'post-build-commands' => ['/usr/local/bin/postbuildscript', '%SBUILD_CHANGES'],
   3                         'chroot-setup-commands' => ['/usr/local/bin/myscript'],
   4                         'chroot-cleanup-commands' => [],
   5                         'pre-build-commands' => []
   6                       };

See the 'EXTERNAL COMMANDS' section of the sbuild man page for more information on external commands.

Using ccache with sbuild

Sometimes you are building a big package and after building (and waiting for too long) you realize that there was a problem in a install file. Using ccache one can avoid to have to wait for that long again for a simple fix.

First, add a bind mount to some directory you want to use for ccache. For example, to add a directory /var/cache/ccache-$arch, do the following.

   1 sudo echo /var/cache/ccache-`dpkg-architecture -qDEB_HOST_ARCH` /var/cache/ccache-`dpkg-architecture -qDEB_HOST_ARCH` none rw,bind 0 0 >>/etc/schroot/sbuild/fstab

Next, perform some modifications to your chroot. Start a chroot session with the 'source:' prefix.

   1 sudo sbuild-shell source:sid-$arch-sbuild

Install ccache inside the chroot

   1 apt-get install ccache

Save the chroot by exiting.

   1 exit

Now add a setup script in /etc/schroot/setup.d so that it may be run upon setting up an sbuild chroot. Let's put the script in /etc/schroot/setup.d/99ccache. Do the following.

   1 sudo touch /etc/schroot/setup.d/99ccache
   2 sudo chmod a+x /etc/schroot/setup.d/99ccache

Now open /etc/schroot/setup.d/99ccache as root with some text editor.

   1 sudo nano /etc/schroot/setup.d/99ccache

Add the following content for your script and save.

   1 #!/bin/sh
   2 if echo $SCHROOT_CHROOT_NAME | grep -e 'sbuild' >/dev/null && test -z "$CCACHE_DISABLE"; then
   3     export CCACHE_DIR=/var/cache/ccache-`dpkg-architecture -qDEB_HOST_ARCH`
   4     export CCACHE_UMASK=002
   5     export CCACHE_COMPRESS=1
   6     export PATH="/usr/lib/ccache:$PATH"
   7     # Set ccache's cache size to 4G, special for big builds.
   8     # Use an appropriate value here.
   9     ccache --max-size 4G
  10 fi

Now your chroot will be setup so that it may use ccache.

Note by mjt: the above does not work. As stated in schroot[-setup] manpage, scripts in /etc/schroot/setup.d/ are _executed_ (run), not _sourced_, so setting of environment variables there has no effect on the resulting chroot environment. So all the ccache setup will be ignored if set up this way. So far I found only one way to actually use ccache with sbuild/schroot -- it is by enabling --preserve-environment and setting all CCACHE_* vars before invoking sbuild, and specifying PATH explicitly in sbuild.conf/.sbuildrc. This is not exactly a good idea, because environment variables should really be cleared, but I know no other easy way to achieve the goal.

Another nitpick: ccache --max-size shouldn't actually run for every sbuild invocation but just once.

Note by ijc: Indeed... I think it should be possible to do by using the command-prefix chroot option to run a script within the chroot which sets the necessary variables and then exec's the remainder of its command line. There are a couple of other problems, first dpkg-architecture -qDEB_HOST_ARCH is evaluated at setup time and not when the chroot is constructed, so it doesn't work for cross arch (even biarch, e.g. i386 on amd64) chroots, second the cache directory is owned by root but the builds do not run as root. I can't see how to inject either $USER or $HOME into the fstab dynamically.

Using eatmydata with sbuild

eatmydata is used when storing data on the system is not that important. This is typically the case during build runs using sbuild. To use eatmydata in sbuild, install the eatmydata package inside the sbuild chroot environment.

   1 sudo sbuild-shell source:sid-$arch-sbuild
   2 apt-get install eatmydata
   3 exit

Then edit your ~/.sbuildrc configuration file and add the command eatmydata to be run as an external command during the chroot setup phase. For example, your $external_commands variable should look like this.

   1 $external_commands = {
   2                         'post-build-commands' => [],
   3                         'chroot-setup-commands' => ['eatmydata'],
   4                         'chroot-cleanup-commands' => [],
   5                         'pre-build-commands' => []
   6                       };

Note by ijc: I expect this doesn't work for similar reasons to why the ccache example is bogus, the chroot-setup-commands are not run as a prefix to the build commands. I expect this would be easyily solved with command-prefix.