Differences between revisions 169 and 170
Revision 169 as of 2020-01-26 00:36:54
Size: 27703
Comment: Update instructions for manual creation, add apt-cacher-ng and remove outdated parts (mention to bug 5 years old)
Revision 170 as of 2020-08-16 12:52:22
Size: 27701
Editor: ?BernhardSchmidt
Comment: Use aspcud resolver for experimental (like the buildd do)
Deletions are marked like this. Additions are marked like this.
Line 498: Line 498:
sbuild --extra-repository='deb http://deb.debian.org/debian experimental main' --build-dep-resolver=aptitude mypkg.dsc sbuild --extra-repository='deb http://deb.debian.org/debian experimental main' --build-dep-resolver=aspcud mypkg.dsc

Translation(s): none


sbuild is used on the official buildd network to build binary packages for all supported architectures. It can also be used by individuals to test that their package builds in a minimal installation of Debian Unstable. In particular, this helps ensure that you haven't missed any build dependencies.

The main alternative to sbuild is pbuilder combined with cowbuilder.

This main part of this page is intended as a short guide to sbuild. It documents how to set up sbuild and build packages with it. The later parts of the page document optional enhancements to the simple setup described in the first section.

If you use propellor to configure your development machine, its Propellor.Property.Sbuild module can perform most of this setup for you.

Setup

Using sbuild-debian-developer-setup

sbuild provides the package sbuild-debian-developer-setup which helps you setup an sbuild environment easily, if you're interested, make sure to take a look at the manpage at https://manpages.debian.org/unstable/sbuild/sbuild-debian-developer-setup.1

Manual creation

Create the chroot

To get started so you may build packages for Debian unstable, run the following. (Lines prefixed with # are needed on jessie and earlier).

   1 sudo apt-get install sbuild schroot debootstrap
   2 sudo sbuild-adduser $LOGNAME 
   3 cp /usr/share/doc/sbuild/examples/example.sbuildrc $HOME/.sbuildrc # copy example config into your home as suggested
   4 sudo apt install apt-cacher-ng
   5  ... *logout* and *re-login* or use `newgrp sbuild` in your current shell
   6 sudo sbuild-createchroot --include=eatmydata,ccache,gnupg unstable /srv/chroot/unstable-amd64-sbuild http://127.0.0.1:3142/deb.debian.org/debian

Now for a brief explanation on what these commands do.

  1. Install sbuild onto the system.
  2. This will add your username so that it may use the sbuild command. Additional users may be added by running sudo sbuild-adduser USER1 USER2 .... sbuild-adduser will prompt you to copy the template sbuild configuration in /usr/share/doc/sbuild/examples/example.sbuildrc to each user's ~/.sbuildrc, to be used as their user sbuild configuration. You can customize sbuild settings here, but you usually won't need to customize anything. This should be done once per user.

  3. Copy the sbuild template configuration to your home folder.
  4. Install apt-cacher-ng, to cache packages needed in your sbuild usage
  5. Update the active user group set to include sbuild.
  6. Use sbuild-createchroot to create a chroot used by sbuild meant for building packages targeting Debian unstable main.

    • The chroot is saved in /srv/chroot/unstable-amd64-sbuild. It installs the packages ccache and eatmydata in the chroot in case you want to use some of the enhancements detailed below. The apt repository used is the mirror service http://deb.debian.org/debian through apt-cacher-ng which will choose a suitable local mirror automatically. This can be changed to use a URL for a different mirror of the Debian archive. You can run this command once per distribution you want, and pass # --arch=i386 to create a chroot for a different architecture (the default is your host architecture).

      The command given above creates a type=directory chroot. If you are short of disc space, you can instead use the following command to create a chroot stored in a tarball at /srv/chroot/unstable-amd64-sbuild.tar.gz. This is not recommended unless you really can't spare the disc space, because several of the enhancements below depend on using a type=directory chroot.

      sudo sbuild-createchroot --make-sbuild-tarball=/srv/chroot/unstable-amd64-sbuild.tar.gz unstable `mktemp -d` http://deb.debian.org/debian

Configuration

If you're setting up sbuild for personal use, instead of as part of a build server, you might want to use the following options in your ~/.sbuildrc. These can also be set on the command line when running sbuild.

   1 $build_arch_all = 1;
   2 $distribution = 'unstable';

The $build_arch_all variable will enable building of architecture independent packages by default. Since official build servers are used to build an existing package for a different architecture (e.g., the uploader builds for i386, uploads arch-i386 and arch-all binary packages, and a build server builds arch-amd64 packages), this is off by default. You can also enable this per build by passing -A to sbuild.

The $distribution variable will set the distribution to build for as 'unstable'. You can set the distribution per build by passing it to the -d option. Be careful not to use -d just to select a specific chroot (use -c for that, see below), as it will override the distribution set in debian/changelog and may lead you to upload a package to a distribution it was not intended for.

Updating

The chroot should be up-to-date before building packages. Use the sbuild-update to perform updates.

First, note the name of the sbuild chroot to be updated. All sbuild chroots built with sbuild-createchroot will have a suffix of '-sbuild'. Thus to find the names of all sbuild chroots, run the following.

   1 schroot -l | grep sbuild

If you followed the setup instructions above, there should be one chroot named source:unstable-$arch-sbuild where $arch is the architecture installed on your machine.

After noting the name of your sbuild chroot, run the following.

   1 sudo sbuild-update -udcar unstable-$arch-sbuild

The arguments '-udcar' will tell sbuild-update to run an apt-get update, dist-upgrade, clean, autoclean, and autoremove in the chroot.

You can also pass --apt-update --apt-distupgrade to the individual sbuild invocation to update the temporary copy of the build chroot, but this won't cause any changes to happen in the persistent copy of the chroot (in the .tar.gz file). So if you are building more than once, you should run sbuild-update instead of relying on this.

Building packages

To build a package from the source directory of a debianized package, simply run the following.

   1 sbuild

Alternatively, you may pass in the '.dsc' file of a package generated by dpkg-buildpackage, git-buildpackage, and so forth so that it may be built with sbuild. For example, to build sbuild from its '.dsc' file, do the following.

   1 sbuild sbuild_*.dsc

sbuild also supports cross-compiling a package: to build a package which is e.g. only buildable on mips, in the amd64 chroot from the above example, you can use:

   1 sbuild --host=mips

To build packages available from the apt repositories used in the sbuild chroot, just pass in a package name (older versions of sbuild required $package_$version). For example to build the latest sbuild:

   1 sbuild -d unstable sbuild

Everything needed to build the latest sbuild version will be downloaded from the repositories and will be saved in your current directory after building of the packages is finished.

If you want parallel build to be used, add a line like the following to ~/.sbuildrc:

$ENV{'DEB_BUILD_OPTIONS'} = 'parallel=5';

Integration with gbp (gbp-buildpackage)

Sbuild can be integrated into gbp's workflow with little effort. For example:

   1 gbp buildpackage --git-builder=sbuild -A -v -d unstable

The only thing we need is to specify "sbuild" as the argument of --git-builder option.

Cleaning up schroot session

Sometimes you can end up with dangling chroot sessions potentially taking up valuable system resources (find them with `schroot -l --all`). This command is useful:

sudo schroot --end-session --all-sessions

Delete a chroot

sudo rm -r /srv/chroot/unstable-amd64-sbuild/
sudo rm /etc/schroot/chroot.d/unstable-amd64-sbuild-* /etc/sbuild/chroot/unstable-amd64-sbuild

Also see sbuild-destroychroot(1).

Enhancements

Using lintian

The use of lintian with sbuild can greatly aid with increasing the quality of Debian packages. To use lintian with sbuild, edit your ~/.sbuildrc configuration file. Open ~/.sbuildrc with your favorite text editor and edit the lines with the following variables.

   1 $run_lintian = 1;
   2 $lintian_opts = ['-i', '-I'];

The $run_lintian variable will enable running of lintian after a successful build with sbuild.

The $lintian_opts variable is an array of options to pass to lintian. Here the options are '-i' which will output information about lintian warnings and errors, and '-I' which will output lintian "info" messages.

Using piuparts

The use of piuparts with sbuild is another feature meant to enhance the quality of Debian packages built with sbuild. To use piuparts, edit your ~/.sbuildrc configuration file. Open ~/.sbuildrc with your favorite text editor and edit the lines with the following variables.

   1 $run_piuparts = 1;
   2 $piuparts_opts = ['--schroot', 'unstable-amd64-sbuild'];

The $run_piuparts variable will enable running piuparts after a successful build with sbuild.

The $piuparts_opts variable is an array of options to pass to piuparts. Here the options instruct piuparts to use sbuild chroot '/srv/chroot/unstable-amd64-sbuild' as the chroot used in its testing of packages.

Using autopkgtest

Add this to your .sbuildrc:

$run_autopkgtest = 1;
$autopkgtest_root_args = '';
$autopkgtest_opts = [ '--', 'schroot', '%r-%a-sbuild' ];

This will run autopkgtest every time you run sbuild. If you don't always want it to run, set the $run_autopkgtest to 0 in the above configuration and use the --run-autopkgtest argument of sbuild to run autopkgtest with sbuild.

See the sbuild.conf manpage for more options.

  • The schroot created using sbuild-createchroot has a few more packages available by default than the ones used by debci (for example build-essential) so if you are adding or removing dependencies, it is recommended to test the autopkgtest using autopkgtest-build-lxc and autopkgtest your_package_dsc.dsc -- lxc -s -e autopkgtest-unstable at least one to make sure your package won't fail when running on debci due to missing dependencies.

On old versions of sbuild where the above was not possible, you needed to add this to .sbuildrc:

$external_commands = {
  'post-build-commands' => [
    [
      'autopkgtest', '%c',
      '--', 'schroot', 'unstable-%a-sbuild;',

      # if autopkgtest's exit code is 8 then the package had no tests
      # but this isn't a failure, so catch it
      'aptexit=$?;',
      'if', 'test', '$aptexit', '=', '8;', 'then',
      'exit', '0;', 'else', 'exit', '$aptexit;', 'fi'
    ],
  ],
};

Consider configuring schroot with tmpfs or eatmydata, or else running adt-run will slow the build a lot.

In order to use the autopkgtest command, you need a version > 4, provided by jessie-backports, or testing.

Apt package caching

To avoid downloading the same packages over and over again when building a package multiple times in a row, install the apt-cacher-ng package and use http://localhost:3142/debian as a proxy.

For an existing sbuild chroot, you can update your sources.list as follows:

   1 sudo sbuild-shell source:unstable-$arch-sbuild
   2 echo 'acquire::http::proxy "http://localhost:3142/debian";' >>  /etc/apt/apt.conf.d/proxy

For new schroot, you can use this:

sudo apt install apt-cacher-ng
sudo sbuild-createchroot --include=eatmydata,ccache,gnupg unstable /srv/chroot/unstable-amd64-sbuild http://127.0.0.1:3142/deb.debian.org/debian

See also AptCacherNg for more details on how to configure the proxy.

Customizations of sbuild chroots

Sometimes it is desirable to further customize your sbuild chroot environment. Typical customizations done are installing more packages inside the chroot, modifying /etc/apt/sources.list, and installing custom scripts to be run inside the chroot.

To modify a chroot, start a session for the chroot with the prefix 'source:'. For example, to modify the unstable-$arch-sbuild chroot, start a session for the chroot as follows.

   1 sudo sbuild-shell source:unstable-$arch-sbuild

This will start a session inside the unstable-$arch-sbuild chroot. Any modifications done inside the chroot will be saved upon exiting. Inside this chroot, you may run apt-get commands to install or remove packages as desired. For example, to install ccache, do the following in the chroot session.

   1 apt-get install ccache

Note that you are already root inside a chroot session. Also, sudo would typically not be installed in the chroot anyway.

Making other modifications such as editing /etc/apt/sources.list or adding scripts inside the chroot is best done from outside the chroot session. In order to do this, leave the current chroot session open and start another terminal session. In the other terminal session, find the path used for the existing chroot session as follows.

   1 schroot --info --all-sessions | grep Path

This should output exactly one line and should specify the path of the chroot session. It should look something like this.

  Path                   /var/lib/schroot/mount/unstable-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c

Doing an ls on this directory should resemble a standard listing of the '/' directory.

With the above path, the sources.list file will be in the following path.

/var/lib/schroot/mount/unstable-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

Open the sources.list file at this path with root privileges using your favorite editor, for example:

   1 sudo -e /var/lib/schroot/mount/unstable-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/etc/apt/sources.list

Proceed to edit the sources.list file as you see fit, then save.

To add scripts inside the chroot, simply place the scripts in the following directory.

/var/lib/schroot/mount/unstable-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/usr/local/bin

Be sure to make the scripts executable.

   1 sudo chmod a+x /var/lib/schroot/mount/unstable-amd64-sbuild-5bad48fe-9823-4454-815f-b869d1d7b22c/usr/local/bin/*

Other modifications may be done to the chroot, either by running commands available within the chroot session, or by running commands with root privileges via the secondary terminal session. Once you are done making modifications to the chroot, simply exit the session. Inside the chroot session, do the following.

   1 exit

After exiting, your modifications will be saved and made available for every new chroot session created afterwards.

External Commands

sbuild supports running external commands at various stages of the build process. This is useful for cases such as running a script inside the chroot after it has been setup. As an example, to run a script in /usr/local/bin/myscript, edit the $external_commands in the sbuild configuration file ~/.sbuildrc as follows.

   1 $external_commands = {
   2                         'post-build-commands' => [],
   3                         'chroot-setup-commands' => ['/usr/local/bin/myscript'],
   4                         'chroot-cleanup-commands' => [],
   5                         'pre-build-commands' => []
   6                       };

sbuild can also translate certain percent escaped keywords for external commands during certain portions of a build. For example, in post build commands, %SBUILD_CHANGES is changed to the path of the '.changes' file for a successfully built package.

Here is an example of adding a post build command to run /usr/local/bin/postbuildscript with %SBUILD_CHANGES as an argument.

   1 $external_commands = {
   2                         'post-build-commands' => ['/usr/local/bin/postbuildscript', '%SBUILD_CHANGES'],
   3                         'chroot-setup-commands' => ['/usr/local/bin/myscript'],
   4                         'chroot-cleanup-commands' => [],
   5                         'pre-build-commands' => []
   6                       };

See the 'EXTERNAL COMMANDS' section of the sbuild man page for more information on external commands.

Using "ccache" with sbuild

ccache is a compiler wrapper that will cache compilation results (produced object files) from gcc and g++; if you repeatedly compile the same source code (or parts of it), ccache will greatly shorten compilation times by avoiding recompilation of files that it has cached earlier.

This is especially useful during package development, when you might have to rebuild a package with a long compilation phase several times. It is also effective with packages that are frequently updated, because often only a few files actually change during updates to a software.

In order to prepare your sbuild environment for ccache, first perform the following setup in the host environment (i.e., outside the chroot), as user root:

   1 dir=/var/cache/ccache-sbuild
   2 install --group=sbuild --mode=2775 -d $dir
   3 env CCACHE_DIR=$dir ccache --max-size 4G
   4 cat >>/etc/schroot/sbuild/fstab <<END
   5 $dir $dir none rw,bind 0 0
   6 END

This assumes that you trust all members of the sbuild group.

It is perfectly fine to share the cache among chroots, even for different architectures. ccache honours the compiler name, size and timestamp as well as the command line when calculating hash values. At least one of these will differ between builds of the same file for different architectures.

Next place the following script into $dir/sbuild-setup:

   1 cat >$dir/sbuild-setup <<END
   2 #!/bin/sh
   3 export CCACHE_DIR=$dir
   4 export CCACHE_UMASK=002
   5 export CCACHE_COMPRESS=1
   6 unset CCACHE_HARDLINK
   7 export PATH="/usr/lib/ccache:\$PATH"
   8 exec "\$@"
   9 END

and make it executable:

chmod a+rx $dir/sbuild-setup

Then for each chroot ($dist-$arch-sbuild) where you want to enable ccache

  1. Install ccache inside the chroot by running

     schroot -c source:$dist-$arch-sbuild -d /home apt-get install ccache
  2. and edit the corresponding configuration file in /etc/schroot/chroot.d/ by appending the line

     command-prefix=/var/cache/ccache-sbuild/sbuild-setup

    (Multiple command-prefix can be joined with commas, in case you already have one configured; see eatmydata below.)

Using eatmydata with sbuild

eatmydata is used when storing data on the system is not that important. This is typically the case during build runs using sbuild. To use eatmydata in sbuild, install the eatmydata package inside the sbuild chroot environment.

To enable first run, as root:

   1 schroot -c source:$dist-$arch-sbuild apt-get install eatmydata

Then edit /etc/schroot/chroot.d/$dist-$arch-sbuild-$suffix to add the line:

command-prefix=eatmydata

If you want to combine this with the ccache instructions from above then use:

command-prefix=/var/cache/ccache-sbuild/sbuild-setup,eatmydata

Note that piuparts invokes eatmydata by default and nested eatmydata invocations don't work. To deal with this pass --no-eatmydata to piuparts in your ~/.sbuildrc, or set up a separate schroot profile for piuparts as described above.

sbuild overlays in tmpfs

If you are using a type=directory chroot as described in "Setup" above, and you have sufficient memory, you can run builds in RAM for a huge speed increase. The build is performed in a tmpfs that is overlaid upon the base chroot. You need union-type=overlay in your /etc/schroot/chroot.d/sbuild-amd64-sbuild-<hash> file but that should be there by default if you followed the setup instructions above.

See also 709774.

Create the executable: /etc/schroot/setup.d/04tmpfs with

   1 cat >/etc/schroot/setup.d/04tmpfs <<"END"
   2 #!/bin/sh
   3 
   4 set -e
   5 
   6 . "$SETUP_DATA_DIR/common-data"
   7 . "$SETUP_DATA_DIR/common-functions"
   8 . "$SETUP_DATA_DIR/common-config"
   9 
  10 
  11 if [ "$STAGE" = "setup-start" ]; then
  12   mount -t tmpfs overlay /var/lib/schroot/union/overlay
  13 elif [ "$STAGE" = "setup-recover" ]; then
  14   mount -t tmpfs overlay /var/lib/schroot/union/overlay
  15 elif [ "$STAGE" = "setup-stop" ]; then
  16   umount -f /var/lib/schroot/union/overlay
  17 fi
  18 END

and make it executable:

chmod a+rx /etc/schroot/setup.d/04tmpfs

Alternatively you could configure fstab to mount /var/lib/schroot/union/overlay as tmpfs

none /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750 0 0

Enabling experimental

The Debian experimental repository can be added dynamically on top of an existing unstable chroot during each sbuild run that requires experimental:

sbuild --extra-repository='deb http://deb.debian.org/debian experimental main' --build-dep-resolver=aspcud mypkg.dsc

Build for experimental

If you want to build for unstable but upload to experimental. Use schroot -l --all-source-chroots to get the name of the chroot (unstable-amd64-sbuild in this case).

sbuild -d experimental -c unstable-amd64-sbuild mypkg.dsc

Alternatively, if you know that you always want to build your packages for experimental in a sid chroot, just add experimental as an alias of your sid schroot to your sid schroot configuration:

aliases=experimental

Enabling incoming.debian.org

Another useful repository to add as --extra-repository option is deb http://incoming.debian.org/debian-buildd/ buildd-unstable main in case you want to build or do a new upload before the next dinstall

sbuild --extra-repository='deb http://incoming.debian.org/debian-buildd/ buildd-unstable main' mypkg.dsc

Disabling network access for dpkg-buildpackage

<!> This doesn't seem to work with recent sbuild (2016/05)

Source packages must be buildable without accessing any remote machines. On the other hand, the build process needs network access for the installation of the build dependencies. Conveniently, the build dependencies are installed by the "root" user while dpkg-buildpackage is run under fakeroot by the user running sbuild. So the following will deny any network access during the build process through blocking traffic originating from any process owned by the sbuild group while the root user will still have network access:

sudo iptables -I OUTPUT -m owner --gid-owner sbuild ! -d 127.0.0.0/8 -j DROP
sudo -u sbuild sbuild mypkg.dsc

A better fix would be if schroot allowed to unshare the network namespace for the dpkg-buildpackage invocation. See bugs 802850 and 802849

source only upload

To obtain a .changes file suitable for a source-only upload, use the --source-only-changes flag when calling sbuild.

Using aliases

In Debian, an unstable chroot is used for building in a number of situations like building an unpacked source package with UNRELEASED in debian/changelog or building packages for experimental. Furthermore, distributions have alternative names like sid/unstable or experimental/rc-buggy. Sbuild selects the chroot to use either from debian/changelog (so it would be nice if UNRELEASED would trigger a build in an unstable chroot) or through the -d option which also sets the Distribution value in the resulting .changes file (so it would be handy if saying -d sid were enough and one wouldn't have to use the -c option to type unstable-amd64-sbuild manually). All of this can be solved by using schroot aliases. My sid schroot config says:

aliases=UNRELEASED,sid,rc-buggy,experimental

This means, that this schroot will be used for packages having UNRELEASED in their debian/changelog, for packages I build with -d sid (because writing out unstable is too long) as well as for packages I build for experimental.

Adding extra packages

It is often necessary to add extra binary packages as build dependencies. For example, you might want to make a package available as a build dependency that is waiting in the NEW queue and so isn't available from the mirrors. To do this, use the --extra-package=./foo.deb option to sbuild.

You might find that the output from the resolver is not helpful in determining which extra package you need to make available. In this case, it can be useful to pass --build-dep-resolver=aptitude which tends to provide more useful output (though you should remove it once you've figured out the problem).

Remote build servers

One advantage of cowbuilder over sbuild is that it supports offloading builds to a remote server with cowpoke. Unfortunately, that command is specifically crafted for cowbuilder, so you need something else for sbuild.

The trick is to create a source package and transfer it to the remote machine for building. The latter can be done with dcmd. Example:

dpkg-buildpackage -S
dcmd scp ../foo-1.0.dsc example.net:build-area
ssh example.net sbuild build-area/foo-1.0.dsc

Troubleshooting

Missing space in /build

It is possible that you lack space for builds even if /srv/chroot has plenty of space. This can happen at build time on large packages (e.g. Libreoffice) because sbuild decompresses the packages in /build in the chroot, which is a bind mount to /var/lib/sbuild/build. If you are on a workstation where /var is shared with the root partition, you may sometimes not have enough space for the largest packages. The symptom will be an error message like this:

E: Disc space is probably not sufficient for building.
I: Source needs 2703644 KiB, while 4089748 KiB is free.)

But then when you actually look at the df, you have plenty of space! This happens because sbuild cleans up after itself on failure and gives back the disk space. If you follow disk space usage more closely during the build, you will notice that /var will take up more and more space until that failure.

A workaround for this is to change the configuration of the /build bind mount. This is done in /etc/schroot/sbuild/fstab. For example, the following configuration uses /home/build instead:

# Mount a large scratch space for the build, so we don't use up
# space on an LVM snapshot of the chroot itself.
#/var/lib/sbuild/build  /build   none    rw,bind         0       0
/home/build  /build   none    rw,bind         0       0