Differences between revisions 29 and 30
Revision 29 as of 2021-10-05 13:08:36
Size: 11323
Comment: misc. tightening of phrases.
Revision 30 as of 2021-10-05 13:09:14
Size: 11215
Comment: drop too specific (and obsolete) note about ext3
Deletions are marked like this. Additions are marked like this.
Line 337: Line 337:

If not, try to use the ext3 filesystem,
for more details, see our [[cowbuilder_benchmark| benchmarks]].

This is a tutorial for git-pbuilder.

git-pbuilder is part of git-buildpackage and the usage is very similar to cowbuilder.

git-pbuilder needs sudo rights!

Usage

First, an environment needs to be created, then it can be used by git-buildpackage, with, for instance, the gbp buildpackage --git-pbuilder option.

Initialization and Variants

You can create base images for any architecture (e.g. i386 or amd64) and distribution (e.g. buster (oldstable), bullseye (stable) or sid (unstable), Ubuntu xxx, ...) that your hardware can run on.

Normal Usage

The easiest is to call with no further options. That will create a Sid (unstable) build environment with the architecture you are currently running on (e.g. if you are using i386, the environment will also be created for i386):

git-pbuilder create

The base build image is created in /var/cache/pbuilder/base.cow/.

Creating other Architecture

If you want to explicitly create an environment for i386 (e.g. while running on architecture amd64) you have to tell git-pbuilder:

ARCH=i386 git-pbuilder create

The base build image is created in /var/cache/pbuilder/base-sid-i386.cow/.

Creating Packages for other Distributions

Packaging is most commonly done in the Sid (unstable) distribution. When you occationally want to create packages for the stable or old-stable release, you have to tell this when git-pbuilder should create the environment.

E.g. to build for the Buster distribution you have to call git-pbuilder like this:

DIST=buster git-pbuilder create

The base image is created in /var/cache/pbuilder/base-buster-[your-platform].cow/.

A base image for the ancient Jessie Distribution is created similarly, only change the argument for DIST.

additional repositories probably needed

If you want to build packages as backport or as packages for stable-security please ensure you have added the correct repositories to the sources.list inside your base environment!

Normally git-pbuilder (pbuilder in the end) will only add a entry for the base repository of the distribution! You will probably need entries for security and [DIST]-update!

Using a Mirror

If you use git-pbuilder (or git-buildpackage) very often and can spare the additional disk space consumed, you can save download time and reduce traffic by using a local mirror or a proxy service.

If you have set up a local apt proxy the you can tell git-pbuilder to use it. Let's say you want to create a base image for Stretch and architecture amd64 with a caching proxy like AptCacherNg with the IP 192.168.110.4 on port 3142, here's the resulting command:

DIST=stretch ARCH=amd64 git-pbuilder create --mirror=http://192.168.110.4:3142/ftp.de.debian.org/debian

The base build image is created in /var/cache/pbuilder/base-stretch-amd64.cow/.

Updating

Make sure your build environment is reasonably up-to-date.

To update the base image just run:

git-pbuilder update

Updating base images for non-default distributions or architectures is similar to creating them. Updating the Buster environment for the same architecture you are currently running on would be:

DIST=buster git-pbuilder update

Or for Stretch on i386:

DIST=stretch ARCH=i386 git-pbuilder update

If you have used a mirror while creating the base images it will be used. So remember if you switch between multiple networks that the update will fail if the mirror isn't reachable. To use another mirror you have to set the variable MIRRORSITE to a valid mirror site:

MIRRORSITE="http://10.101.0.10:3142/ftp.debian.org/debian" git-pbuilder update

Installing Extra Packages

Sometimes you need extra packages installed on the base image. This is helpful for working off-line or to speed up the packaging. The workflow for that is similar to cowbuilder.

git-pbuilder login --save-after-login
# first step, update the package list
root@host:/# apt-get update
# then you can install any package
root@host:/# apt-get install vim screen less

You have to repeat those steps for each base image you use:

DIST=buster ARCH=amd64 git-pbuilder login --save-after-login
...

Using Local Packages

Sometimes the package you are trying to build build-depends on a library you just packaged, and is not available in the official repositories. PbuilderTricks explains how to solve this with pbuilder in general. With git-pbuilder, after having created the /etc/pbuilderrc (or /root/.pbuilderrc) and D05deps as instructed, you need to call this:

git-pbuilder update --override-config

Tips

Use of eatmydata

You can install eatmydata to improve the speed of your builds.

git-pbuilder login --save-after-login
root@host:/# apt-get update
root@host:/# apt-get install eatmydata

Using ccache

If you are often building the same package with a big source then it is useful to speed up a second build with ccache. To do so you have to tell pbuilder the needed environment for the use of ccache inside the chroot.

The ccache cache directory has to be placed somewhere in your file system. Suggested place is /var/cache/pbuilder/ccache but you can also put it e.g. under /home/ccache in case you have more free space there.

Beware to not use a NFS or CIFS share!

The executing right for this directory needs to be set to ga+w so the user pbuilder (who the ccache will be run by) can create the needed subdirectories there. If the directory doesn't exist create and set/correct the permissions on it.

The following one-liners should do the trick, even if the directory exists:

sudo install -d -m 2775 -o 1234 -g 1234 /var/cache/pbuilder/ccache
echo "for i in $(echo {0..9}) $(echo {a..f}) tmp; do sudo mkdir -p /var/cache/pbuilder/ccache/$i; done" | bash
sudo chown -R 1234:1234 /var/cache/pbuilder/ccache

Next you have to tweak your /etc/pbuilderrc (or $HOME/.pbuilderrc). Fill in the following part:

export CCACHE_DIR="/var/cache/pbuilder/ccache"
export PATH="/usr/lib/ccache:${PATH}"
EXTRAPACKAGES="ccache"
BINDMOUNTS="${CCACHE_DIR}"

Error while configure script is running

If you running a build for the first time after setting up the ccache environment sometimes an error occurs while the configure script is trying to detect the C (or C++) Compiler for creating object files. It will look similar to this snippet:

checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... configure: error: in `/build/package/foo':
configure: error: cannot compute suffix of object files: cannot compile
See `config.log' for more details
...
make: *** [build] Error 2
...
E: Failed autobuilding of package

This is related to wrong permissions on the folder where ccache is storing the temporary files (based on the setting from above take a look at /var/cache/pbuilder/ccache and correct the file permissions by calling sudo chown -R 1234:1234 /var/cache/pbuilder/ccache.

Changing standard ccache options

Without further options ccache will use a cache size of 1GB and an unlimited amount of cached files. Depending on the package you build you will want to adjust those or other limits. The needed caching size depends on the size of object files the build will produce. You have to investigate here. So maybe you want to set the caching size to 4GB. This has to be done in the chroot, so the only way to do this is a hook script. You need a hook script of type A because it has to be set before the build starts.

Put something like the following as file A10set_ccache_options in your hook directory:

# A10_set_ccache_options
# setting needed options to ccache
# possible options can be found on http://ccache.samba.org/manual.html#_options

# increase the ccache caching size
ccache -M 4G
# output the current statistics
ccache -s

With above hook script you will get an output right before build starts similar to this:

cache directory             /home/ccache
cache hit (direct)                     6
cache hit (preprocessed)               1
cache miss                           982
called for link                       57
called for preprocessing              26
compile failed                        18
preprocessor error                     8
bad compiler arguments                 2
unsupported source language            9
autoconf compile/link                133
unsupported compiler option            3
no input file                         24
files in cache                      2300
cache size                         619.5 Mbytes
max cache size                       4.0 Gbytes

Creating a specific base chroot

You can spend a lot of time waiting for the prepared chroot if you are building packages with a big list of dependencies, even if the packages are cached inside the pbuilder apt directory. That is annoying, and unnecessary for developing and tuning the package. The build would be much quicker if the used chroot had already installed all dependencies. As git-pbuilder can pass cowbuilder arguments as well the easiest way is to tell git-pbuilder which base chroot cowbuilder to use. But before that you have to create your desired chroot.

Copy the base directory to a new directory and name it as you want - except for one rule: the new directory must start with 'base-'. So for example if you want to create a new base chroot based on the default sid/unstable chroot copy the /var/cache/pbuilder/buildd/base.cow to /var/cache/pbuilder/buildd/base-my_package.cow:

sudo cp -a /var/cache/pbuilder/base.cow /var/cache/pbuilder/base-my_package.cow

Next you need to log into this new chroot and install all the needed dependencies persistently. For this you need a package list for apt-get or dpkg --set-selections:

DIST=my_package git-pbuilder login --save-after-login

Now install the needed packages and log out:

apt-get install $(list of packages)

Or alternatively with dpkg --set-selections:

dpkg --set-selections < packagelist # created with 'dpkg --get selections \* > /tmp/packagelist'

You can now use this prepared chroot with the git-buildpackage option --git-dist=:

gbp --git-dist=my_package ... other options ...

Troubleshooting

Slow copying and removing of the COW directory

What cowbuilder does is:

cp -al /var/cache/pbuilder/base.cow /tmp/new
rm -rf /tmp/[new]

Of course cowbuilder uses a different location than /tmp/[new]. You need to optimize those 2 commands on your computer. They should take no more than a few milliseconds each.