This is a tutorial for git-pbuilder.
git-pbuilder is part of git-buildpackage and the usage is very similar to cowbuilder.
git-pbuilder needs sudo rights!
Contents
Usage
First, an environment needs to be created, then it can be used by git-buildpackage, with, for instance, the gbp buildpackage --git-pbuilder option.
Initialization and Variants
You can create base images for any architecture (e.g. i386 or amd64) and distribution (e.g. bullseye (oldstable), bookworm (stable) or sid (unstable), Ubuntu xxx, ...) that your hardware can run on.
Normal Usage
The easiest is to call with no further options. That will create a Sid (unstable) build environment with the architecture you are currently running on (e.g. if you are using i386, the environment will also be created for i386):
git-pbuilder create
The base build image is created in /var/cache/pbuilder/base.cow/.
Creating other Architecture
If you want to explicitly create an environment for i386 (e.g. while running on architecture amd64) you have to tell git-pbuilder:
ARCH=i386 git-pbuilder create
The base build image is created in /var/cache/pbuilder/base-sid-i386.cow/.
Creating Packages for other Distributions
Packaging is most commonly done in the Sid (unstable) distribution. When you occationally want to create packages for the stable or old-stable release, you have to tell this when git-pbuilder should create the environment.
E.g. to build for the Buster distribution you have to call git-pbuilder like this:
DIST=buster git-pbuilder create
The base image is created in /var/cache/pbuilder/base-buster-[your-platform].cow/.
A base image for the ancient Jessie Distribution is created similarly, only change the argument for DIST.
additional repositories probably needed
If you want to build packages as backport or as packages for stable-security please ensure you have added the correct repositories to the sources.list inside your base environment!
Normally git-pbuilder (pbuilder in the end) will only add a entry for the base repository of the distribution! You will probably need entries for security and [DIST]-update!
Using a Mirror
If you use git-pbuilder (or git-buildpackage) very often and can spare the additional disk space consumed, you can save download time and reduce traffic by using a local mirror or a proxy service.
If you have set up a local apt proxy the you can tell git-pbuilder to use it. Let's say you want to create a base image for Stretch and architecture amd64 with a caching proxy like AptCacherNg with the IP 192.168.110.4 on port 3142, here's the resulting command:
DIST=stretch ARCH=amd64 git-pbuilder create --mirror=http://192.168.110.4:3142/ftp.de.debian.org/debian
The base build image is created in /var/cache/pbuilder/base-stretch-amd64.cow/.
Updating
Make sure your build environment is reasonably up-to-date.
To update the base image just run:
git-pbuilder update
Updating base images for non-default distributions or architectures is similar to creating them. Updating the Buster environment for the same architecture you are currently running on would be:
DIST=buster git-pbuilder update
Or for Stretch on i386:
DIST=stretch ARCH=i386 git-pbuilder update
If you have used a mirror while creating the base images it will be used. So remember if you switch between multiple networks that the update will fail if the mirror isn't reachable. To use another mirror you have to set the variable MIRRORSITE to a valid mirror site:
MIRRORSITE="http://10.101.0.10:3142/ftp.debian.org/debian" git-pbuilder update
Installing Extra Packages
Sometimes you need extra packages installed on the base image. This is helpful for working off-line or to speed up the packaging. The workflow for that is similar to cowbuilder.
git-pbuilder login --save-after-login # first step, update the package list root@host:/# apt-get update # then you can install any package root@host:/# apt-get install vim screen less
You have to repeat those steps for each base image you use:
DIST=buster ARCH=amd64 git-pbuilder login --save-after-login ...
Using Local Packages
Sometimes the package you are trying to build build-depends on a library you just packaged, and is not available in the official repositories. PbuilderTricks explains how to solve this with pbuilder in general. With git-pbuilder, after having created the /etc/pbuilderrc (or /root/.pbuilderrc) and D05deps as instructed, you need to call this:
git-pbuilder update --override-config
Tips
Use of eatmydata
You can install eatmydata to improve the speed of your builds.
git-pbuilder login --save-after-login root@host:/# apt-get update root@host:/# apt-get install eatmydata
Using ccache
If you are often building the same package with a big source then it is useful to speed up a second build with ccache. To do so you have to tell pbuilder the needed environment for the use of ccache inside the chroot.
The ccache cache directory has to be placed somewhere in your file system. Suggested place is /var/cache/pbuilder/ccache but you can also put it e.g. under /home/ccache in case you have more free space there.
Beware to not use a NFS or CIFS share!
The executing right for this directory needs to be set to ga+w so the user pbuilder (who the ccache will be run by) can create the needed subdirectories there. If the directory doesn't exist create and set/correct the permissions on it.
The following one-liners should do the trick, even if the directory exists:
sudo install -d -m 2775 -o 1234 -g 1234 /var/cache/pbuilder/ccache echo 'for i in $(echo {0..9}) $(echo {a..f}) tmp; do sudo mkdir -p /var/cache/pbuilder/ccache/$i; done' | bash sudo chown -R 1234:1234 /var/cache/pbuilder/ccache
Next you have to tweak your /etc/pbuilderrc (or $HOME/.pbuilderrc). Fill in the following part:
export CCACHE_DIR="/var/cache/pbuilder/ccache" export PATH="/usr/lib/ccache:${PATH}" EXTRAPACKAGES="ccache" BINDMOUNTS="${CCACHE_DIR}"
Error while configure script is running
If you running a build for the first time after setting up the ccache environment sometimes an error occurs while the configure script is trying to detect the C (or C++) Compiler for creating object files. It will look similar to this snippet:
checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... configure: error: in `/build/package/foo': configure: error: cannot compute suffix of object files: cannot compile See `config.log' for more details ... make: *** [build] Error 2 ... E: Failed autobuilding of package
This is related to wrong permissions on the folder where ccache is storing the temporary files (based on the setting from above take a look at /var/cache/pbuilder/ccache and correct the file permissions by calling sudo chown -R 1234:1234 /var/cache/pbuilder/ccache.
Changing standard ccache options
Without further options ccache will use a cache size of 1GB and an unlimited amount of cached files. Depending on the package you build you will want to adjust those or other limits. The needed caching size depends on the size of object files the build will produce. You have to investigate here. So maybe you want to set the caching size to 4GB. This has to be done in the chroot, so the only way to do this is a hook script. You need a hook script of type A because it has to be set before the build starts.
Put something like the following as file A10set_ccache_options in your hook directory:
# A10_set_ccache_options # setting needed options to ccache # possible options can be found on http://ccache.samba.org/manual.html#_options # increase the ccache caching size ccache -M 4G # output the current statistics ccache -s
With above hook script you will get an output right before build starts similar to this:
cache directory /home/ccache cache hit (direct) 6 cache hit (preprocessed) 1 cache miss 982 called for link 57 called for preprocessing 26 compile failed 18 preprocessor error 8 bad compiler arguments 2 unsupported source language 9 autoconf compile/link 133 unsupported compiler option 3 no input file 24 files in cache 2300 cache size 619.5 Mbytes max cache size 4.0 Gbytes
Creating a specific base chroot
You can spend a lot of time waiting for the prepared chroot if you are building packages with a big list of dependencies, even if the packages are cached inside the pbuilder apt directory. That is annoying, and unnecessary for developing and tuning the package. The build would be much quicker if the used chroot had already installed all dependencies. As git-pbuilder can pass cowbuilder arguments as well the easiest way is to tell git-pbuilder which base chroot cowbuilder to use. But before that you have to create your desired chroot.
Copy the base directory to a new directory and name it as you want - except for one rule: the new directory must start with 'base-'. So for example if you want to create a new base chroot based on the default sid/unstable chroot copy the /var/cache/pbuilder/buildd/base.cow to /var/cache/pbuilder/buildd/base-my_package.cow:
sudo cp -a /var/cache/pbuilder/base.cow /var/cache/pbuilder/base-my_package.cow
Next you need to log into this new chroot and install all the needed dependencies persistently. For this you need a package list for apt-get or dpkg --set-selections:
DIST=my_package git-pbuilder login --save-after-login
Now install the needed packages and log out:
apt-get install $(list of packages)
Or alternatively with dpkg --set-selections:
dpkg --set-selections < packagelist # created with 'dpkg --get selections \* > /tmp/packagelist'
You can now use this prepared chroot with the git-buildpackage option --git-dist=:
gbp --git-dist=my_package ... other options ...
Troubleshooting
Slow copying and removing of the COW directory
What cowbuilder does is:
cp -al /var/cache/pbuilder/base.cow /tmp/new rm -rf /tmp/[new]
Of course cowbuilder uses a different location than /tmp/[new]. You need to optimize those 2 commands on your computer. They should take no more than a few milliseconds each.