Differences between revisions 13 and 30 (spanning 17 versions)
Revision 13 as of 2014-04-01 08:48:26
Size: 9180
Comment: creating a customized specific chroot for package building
Revision 30 as of 2021-10-05 13:09:14
Size: 11215
Comment: drop too specific (and obsolete) note about ext3
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
This is a tutorial for `git-pbuilder`. 

`git-pbuilder` is part of the package `git-buildpackage` and the usage is very similar to `cowbuilder` (see also [[cowbuilder]]). 
This is a tutorial for `git-pbuilder`.

`git-pbuilder` is part of DebianPkg:git-buildpackage
and the usage is very similar to [[cowbuilder]].
Line 11: Line 12:
First, an environment needs to be created, then it can be used by git-buildpackage, with, for instance, the `gbp buildpackage --git-pbuilder` option. First, an environment needs to be created,
then it can be used by git-buildpackage,
with, for instance, the `gbp buildpackage --git-pbuilder` option.
Line 14: Line 17:
You can create base images for all architectures (mostly i386 or amd64) and distribution (mostly Squeeze, Wheezy or Testing/SID, Ubuntu xxx, ...) that your hardware can run on.
You can create base images
for any architecture (e.g. i386 or amd64) and distribution
(e.g. <<DebianCodename(oldstable)>> (oldstable)
,
<<
DebianCodename(stable)>> (stable) or sid (unstable), Ubuntu xxx, ...)
that your hardware can run on.
Line 17: Line 25:
The easiest usage is to call no further options, that will create a Testing/SID build environment with the architecture you are currently running on (if you are using i386, the environment will also be created for i386): The easiest is to call with no further options.
That will create a Sid (unstable) build environment
with the architecture you are currently running on
(e.g. if you are using i386,
the environment will also be created for i386):
Line 21: Line 33:
The base build image is created in `/var/cache/pbuilder/base.cow/`
The base build image is created in `/var/cache/pbuilder/base.cow/`.
Line 24: Line 37:
If you want explicitly create a environment for i386 (but running on architecture `amd64`) you have to tell that `git-pbuilder`:
If you want to explicitly create an environment for i386
(e.g.
while running on architecture `amd64`)
you have to tell `git-pbuilder`:
Line 28: Line 44:
The base build image is created in `/var/cache/pbuilder/base-sid-i386.cow/`
The base build image is created in `/var/cache/pbuilder/base-sid-i386.cow/`.
Line 31: Line 48:
Mainly package creating is done in the Testing distribution. But sometimes you want to create packages for the stable or old-stable release, so you have to tell this also while `git-pbuilder` is creating the environment. If you want to build for the Wheezy distribution then you have to call `git-pbuilder` like this:
{{{
DIST=wheezy git-pbuilder create
}}}
The base image is created in `/var/cache/pbuilder/base-wheezy-[your-platform].cow/`. The creating of a base image for the (old-old-stable) Squeeze Distribution is similar, just change the argument for `DIST`.
Packaging is most commonly done in the Sid (unstable) distribution.
When you occationally want to create packages for the stable or old-stable release,
you have to tell this when `git-pbuilder` should create the environment.

E.g. to build for the Buster distribution
you have to call `git-pbuilder` like this:
{{{
DIST=buster git-pbuilder create
}}}

The base image is created in `/var/cache/pbuilder/base-buster-[your-platform].cow/`.

A base image for the ancient Jessie Distribution is created similarly,
only change the argument for `DIST`.

{{{#!wiki note
'''additional repositories probably needed'''

If you want to build packages as backport or as packages for stable-security
please ensure you have added the correct repositories to the sources.list
inside your base environment!

Normally git-pbuilder (pbuilder in the end) will only add a entry
for the base repository of the distribution!
You will probably need entries for security and [DIST]-update!
}}}
Line 38: Line 76:
If you use `git-pbuilder` (or `git-buildpackage`) very offen it's better to use a local mirror to save downloadtime and reduce traffic. If you have set up a local Apt proxy the you can tell `git-pbuilder` to use it. Let's say you want to create a base image for `Squeeze` and architecture `amd64` with a caching proxy with the IP 192.168.110.4 on port 3142 (like [[DebianEdu/HowTo/NetinstallWithAptCacher|apt-cacher-ng]]), so it's a combination of all variants from above:
{{{
DIST=squeeze ARCH=amd64 git-pbuilder create --mirror=http://192.168.110.4:3142/ftp.de.debian.org/debian
}}}
The base build image is created in `/var/cache/pbuilder/base-squeeze-amd64.cow/`

If you use `git-pbuilder` (or `git-buildpackage`) very often
and can spare the additional disk space consumed,
you can
save download time and reduce traffic
by using a local mirror or a proxy service.

If you have set up a local apt proxy
the you can tell `git-pbuilder` to use it.
Let's say you want to create a base image for `Stretch` and architecture `amd64`
with a caching proxy like AptCacherNg with the IP 192.168.110.4 on port 3142,
here
's the resulting command:
{{{
DIST=stretch ARCH=amd64 git-pbuilder create --mirror=http://192.168.110.4:3142/ftp.de.debian.org/debian
}}}

The base build image is created in `/var/cache/pbuilder/base-stretch-amd64.cow/`.
Line 45: Line 94:
If the date of creation of your build environment is quite older you have to update it before you can use it. 
Mak
e sure your build environment is reasonably up-to-date.
Line 50: Line 101:
The update for the possible other distributions or architectures is similar like the creation of it. Update the Wheezy environmnt for the same architecture you are currently running on would be:
{{{
DIST=wheezy git-pbuilder update
}}}
or for Squeeze on i386
{{{
DIST=squeeze ARCH=i368 git-pbuilder update
}}}

If you have used a mirror while creating the base images it will be used! So remember that if you have various networks you working, the update will fail if the mirror isn't reachable.

Updating base images for non-default distributions or architectures
is similar to creating them.
Updating the Buster environment for the same architecture you are currently running on would be:
{{{
DIST=buster git-pbuilder update
}}}

Or for Stretch on i386:
{{{
DIST=stretch ARCH=i386 git-pbuilder update
}}}

If you have used a mirror while creating the base images it will be used.
So remember if you switch between multiple networks
that the update will fail if the mirror isn't reachable.
To use another mirror
you have to set the variable `MIRRORSITE` to a valid mirror site:
{{{
MIRRORSITE="http://10.101.0.10:3142/ftp.debian.org/debian" git-pbuilder update
}}}
Line 62: Line 124:
Sometimes you have to install extra packages to the base image. This is helpful if you work offline in some cases or you want to speed up the packaging. The workflow for that is similar to cowbuilder.
Sometimes you need extra packages installed on the base image.
This is helpful for working off-line
or to speed up the packaging.
The workflow for that is similar to cowbuilder.
Line 71: Line 137:
You have to repeat this steps for every base image you use if you needs the same behaviour in your various build environments.
{{{
DIST=wheezy ARCH=amd64 git-pbuilder login --save-after-login
You have to repeat those steps for each base image you use:
{{{
DIST=buster ARCH=amd64 git-pbuilder login --save-after-login
Line 77: Line 143:
== Using Local Packages ==

Sometimes the package you are trying to build
build-depends on a library you just packaged,
and is not available in the official repositories.
PbuilderTricks explains how to solve this with pbuilder in general.
With git-pbuilder,
after having created the /etc/pbuilderrc (or /root/.pbuilderrc) and D05deps as instructed,
you need to call this:
{{{
git-pbuilder update --override-config
}}}
Line 80: Line 159:
You can install the package [[DebianPts:libeatmydata|eatmydata]] to improve the speed of your builds. You can install DebianPkg:eatmydata to improve the speed of your builds.
Line 88: Line 167:
If you building often the same package which has a bigger source then it useful to speed up a second build with DebianPts:ccache. To do so you have to tell pbuilder the needed environment for the use of ccache inside the chroot. The ccache cache directory has to be placed somethere in your file system, suggest place is `/var/cache/pbuilder/ccache` but you can put it also under `/home/ccache` for example in case you have more free space there. But please do not use a NFS or CIFS share! The executing right for this directory needs to be set to a+w so the user pbuilder (which the ccache will run) can create the needed subdirectories there. If not exists create the directory and set/correct the permissions on it.

{{{
sudo mkdir -p /var/cache/pbuilder/ccache
sudo chmod a+
w /var/cache/pbuilder/ccache
}}}

Next you have to tweak your `/etc/pbuilderrc` (or `$HOME/.pbuilderrc`). Fill in the following part.

If you are often building the same package with a big source
then it is
useful to speed up a second build with DebianPkg:ccache.
To do so you have to tell pbuilder the needed environment
for the use of ccache inside the chroot.

The ccache cache directory has to be placed somewhere in your file system.
Suggested
place is `/var/cache/pbuilder/ccache`
but you can also put it e.g.
under `/home/ccache`
in case you have
more free space there.

Beware to not use a NFS or CIFS share!

The executing right for this directory needs to be set to ga+w
so the user `pbuilder`
(who the ccache will be run by) can create the needed subdirectories there.
If the directory doesn't exist
create
and set/correct the permissions on it.

The following one-liners should do the trick,
even if the directory exists:

{{{
sudo install -d -m 2775 -o 1234 -g 1234 /var/cache/pbuilder/ccache
echo "for i in $(echo {0..9}) $(echo {a..f}) tmp; do sudo
mkdir -p /var/cache/pbuilder/ccache/$i; done" | bash
sudo cho
wn -R 1234:1234 /var/cache/pbuilder/ccache
}}}

Next you have to tweak your `/etc/pbuilderrc` (or `$HOME/.pbuilderrc`).
Fill in the following part:
Line 104: Line 202:
That's all, next run while using git-pbuilder it will use ccache. === Error while configure script is running ===

If you running a build for the first time after setting up the ccache environment
sometimes an error occurs
while the configure script is trying to detect the C (or C++) Compiler
for creating object files.
It will look similar to this snippet:
{{{
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... configure: error: in `/build/package/foo':
configure: error: cannot compute suffix of object files: cannot compile
See `config.log' for more details
...
make: *** [build] Error 2
...
E: Failed autobuilding of package
}}}

This is related to wrong permissions
on the folder where ccache is storing the temporary files
(based on the setting from above
take a look at `/var/cache/pbuilder/ccache`
and correct the file permissions
by calling `sudo chown -R 1234:1234 /var/cache/pbuilder/ccache`.
Line 107: Line 230:
Without further options ccache will use a cache size of 1GB and endlessly amount of cached files. Depending on the package you build you will set up other maximums there. The needed caching size depends on the size of object files the build will produce. You have to investigate here. So maybe you wanna set the caching size to 4GB. This has to done in the chroot, so the only way to do this is a hook script. You need a hook script from typ A because the setting has to be set before the build starts. The script is right easy, put the following context as file `A10set_ccache_options` in your hook directory.

Without further options
ccache will use a cache size of 1GB
and an unli
mited amount of cached files.
Depending on the package you build
you will want to adjust those or other limits.
The needed caching size depends on the size of object files the build will produce.
You have to investigate here.
So
maybe you want to set the caching size to 4GB.
This has to be done in the chroot,
so the only way to do this is a hook script.
You need a hook script of type A
because it has to be set before the build starts.

Put s
omething like the following
as file `A10set_ccache_options` in your hook directory:
Line 121: Line 258:
With this hook script get a similar output right before build starts like this.
{{{
cache directory         /home/ccache
With above hook script you will get an output right before build starts
similar
to this:
{{{
cache directory /home/ccache
Line 142: Line 280:
You can spend a lot of time with waiting for the prepared chroot if you building packages with a big list of dependencies, even if the packages are cached inside the pbuilder apt directory. This annoying and in case of developing an tuning the package unnecessary. The build would be much quicker if the used chroot has already installed all dependencies.
As `git-pbuilder` can pass cowbuilder arguments as well the easiest way is to tell `git-pbuilder` which base chroot cowbuilder should use. But before that you have to create your desired chroot. To do this just copy the base directory to a new directory and name it as you want. You have to respect on rule, the new directory must start with 'base-'. So for example if you want to create a new base chroot based on the default sid/unstable chroot copy the `/var/cache/pbuilder/buildd/base.cow` to `/var/cache/pbuilder/buildd/base-$your_package.cow`.

You can spend a lot of time waiting for the prepared chroot
if you are
building packages with a big list of dependencies,
even if the packages are cached inside the pbuilder apt directory.
That is annoying, and unnecessary for developing and tuning the package.
The build would be much quicker
if the used chroot had
already installed all dependencies.
As `git-pbuilder` can pass cowbuilder arguments as well
the easiest way is to tell `git-pbuilder` which base chroot cowbuilder to use.
But before that you have to create your desired chroot.

Copy the base directory to a new directory
and name it as you want - except for one rule:
the new directory must start with 'base-'.
So for example if you want to create a new base chroot
based on the default sid/unstable chroot
copy the `/var/cache/pbuilder/buildd/base.cow`
to `/var/cache/pbuilder/buildd/base-my_package.cow`:
Line 149: Line 302:
Next you need to login into this new chroot and install all the needed dependencies persistently. For this you need a package list for `apt-get` or `dpkg --set-selections`.
Next you need to log into this new chroot
and install all the needed dependencies persistently.
For this you need a package list for `apt-get` or `dpkg --set-selections`:
Line 155: Line 309:
Now install the needed packages and log out.

{{{
#
apt-get install $(list of packages)
# #or with `dpkg --set-selections`
#
dpkg --set-selections < packagelist # created with 'dpkg --get selections \* > /tmp/packagelist'
}}}

You can now use this prepared chroot with the option `--git-dist=` of `git-buildpackage`.
{{{
git-
buildpackage --git-dist=my_package ... other options ...
Now install the needed packages and log out:
{{{
apt-get install $(list of packages)
}}}

Or alternatively with `dpkg --set-selections`:
{{{
dpkg --set-selections < packagelist # created with 'dpkg --get selections \* > /tmp/packagelist'
}}}

You can now use this prepared chroot with the `git-buildpackage` option `--git-dist=`:
{{{
g
bp --git-dist=my_package ... other options ...
Line 177: Line 333:
Of course `cowbuilder` uses a different location than `/tmp/[new]`. You need to optimize those 2 commands on your computer. They should take around 0.2s each. If not, try to use the ext3 filesystem, for more details, see our [[cowbuilder_benchmark| benchmarks]].
Of course `cowbuilder` uses a different location than `/tmp/[new]`.
You need to optimize those 2 commands on your computer.
They should take no more than a few milliseconds each.

This is a tutorial for git-pbuilder.

git-pbuilder is part of git-buildpackage and the usage is very similar to cowbuilder.

git-pbuilder needs sudo rights!

Usage

First, an environment needs to be created, then it can be used by git-buildpackage, with, for instance, the gbp buildpackage --git-pbuilder option.

Initialization and Variants

You can create base images for any architecture (e.g. i386 or amd64) and distribution (e.g. buster (oldstable), bullseye (stable) or sid (unstable), Ubuntu xxx, ...) that your hardware can run on.

Normal Usage

The easiest is to call with no further options. That will create a Sid (unstable) build environment with the architecture you are currently running on (e.g. if you are using i386, the environment will also be created for i386):

git-pbuilder create

The base build image is created in /var/cache/pbuilder/base.cow/.

Creating other Architecture

If you want to explicitly create an environment for i386 (e.g. while running on architecture amd64) you have to tell git-pbuilder:

ARCH=i386 git-pbuilder create

The base build image is created in /var/cache/pbuilder/base-sid-i386.cow/.

Creating Packages for other Distributions

Packaging is most commonly done in the Sid (unstable) distribution. When you occationally want to create packages for the stable or old-stable release, you have to tell this when git-pbuilder should create the environment.

E.g. to build for the Buster distribution you have to call git-pbuilder like this:

DIST=buster git-pbuilder create

The base image is created in /var/cache/pbuilder/base-buster-[your-platform].cow/.

A base image for the ancient Jessie Distribution is created similarly, only change the argument for DIST.

additional repositories probably needed

If you want to build packages as backport or as packages for stable-security please ensure you have added the correct repositories to the sources.list inside your base environment!

Normally git-pbuilder (pbuilder in the end) will only add a entry for the base repository of the distribution! You will probably need entries for security and [DIST]-update!

Using a Mirror

If you use git-pbuilder (or git-buildpackage) very often and can spare the additional disk space consumed, you can save download time and reduce traffic by using a local mirror or a proxy service.

If you have set up a local apt proxy the you can tell git-pbuilder to use it. Let's say you want to create a base image for Stretch and architecture amd64 with a caching proxy like AptCacherNg with the IP 192.168.110.4 on port 3142, here's the resulting command:

DIST=stretch ARCH=amd64 git-pbuilder create --mirror=http://192.168.110.4:3142/ftp.de.debian.org/debian

The base build image is created in /var/cache/pbuilder/base-stretch-amd64.cow/.

Updating

Make sure your build environment is reasonably up-to-date.

To update the base image just run:

git-pbuilder update

Updating base images for non-default distributions or architectures is similar to creating them. Updating the Buster environment for the same architecture you are currently running on would be:

DIST=buster git-pbuilder update

Or for Stretch on i386:

DIST=stretch ARCH=i386 git-pbuilder update

If you have used a mirror while creating the base images it will be used. So remember if you switch between multiple networks that the update will fail if the mirror isn't reachable. To use another mirror you have to set the variable MIRRORSITE to a valid mirror site:

MIRRORSITE="http://10.101.0.10:3142/ftp.debian.org/debian" git-pbuilder update

Installing Extra Packages

Sometimes you need extra packages installed on the base image. This is helpful for working off-line or to speed up the packaging. The workflow for that is similar to cowbuilder.

git-pbuilder login --save-after-login
# first step, update the package list
root@host:/# apt-get update
# then you can install any package
root@host:/# apt-get install vim screen less

You have to repeat those steps for each base image you use:

DIST=buster ARCH=amd64 git-pbuilder login --save-after-login
...

Using Local Packages

Sometimes the package you are trying to build build-depends on a library you just packaged, and is not available in the official repositories. PbuilderTricks explains how to solve this with pbuilder in general. With git-pbuilder, after having created the /etc/pbuilderrc (or /root/.pbuilderrc) and D05deps as instructed, you need to call this:

git-pbuilder update --override-config

Tips

Use of eatmydata

You can install eatmydata to improve the speed of your builds.

git-pbuilder login --save-after-login
root@host:/# apt-get update
root@host:/# apt-get install eatmydata

Using ccache

If you are often building the same package with a big source then it is useful to speed up a second build with ccache. To do so you have to tell pbuilder the needed environment for the use of ccache inside the chroot.

The ccache cache directory has to be placed somewhere in your file system. Suggested place is /var/cache/pbuilder/ccache but you can also put it e.g. under /home/ccache in case you have more free space there.

Beware to not use a NFS or CIFS share!

The executing right for this directory needs to be set to ga+w so the user pbuilder (who the ccache will be run by) can create the needed subdirectories there. If the directory doesn't exist create and set/correct the permissions on it.

The following one-liners should do the trick, even if the directory exists:

sudo install -d -m 2775 -o 1234 -g 1234 /var/cache/pbuilder/ccache
echo "for i in $(echo {0..9}) $(echo {a..f}) tmp; do sudo mkdir -p /var/cache/pbuilder/ccache/$i; done" | bash
sudo chown -R 1234:1234 /var/cache/pbuilder/ccache

Next you have to tweak your /etc/pbuilderrc (or $HOME/.pbuilderrc). Fill in the following part:

export CCACHE_DIR="/var/cache/pbuilder/ccache"
export PATH="/usr/lib/ccache:${PATH}"
EXTRAPACKAGES="ccache"
BINDMOUNTS="${CCACHE_DIR}"

Error while configure script is running

If you running a build for the first time after setting up the ccache environment sometimes an error occurs while the configure script is trying to detect the C (or C++) Compiler for creating object files. It will look similar to this snippet:

checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... configure: error: in `/build/package/foo':
configure: error: cannot compute suffix of object files: cannot compile
See `config.log' for more details
...
make: *** [build] Error 2
...
E: Failed autobuilding of package

This is related to wrong permissions on the folder where ccache is storing the temporary files (based on the setting from above take a look at /var/cache/pbuilder/ccache and correct the file permissions by calling sudo chown -R 1234:1234 /var/cache/pbuilder/ccache.

Changing standard ccache options

Without further options ccache will use a cache size of 1GB and an unlimited amount of cached files. Depending on the package you build you will want to adjust those or other limits. The needed caching size depends on the size of object files the build will produce. You have to investigate here. So maybe you want to set the caching size to 4GB. This has to be done in the chroot, so the only way to do this is a hook script. You need a hook script of type A because it has to be set before the build starts.

Put something like the following as file A10set_ccache_options in your hook directory:

# A10_set_ccache_options
# setting needed options to ccache
# possible options can be found on http://ccache.samba.org/manual.html#_options

# increase the ccache caching size
ccache -M 4G
# output the current statistics
ccache -s

With above hook script you will get an output right before build starts similar to this:

cache directory             /home/ccache
cache hit (direct)                     6
cache hit (preprocessed)               1
cache miss                           982
called for link                       57
called for preprocessing              26
compile failed                        18
preprocessor error                     8
bad compiler arguments                 2
unsupported source language            9
autoconf compile/link                133
unsupported compiler option            3
no input file                         24
files in cache                      2300
cache size                         619.5 Mbytes
max cache size                       4.0 Gbytes

Creating a specific base chroot

You can spend a lot of time waiting for the prepared chroot if you are building packages with a big list of dependencies, even if the packages are cached inside the pbuilder apt directory. That is annoying, and unnecessary for developing and tuning the package. The build would be much quicker if the used chroot had already installed all dependencies. As git-pbuilder can pass cowbuilder arguments as well the easiest way is to tell git-pbuilder which base chroot cowbuilder to use. But before that you have to create your desired chroot.

Copy the base directory to a new directory and name it as you want - except for one rule: the new directory must start with 'base-'. So for example if you want to create a new base chroot based on the default sid/unstable chroot copy the /var/cache/pbuilder/buildd/base.cow to /var/cache/pbuilder/buildd/base-my_package.cow:

sudo cp -a /var/cache/pbuilder/base.cow /var/cache/pbuilder/base-my_package.cow

Next you need to log into this new chroot and install all the needed dependencies persistently. For this you need a package list for apt-get or dpkg --set-selections:

DIST=my_package git-pbuilder login --save-after-login

Now install the needed packages and log out:

apt-get install $(list of packages)

Or alternatively with dpkg --set-selections:

dpkg --set-selections < packagelist # created with 'dpkg --get selections \* > /tmp/packagelist'

You can now use this prepared chroot with the git-buildpackage option --git-dist=:

gbp --git-dist=my_package ... other options ...

Troubleshooting

Slow copying and removing of the COW directory

What cowbuilder does is:

cp -al /var/cache/pbuilder/base.cow /tmp/new
rm -rf /tmp/[new]

Of course cowbuilder uses a different location than /tmp/[new]. You need to optimize those 2 commands on your computer. They should take no more than a few milliseconds each.