Differences between revisions 4 and 51 (spanning 47 versions)
Revision 4 as of 2007-09-15 21:58:05
Size: 734
Editor: OndrejCertik
Comment:
Revision 51 as of 2020-04-02 14:18:18
Size: 5429
Editor: RhondaDVine
Comment: small updates
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== cowbuilder == This is a tutorial for cowbuilder. See also [[qemubuilder]], [[sbuild]].
Line 3: Line 3:
This is the tutorial for cowbuilder. See also [:qemubuilder: qemubuilder]. <<TableOfContents>>
Line 9: Line 9:
Create the base repository: Create the base image:
Line 13: Line 13:
And a new and shiny build image is created in `/var/cache/pbuilder/base.cow/`
Line 16: Line 17:
Update the repository Update the base image
Line 26: Line 27:
= troubleshooting = = Tips =

== Bash and SVN tips ==

Setup your `.bashrc` according to: http://upsilon.cc/~zack/blog/posts/2007/09/svn-cowbuilder/

== Building your package for many distributions at once ==

`cowbuilder` is really handy to build your pet project for many distributions at once. If you have got an `amd64` system, you can easily build a package for `i386` and `amd64` architectures and for say `stable`, `sid` and `lucid`.

  * The first step involves choosing as many basepaths as you need. I chose to put them in `/var/cache/pbuilder/$DIST-$ARCH/base.cow`. Create the base directory, example:
    {{{
mkdir /var/cache/pbuilder/squeeze-i386
}}}

  * Then create the base image :

    {{{
sudo cowbuilder --create --basepath /var/cache/pbuilder/squeeze-i386/base.cow --distribution squeeze --debootstrapopts --arch --debootstrapopts i386
}}}

    {{{{#!wiki note
Creating an ubuntu base path is trickier because one must :
  1. Obtain ubuntu signing keys
  1. Use --keyring option to specific ubuntu signing keys
  1. add `universe` to complete creation of chroot because the `cowdancer` package is in universe.
    {{{
apt install ubuntu-archive-keyring
DIST=gutsy sudo cowbuilder --create --distribution gutsy --components "main universe" --basepath /var/cache/pbuilder/gutsy-amd64.cow --mirror http://apt:9999/ubuntu --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg
}}}
}}}}

  * Create a `~/.pbuilderrc` file, taking the one from [[https://wiki.ubuntu.com/PbuilderHowto|Ubuntu pbuilder howto]], in the `Multiple pbuilders` section, uncommenting the line
    {{{
BASEPATH="/var/cache/pbuilder/$NAME/base.cow/
}}}

  * Bring your cowbuilder chroot up to date by running
   {{{
sudo HOME=$HOME DIST=sid cowbuilder --update
}}}
  Setting HOME is necessary as sudo strips environment variables for security reasons (see [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567995|Debian bug]]) - otherwise cowbuilder will not find the .pbuilderrc you have just created.

  * You can now build up a package. For instance a backport of nano for squeeze:
   {{{
dget -x http://ftp.de.debian.org/debian/pool/main/n/nano/nano_2.2.6-1.dsc
# For a Debian Squeeze on i386 target
sudo DIST=squeeze ARCH=i386
# Ubuntu a Hardy on amd64 target
sudo DIST=hardy ARCH=amd64 cowbuilder --build nano_2.2.6-1.dsc
}}}

The result will be available in: `/var/cache/pbuilder/squeeze-i386/result`.

=== Optional ===
  * Once everything is set up, you can use a script in a debian source package directory to build it for every distribution.
{{{
#!/bin/bash
set -e

RESULTDIR=$1
[ -d $RESULTDIR ] || mkdir $RESULTDIR

DISTLIST=$(ls -d /var/cache/pbuilder/*.cow)

for chrootdir in $DISTLIST
do
    sudo cowbuilder --update --basepath $chrootdir
    DIST=$(basename $chrootdir | cut -d'-' -f1)
    [ -d $RESULTDIR/$DIST ] || mkdir $RESULTDIR/$DIST
    DIST=$DIST pdebuild --pbuilder cowbuilder\
                        --buildresult $RESULTDIR/$DIST\
                        -- --basepath $chrootdir
done
}}}

  * You can use an `apt` cache to reduce the downloads on mirrors, so if the mirror is `http://apt:9999/debian` or `http://apt:9999/ubuntu`, you shoud then add `--mirror http://apt:9999/debian`. A good and simple proxy to use is [[approx]].

  * You can use eatmydata inside and outside of the cowbuilder chroot to speed up dpkg
{{{
DIST=sid cowbuilder --login --save
apt-get install eatmydata
}}}
  * If you have pbuilder >= 0.225, just add this to your pbuilderrc:
{{{
EXTRAPACKAGES=eatmydata
EATMYDATA=yes
}}}
  * For eatmydata (<=26-2.1), add this /etc/pbuilderrc for :
{{{
export LD_PRELOAD=${LD_PRELOAD:+$LDPRELOAD:}/usr/lib/libeatmydata/libeatmydata.so
}}}
  * For eatmydata (>=82-2), add this /etc/pbuilderrc
{{{
export LD_PRELOAD=${LD_PRELOAD:+$LDPRELOAD:}libeatmydata.so
}}}

Please note eatmydata inside and outside of the cowbuilder chroot should be the same type.

== Using with git-buildpackage ==

The above config can be adapted to work with the defaults of git-buildpackage:

{{{
-BASEPATH="/var/cache/pbuilder/$NAME/base.cow/"
+BASEPATH="/var/cache/pbuilder/base-$NAME.cow/"
}}}

Then the images can be created with:

{{{
DIST=sid ARCH=amd64 git-pbuilder create
}}}

And packages be built with:

{{{
DIST=sid ARCH=amd64 git-buildpackage --git-pbuilder
}}}

A bug was opened to make sure the ARCH argument is coherent with the pbuilder way (ie. that it's optional), see: [[DebianBug:646882|#646882]]

= Troubleshooting =
Line 35: Line 158:
Of course `cowbuilder` uses a different location than `/tmp/new`. Try those commands and see how fast they are on your computer. You can compare them to our [:cowbuilder_benchmark: benchmarks]. Of course `cowbuilder` uses a different location than `/tmp/new`. You need to optimize those 2 commands on your computer. They should take around 0.2s each. If not, try to use the ext3 filesystem, for more details, see our [[cowbuilder_benchmark| benchmarks]].

This is a tutorial for cowbuilder. See also qemubuilder, sbuild.

Usage

Initialization

Create the base image:

sudo cowbuilder --create

And a new and shiny build image is created in /var/cache/pbuilder/base.cow/

Every day usage

Update the base image

sudo cowbuilder --update

Build a package:

sudo cowbuilder --build somepackage.dsc

Tips

Bash and SVN tips

Setup your .bashrc according to: http://upsilon.cc/~zack/blog/posts/2007/09/svn-cowbuilder/

Building your package for many distributions at once

cowbuilder is really handy to build your pet project for many distributions at once. If you have got an amd64 system, you can easily build a package for i386 and amd64 architectures and for say stable, sid and lucid.

  • The first step involves choosing as many basepaths as you need. I chose to put them in /var/cache/pbuilder/$DIST-$ARCH/base.cow. Create the base directory, example:

    • mkdir /var/cache/pbuilder/squeeze-i386
  • Then create the base image :
    • sudo cowbuilder --create --basepath /var/cache/pbuilder/squeeze-i386/base.cow --distribution squeeze --debootstrapopts --arch --debootstrapopts i386

      Creating an ubuntu base path is trickier because one must :

      1. Obtain ubuntu signing keys
      2. Use --keyring option to specific ubuntu signing keys
      3. add universe to complete creation of chroot because the cowdancer package is in universe.

        • apt install ubuntu-archive-keyring
          DIST=gutsy sudo cowbuilder --create --distribution gutsy --components "main universe" --basepath /var/cache/pbuilder/gutsy-amd64.cow --mirror http://apt:9999/ubuntu --debootstrapopts --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg
  • Create a ~/.pbuilderrc file, taking the one from Ubuntu pbuilder howto, in the Multiple pbuilders section, uncommenting the line

    • BASEPATH="/var/cache/pbuilder/$NAME/base.cow/
  • Bring your cowbuilder chroot up to date by running
    • sudo HOME=$HOME DIST=sid cowbuilder --update

    Setting HOME is necessary as sudo strips environment variables for security reasons (see Debian bug) - otherwise cowbuilder will not find the .pbuilderrc you have just created.

  • You can now build up a package. For instance a backport of nano for squeeze:
    • dget -x http://ftp.de.debian.org/debian/pool/main/n/nano/nano_2.2.6-1.dsc
      # For a Debian Squeeze on i386 target
      sudo DIST=squeeze ARCH=i386 
      # Ubuntu a Hardy on amd64 target
      sudo DIST=hardy ARCH=amd64 cowbuilder --build nano_2.2.6-1.dsc

The result will be available in: /var/cache/pbuilder/squeeze-i386/result.

Optional

  • Once everything is set up, you can use a script in a debian source package directory to build it for every distribution.

set -e

RESULTDIR=$1
[ -d $RESULTDIR ] || mkdir $RESULTDIR

DISTLIST=$(ls -d /var/cache/pbuilder/*.cow)

for chrootdir in $DISTLIST
do
    sudo cowbuilder --update --basepath $chrootdir
    DIST=$(basename $chrootdir | cut -d'-' -f1)
    [ -d $RESULTDIR/$DIST ] || mkdir $RESULTDIR/$DIST
    DIST=$DIST pdebuild --pbuilder cowbuilder\
                        --buildresult $RESULTDIR/$DIST\
                        -- --basepath $chrootdir
done
  • You can use an apt cache to reduce the downloads on mirrors, so if the mirror is http://apt:9999/debian or http://apt:9999/ubuntu, you shoud then add --mirror http://apt:9999/debian. A good and simple proxy to use is ?approx.

  • You can use eatmydata inside and outside of the cowbuilder chroot to speed up dpkg

DIST=sid cowbuilder --login --save
apt-get install eatmydata
  • If you have pbuilder >= 0.225, just add this to your pbuilderrc:

EXTRAPACKAGES=eatmydata
EATMYDATA=yes
  • For eatmydata (<=26-2.1), add this /etc/pbuilderrc for :

export LD_PRELOAD=${LD_PRELOAD:+$LDPRELOAD:}/usr/lib/libeatmydata/libeatmydata.so
  • For eatmydata (>=82-2), add this /etc/pbuilderrc

export LD_PRELOAD=${LD_PRELOAD:+$LDPRELOAD:}libeatmydata.so

Please note eatmydata inside and outside of the cowbuilder chroot should be the same type.

Using with git-buildpackage

The above config can be adapted to work with the defaults of git-buildpackage:

-BASEPATH="/var/cache/pbuilder/$NAME/base.cow/"
+BASEPATH="/var/cache/pbuilder/base-$NAME.cow/"

Then the images can be created with:

DIST=sid ARCH=amd64 git-pbuilder create

And packages be built with:

DIST=sid ARCH=amd64 git-buildpackage --git-pbuilder

A bug was opened to make sure the ARCH argument is coherent with the pbuilder way (ie. that it's optional), see: #646882

Troubleshooting

Slow copying and removing of the COW directory

What cowbuilder does is:

cp -al /var/cache/pbuilder/base.cow /tmp/new
rm -rf /tmp/new

Of course cowbuilder uses a different location than /tmp/new. You need to optimize those 2 commands on your computer. They should take around 0.2s each. If not, try to use the ext3 filesystem, for more details, see our benchmarks.