cowbuilder is a nice tool which builds packages (optionally in parallel) in clean chroots quickly by making use of copy-on-write technology. This document will describe how to set up your Debian/m68k ARAnyM to use an already prepared base.cow until such time as an updated Aranym/Quick image has been uploaded. (This should not take too long any more.)

Set up the repository

Make sure you have the following two lines in your /etc/apt/sources.list:

deb http://ftp.debian-ports.org/debian unstable main
deb http://ftp.debian-ports.org/debian unreleased main

If not, add them. You can also use one of the mirrors, of course. If you have to change /etc/apt/sources.list, I suggest to install debian-archive-keyring and debian-ports-archive-keyring beforehand. Run sudo apt-get update afterwards.

So, for example, do this:

$ sudo mv /etc/apt/sources.list /etc/apt/sources.list.old
$ sudo dd of=/etc/apt/sources.list <<EOF
deb http://ftp.debian-ports.org/debian unstable main
deb http://ftp.debian-ports.org/debian unreleased main
EOF
$ sudo env LC_ALL=C apt-get update
$ sudo apt-get --purge --no-install-recommends install debian-archive-keyring debian-ports-archive-keyring
$ sudo env LC_ALL=C apt-get update

If you come from Aranym/Quick

https://pfau.mirbsd.org/~tg/pub/dranym.png

Use apt-get --purge --no-install-recommends install FOO generally. Add the two keyring packages and openssh-server so you can ssh in instead of using the console.

You can add --no-upgrade to the apt-get command line to prevent more changes to the host OS than necessary. (Or you can just try and a-g d-u the host system too. Takes some hours…)

Install the necessary packages

You will need to install at least cowbuilder and a TLS capable kernel image of your choice (the ones in Debian include the modules, so even if you use Aranym which has the kernels “outside”, like classical Xen without pygrub, you must install a kernel image in the m68k VM for the LKMs to be available. An example image is linux-image-2.6.38-2-atari (which the author of this article uses on Aranym). You also do want to install debian-archive-keyring and debian-ports-archive-keyring (re-run sudo apt-get update to get rid of “WARNING: the following packages cannot be authenticated!” style messages).

It’s also good to install eatmydata as it yields a (small, but on m68k we’re thankful for every bit) speed-up (whose cost is the potential loss of data integrity, but as the chroots are thrown away after building, this doesn’t matter).

You also need tar and gzip (which you have anyway) to extract the tarball.

Again, an example (after the one above):

$ sudo apt-get --purge --no-install-recommends --no-upgrade install cowbuilder eatmydata linux-image-atari openssh-server tar gzip #xz-utils

Configure pbuilder (cowbuilder backend)

You want a user. Pray that uid and gid 1234 are not yet used on your system (otherwise, pbuilderrc needs more changes):

$ sudo addgroup --system --gid 1234 pbuilder
$ sudo adduser --system --no-create-home --uid 1234 --gecos pbuilder --disabled-login pbuilder

The file /etc/pbuilderrc wants love, unfortunately the defaults are not very good. Here’s one you can use (it’s basically a POSIX shell script, as to its syntax) and customise:

# to have a different shell prompt inside the chroot
export debian_chroot="pbuild$$"

# set this to your favourite debian-ports.org mirror
MIRRORSITE=http://ftp.debian-ports.org/debian

# desirable, unless you *want* to run most time-consuming
# testsuites, which usually fail anyway… (yay for optimism)
export DEB_BUILD_OPTIONS='nobench nocheck'

# either disable ccache…
CCACHEDIR=""
# … or enable it; yes, all of this is needed, and do ensure
# the directory and all of its content is writable by the
# pbuilder user
#CCACHEDIR="/var/cache/pbuilder/ccache"
#export CCACHE_DIR="/var/cache/pbuilder/ccache"
#export PATH="/usr/lib/ccache:${PATH}"

# which dependency resolver to use? default is aptitude,
# a package in very bad shape in m68k (but not rebuilt
# due to its heavy dependencies); it’s not usable atm.
# uncomment this to use one internal to pbuilder (shell script):
PBUILDERSATISFYDEPENDSCMD="/usr/lib/pbuilder/pbuilder-satisfydepends-classic"

# various settings
export DEBIAN_FRONTEND=noninteractive
#export ARCHITECTURE=$(dpkg --print-architecture)
export ARCHITECTURE=m68k
export PKGNAME_LOGFILE_EXTENTION=_${ARCHITECTURE}.build
export PKGNAME_LOGFILE=yes
#REMOVEPACKAGES="debconf-i18n"
export LC_ALL=C

Set up the cowbuilder “base” chroot

Download the base.cow tarball…

What’s in it?

I have debootstrapped this using cowbuilder with the three keyring packages (Debian, debian-ports.org, my repo) as well as the helpers’ in-chroot parts eatmydata and fakeroot, plus pbuilder’s helpers apt, ccache (see below). Then I set up its etc/apt/sources.list and etc/apt/apt.conf, purged everything not in Debian proper (i.e. the keyring part) and otherwise obsolete (atari and mac fdisk, atari bootstrap), upgraded it, copied it to my work system, cleaned it up, made a tarball. So it contains only Debian/m68k material and configuration files.

download, extract, set up

Get one of these tarballs (updated 2012-04-20):

Note: this includes the wtf-debian-keyring package and sources.list entry.

$ cd /var/cache/pbuilder
$ sudo mv base.cow base.old
$ gzip -dc ~/base.cow-m68k-*.tgz | sudo tar xpf -
$ sudo eatmydata cowbuilder --update

If you enabled/disabled ccache or switch the dependency resolver in /etc/pbuilderrc (or ~/.pbuilderrc of course), or change other settings such as EXTRAPACKAGES and REMOVEPACKAGES, you must run sudo cowbuilder --update (or sudo eatmydata cowbuilder --update which is perfectly safe) afterwards. This gives pbuilder a chance to install/purge its helper packages. The update will also run an apt-get update and upgrade inside the chroot. Since I omitted the lists and package database from the tarball, it’s always needed at least once after extraction.

… or make your own

At the moment, Debian/m68k requires packages from both the unstable and unreleased suites (we hope to be able to get rid of using unreleased again, eventually, but due to the squeeze freeze, bugs, slowness, etc. we’re not quite there yet). However, debootstrap, which is used by cowbuilder, cannot pull from more than one repository. So, you’d need to use an inofficial one instead (which also involves installing its keyring package into the host OS first, etc). Really, save yourself the hassle and use the pre-made tarball.

Also, Debian unstable is a moving target. I actually failed the first time trying to bootstrap, due to some dependency issue… once you have it done, upgrading can be partial (with packages held back until the architecture caught up) and things continue to work.

If you’re daring, a posting to the debian-68k@ mailing list describes the process. Or just ask me.

Bonus: The inofficial repository also contains some notes for advanced use/configuration of things, scripts to wrap cowbuilder, etc. If you know things about Debian, look at it.

Compile packages

Prepare a *.dsc file and run something like this: sudo cowbuilder --build mypkg_1.2-3.dsc

If you want to build only the binary parts, add: --debbuildopts -b

If you want to build only the binary-arch parts, add this (yes, in my experience, you need both) instead: --debbuildopts -B --binary-arch

If you want to force it to include the source (e.g. for uploads to Debian backports, Debian-Ports unreleased, etc.) use: --debbuildopts -sa

Real world examples (see below on Porter NMU for the use of the -m option):

$ sudo cowbuilder --debbuildopts "-m'$DEBEMAIL'" --debbuildopts -B --binary-arch --build gcc-4.4_4.4.5-10.dsc
$ sudo eatmydata cowbuilder --debbuildopts -sa --build util-linux_2.17.2-5+m68k.dsc

Do not bother with debuild/pdebuild, it’s useless (it first creates a .dsc, then runs pbuilder on that, instead of building directly from the unpacked source package). Instead, just create your *.dsc as usual (usually dpkg-buildpackage -rfakeroot -S or something like that) and proceed as above.

Pick up the results

Look in /var/cache/pbuilder/result/ for your *.changes files and the ones listed in it. There will also be *_m68k.build files containing the cowbuilder log.

Run debsign -k01234567 *.changes or debrsign -k01234567 user@host.domain *.changes afterwards, where 01234567 is your PGP Key ID, to get the *.changes and *.dsc, if any (not -b/-B), files signed. (Note that some debrsign versions may support only one file argument.)

You can upload to Debian-Ports using dput with this in ~/.dput.cf using dput dpo *.changes style commands:

[dpo]
fqdn                    = ftp.debian-ports.org
incoming                = /incoming/
login                   = anonymous
allow_dcut              = 1
method                  = ftp

If you want to upload by copying out to another box (for signing and processing) first, use something like this (the label, here p, is arbitrary but unique):

[p]
login                   = tg
fqdn                    = servername.domain
method                  = rsync
allow_unsigned_uploads  = 1
incoming                = ~tg/iS/

Advanced techniques

ccache

IMPORTANT when using ccache: If the toolchain (binutils, gcc, …) changes, you should reset the cache. It can only be done when no builds are running (which also gives you a chance to remove stale /var/cache/pbuilder/build/cow.* directories). This should work: sudo -u pbuilder env CCACHE_DIR=/var/cache/pbuilder/ccache /usr/bin/ccache -s

However, I could never get it to actually use ccache during compilation, even with the three variables (CCACHEDIR, export CCACHE_DIR, change PATH) set… maybe you ❦ have more luck and document that here ☺

When doing porter uploads…

You do not want to spam people. Add export DEBEMAIL='My Name <mylogin@debian.org>' to ~/.profile, and every time you build a package for uploading (Porter NMU), use a command line like this (as it’s a porter NMU, it targets unstable, and thus you only want the binary-arch part):

$ sudo eatmydata cowbuilder --debbuildopts -B --debbuildopts "-m'$DEBEMAIL'" --build --binary-arch foo_1.2.dsc

Use a hook to drop into a shell upon build failure

$ mkdir -p $HOME/pb-hook
$ cat >$HOME/pb-hook/C99fu <<'EOF'
#!/bin/sh
# $MirOS: contrib/hosted/tg/deb/hookdir/C99fu,v 1.1 2010/03/18 19:46:22 tg Exp $

echo
echo Build failed. Trying to invoke a shell.
echo
# optionally comment out the next two lines, will give you a dash instead
apt-get -y --force-yes install less mksh
ENV=/etc/skel/.mkshrc /bin/mksh -l 0<>/dev/tty >&0 2>&0 || \
    /bin/sh 0<>/dev/tty >&0 2>&0
EOF
$ chmod +x $HOME/pb-hook/*

Afterwards, add --hookdir $HOME/pb-hook to all cowbuilder invocations. This is nice to inspect the chroot to find out just why a build may have failed. (I’m not sure you can just chroot into /var/cache/pbuilder/build/cow.〈PID〉 or whether that may circumvent cowdancer, so don’t do it, just run on that shell.) Once you leave it, the chroot will be cleaned up.

Wrapper scripts

These must be chmod +x’d, of course.

Some explanations: * Resetting the locale, because sometimes, some things get passed from outside the chroot to the inside, yielding weird error messages about locales and other things. Probably should use env -i and a whitelist of allowed environment variables, which would also make stuff more deterministic (think of ${DEB_*} flags), but hinder easy customisation (not a problem though…). * Checking whether $DEBEMAIL is of the form User Name <login@debian.org> since we assume that when passing it to dpkg-buildpackage’s -m option – $DEBEMAIL has other valid forms, notably just the eMail address, but then you have to set other variables, so I never bothered and ever only set this one. * Feel free to improve upon those ☻

chroot maintenance

/usr/local/bin/eW

#!/bin/sh
# eatmydata (and C locale) Wrapper
# e.g. eW cowbuilder --update (if that then works…)
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata "$@"

/usr/local/bin/bL

#!/bin/sh
# cowBuilder Login and save afterwards
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --login --save-after-login

Development

/usr/local/bin/bS

#!/bin/sh
# cowBuilder Shell and discard afterwards
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --login --bindmounts $HOME

Package building

/usr/local/bin/ba

#!/bin/sh
# build all
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build "$@"

/usr/local/bin/bb

#!/bin/sh
# build binary arch and indep
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts -b --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build "$@"

/usr/local/bin/bB

#!/bin/sh
# build binary arch only
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts -B --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build --binary-arch "$@"

speed up

While no cowbuilder/pbuilder is running (the directory /var/cache/pbuilder/build is empty), you can edit files inside /var/cache/pbuilder/base.cow to your heart’s desire. The etc/apt/apt.conf and etc/apt/sources.list have already been customised; if you want, here’s some more:

You don’t really need deb-src lines in the in-chroot etc/apt/sources.list (unless you use apt-get build-dep commands inside it, either from a hook (e.g. as architecture wildcard workaround) or when logging in (bL and bS scripts above). They’re nice outside though: * dget -d http://…/foo_1.2-3.dsc * apt-get -d source foo=1.2-3 * apt-get -s build-dep foo=1.2-3

See whether you want Acquire::http::Pipeline-Depth "0"; in etc/apt/apt.conf or can do without.

prevent in-chroot man-db from updating its cache

TODO: same for info?

var/cache/debconf/config.dat

Name: man-db/auto-update
Template: man-db/auto-update
Value: false
Owners: man-db

Also nice / related to this:

touch …/base/cow/CurrentlyBuilding

force specific versions of packages to be installed

… for example, when we have a too-new arch:all package depending on an arch-dep package that’s not built yet, and we don’t want to recompile everything just to get this working. Note: wildcards are not supported, contrary to the pinning documentation, prior to, IIRC, version 0.8.14.1 of apt.

wget -O - https://www.freewrt.org/~tg/debs68k/2-prefs.txt | sudo dd of=/var/cache/pbuilder/base.cow/etc/apt/preferences

share APT package cache with host system

Do not run apt-get autoclean when you do this. Manual, if any, cleanup required.

$ sudo mv /var/cache/apt/archives/* /var/cache/pbuilder/aptcache/
$ cd /var/cache/apt
$ sudo rmdir archives
$ sudo ln -s ../pbuilder/aptcache archives
$ sudoedit /etc/apt/apt.conf

Then, append the following line (yes, that’s the apt.conf outside the chroot):

Dir::Cache::Archives "/var/cache/pbuilder/aptcache";

We keep the symlink just to stay out of trouble.

multi-distribution/-suite pbuilderrc

Not just for m68k but in general (not really useful on m68k). Can also build i386 packages on an amd64 system. Is multi-tenant capable. CVSweb link to latest version – Remember that packages uploaded to Debian proper MUST be built on bare metal, not in an emulated environment, except if the porter team for the target architecture allows otherwise (m68k explicitly allows ARAnyM, but e.g. ARM explicitly disallows emulated/virtualised environments)!