Differences between revisions 13 and 14
Revision 13 as of 2010-12-31 23:33:47
Size: 19213
Comment: new issue
Revision 14 as of 2011-01-30 21:23:49
Size: 19701
Comment: improve copy-paste parts
Deletions are marked like this. Additions are marked like this.
Line 96: Line 96:
$ xz -dc /path/to/base.cow-m68k-*.txz | sudo tar xpf - $ xz -dc ~/base.cow-m68k-*.txz | sudo tar xpf -
Line 164: Line 164:
}}}

If you want to upload by copying out to another box (for signing and processing) first, use something like this (the label, here '''p''', is arbitrary but unique):
{{{
[p]
login = tg
fqdn = servername.domain
method = rsync
allow_unsigned_uploads = 1
incoming = ~tg/iS/
Line 323: Line 333:
Package: libxmu-headers
Pin: version 2:1.0.4-1
Pin-Priority: 1001

Package: mesa-common-dev
Pin: version 7.0.3-6
Pin-Priority: 1001

cowbuilder is a nice tool which builds packages (optionally in parallel) in clean chroots quickly by making use of copy-on-write technology. This document will describe how to set up your Debian/m68k ARAnyM to use an already prepared base.cow until such time as an updated Aranym/Quick image has been uploaded. (This should not take too long any more.)

Set up the repository

Make sure you have the following two lines in your /etc/apt/sources.list:

deb http://ftp.debian-ports.org/debian unstable main
deb http://ftp.debian-ports.org/debian unreleased main

If not, add them. You can also use one of the mirrors, of course. If you have to change /etc/apt/sources.list, I suggest to install debian-archive-keyring and debian-ports-archive-keyring beforehand. Run sudo apt-get update afterwards.

So, for example, do this:

$ sudo mv /etc/apt/sources.list /etc/apt/sources.list.old
$ sudo dd of=/etc/apt/sources.list <<EOF
deb http://ftp.debian-ports.org/debian unstable main
deb http://ftp.debian-ports.org/debian unreleased main
EOF
$ sudo env LC_ALL=C apt-get update
$ sudo apt-get --purge --no-install-recommends install debian-archive-keyring debian-ports-archive-keyring
$ sudo env LC_ALL=C apt-get update

If you come from Aranym/Quick

Use apt-get --purge --no-install-recommends install FOO generally. Add the two keyring packages and openssh-server so you can ssh in instead of using the console. Append video=atafb:vga2 to the [LILO] Args=… when using the newer kernels, otherwise you won’t see a thing (see this gmane.linux.ports.m68k thread for details).

You can add --no-upgrade to the apt-get command line to prevent more changes to the host OS than necessary. (Or you can just try and a-g d-u the host system too. Takes some hours…)

Install the necessary packages

You will need to install at least cowbuilder and a TLS capable kernel image of your choice (the ones in Debian include the modules, so even if you use Aranym which has the kernels “outside”, like classical Xen without pygrub, you must install a kernel image in the m68k VM for the LKMs to be available. An example image is linux-image-2.6.32-5+m68k.3-atari (which the author of this article uses on Aranym). You also do want to install debian-archive-keyring and debian-ports-archive-keyring (re-run sudo apt-get update to get rid of “WARNING: the following packages cannot be authenticated!” style messages).

It’s also good to install eatmydata as it yields a (small, but on m68k we’re thankful for every bit) speed-up (whose cost is the potential loss of data integrity, but as the chroots are thrown away after building, this doesn’t matter).

You also need tar (which you have anyway) and xz-utils to extract the tarball (xz -7e compression needs “only” around 17 MiB RAM for decompression, so I used that, except I fscked up and forgot to switch the checksum algorithm to CRC32…).

Again, an example (after the one above):

$ sudo apt-get --purge --no-install-recommends --no-upgrade install cowbuilder eatmydata linux-image-2.6.32-5+m68k.3-atari openssh-server tar xz-utils

Configure pbuilder (cowbuilder backend)

You want a user. Pray the uid 1234 is not yet used on your system (otherwise, pbuilderrc needs more changes):

$ sudo adduser --system --no-create-home --uid 1234 --gecos pbuilder --disabled-login pbuilder

The file /etc/pbuilderrc wants love, unfortunately the defaults are not very good. Here’s one you can use (it’s basically a POSIX shell script, as to its syntax) and customise:

# to have a different shell prompt inside the chroot
export debian_chroot="pbuild$$"

# set this to your favourite debian-ports.org mirror
MIRRORSITE=http://ftp.debian-ports.org/debian

# desirable, unless you *want* to run most time-consuming
# testsuites, which usually fail anyway… (yay for optimism)
export DEB_BUILD_OPTIONS='nobench nocheck'

# either disable ccache…
CCACHEDIR=""
# … or enable it; yes, all of this is needed, and do ensure
# the directory and all of its content is writable by the
# pbuilder user
#CCACHEDIR="/var/cache/pbuilder/ccache"
#export CCACHE_DIR="/var/cache/pbuilder/ccache"
#export PATH="/usr/lib/ccache:${PATH}"

# which dependency resolver to use? default is aptitude,
# a package in very bad shape in m68k (but not rebuilt
# due to its heavy dependencies); it’s usable though.
# uncomment this to use one internal to pbuilder (shell script):
#PBUILDERSATISFYDEPENDSCMD="/usr/lib/pbuilder/pbuilder-satisfydepends-classic"

Set up the cowbuilder “base” chroot

Download the base.cow tarball…

What’s in it?

I have debootstrapped this using cowbuilder with the three keyring packages (Debian, debian-ports.org, my repo) as well as the helpers’ in-chroot parts eatmydata and fakeroot, plus pbuilder’s helpers apt, aptitude (see below), ccache (see below). Then I set up its etc/apt/sources.list and etc/apt/apt.conf, purged everything not in Debian proper (i.e. the keyring part, but not the helper to keep aptitude installable), upgraded it, copied it to my work system, cleaned it up, made a tarball. So it contains only Debian/m68k material and two APT configuration files.

download, extract, set up

Last changed: 2010-12-31PGP signature with my DD key

Get it from people.d.o or my ~ on FreeWRT.

$ cd /var/cache/pbuilder
$ xz -dc ~/base.cow-m68k-*.txz | sudo tar xpf -

If you enabled/disabled ccache or switch the dependency resolver in /etc/pbuilderrc (or ~/.pbuilderrc of course), you must run sudo cowbuilder --update (or sudo eatmydata cowbuilder --update which is perfectly safe) after extracting the base tarball. This gives pbuilder a chance to install/purge its helper packages.

… or make your own

At the moment, Debian/m68k requires packages from both the unstable and unreleased suites (we hope to be able to get rid of using unreleased again, eventually, but due to the squeeze freeze, bugs, slowness, etc. we’re not quite there yet). However, debootstrap, which is used by cowbuilder, cannot pull from more than one repository. So, you’d need to use an inofficial one instead (which also involves installing its keyring package into the host OS first, etc). Really, save yourself the hassle and use the pre-made tarball.

Also, Debian unstable is a moving target. I actually failed the first time trying to bootstrap, due to some dependency issue… once you have it done, upgrading can be partial (with packages held back until the architecture caught up) and things continue to work.

If you’re daring, a posting to the debian-68k@ mailing list describes the process. Or just ask me.

Bonus: The inofficial repository also contains some notes for advanced use/configuration of things, scripts to wrap cowbuilder, etc. If you know things about Debian, look at it.

Finishing touches

Running sudo eatmydata cowbuilder --update right now (and every once in a while, immediately before using it) is a good idea anyway, even if pbuilderrc was not changed, as it involves an apt-get update inside the chroot, which is needed for it to pick up the most up-to-date build dependencies when compiling.

Compile packages

Prepare a *.dsc file and run something like this: sudo cowbuilder --build mypkg_1.2-3.dsc

If you want to build only the binary parts, add: --debbuildopts -b

If you want to build only the binary-arch parts, add this (yes, in my experience, you need both) instead: --debbuildopts -B --binary-arch

If you want to force it to include the source (e.g. for uploads to Debian backports, Debian-Ports unreleased, etc.) use: --debbuildopts -sa

Real world examples (see below on Porter NMU for the use of the -m option):

$ sudo cowbuilder --debbuildopts "-m'$DEBEMAIL'" --debbuildopts -B --binary-arch --build gcc-4.4_4.4.5-10.dsc
$ sudo eatmydata cowbuilder --debbuildopts -sa --build util-linux_2.17.2-5+m68k.dsc

Do not bother with debuild/pdebuild, it’s useless (it first creates a .dsc, then runs pbuilder on that, instead of building directly from the unpacked source package). Instead, just create your *.dsc as usual (usually dpkg-buildpackage -rfakeroot -S or something like that) and proceed as above.

Problems

At the moment, pbuilder still does not support architecture wildcards, that is, things like this:

Source: dpkg
Version: 1.15.8.7
Build-Depends: debhelper (>= 6.0.7), pkg-config, gettext (>= 0.18), po4a (>= 0.33.1), libncursesw5-dev, zlib1g-dev (>= 1:1.1.3-19.1), libbz2-dev, flex, libselinux1-dev (>= 1.28-4) [linux-any], libtimedate-perl, libio-string-perl

Here, libselinux1-dev will not get installed by pbuilder-satisfydepends, and dpkg-buildpackage will error out later. This is filed as bug, but somehow there seems to be no progress despite several patches and suggestions… but then, pbuilder has a lot of bugs. Still, this is troubling, especially when buildd doesn’t work. Workaround: use a (separate) hook dir to install the dependencies first… *sigh*

$ cat pb-dpkg/D00deps
#!/bin/sh
apt-get -y --force-yes -o Dpkg::Options::=--force-confnew,confmiss install libselinux1-dev

Pick up the results

Look in /var/cache/pbuilder/result/ for your *.changes files and the ones listed in it.

Run debsign -k01234567 *.changes or debrsign -k01234567 user@host.domain *.changes afterwards, where 01234567 is your PGP Key ID, to get the *.changes and *.dsc, if any (not -b/-B), files signed. (Note that some debrsign versions may support only one file argument.)

You can upload to Debian-Ports using dput with this in ~/.dput.cf using dput dpo *.changes style commands:

[dpo]
fqdn                    = ftp.debian-ports.org
incoming                = /incoming/
login                   = anonymous
allow_dcut              = 1
method                  = ftp

If you want to upload by copying out to another box (for signing and processing) first, use something like this (the label, here p, is arbitrary but unique):

[p]
login                   = tg
fqdn                    = servername.domain
method                  = rsync
allow_unsigned_uploads  = 1
incoming                = ~tg/iS/

Advanced techniques

ccache

IMPORTANT when using ccache: If the toolchain (binutils, gcc, …) changes, you should reset the cache. It can only be done when no builds are running (which also gives you a chance to remove stale /var/cache/pbuilder/build/cow.* directories). This should work: sudo -u pbuilder env CCACHE_DIR=/var/cache/pbuilder/ccache /usr/bin/ccache -s

However, I could never get it to actually use ccache during compilation, even with the three variables (CCACHEDIR, export CCACHE_DIR, change PATH) set… maybe you ❦ have more luck and document that here ☺

When doing porter uploads…

You do not want to spam people. Add export DEBEMAIL='My Name <mylogin@debian.org>' to ~/.profile, and every time you build a package for uploading (Porter NMU), use a command line like this (as it’s a porter NMU, it targets unstable, and thus you only want the binary-arch part):

$ sudo eatmydata cowbuilder --debbuildopts -B --debbuildopts "-m'$DEBEMAIL'" --build --binary-arch foo_1.2.dsc

Use a hook to drop into a shell upon build failure

$ mkdir -p $HOME/pb-hook
$ cat >$HOME/pb-hook/C99fu <<'EOF'
#!/bin/sh
# $MirOS: contrib/hosted/tg/deb/hookdir/C99fu,v 1.1 2010/03/18 19:46:22 tg Exp $

echo
echo Build failed. Trying to invoke a shell.
echo
# optionally comment out the next two lines, will give you a dash instead
apt-get -y --force-yes install less mksh
ENV=/etc/skel/.mkshrc /bin/mksh -l 0<>/dev/tty >&0 2>&0 || \
    /bin/sh 0<>/dev/tty >&0 2>&0
EOF
$ chmod +x $HOME/pb-hook/*

Afterwards, add --hookdir $HOME/pb-hook to all cowbuilder invocations. This is nice to inspect the chroot to find out just why a build may have failed. (I’m not sure you can just chroot into /var/cache/pbuilder/build/cow.〈PID〉 or whether that may circumvent cowdancer, so don’t do it, just run on that shell.) Once you leave it, the chroot will be cleaned up.

Wrapper scripts

These must be chmod +x’d, of course.

Some explanations: * Resetting the locale, because sometimes, some things get passed from outside the chroot to the inside, yielding weird error messages about locales and other things. Probably should use env -i and a whitelist of allowed environment variables, which would also make stuff more deterministic (think of ${DEB_*} flags), but hinder easy customisation (not a problem though…). * Checking whether $DEBEMAIL is of the form User Name <login@debian.org> since we assume that when passing it to dpkg-buildpackage’s -m option – $DEBEMAIL has other valid forms, notably just the eMail address, but then you have to set other variables, so I never bothered and ever only set this one. * Feel free to improve upon those ☻

chroot maintenance

/usr/local/bin/eW

#!/bin/sh
# eatmydata (and C locale) Wrapper
# e.g. eW cowbuilder --update (if that then works…)
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata "$@"

/usr/local/bin/bL

#!/bin/sh
# cowBuilder Login and save afterwards
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --login --save-after-login

Development

/usr/local/bin/bS

#!/bin/sh
# cowBuilder Shell and discard afterwards
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --login --bindmounts $HOME

Package building

/usr/local/bin/ba

#!/bin/sh
# build all
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build "$@"

/usr/local/bin/bb

#!/bin/sh
# build binary arch and indep
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts -b --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build "$@"

/usr/local/bin/bB

#!/bin/sh
# build binary arch only
case $DEBEMAIL in
*' <'*'@'*'>') ;;
*) echo >&2 "\$DEBEMAIL ($DEBEMAIL) not correct"; exit 1 ;;
esac
exec env LANG=C LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C LC_ALL=C eatmydata cowbuilder --debbuildopts -B --debbuildopts "-m'$DEBEMAIL'" --hookdir $HOME/pb-hook --build --binary-arch "$@"

speed up

While no cowbuilder/pbuilder is running (the directory /var/cache/pbuilder/build is empty), you can edit files inside /var/cache/pbuilder/base.cow to your heart’s desire. The etc/apt/apt.conf and etc/apt/sources.list have already been customised; if you want, here’s some more:

You don’t really need deb-src lines in the in-chroot etc/apt/sources.list (unless you use apt-get build-dep commands inside it, either from a hook (e.g. as architecture wildcard workaround) or when logging in (bL and bS scripts above). They’re nice outside though: * dget -d http://…/foo_1.2-3.dsc * apt-get -d source foo=1.2-3 * apt-get -s build-dep foo=1.2-3

See whether you want Acquire::http::Pipeline-Depth "0"; in etc/apt/apt.conf or can do without.

prevent in-chroot man-db from updating its cache

TODO: same for info?

var/cache/debconf/config.dat

Name: man-db/auto-update
Template: man-db/auto-update
Value: false
Owners: man-db

force specific versions of packages to be installed

… for example, when we have a too-new arch:all package depending on an arch-dep package that’s not built yet, and we don’t want to recompile everything just to get this working. Note: wildcards are not supported, contrary to the pinning documentation.

etc/apt/preferences

Package: kdelibs5-data
Pin: version 4:4.2.2-2
Pin-Priority: 1001

Package: libxi-dev
Pin: version 1:1.0.1-4
Pin-Priority: 1001

Package: libxi6
Pin: version 1:1.0.1-4
Pin-Priority: 1001

Package: libxmu-headers
Pin: version 2:1.0.4-1
Pin-Priority: 1001

Package: mesa-common-dev
Pin: version 7.0.3-6
Pin-Priority: 1001

Package: x11proto-fixes-dev
Pin: version 4.0-2
Pin-Priority: 1001

Package: x11proto-input-dev
Pin: version 1.3.2-4
Pin-Priority: 1001

Package: x11proto-record-dev
Pin: version 1.13.2-4
Pin-Priority: 1001

Package: x11proto-scrnsaver-dev
Pin: version 1.1.0.0-1
Pin-Priority: 1001

Package: x11proto-xext-dev
Pin: version 7.0.2-5
Pin-Priority: 1001

Package: x11proto-xinerama-dev
Pin: version 1.1.2-4
Pin-Priority: 1001

share APT package cache with host system

Do not run apt-get autoclean when you do this. Manual, if any, cleanup required.

$ sudo mv /var/cache/apt/archives/* /var/cache/pbuilder/aptcache/
$ cd /var/cache/apt
$ sudo rmdir archives
$ sudo ln -s ../pbuilder/aptcache archives
$ sudoedit /etc/apt/apt.conf

Then, append the following line (yes, that’s the apt.conf outside the chroot):

Dir::Cache::Archives "/var/cache/pbuilder/aptcache";

We keep the symlink just to stay out of trouble.