Cross-installing Debian using debootstrap/multistrap

(Nothing on this page is specific to emdebian, rather than debian)

From debootstrap's manpage:

Basically, bootstrapping a Debian system involves four steps (some were ommitted for simplicity):

  1. Download the necessary .deb packages from a repository.
  2. Unpack them into the target directory.
  3. Chroot into the target directory.
  4. Run the installation and configuration scripts from each package, finishing the setup.

Usually, when doing cross-bootstrapping, steps 3 and 4 need to be done on the target side. This document explains how to bootstrap a Debian system for a target architecture different from the host's one, doing all steps on the host side and, optionally, using QEMU to run target executables.

This is useful, for example, to create filesystem images without depending on the target board to be available.

Requirements

There are lots of ways of doing this. You can use multistrap (recommended) or debootstrap or cdebootstrap for stages 1 and 2. And QEMU or real hardware for stages 3 & 4.

Four variant approaches are described on this page:

  1. Multistrap + real hardware
    • Multistrap + config script for rootfs creation
    • hardware or QEMU for configuration.
  2. QEMU
    • QEMU with transparent user mode emulation (see QemuUserEmulation)

    • debootstrap, with patch (355801) applied to support cross-bootstrapping

  3. Real hardware
    • An ARM machine which can mount an NFS root
    • cdebootstrap (in squeeze)
  4. Unpacking method
    • Any Debian box, to support any ARM machine with serial, network or USB connectivity
    • emsandbox (debootstrap --foreign).

This page will try to stick to general issues around how a cross bootstrap differs from typical usage of debootstrap. See also EmDebian/DeBootstrap which deals with specific issues with particular boards or devices.

Now there’s also 'debootstrap --foreign' + 'debootstrap --second-stage', which should erase the need to “re-install everything to get the configuration scripts run”. The author of this document is asked to look into this. --mirabilos

Multistrap

Multistrap is a tool which does essentially the same job as Debootstrap but with a different design, giving a lot more flexibility, and a different set of pros/cons. It works in a completely different way by simply using apt and dpkg, rather than avoiding using them, which is how debootstrap works. It is focussed on producing rootfs images for devices, as opposed to chroots for existing machines, but in fact (like debootstrap) does both of these jobs almost equally well.

Advantages:

Limitations:

Multistrap creates the image from the packages needed; it does not make device/chroot-specific modifications. Most real devices need a set of such changes to operate (console, inittab, fstab, devicenodes, etc) and a script or manual changes are needed. It has hooks to help automate this sort of config.

apt-get install multistrap

Example use

Basic usage

To create a new image in targetdir for arch armel using repositories specified in multistrap.conf: (both arch and dir can be specified in the config file too, if you prefer)

# multistrap -a armel -d targetdir -f multistrap.conf

This is a minimal config which will make a grip rootfs using just the packages which are Priority: required (essential is ignored by multistrap, making it much easier to only get what you want/need). Noauth stops secure-apt complaining about keys if no keyring package is specified.

[General]
noauth=true
unpack=true
debootstrap=Grip
aptsources=Grip

[Grip]
# space separated package list
source=http://www.emdebian.org/grip
suite=lenny

The (tared-up) result is a 30Mb tarball (approx 50% apt cache contents). Adding

cleanup=true

to the config removes the apt cache debs and lists, resulting in a 15Mb tarball.

multiple repository usage

This is a more complex example which uses four repositories: grip, debian (for extra packages not in grip), grip updates from proposed-updates, and a local repository, as well as specifying some optional packages from those repositories. The important thing is that for every name listed in 'aptsources' you need a correspondingly named section containing at least source and suite. The use of keyring packages and specifying components other than the default of 'main' is also shown.

I've set the cleanup option (which tidies up /var/cache/apt/ ), the retainsources option (which keeps all downloaded debs in a directory so you have a copy of what was used to make this rootfs) and an example of setting the arch and target dir in the config instead of on the command line:

[General]
unpack=true
debootstrap=Grip
aptsources=Grip Updates Debian Localserver
cleanup=true
arch=armel
directory=/tmp/targetdir/
retainsources=/tmp/targetdir/sources

[Grip]
#space separated package list
packages=udev ntpdate
source=http://www.emdebian.org/grip
keyring=emdebian-archive-keyring
suite=lenny

[Updates]
packages=apt
source=http://www.emdebian.org/grip
keyring=emdebian-archive-keyring
suite=lenny-proposed-updates

[Debian]
packages=erlang-nox
source=http://ftp.uk.debian.org/debian
keyring=debian-archive-keyring
suite=lenny

[Localserver]
packages=sl40-platform-debian balloon3-config
source=http://aptcache.localhost:4132/debian
keyring=localrepo-archive-keyring
suite=development

Running

sudo multistrap -f multistrap.conf

updates package lists, takes packages from all 4 repositories (checking them against the specified keyrings) and builds the rootfs, clears out the apt-cache and saves all the debs in the directory specified in retainsources.

packages from multiple components

As of multistrap version v2.0.6, it is possible to use the "components=" option to load packages from components other than main. Here is an example to include a firmware package from non-free:

[General]
noauth=true
unpack=true
debootstrap=Squeeze
aptsources=Squeeze
arch=armel

[Squeeze]
packages=wpasupplicant wireless-tools firmware-ralink
source=http://ftp.au.debian.org/debian/
keyring=debian-archive-keyring
components=main non-free
suite=squeeze

Configuring the rootfs

The rootfs produced is chrootable, but often not bootable without fixing up a few files. See the QEMU/DeBootstrap section below for examples of the changes usually needed. Normaly it's best to script those changes for repeatability.

If you can, it usually easier to chroot into the rootfs (on an ARM machine, or using qemu) to configure it as less work is required than making it bootable.

chroot /tmp/targetdir
dpkg --configure -a

will run all the postinst scripts of the installed packages and complete the rootfs generation process. Tar up the result and copy it onto your device.

Alternatively, if you can boot the image on the device then you can do the

dpkg --configure -a

on the device directly.

In order to complete the configure of dash you should

/var/lib/dpkg/info/dash.preinst install

as described on the Multistrap page.

QEMU/debootstrap approach

Installing required packages

Install required packages using the following command:

# apt-get install binfmt-support qemu qemu-user-static debootstrap

Generating cross images as root user

The following steps need to be done as root, because debootstrap will create device nodes (using mknod) as well as chroot into the newly created system, which require root privileges.

  1. Choose the target's architecture and Debian suite (e.g. stable, testing, sid) you want to bootstrap. For this tutorial, we will choose the ARM target architecture and the stretch (AKA Debian testing) suite.
  2. Choose a (empty) directory where you want the new system to be populated. This directory will reflect the root filesystem of the new system. Note that the host system is not affected in any way, and all modifications are restricted to the directory you have chosen.

    Note: For the purposes of this document, we will use "debian_armel_stretch" as the target directory. Create this directory, if it does not exist already:

    # mkdir debian_armel_stretch
  3. To bootstrap the new system, just run qemu-debootstrap. E.g.:

    # qemu-debootstrap --arch armel stretch debian_armel_stretch http://deb.debian.org/debian/
    where "debian_armel_stretch" is the directory you have created and "armel" is the target architecture. See debootstrap(8) manpage for more details about each parameter used above.

    "http://deb.debian.org/debian/" is the Debian mirror from which the necessary .deb packages will be downloaded. You are free to choose any mirror you like, as long as it has the architecture you are trying to bootstrap. See http://www.debian.org/mirror/list for the list of available Debian mirrors.

  4. The new system is now created. By default, debootstrap creates a very minimal system, so you will probably want to extend it (see below 'Configuring the new system' for some instructions).

Generating cross images as non-root user

[General]
# arch and directory can be specified on the command line.
arch=armel
directory=/tmp/chroot_armel
omitrequired=false
# same as --tidy-up option if set to true
cleanup=true
# retain the sources outside the rootfs for distribution
# specify a directory to which all the .debs can be moved.
# or override with the --source-dir option.
retainsources=
# same as --no-auth option if set to true
# keyring packages listed in each debootstrap will
# still be installed.
noauth=true
# extract all downloaded archives
unpack=true
# the order of sections is no longer important.
# debootstrap determines which repository is used to
# calculate the list of Priority: required packages
debootstrap=Grip
# the order of sections is no longer important.
# aptsources is a list of sections to be listed
# in the /etc/apt/sources.list.d/multistrap.sources.list
# of the target.
aptsources=Grip
#configscript=config.sh
#setupscript=setup.sh

[Grip]
packages=udev aptitude
source=http://www.emdebian.org/grip
keyring=emdebian-archive-keyring
suite=lenny
omitdebsrc=true

$ cd /tmp
$ multistrap -a armel -f armel.conf -d chroot_armel

$ fakeroot debootstrap --foreign --arch=armel sid $ROOTDIR

$ cp /usr/bin/qemu-arm-static chroot_armel/usr/bin

Using schroot

[chroot_armel]
description=Debian armel cross chroot
directory=/tmp/chroot_armel
users=$USER
groups=sbuild
root-groups=root
aliases=default

$ sudo sh -c "echo 0 > /proc/sys/vm/mmap_min_addr"

$ schroot -c chroot_armel

Using fakechroot

$ sudo ln -sf /tmp/chroot_armel/lib/ld-linux.so.3 /lib/ld-linux.so.3

$ LD_LIBRARY_PATH=$(pwd)/chroot_armel/usr/lib/fakechroot fakechroot /usr/sbin/chroot $(pwd)/chroot_armel/ /bin/pwd

Configuring the new system (target specific)

The system you have just created needs a few tweaks so you can use it for specific tasks. The following steps need to be done as root, as the files touched are owned by root.

  1. First create a fixed symlink to the directory where the new system was bootstrapped:

    # ln -sfn /full/path/to/debian_armel_wheezy/ /armelfs_debian
  2. Mount the proc filesystem on the target so the next commands can run properly:

    # mount -t proc proc /armelfs_debian/proc

Serial console

  1. Edit /armelfs_debian/etc/inittab and add the following line at the bottom:

    T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100

    also comment out the following lines (virtual consoles are not needed for our purposes):

    1:2345:respawn:/sbin/getty 38400 tty1
    2:23:respawn:/sbin/getty 38400 tty2
    3:23:respawn:/sbin/getty 38400 tty3
    4:23:respawn:/sbin/getty 38400 tty4
    5:23:respawn:/sbin/getty 38400 tty5
    6:23:respawn:/sbin/getty 38400 tty6
  2. Create the "ttyS0" device node on the target so the serial console can be used:

    # chroot /armelfs_debian sh -c "cd /dev; ./MAKEDEV ttyS0"

Networking

Copy the /etc/hosts file from the host to the target system, so networking can work properly inside the chroot:

# cp /etc/hosts /armelfs_debian/etc/

Apt

  1. Run apt-setup to create a new /etc/apt/sources.list for the target:

    # LC_ALL=C chroot /armelfs_debian apt-setup
  2. Select "edit sources list by hand" and add the following entries:

    deb http://ftp.debian.org/debian/ wheezy main contrib non-free
    deb-src http://ftp.debian.org/debian/ wheezy main contrib non-free
    Feel free to use other mirrors (as long as it supports the target platform).
  3. After downloading the packages list, it will ask whether you want to add other sources. Select "no".

Done! You can now install other packages using apt-get install <package> as usual.

References

Real Hardware/cdebootstrap approach

We will:

This is a rather brute-force approach but it works. This was tested with the Oct 2006 version of cdebootstrap in unstable.

# cdebootstrap --arch=armel testing /testing-armel-chroot http://ftp.uk.debian.org/debian/

I found that if you didn't specify a mirror then it just stopped after 4 lines and failed ot find a Packages file.

this will unpack everything (into '/testing-armel-chroot') then error out.

make sure that your testing-armel-chroot dir is accessible over NFS by editing /etc/exports and running exportfs -r to re-read the config then exportfs to check it is exported. Export it with options rw,no-root-squash,async.

No device files have been created so you will need to create /dev/ttyS0 at least:

# mknod /testing-armel-chroot/dev/ttyS0 c 4 64"

or in my case

# mknod /testing-armel-chroot/dev/ttyAMA0 c 204 16

Now start up your hardware with an NFS-capable kernel with kernel arguments something like:

# root=/dev/nfs nfsroot=hostname:/testing-armel-chroot init=/bin/sh console=/dev/ttyS0

You need to specify /bin/sh as the init because there is no inittab file and even if you add one you won't be able to login because login/pam are not set up properly.

Once you have a prompt you can

# cd /var/cache/boostrap

and run all the install scripts.

One way to do this (there may well be a better way) is just to do dpkg -i *.deb which re-installs everything. In order to avoid base-files failing to install over itself I did rm /var/mail before this. I found that dpkg gave the error dpkg: dpkg - error: PATH is not set so in fact the command line to use is

# PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin dpkg -i *.deb

(get the path by doing set )

This mostly works but some packages fail to install and you get to go back and do

# PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin dpkg -i <packages which failed>

once you have got everything in, you should have a standard debian rootfs for the target arch.

And it works for me. There is no editor or networking installed after this so you can't do much without installing some more packages.

Second attempt using debootstrap

deboostrap is equivalent to cdebootstrap, but is shell rather than c. Again you need to give a mirror that has the packages for your architecture otherwise it will give about 4 lines of output and then stop when it fails to find packages files. Your existing sources.list does not appear to get used as a default.

# debootstrap --verbose --arch armel --foreign --variant=buildd sid sid-armel-chroot http://ftp.uk.debian.org/debian/

So, as above create a suitable serial console dev file:

# mknod /sid-armel-chroot/dev/ttyAMA0 c 204 16

Then boot into the new chroot, and install everything again, as above. Making debootstrap at least fill in the package database so that we could run all the install scripts more neatly would be good. I'm sure debootstrap could do this for us.

Unpacking method

We will:

emsandbox is a wrapper for debootstrap that tries to handle the issues around --foreign. The weakness of the normal --second-stage support in debootstrap is that it puts a large burden on the target device. Downloaded packages are copied into the root filesystem unchanged, then unpacked, configured and installed on the real device. Recent versions of emsandbox use an "unpacking" method that avoids the need to carry the .deb package files into the actual installed root filesystem tarball. This reduces the size of the rootfs tarball and decreases the workload for the target device.

emsandbox uses the Emdebian repository where package dependencies have been modified to support a much smaller root filesystem - around 24Mb installed instead of 180Mb - and the method can be extended to make even smaller images by using uClibc or not using apt etc.

This method can only support such small images if the appropriate packages already exist in the Emdebian repository or some other repository that can be used by debootstrap. emsandbox uses debootstrap to resolve the dependencies of the base and required packages but the normal --second-stage is much less work (consisting primarily of dpkg --configure -a). --second-stage itself is still used but is generally just a hook for machine:variant customisation scripting.

Issues with the final filesystem image

By modifiying the basic set of packages used by debootstrap, the normal Debian init support needs to be hacked. Principally, using busybox instead of coreutils and sysv-rc means that runlevels are not supported and /etc/inittab needs to be compatible with busybox.

Other components of a standard Debian debootstrap environment may also be missing, e.g. /sys/. emsandbox can create these directories for you via the suite script that is used for debootstrap.

Issues with replacing debconf with cdebconf

See Debian bug 451130. emsandbox will use a workaround to ensure that cdebconf initialises the debconf database until 451130 is closed. The command is:

chroot $BUILDPLACE /usr/lib/cdebconf/debconf-loadtemplate /usr/share/debconf/demo /usr/share/debconf/demo.templates

To use cdebconf at all, an environment variable also needs to be exported:

DEBCONF_USE_CDEBCONF=true
export DEBCONF_USE_CDEBCONF

Normal debconf behaviour can be disabled using the expected debconf environment variable support:

DEBIAN_FRONTEND=noninteractive
DEBCONF_NONINTERACTIVE_SEEN=true
export DEBIAN_FRONTEND DEBCONF_NONINTERACTIVE_SEEN

/etc/group and /etc/passwd

Minimal versions of these files must exist before dpkg --configure -a is run, principally because the unpacking method avoids the need for packages to be processed in a particular sequence so emsandbox cannot rely on files being created in the postinst or preinst maintainer scripts. This also allows emsandbox to support arbitrary package sets where the packages that would create these files on a normal Debian box are simply omitted.

Support for creating these files is available via emsandbox using the suite script support in debootstrap.

ash, not bash

busybox does not support bash, so all shell code used by emsandbox must be POSIX compliant - do not use bashisms.

busybox applets, not coreutils

Various commands in Debian have a wider option set than the busybox equivalent. Examples are numerous but some of the most common include:

Packages using such options will be patched in the Emdebian version - note that this may cause unexpected behaviour, in which case a bug should be filed against the package in Debian, requesting support for the busybox implementation, using embug --prepare.

starting init processes

Various packages will put scripts into /etc/init.d/ and expect those scripts to be symlinked in directories like /etc/rc2.d/ etc. which will not exist in a busybox init due to the lack of runlevel support. Work is ongoing in emsandbox to create a usable support mechanism for this issue but will probably involve using /etc/rcS.d/ to hold the symlinks and calling each via the /etc/init.d/rcS script. Again, the suite script will provide support for setting up these symlinks.

Outputting messages to the user

This method still uses debootstrap so the normal STDOUT file descriptor is wrapped within debootstrap functions.

To output to the user, use debootstrap support functions:

info INSTCORE "message"

Calling processes within debootstrap

Other functions are also available, like in_target to ensure that a process is actioned within the filesystem image using the chroot command.

Method

Instead of a complicated set of shell instructions in the --second-stage function, the intention is to provide a series of shell functions accessible via emsandbox and empbuilderlib from emdebian-tools to provide the necessary support. The suite script is a shell script called by debootstrap and which normally describes how debootstrap should create a chroot for the Debian suite, i.e. testing, unstable, etc. Emdebian co-opts this debootstrap support to describe how debootstrap should create a filesystem image using the package set available from Emdebian. In effect, emsandbox tells debootstrap that each customised emsandbox filesystem image is a different suite of Debian. The first part of the suite script simply sets out the names of the packages to be downloaded from the Emdebian repository by debootstrap. This is the work_out_debs () function.

work_out_debs () {

        required="busybox dpkg libstdc++6 libgcc1 libc6 cdebconf
        libdebian-installer4 zlib1g libnewt0.52 libslang2"

        base="apt gpgv libncurses5 libreadline5 readline-common mktemp
        debconf-shell debianutils makedev base-passwd whiptail
        gnupg udev base-files debian-archive-keyring udhcpc udhcpd sysv-rc
        util-linux module-init-tools initscripts"

}

The distinction between required and base is lost if the unpacking method is used but debootstrap does require that some packages exist in each section. (In normal debootstrap, required packages are partially unpacked, base are not but the .debs are retained for both sets.)

To solve problems of code duplication, a set of common functions is available in empbuilderlib that can be called as part of the first_stage_install ()  function in the suite script.

The eventual suite script can look something like this:

mirror_style release
download_style apt

work_out_debs () {

        required="busybox dpkg libstdc++6 libgcc1 libc6 cdebconf
        libdebian-installer4 zlib1g libnewt0.52 libslang2"

        base="apt gpgv libncurses5 libreadline5 readline-common mktemp
        debconf-shell debianutils makedev base-passwd whiptail
        gnupg udev base-files debian-archive-keyring udhcpc udhcpd sysv-rc
        util-linux module-init-tools initscripts"
}

first_stage_install () {
        . /usr/lib/emdebian-tools/empbuilderlib
        extract $required
        dpkg_support
        dummy_fstab
        dummy_resolvconf
        dummy_hostname
        dummy_hosts
        setup_devices
        make_proc
        make_varlog
        make_varspool
        info INSTCORE "Setting up dpkg"
        x_feign_install dpkg
        info INSTCORE "Unpacking debootstrapped environment"
        unpack_debootstrap
}

second_stage_install () {
        if [ -f datestring ]; then
                TIME=`cat datestring`
                info INSTCORE "Setting approximate time of $TIME"
                date -s $TIME
        fi
        info INSTCORE "Running ldconfig..."
        in_target /sbin/ldconfig
        info INSTCORE "Running depmod..."
        in_target depmod
        setup_cdebconf
        info INSTCORE "Configuring ..."
        in_target dpkg --configure -a
        if [ ! -e "$TARGET/etc/localtime" ]; then
                ln -sf /usr/share/zoneinfo/UTC "$TARGET/etc/localtime"
        fi
        if [ -f 0x97BB3B58.txt ]; then
                rm 0x97BB3B58.txt
        fi
        info BASESUCCESS "Emdebian base system installed successfully."
}

The main work is carried out in unpack_debootstrap which feigns the unpacking and installation of the .deb packages using dpkg support but without running any maintainer scripts, including preinst. This allows emsandbox to create an almost functional filesystem image, leaving only the absolute minimum amount of work to be done on the target device before rebooting.

Via support for emsecondstage, each stage can be customised with your own shell functions - as long as no bashisms are used. Each emsandbox filesystem image can also use a different suite script so each machine:variant has full support over the contents of the root filesystem.

Note that none of the emsandbox / empbuilderlib support functions are available within second_stage_install () - only standard debootstrap functions exist (and those are removed once debootstrap --second-stage completes).

approximate time and SecureApt

The above snippet includes a time handler in second_stage_install ()  which is a bit of an oddity. It uses support in emsandbox to create an approximate time string within the filesystem image as a file called /datestring. The intention is to allow a sensible, approximate, time to be set on the device without requiring network access. Otherwise, getting the correct time would be the only stage in the installation that would require a network connection. ntp is available to any Emdebian device and includes ntpdate-debian so that once a working network connection becomes available, the current time can be set precisely. Without even an approximate time, various parts of the Debian setup can get confused - especially relating to GnuPG keys or signature verification (gpg doesn't like timestamps in 1970 when verifying signatures made 30 years 'into the future'). This support is needed for SecureApt configuration.

0x97BB3B58.txt is the Emdebian Archive Keyring as gpg armoured text, it is added to apt-key during the postinst configuration of the debian-archive-keyring package in Emdebian to provide SecureApt support.

/datestring uses a time format supported by the busybox implementation of date:

        date +%m%d%H%M%Y

Installing the image

Current methods involve using existing support on the device (in my case a balloon3). The intention is to be able to support any method of copying the filesystem image tarball onto the device, including using the Debian Installer. For this to work, all necessary setup needs to be retained within the packages themselves or the suite script via emsandbox machine:variant support.

D-I would then handle the filesystem image created by emsandbox and calls emsecondstage which is installed by emsandbox to merge the machine:variant support with debootstrap --second-stage. D-I would use the interactive form of cdebconf support. For this to work, Emdebian needs to rebuild the D-I udebs so that the D-I code itself is reduced in size.


CategoryEmdebian