• Status: draft

Summary

Currently the logic and implicit dependencies to bootstrap a distribution installation is hardcoded in packages such as debootstrap or cdebootstrap (but not mmdebstrap or multistrap which are trying to solve this in a cleaner way). This is a problem because it is fragile, needs to be duplicated on each project that wants to bootstrap a distribution, it is not self-contained, it is also distribution specific, and can easily get out-of-sync with the packages found in the archive, their implied dependencies or the order they might get unpacked or configured by dpkg.

Recent examples showing the fragility of this approach are 760568, 766459, 767999, regardless of what package was at fault here.

Further detailed description of the problem.

Proposal: full declarative pseudo-Essential

Remove any maintainer script usage for the bootstrap case ("configure"/"install" w/o a previous version argument) in all of the pseudo-Essential set. And rely on declarative metadata. This requires supporting:

A possible implementation for such template initialized files could be adding metadata to binary packages (e.g. mtree) that tells dpkg which paths to initialize from which template files (e.g. /etc/passwd from /usr/share/base-passwd/passwd.master). For backwards-compatibility with older dpkg that do not support this feature, the maintainer scripts can continue to install such files.

The rest might require fixing code to:

This has the nicer property of requiring no execution of maintainer scripts, so no complications with chrootless runs, and ending up with a finished image not requiring a second-stage configuration for foreign architectures.

Current maintscript usage in pseudo-Essential set

So, when all of the above have been removed, we could update policy to state that the bootstrap case requires 0 code execution from maintscripts. To track progress, for maintscripts that need to handle also non-bootstrap code, once the bootstrap code has been removed we could mark it with something like: 

   1    # Package in pseudo-Essential set with no bootstrap maintscript code.
   2     if [ "$1" = configure ] && [ -z "$2" ]; then
   3       exit 0
   4     fi

The current list of uses is: 

   1 dpkg-query -f '${binary:Package}\n' -W \
   2   $(/usr/sbin/debootstrap --variant=minbase --print-debs \
   3       --exclude=apt,e2fsprogs sid /tmp/sid-tmp 2>/dev/null) \
   4   | grep -v :i386 \
   5   | while read pkg; do
   6      ls -1 /var/lib/dpkg/info/$pkg.*inst 2>/dev/null; \
   7 done

Proposal: chrootless maintscripts

The installation bootstrap logic for any pseudo-essential package currently handled in tools such as debootstrap or cdebootstrap would be moved into a new package maintainer script or similar. Those would need to be run from outside the chroot, so that we are not back to the problem of implicit assumptions and ordering though. And the expectations on the external environment would need to be specified, for example assuming just POSIX utilities (or a subset of it).

This could also be used to bootstrap a foreign architecture, as the setup would be done by the native system, but in this case it would require not-chroot'ing and passing to the maintainer script the path of the root directory.

Detached chroot handling

A related topic, is the handling of chroots, be them native or foreign, without requiring chroot(2)ing into the directory. This required adding a new environment variable set by dpkg named DPKG_ROOT, which can be used by maintainer scripts when needing to interact with the chroot filesystem.

Starting with dpkg 1.18.5, maintainer scripts get the environment variable DPKG_ROOT set by default to the empty string. With older dpkg it will be unset, so if the script uses set -u you might need to do something like «: "${DPKG_ROOT=}"» for backwards compatibility. If the user requests the new mode of operation the DPKG_ROOT environment variable will contain a chroot path that can be prepended to the pathnames accessed. For example

The only way to test this currently is to force the new mode with the --root and --force-script-chrootless, but that will force the mode even when the affected packages do not support it, which might damage your host system, do not use without very careful consideration; running dpkg as a nonpriviledged user can mitigate that issue (possibly requiring fakeroot).

Support for DPKG_ROOT utilization in dpkg is complete as of 1.21.0 (with all relevant tools also supporting a --root option), and support for other tools and packages being tracked under the dpkg-root-support usertag of debian-dpkg@lists.debian.org.

Note that DPKG_ROOT does not generally solve the installation bootstrap use case, because it requires a working dpkg on the outside whereas debootstrap is meant to be run on non-Debian systems.

Progress on DPKG_ROOT support

  1. /!\ base-files is blocked on 824594. Cleanup 989725 and 989726.

  2. /!\ base-passwd uses debconf and is blocked on 983425.

  3. /!\ bash has complex maintainer scripts. Adding support for DPKG_ROOT as is is not reasonable. Their complexity is no longer needed. Cleanup is blocked on 989334. /etc/shells support is blocked on 990440. More patches are needed.

  4. /!\ dash uses debconf and is blocked on 983425. Additionally, its maintainer scripts do more than they need to do. Cleanup reported as 989419 and 989632. /etc/shells management reported as 990440.

  5. /!\ debconf is blocked on 983425.

  6. /!\ debhelper needs to be updated for util-linux 983566

  7. /!\ debianutils should support trigger-based management of /etc/shells 990440.

  8. /!\ coreutils needs some paths prefixed with DPKG_ROOT and is blocked on 983565.

  9. /!\ libc-bin invokes ldconfig without -r and is blocked on 983412.

  10. /!\ libgssapi-krb5-2 uses debconf and is blocked on 983425.

  11. /!\ libkrb5-3 uses debconf and is blocked on 983425.

  12. /!\ libnsl2 uses debconf and is blocked on 983425.

  13. /!\ src:pam uses debconf and is blocked on 983425.

  14. /!\ libssl1.1 uses debconf and is blocked on 983425.

  15. /!\ libtirpc3 uses debconf and is blocked on 983425.

  16. /!\ login uses debconf and is blocked on 983425.

  17. /!\ src:shadow needs significant changes. In particular it uses getent, which queries the outer system database. It also needs cruft removal 989712.

  18. /!\ util-linux uses debconf and is blocked on 983425. It also suffers from the debhelper issue mentioned earlier.

Deferred: Maintainer script dependencies

Because this is not limited to packages in the Essential set, any Depends and Pre-Depends would need to be installed on the host systems. To avoid having to install all of those in the host, Helmut Grohne proposed adding a new Maint-Depends field, which would declare dependencies required by the maintainer scripts, and those and only those would need to be present on the host system. A companion field Runtime-Depends would list the packages that are required for using the package after its installation is complete. Any package listed in Depends would be considered part of both Maint-Depends and Runtime-Depends.

This is problematic though, and several reasons have been presented in 804624, but the hope is to eventually find a solution to this problem.

Julian Andres Klode proposed an alternative representation of the same concept. Instead of introducing a new field, the binary package relation syntax could be extended using a notation similar to build profiles. As such, dependencies could be tagged for their relevant use in even more detail (on a maintainer script granularity). Consider for example Depends: foo <postinst>.

If we allow removal of packages listed in Maint-Depends, we may end up in situations where removing another package becomes impossible without (temporarily) installing its Maint-Depends. When that becomes impossible, such packages may become unremovable. This would also make our dependency graph incompatible with CUDF solving. As a consequence, removing Maint-Depends should not normally be allowed.

The proposed splitting of Depends may help with breaking dependency loops. A looping dependency can sometimes be broken by moving it to Runtime-Depends. The relevant maintainer scripts can then be run without satisfying Runtime-Depends.

While this is all nice and such, it does not help installation bootstrap all that much. The only case where it does help is foreign bootstraps without qmeu.

Deferred: declarative ordering

The biggest piece of knowledge about install bootstrap that resides in debootstrap is the ordering or package installations. For instance, it knows that base-passwd must be installed first and base-files must be next. Moving this ordering information to metadata could be an option. A frequent suggestion is adding dependencies between essential packages for this purpose as apt does honour. Adding such dependencies is not universally agreed upon though as policy section 3.8 declares such dependencies as unnecessary. 924401

The apt solver deals badly with relations between essential packages. Workarounds had to be added e.g. for dealing with Breaks introduced by in glibc. Beyond this, apt does not know about maintainer scripts (or their ordering) and it really does not want to know. Ordering the packages by Pre-Depends for dpkg already is painful. The only place ordering really would matter is base-passwd vs base-files vs the rest. And for these cases, declarative solutions are attainable.

Specialize bootstrap ordering

Instead of using dependency fields, which can be problematic and add constraints that were explicitly avoided by declaring packages as Essential:yes as mentioned above, a new field could be added to declare parts of the bootstrap order. For example a new Bootstrap, Bootstrap-Phase or Bootstrap-Order (or another name) field with values of for example fsys-tree, fsys-meta.

The correct order here would seem to be: fsys-tree → fsys-meta → rest of essential + dependencies

See 1071078

Non-Proposal: bootstrap scripts

This is a variant of chrootless maintainer scripts. In essence it removes the chrootless part and leaves the idea of having separate maintainer scripts for the installation bootstrap to ones used for regular installation or upgrades. In essence regular preinst and postinst invocations would be skipped entirely during installation bootstrap and a separate bootstrap script would be run after all relevant packages have been unpacked in unspecified order. It should have the same effect as if a package were regularly installed, under a number of stronger assumptions:

This is a no-brainer as all of it can be implemented without the need for a separate script. The order of maintainer scripts already is practically undefined and a maintainer script can detect is first execution by checking for the empty version. Sufficient conditionalization can achieve the same effect.

CategorySpec