Differences between revisions 37 and 39 (spanning 2 versions)
Revision 37 as of 2021-12-06 00:42:01
Size: 18365
Editor: GuillemJover
Comment: Update checklist
Revision 39 as of 2021-12-06 06:41:50
Size: 18441
Editor: ?HelmutGrohne
Comment: More updates
Deletions are marked like this. Additions are marked like this.
Line 80: Line 80:
  * [ ] /var/lib/dpkg/info/bash.preinst (compiled binary, DebianBug:958083)   * (./) /var/lib/dpkg/info/bash.preinst (compiled binary, DebianBug:958083)
Line 94: Line 94:
  * [ ] /var/lib/dpkg/info/debconf.postinst (removed in DebianBug:989567)

    * Removes an obsolete conffile.
  * [ ] /var/lib/dpkg/info/debconf.postinst

    * Removed an obsolete conffile (until DebianBug:989567). Now an empty script.
Line 205: Line 205:
Support for `DPKG_ROOT` utilization is being added to dpkg's own tools first (eg. `update-alternatives`), and being tracked under the [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-dpkg@lists.debian.org;tag=dpkg-root-support|dpkg-root-support usertag of debian-dpkg@lists.debian.org]]. Support for `DPKG_ROOT` utilization in dpkg is complete as of 1.21.0 (with all relevant tools also supporting a --root option), and support for other tools and packages being tracked under the [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-dpkg@lists.debian.org;tag=dpkg-root-support|dpkg-root-support usertag of debian-dpkg@lists.debian.org]].

  • Status: draft


Currently the logic and implicit dependencies to bootstrap a distribution installation is hardcoded in packages such as debootstrap or cdebootstrap (but not mmdebstrap or multistrap which are trying to solve this in a cleaner way). This is a problem because it is fragile, needs to be duplicated on each project that wants to bootstrap a distribution, it is not self-contained, it is also distribution specific, and can easily get out-of-sync with the packages found in the archive, their implied dependencies or the order they might get unpacked or configured by dpkg.

Recent examples showing the fragility of this approach are 760568, 766459, 767999, regardless of what package was at fault here.

Further detailed description of the problem.

Proposal: full declarative pseudo-Essential

Remove any maintainer script usage for the bootstrap case ("configure"/"install" w/o a previous version argument) in all of the pseudo-Essential set. And rely on declarative metadata. This requires supporting:

  • Declarative alternatives.
  • Declarative diversions.
  • Declarative file metadata.

    • With support for ghost files.
    • With support for files initialized from a template. Such files or directories are only installed once during the initial package installation and never updated. The major user would be base-files.postinst.

  • Declarative config files / conffiles.
    • With support for files initialized from a template. Such files would only be installed once during initial package installation and never updated. A major user would be base-passwd.preinst.

A possible implementation for such template initialized files could be adding metadata to binary packages (e.g. mtree) that tells dpkg which paths to initialize from which template files (e.g. /etc/passwd from /usr/share/base-passwd/passwd.master). For backwards-compatibility with older dpkg that do not support this feature, the maintainer scripts can continue to install such files.

The rest might require fixing code to:

  • Work on /etc/shells? Discussion at <YMJTIrKQbjDjyZbP@alf.mars>, 990440

  • Fixing programs to generate required, but missing files.
  • debconf is mostly used for handling upgrades and can be often skipped entirely during installation bootstrap.

  • dash presently supports changing /bin/sh via a local diversion and via debconf. Try discarding the debconf method and see whether anyone objects. A major user is the Debian installer where you can preseed this question. Adding special support to the installer to add the local diversion may be feasible.

This has the nicer property of requiring no execution of maintainer scripts, so no complications with chrootless runs, and ending up with a finished image not requiring a second-stage configuration for foreign architectures.

Current maintscript usage in pseudo-Essential set

So, when all of the above have been removed, we could update policy to state that the bootstrap case requires 0 code execution from maintscripts. To track progress, for maintscripts that need to handle also non-bootstrap code, once the bootstrap code has been removed we could mark it with something like:

  • # Package in pseudo-Essential set with no bootstrap maintscript code.

    if [ "$1" = configure ] && [ -z "$2" ]; then

    • exit 0

The current list of uses is:

  • $ dpkg-query -f '${binary:Package}\n' -W \
    • $(/usr/sbin/debootstrap --variant=minbase --print-debs \
      • --exclude=apt,e2fsprogs sid /tmp/sid-tmp 2>/dev/null) \

      | grep -v :i386 \ | while read pkg; do
      • ls -1 /var/lib/dpkg/info/$pkg.*inst 2>/dev/null; \

      • done
  • [ ] /var/lib/dpkg/info/base-files.postinst
    • Creates directories with proper owner/permissions.
    • Creates symlinks for the default dpkg origin file.
    • Creates compat symlinks for some directories.
    • Initializes config files from templates.
    • Initializes state files from templates.
    • Initializes "log" files.
    • Initializes /run file. This should be removed, as /run is supposed to be clean after each boot. 989725

    • Initializes various dpkg files. This should probably be removed. 989726

  • [ ] /var/lib/dpkg/info/base-passwd.postinst
    • Initializes config files from templates.
    • Runs update-passwd, although I think this should not need to be run on the bootstrap case.
  • [ ] /var/lib/dpkg/info/base-passwd.preinst
    • Initializes config files from embedded templates in the maintscript. Duplicates the logic in the postinst script.
  • [ ] /var/lib/dpkg/info/bash.postinst
    • Installs alternatives.
    • Adds itself into /etc/shells via add-shell. 990440

  • (./) /var/lib/dpkg/info/bash.preinst (compiled binary, 958083)

    • Handles the /bin/sh symlink creation if missing, and diversion installation. But perhaps this can be removed now, ISTR a bug report requesting just that, but cannot find it.
  • [ ] /var/lib/dpkg/info/coreutils.postinst
    • Creates /usr/bin/ to /bin/ transition symlink. [ Due to merged-usr-via-symlinks. :( … ]

  • [ ] /var/lib/dpkg/info/dash.postinst
    • Handles /bin/sh symlink creation if missing, and diversion installation.
    • Adds itself into /etc/shells via add-shell. 990440

    • Proposed reductions 989419 and 989632

  • [ ] /var/lib/dpkg/info/debconf.postinst
    • Removed an obsolete conffile (until 989567). Now an empty script.

  • (./) /var/lib/dpkg/info/debconf.preinst (removed in https://salsa.debian.org/pkg-debconf/debconf/-/merge_requests/4)

    • Removes an obsolete conffile.
    • Renames a conffile.
  • [ ] /var/lib/dpkg/info/debianutils.postinst
    • Initializes /etc/shells from a template.
    • Creates /usr/bin/ to /bin/ transition symlink. [ Due to merged-usr-via-symlinks. :( … ]

  • (./) /var/lib/dpkg/info/dpkg.postinst (removed in 1.20.0)

    • Initializes its database and log files with owner and permissions. This should probably be done at run-time instead, there's already a report about this. This is fixed now in the next/bootstrap branch.
  • [ ] /var/lib/dpkg/info/libc-bin.postinst
    • Initialize config file from template.
    • Runs ldconfig.
  • [ ] /var/lib/dpkg/info/libc6:amd64.postinst
    • Removes hwcappkgs registry. Only relevant on upgrades.
    • Handles nohwcap file removal. Only relevant on upgrades.
    • Restarts init. Only relevant on upgrades.
  • [ ] /var/lib/dpkg/info/libc6:amd64.preinst
    • LD_LIBRARY_PATH sanity checks. Only relevant on upgrades.
    • Kernel version requirement checks.
  • (./) /var/lib/dpkg/info/libgcc1:amd64.postinst (removed in the process of renaming to libgcc-s1)

    • Handles directory to symlink transition. Only relevant on upgrades.
  • [ ] /var/lib/dpkg/info/libpam-modules:amd64.postinst
    • Creates empty config files.
  • [ ] /var/lib/dpkg/info/libpam-modules:amd64.preinst
    • Does nothing during bootstrapping.
  • [ ] /var/lib/dpkg/info/libpam-runtime.postinst
    • Run pam-auth-update.
    • Initialize config file from template.
  • [ ] /var/lib/dpkg/info/libpam0g:amd64.postinst
    • Does nothing during bootstrapping.
  • [ ] /var/lib/dpkg/info/login.postinst
    • Obsolete init script conffile removal.
    • Create config files with owner and permissions.
    • Cruft reduction 989712

  • (./) /var/lib/dpkg/info/login.preinst (removed in https://salsa.debian.org/debian/shadow/-/merge_requests/10)

    • Does nothing during bootstrapping.
  • [ ] /var/lib/dpkg/info/mawk.postinst
    • Installs alternatives.
  • [ ] /var/lib/dpkg/info/passwd.postinst
    • Fixes permissions for log files, should only be needed on upgrade, and it should actually be removed now.
    • Handles shadow group addition, only relevant for upgrades.
    • Runs shadownconfig.
    • Runs systemd-tmpfiles.
    • Cruft reduction 989712

  • [ ] /var/lib/dpkg/info/passwd.preinst (removed in https://salsa.debian.org/debian/shadow/-/merge_requests/11)

    • Does nothing during bootstrapping.
  • [ ] /var/lib/dpkg/info/tar.postinst
    • Installs alternatives.
  • [ ] /var/lib/dpkg/info/tzdata.postinst
    • Does nothing during bootstrapping. The actual initial timezone setup is off-loaded to the bootstrapping programs. :/
  • [ ] /var/lib/dpkg/info/util-linux.postinst
    • Installs alternatives.
    • Removes obsolete alternatives.
    • Runs helpers for sysvinit/systemd setup.

Proposal: chrootless maintscripts

The installation bootstrap logic for any pseudo-essential package currently handled in tools such as debootstrap or cdebootstrap would be moved into a new package maintainer script or similar. Those would need to be run from outside the chroot, so that we are not back to the problem of implicit assumptions and ordering though. And the expectations on the external environment would need to be specified, for example assuming just POSIX utilities (or a subset of it).

This could also be used to bootstrap a foreign architecture, as the setup would be done by the native system, but in this case it would require not-chroot'ing and passing to the maintainer script the path of the root directory.

Detached chroot handling

A related topic, is the handling of chroots, be them native or foreign, without requiring chroot(2)ing into the directory. This required adding a new environment variable set by dpkg named DPKG_ROOT, which can be used by maintainer scripts when needing to interact with the chroot filesystem.

Starting with dpkg 1.18.5, maintainer scripts get the environment variable DPKG_ROOT set by default to the empty string. With older dpkg it will be unset, so if the script uses set -u you might need to do something like «: "${DPKG_ROOT=}"» for backwards compatibility. If the user requests the new mode of operation the DPKG_ROOT environment variable will contain a chroot path that can be prepended to the pathnames accessed. For example

  • echo config > $DPKG_ROOT/etc/package/package.conf

The only way to test this currently is to force the new mode with the --root and --force-script-chrootless, but that will force the mode even when the affected packages do not support it, which might damage your host system, do not use without very careful consideration; running dpkg as a nonpriviledged user can mitigate that issue (possibly requiring fakeroot).

Support for DPKG_ROOT utilization in dpkg is complete as of 1.21.0 (with all relevant tools also supporting a --root option), and support for other tools and packages being tracked under the dpkg-root-support usertag of debian-dpkg@lists.debian.org.

Note that DPKG_ROOT does not generally solve the installation bootstrap use case, because it requires a working dpkg on the outside whereas debootstrap is meant to be run on non-Debian systems.

Progress on DPKG_ROOT support

  1. /!\ base-files is blocked on 824594. Cleanup 989725 and 989726.

  2. /!\ base-passwd uses debconf and is blocked on 983425.

  3. /!\ bash has complex maintainer scripts. Adding support for DPKG_ROOT as is is not reasonable. Their complexity is no longer needed. Cleanup is blocked on 989334. /etc/shells support is blocked on 990440. More patches are needed.

  4. /!\ dash uses debconf and is blocked on 983425. Additionally, its maintainer scripts do more than they need to do. Cleanup reported as 989419 and 989632. /etc/shells management reported as 990440.

  5. /!\ debconf is blocked on 983425.

  6. /!\ debhelper needs to be updated for util-linux 983566

  7. /!\ debianutils should support trigger-based management of /etc/shells 990440.

  8. /!\ coreutils needs some paths prefixed with DPKG_ROOT and is blocked on 983565.

  9. /!\ libc-bin invokes ldconfig without -r and is blocked on 983412.

  10. /!\ libgssapi-krb5-2 uses debconf and is blocked on 983425.

  11. /!\ libkrb5-3 uses debconf and is blocked on 983425.

  12. /!\ libnsl2 uses debconf and is blocked on 983425.

  13. /!\ src:pam uses debconf and is blocked on 983425.

  14. /!\ libssl1.1 uses debconf and is blocked on 983425.

  15. /!\ libtirpc3 uses debconf and is blocked on 983425.

  16. /!\ login uses debconf and is blocked on 983425.

  17. /!\ src:shadow needs significant changes. In particular it uses getent, which queries the outer system database. It also needs cruft removal 989712.

  18. /!\ util-linux uses debconf and is blocked on 983425. It also suffers from the debhelper issue mentioned earlier.

Deferred: Maintainer script dependencies

Because this is not limited to packages in the Essential set, any Depends and Pre-Depends would need to be installed on the host systems. To avoid having to install all of those in the host, Helmut Grohne proposed adding a new Maint-Depends field, which would declare dependencies required by the maintainer scripts, and those and only those would need to be present on the host system. A companion field Runtime-Depends would list the packages that are required for using the package after its installation is complete. Any package listed in Depends would be considered part of both Maint-Depends and Runtime-Depends.

This is problematic though, and several reasons have been presented in 804624, but the hope is to eventually find a solution to this problem.

Julian Andres Klode proposed an alternative representation of the same concept. Instead of introducing a new field, the binary package relation syntax could be extended using a notation similar to build profiles. As such, dependencies could be tagged for their relevant use in even more detail (on a maintainer script granularity). Consider for example Depends: foo <postinst>.

If we allow removal of packages listed in Maint-Depends, we may end up in situations where removing another package becomes impossible without (temporarily) installing its Maint-Depends. When that becomes impossible, such packages may become unremovable. This would also make our dependency graph incompatible with CUDF solving. As a consequence, removing Maint-Depends should not normally be allowed.

The proposed splitting of Depends may help with breaking dependency loops. A looping dependency can sometimes be broken by moving it to Runtime-Depends. The relevant maintainer scripts can then be run without satisfying Runtime-Depends.

While this is all nice and such, it does not help installation bootstrap all that much. The only case where it does help is foreign bootstraps without qmeu.

Non-Proposal: declarative ordering

The biggest piece of knowledge about install bootstrap that resides in debootstrap is the ordering or package installations. For instance, it knows that base-passwd must be installed first and base-files must be next. Moving this ordering information to metadata could be an option. A frequent suggestion is adding dependencies between essential packages for this purpose as apt does honour. Adding such dependencies is not universally agreed upon though as policy section 3.8 declares such dependencies as unnecessary. 924401

The apt solver deals badly with relations between essential packages. Workarounds had to be added e.g. for dealing with Breaks introduced by in glibc. Beyond this, apt does not know about maintainer scripts (or their ordering) and it really does not want to know. Ordering the packages by Pre-Depends for dpkg already is painful. The only place ordering really would matter is base-passwd vs base-files vs the rest. And for these cases, declarative solutions are attainable.

Non-Proposal: bootstrap scripts

This is a variant of chrootless maintainer scripts. In essence it removes the chrootless part and leaves the idea of having separate maintainer scripts for the installation bootstrap to ones used for regular installation or upgrades. In essence regular preinst and postinst invocations would be skipped entirely during installation bootstrap and a separate bootstrap script would be run after all relevant packages have been unpacked in unspecified order. It should have the same effect as if a package were regularly installed, under a number of stronger assumptions:

  • No init system or any kind of service is being run from the root directory of the installation being operated on.
  • No packages outside the transitive essential set have been unpacked.
  • The bootstrap script is only run once.

This is a no-brainer as all of it can be implemented without the need for a separate script. The order of maintainer scripts already is practically undefined and a maintainer script can detect is first execution by checking for the empty version. Sufficient conditionalization can achieve the same effect.