Differences between revisions 25 and 27 (spanning 2 versions)
Revision 25 as of 2021-06-11 16:13:59
Size: 14487
Editor: ?HelmutGrohne
Comment: replace item about /etc/shells with discussion reference
Revision 27 as of 2021-06-16 11:18:29
Size: 15952
Editor: ?HelmutGrohne
Comment: incorporate feedback from Julian Andres Klode
Deletions are marked like this. Additions are marked like this.
Line 27: Line 27:
  * Work on {{{/etc/shells}}}? Discussion at https://lists.debian.org/debian-devel/2021/06/msg00097.html   * Work on {{{/etc/shells}}}? Discussion at <<mid(YMJTIrKQbjDjyZbP@alf.mars)>>
Line 226: Line 226:
Because this is not limited to packages in the Essential set, any '''Depends''' and '''Pre-Depends''' would need to be installed on the host systems. To avoid having to install all of those in the host, Helmut Grohne proposed adding a new '''Maint-Depends''' field, which would declare dependencies required by the maintainer scripts, and those and only those would need to be present on the host system. Because this is not limited to packages in the Essential set, any '''Depends''' and '''Pre-Depends''' would need to be installed on the host systems. To avoid having to install all of those in the host, Helmut Grohne proposed adding a new '''Maint-Depends''' field, which would declare dependencies required by the maintainer scripts, and those and only those would need to be present on the host system. A companion field '''Runtime-Depends''' would list the packages that are required for using the package after its installation is complete. Any package listed in '''Depends''' would be considered part of both '''Maint-Depends''' and '''Runtime-Depends'''.
Line 230: Line 230:
Julian Andres Klode proposed an alternative representation of the same concept. Instead of introducing a new field, the binary package relation syntax could be extended using a notation similar to build profiles. As such, dependencies could be tagged for their relevant use in even more detail (on a maintainer script granularity). Consider for example {{{Depends: foo <postinst>}}}.

If we allow removal of packages listed in {{{Maint-Depends}}}, we may end up in situations where removing another package becomes impossible without (temporarily) installing its {{{Maint-Depends}}}. When that becomes impossible, such packages may become unremovable. This would also make our dependency graph incompatible with CUDF solving. As a consequence, removing {{{Maint-Depends}}} should not normally be allowed.

The proposed splitting of {{{Depends}}} may help with breaking dependency loops. A looping dependency can sometimes be broken by moving it to {{{Runtime-Depends}}}. The relevant maintainer scripts can then be run without satisfying {{{Runtime-Depends}}}.
Line 233: Line 239:

The apt solver deals badly with relations between essential packages. Workarounds had to be added e.g. for dealing with {{{Breaks}}} introduced by in {{{glibc}}}.

Status: draft

Summary

Currently the logic and implicit dependencies to bootstrap a distribution installation is hardcoded in packages such as debootstrap or cdebootstrap (but not mmdebstrap or multistrap which are trying to solve this in a cleaner way). This is a problem because it is fragile, needs to be duplicated on each project that wants to bootstrap a distribution, it is not self-contained, it is also distribution specific, and can easily get out-of-sync with the packages found in the archive, their implied dependencies or the order they might get unpacked or configured by dpkg.

Recent examples showing the fragility of this approach are 760568, 766459, 767999, regardless of what package was at fault here.

Further detailed description of the problem.

Proposal: full declarative pseudo-Essential

Remove any maintainer script usage for the bootstrap case ("configure"/"install" w/o a previous version argument) in all of the pseudo-Essential set. And rely on declarative metadata. This requires supporting:

  • Declarative alternatives.
  • Declarative diversions.
  • Declarative file metadata.
    • With support for ghost files.
    • With support for files initialized from a template.
  • Declarative config files / conffiles.
    • With support for files initialized from a template.

The rest might require fixing code to:

This has the nicer property of requiring no execution of maintainer scripts, so no complications with chrootless runs, and ending up with a finished image not requiring a second-stage configuration for foreign architectures.

Current maintscript usage in pseudo-Essential set

So, when all of the above have been removed, we could update policy to state that the bootstrap case requires 0 code execution from maintscripts. To track progress, for maintscripts that need to handle also non-bootstrap code, once the bootstrap code has been removed we could mark it with something like:

  • # Package in pseudo-Essential set with no bootstrap maintscript code.

    if [ "$1" = configure ] && [ -z "$2" ]; then

    • exit 0
    fi

The current list of uses is:

  • $ dpkg-query -f '${binary:Package}\n' -W \
    • $(/usr/sbin/debootstrap --variant=minbase --print-debs \
      • --exclude=apt,e2fsprogs sid /tmp/sid-tmp 2>/dev/null) \

      | grep -v :i386 \ | while read pkg; do
      • ls -1 /var/lib/dpkg/info/$pkg.*inst 2>/dev/null; \

      • done
  • /var/lib/dpkg/info/base-files.postinst
    • Creates directories with proper owner/permissions.
    • Creates symlinks for the default dpkg origin file.
    • Creates compat symlinks for some directories.
    • Initializes config files from templates.
    • Initializes state files from templates.
    • Initializes "log" files.
    • Initializes /run file. This should be removed, as /run is supposed to be clean after each boot. 989725

    • Initializes various dpkg files. This should probably be removed. 989726

  • /var/lib/dpkg/info/base-passwd.postinst
    • Ininitializes config files from templates.
    • Runs update-passwd, although I think this should not need to be run on the bootstrap case.
  • /var/lib/dpkg/info/base-passwd.preinst
    • Ininitializes config files from embedded templates in the maintscript. Duplicates the logic in the postinst script.
  • /var/lib/dpkg/info/bash.postinst
    • Installs alternatives.
    • Adds itself into /etc/shells via add-shell.
  • /var/lib/dpkg/info/bash.preinst (compiled binary, 958083)

    • Handles the /bin/sh symlink creation if missing, and diversion installation. But perhaps this can be removed now, ISTR a bug report requesting just that, but cannot find it.
  • /var/lib/dpkg/info/coreutils.postinst
    • Creates /usr/bin/ to /bin/ transition symlink. [ Due to merged-usr-via-symlinks. :( … ]

  • /var/lib/dpkg/info/dash.postinst
    • Handles /bin/sh symlink creation if missing, and diversion installation.
    • Adds itself into /etc/shells via add-shell.
    • Proposed reductions 989419 and 989632

  • /var/lib/dpkg/info/debconf.postinst (removed in 989567)

    • Removes an obsolete conffile.
  • (./) /var/lib/dpkg/info/debconf.preinst (removed in https://salsa.debian.org/pkg-debconf/debconf/-/merge_requests/4)

    • Removes an obsolete conffile.
    • Renames a conffile.
  • /var/lib/dpkg/info/debianutils.postinst
    • Initializes /etc/shells from a template.
    • Creates /usr/bin/ to /bin/ transition symlink. [ Due to merged-usr-via-symlinks. :( … ]

  • (./) /var/lib/dpkg/info/dpkg.postinst (removed in 1.20.0)

    • Initializes its database and log files with owner and permissions. This should probably be done at run-time instead, there's already a report about this. This is fixed now in the next/bootstrap branch.
  • /var/lib/dpkg/info/libc-bin.postinst
    • Initialize config file from template.
    • Runs ldconfig.
  • /var/lib/dpkg/info/libc6:amd64.postinst
    • Removes hwcappkgs registry. Only relevant on upgrades.
    • Handles nohwcap file removal. Only relevant on upgrades.
    • Restarts init. Only relevant on upgrades.
  • /var/lib/dpkg/info/libc6:amd64.preinst
    • LD_LIBRARY_PATH sanity checks. Only relevant on upgrades.
    • Kernel version requirement checks.
  • (./) /var/lib/dpkg/info/libgcc1:amd64.postinst (removed in the process of renaming to libgcc-s1)

    • Handles directory to symlink transition. Only relevant on upgrades.
  • /var/lib/dpkg/info/libpam-modules:amd64.postinst
    • Creates empty config files.
  • /var/lib/dpkg/info/libpam-modules:amd64.preinst
    • Does nothing during bootstrapping.
  • /var/lib/dpkg/info/libpam-runtime.postinst
    • Run pam-auth-update.
    • Initialize config file from template.
  • /var/lib/dpkg/info/libpam0g:amd64.postinst
    • Does nothing during bootstrapping.
  • /var/lib/dpkg/info/login.postinst
    • Obsolete init script conffile removal.
    • Create config files with owner and permissions.
    • Cruft reduction 989712

  • (./) /var/lib/dpkg/info/login.preinst (removed in https://salsa.debian.org/debian/shadow/-/merge_requests/10)

    • Does nothing during bootstrapping.
  • /var/lib/dpkg/info/mawk.postinst
    • Installs alternatives.
  • /var/lib/dpkg/info/passwd.postinst
    • Fixes permissions for log files, should only be needed on upgrade, and it should actually be removed now.
    • Handles shadow group addition, only relevant for upgrades.
    • Runs shadownconfig.
    • Runs systemd-tmpfiles.
    • Cruft reduction 989712

  • /var/lib/dpkg/info/passwd.preinst (removed in https://salsa.debian.org/debian/shadow/-/merge_requests/11)

    • Does nothing during bootstrapping.
  • /var/lib/dpkg/info/tar.postinst
    • Installs alternatives.
  • /var/lib/dpkg/info/tzdata.postinst
    • Does nothing during bootstrapping. The actual initial timezone setup is off-loaded to the bootstrapping programs. :/
  • /var/lib/dpkg/info/util-linux.postinst
    • Installs alternatives.
    • Removes obsolete alternatives.
    • Runs helpers for sysvinit/systemd setup.

Proposal: chrootless maintscripts

The installation bootstrap logic for any pseudo-essential package currently handled in tools such as debootstrap or cdebootstrap would be moved into a new package maintainer script or similar. Those would need to be run from outside the chroot, so that we are not back to the problem of implicit assumptions and ordering though. And the expectations on the external environment would need to be specified, for example assuming just POSIX utilities (or a subset of it).

This could also be used to bootstrap a foreign architecture, as the setup would be done by the native system, but in this case it would require not-chroot'ing and passing to the maintainer script the path of the root directory.

Detached chroot handling

A related topic, is the handling of chroots, be them native or foreign, without requiring chroot(2)ing into the directory. This required adding a new environment variable set by dpkg named DPKG_ROOT, which can be used by maintainer scripts when needing to interact with the chroot filesystem.

Starting with dpkg 1.18.5, maintainer scripts get the environment variable DPKG_ROOT set by default to the empty string. With older dpkg it will be unset, so if the script uses set -u you might need to do something like «: "${DPKG_ROOT=}"» for backwards compatibility. If the user requests the new mode of operation the DPKG_ROOT environment variable will contain a chroot path that can be prepended to the pathnames accessed. For example

  • echo config > $DPKG_ROOT/etc/package/package.conf

The only way to test this currently is to force the new mode with the --root and --force-script-chrootless, but that will force the mode even when the affected packages do not support it, which might damage your host system, do not use without very careful consideration; running dpkg as a nonpriviledged user can mitigate that issue (possibly requiring fakeroot).

Support for DPKG_ROOT utilization is being added to dpkg's own tools first (eg. update-alternatives), and being tracked under the dpkg-root-support usertag of debian-dpkg@lists.debian.org.

Progress on DPKG_ROOT support

  1. /!\ base-files is blocked on 824594. Cleanup 989725 and 989726.

  2. /!\ base-passwd uses debconf and is blocked on 983425.

  3. /!\ bash has complex maintainer scripts. Adding support for DPKG_ROOT as is is not reasonable. Their complexity is no longer needed. Cleanup is blocked on 989334. More patches are needed.

  4. /!\ dash uses debconf and is blocked on 983425. Additionally, its maintainer scripts do more than they need to do. Cleanup reported as 989419 and 989632.

  5. /!\ debconf is blocked on 983425.

  6. /!\ debhelper needs to be updated for util-linux 983566

  7. {X} debianutils needs to add support for DPKG_ROOT to add-shell, remove-shell and its maintainer scripts. This is non-trivial due to the use of realpath.

  8. /!\ coreutils needs some paths prefixed with DPKG_ROOT and is blocked on 983565.

  9. {X} libc-bin invokes ldconfig without -r.

  10. /!\ libgssapi-krb5-2 uses debconf and is blocked on 983425.

  11. /!\ libkrb5-3 uses debconf and is blocked on 983425.

  12. /!\ libnsl2 uses debconf and is blocked on 983425.

  13. /!\ src:pam uses debconf and is blocked on 983425.

  14. /!\ libssl1.1 uses debconf and is blocked on 983425.

  15. /!\ libtirpc3 uses debconf and is blocked on 983425.

  16. /!\ login uses debconf and is blocked on 983425.

  17. /!\ src:shadow needs significant changes. In particular it uses getent, which queries the outer system database. It also needs cruft removal 989712.

  18. /!\ util-linux uses debconf and is blocked on 983425. It also suffers from the debhelper issue mentioned earlier.

Maintainer script dependencies

Because this is not limited to packages in the Essential set, any Depends and Pre-Depends would need to be installed on the host systems. To avoid having to install all of those in the host, Helmut Grohne proposed adding a new Maint-Depends field, which would declare dependencies required by the maintainer scripts, and those and only those would need to be present on the host system. A companion field Runtime-Depends would list the packages that are required for using the package after its installation is complete. Any package listed in Depends would be considered part of both Maint-Depends and Runtime-Depends.

This is problematic though, and several reasons have been presented in 804624, but the hope is to eventually find a solution to this problem.

Julian Andres Klode proposed an alternative representation of the same concept. Instead of introducing a new field, the binary package relation syntax could be extended using a notation similar to build profiles. As such, dependencies could be tagged for their relevant use in even more detail (on a maintainer script granularity). Consider for example Depends: foo <postinst>.

If we allow removal of packages listed in Maint-Depends, we may end up in situations where removing another package becomes impossible without (temporarily) installing its Maint-Depends. When that becomes impossible, such packages may become unremovable. This would also make our dependency graph incompatible with CUDF solving. As a consequence, removing Maint-Depends should not normally be allowed.

The proposed splitting of Depends may help with breaking dependency loops. A looping dependency can sometimes be broken by moving it to Runtime-Depends. The relevant maintainer scripts can then be run without satisfying Runtime-Depends.

Proposal: declarative ordering

The biggest piece of knowledge about install bootstrap that resides in debootstrap is the ordering or package installations. For instance, it knows that base-passwd must be installed first and base-files must be next. Moving this ordering information to metadata could be an option. A frequent suggestion is adding dependencies between essential packages for this purpose as apt does honour. Adding such dependencies is not universally agreed upon though as policy section 3.8 declares such dependencies as unnecessary.

The apt solver deals badly with relations between essential packages. Workarounds had to be added e.g. for dealing with Breaks introduced by in glibc.

Proposal: bootstrap scripts

This is a variant of chrootless maintainer scripts. In essence it removes the chrootless part and leaves the idea of having separate maintainer scripts for the installation bootstrap to ones used for regular installation or upgrades. In essence regular preinst and postinst invocations would be skipped entirely during installation bootstrap and a separate bootstrap script would be run after all relevant packages have been unpacked in unspecified order. It should have the same effect as if a package were regularly installed, under a number of stronger assumptions:

  • No init system or any kind of service is being run from the root directory of the installation being operated on.
  • No packages outside the transitive essential set have been unpacked.
  • The bootstrap script is only run once.