Differences between revisions 40 and 42 (spanning 2 versions)
Revision 40 as of 2014-01-04 00:34:41
Size: 20942
Editor: wookey
Comment:
Revision 42 as of 2014-01-15 05:57:25
Size: 22996
Editor: GeoffSimmons
Comment: InterWiki, use internal links, limit automatic linking, formatting.
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#language en
Line 4: Line 5:
This wiki page is intended to describe the issues of, and mechanisms for, bootstrapping Debian from zero to a full archive. Otherwise known as '''profile builds''' (and previously '''staged builds'''). It has developed somewhat over a couple of years as practical experience has grown, and is now fairly mature. This wiki page is intended to describe the issues of, and mechanisms for, bootstrapping Debian from zero to a full archive. Otherwise known as '''profile builds''' (and previously '''staged builds'''). It has developed somewhat over a couple of years as practical experience has grown, and is now technically useable, and making its way into the archive.
Line 12: Line 13:
Recent new bootstraps have been done for [[SH4|sh4]], [[ArmHardFloatPort|armhf]], [[https://alioth.debian.org/projects/i386-uclibc|uclibc]] and [[http://avr32.debian.net/|avr32]]. More are coming down the line, including the imminent [[Arm64Port|arm64]]. Subarch flavoured rebuilds (e.g. to optimise for a particular CPU) are particularly useful on ARM and MIPS architectures. New bootstraps are done every year. At the time of writing [[SH4|sh4]], [[ArmHardFloatPort|armhf]], [[https://alioth.debian.org/projects/i386-uclibc|uclibc]] and [[http://avr32.debian.net/|avr32]], and x32 have been done, and [[Arm64Port|arm64]] and mips64el are in progress. Subarch flavoured rebuilds (e.g. to optimise for a particular CPU) are particularly useful on ARM and MIPS architectures (Raspberry Pi is most popular recent example).
Line 20: Line 21:
This work does need build-system and policy changes, which are detailed on this page.

== Packaging principles ==

An important principle is that the packaging changes necessary for this to work are reasonably clear and transparent. A Debian packager should not have to understand this stuff (staged builds and cross-building) in loving detail to avoid breaking things whilst making maintenance changes. Considering this principle helps when deciding between different technically-satisfactory ways of achieving things.
This work needs build-system and policy changes, which are detailed on this page.

The goal is to be able to run regular automated bootstrap builds of the archive to check that it works, and allow porters to concentrate on actual porting work, rather than a lot of associated aggravation
.

== Status ==

The current (Jan 2014) status is:

 * build profiles are supported in dpkg 1.17.2 onwards.
 * Support in the buildd infrastructure is still needed
 * Suite bootstrappability is tracked at https://bootstrap.debian.net/
 * Bugs are tracked at http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=bootstrap@debian.org;tag=bootstrap

== Packaging/Design principles ==

An important principle in the design is that the packaging changes necessary for this to work are reasonably clear and transparent. A Debian packager should not have to understand this stuff (staged builds and cross-building) in loving detail to avoid breaking things whilst making maintenance changes.
Line 28: Line 40:
These principles informed the design when deciding between different technically-satisfactory ways of achieving things.
Line 33: Line 47:
Working out which packages to modify, and how, has so far been a manual process, done by manually examining build-dep loops and choosing which packages are most easily and cleanly modified. Since the "[[http://blog.mister-muffin.de/2013/01/25/bootstrappable-debian---new-milestone/|Port bootstrap build-ordering tool]]" [[SummerOfCode2012/StudentApplications/JohannesSchauer|project]] in the [[SummerOfCode2012|GSoC 2012]], the process of finding source packages which make sense to modify can be semi-automated. Once that is done, building bootstrap-able packages can be a fully automatable process. Working out which packages to modify, and how, is fundamentally a manual process, done by manually examining build-dep loops and choosing which packages are most easily and cleanly modified. Since the "[[http://blog.mister-muffin.de/2013/01/25/bootstrappable-debian---new-milestone/|Port bootstrap build-ordering tool]]" [[SummerOfCode2012/StudentApplications/JohannesSchauer|project]] in the [[SummerOfCode2012|GSoC 2012]], the process of finding source packages which make sense to modify can be semi-automated. Once that is done, building bootstrap-able packages can be a fully automated process.
Line 37: Line 51:
The reduced dependencies are specified in the control file, using a concept which is called build profiles. The build profile format was proposed by Guillem Jover together with other solutions he presented in bug#DebianBug:661538. Build profiles extend the Build-Depends format with a syntax similar to architecture restrictions but using < and > instead. The current Spec is detailed in BuildProfileSpec. Previous revisions were discussed on debian-devel and debian-dpkg. The reduced dependencies are specified in the control file, using a concept which is called '''build profiles'''. The build profile format was proposed by Guillem Jover together with other solutions he presented in bug#DebianBug:661538. Build profiles extend the Build-Depends format with a syntax similar to architecture restrictions but using < and > instead. The current Spec is detailed in BuildProfileSpec. Previous revisions were discussed on debian-devel and debian-dpkg.
Line 40: Line 54:
Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny Build-Depends: huge (>= 1.0) [i386 arm] <!cross !stage1>, tiny
Line 43: Line 57:
The build dependency "huge" would not be required by the source package if it is built in the "embedded" or "stage1" profile. This mechanism neatly allows for removed build-deps, replaced build-deps and added build-deps, and an arbitrary number of possible 'profiles'.

Besides bootstrapping, these build profiles could also be used for embedded builds, and to allow for changed buil-deps when cross-building. Patches for dpkg (bug#DebianBug:661538) and dose3 implementing this syntax already exist.

This scheme supersedes an earlier version, (referred-to as 'staged' builds), which used repeated Build-depends-StageN: lines. See the dpkg bug#DebianBug:661538 for the evolution of this. The profile labels are arbitrary but agreement on label usage is necessary. For bootstrap automation we have been using 'stage1', 'stage2', etc which fits with existing custom in packages which already have such internal mechanisms using DEB_STAGE (currently gcc, eglibc, libselinux, gcj, gnat, gdc, linux [9]) These seem like sensible names so we propose to stick with them. Other useful profiles can be defined in the future.

An environment variable (DEB_BUILD_PROFILES) is used to control when packages are built in reduced staged/bootstrap mode, and at what stage.. debian/rules can check this variable and miss out some optional features to reduce the dependency tree (e.g. building kerberos without LDAP support). dpkg-buildpackage/dpkg-checkbuilddeps also checks the reduced/changed build-dependencies instead of the normal ones.

DEB_BUILD_PROFILES is used because it is aready preserved and passed on by all build tools, and this is a build option like other things it is used to control.
The build dependency "huge" would not be required by the source package if it is built in the "cross" or "stage1" profile. This mechanism neatly allows for removed build-deps, replaced build-deps and added build-deps, and an arbitrary number of possible 'profiles'.

Besides bootstrapping, these build profiles can also be used for embedded builds, to allow for changed build-deps when cross-building, and build-dependencies only needed for running tests.

This scheme supersedes an earlier version, (referred-to as 'staged' builds), which used repeated Build-depends-StageN: lines. See the dpkg bug#DebianBug:661538 for the evolution of this. The profile labels are arbitrary but agreement on label usage is necessary. For bootstrap automation we have been using 'stage1', 'stage2', etc which fits with existing custom in packages which already have such internal mechanisms using DEB_STAGE (currently gcc, eglibc, libselinux, gcj, gnat, gdc, linux [9]) These seem like sensible names so we propose to stick with them. Other useful profiles are 'cross' and 'notest'.

An environment variable (DEB_BUILD_PROFILES) is used to control when packages are built in reduced staged/bootstrap mode, and at what stage.. debian/rules can check this variable and miss out some optional features to reduce the dependency tree (e.g. building kerberos without LDAP support, avahi without gtk and qt, etc). dpkg-buildpackage/dpkg-checkbuilddeps also checks the reduced/changed build-dependencies instead of the normal ones.

DEB_BUILD_PROFILES is used because it is preserved and passed on by all build tools, and this is a build option like other things it is used to control.
Line 55: Line 69:
Bootstrapped/Staged packages are marked as such (in the control file) either with a new field or with a version having ~stage1 appended to it. They are not uploaded to normal repositories. It is important to avoid accidentally mistaking a bootstrap/staged package for a 'real' (normally-built) package. As soon as possible a bootstrap package should be rebuilt as a full package, to avoid having to rebuild many packages aginst the full version once it is available. The details for this are not fleshed-out, but an extra control header seems the obvious thing to do. A version suffix may be useful too, mostly to help humans. Bootstrapped/Staged packages are marked as such (in the control file) with a new field called Built-for-profiles:. They are not uploaded to normal repositories. It is important to avoid accidentally mistaking a bootstrap/staged package for a 'real' (normally-built) package. As soon as possible a bootstrap package should be rebuilt as a full package, to avoid having to rebuild many packages against the full version once it is available. A version suffix is useful too, mostly to help humans, but is not enforced by the tools.
Line 59: Line 73:
This process is usually called 'staged' builds as well as 'bootstrap' or 'profile' builds. Exact field names and variable names is a subject for bikeshedding. Whatever is most likely to be clear to developers and not clash with other purposes is best. This process is usually called 'profile' builds, but the terms 'staged' builds and 'bootstrap' builds have also been used as it developed.
Line 63: Line 77:
Proof-of-concept patches for packages that need to understand the new fields have been made. They are here:
http://wookware.org/software/cyclicdeps/patches/
These are the main patches that have implemented this, going from proof-of-concept to final archive uploads.
Line 72: Line 85:
Here are some early patches for the tools:
http://people.debian.org/~wookey/bootstrap/patches/profiles/tools/

Here are various patches for packages, adding support for profiled builds of packages:
http://people.debian.org/~wookey/bootstrap/patches/profiles/packages/
Line 82: Line 101:
For cross-building to be reliable cross-dependency metadata needs to be in packages, so that it is clear whether a build dependency should be satisfied by the build architecture or the host architecture. Multiarch information can be used to provide this information along with build-dependency decoration for the farily rare exceptions. Details are specified here: https://wiki.ubuntu.com/MultiarchCross For cross-building to be reliable cross-dependency metadata needs to be in packages, so that it is clear whether a build dependency should be satisfied by the build architecture or the host architecture. Multiarch information can be used to provide this information along with build-dependency decoration for the farily rare exceptions. Details are specified here: UbuntuWiki:MultiarchCross
Line 86: Line 105:
Prior to useful amounts of multiarch metadata is in packages, heurisitics must be used, as implemented in xdeb (and the now-deprecated apt-cross), or all dependencies must be installed for both host and native, as implemented in xapt. These are all ugly and horrid, but better than nothing. xdeb or xapt are better than multiarch-cross in whezzy and precise, but multiarch-cross is where all the current work is, and as of end 2012 it works for much of the base system in Debian unstable and Ubuntu raring. Prior to useful amounts of multiarch metadata is in packages, heurisitics must be used, as implemented in xdeb (and the now-deprecated apt-cross), or all dependencies must be installed for both host and native, as implemented in xapt. These are all ugly and horrid, but better than nothing. xdeb or xapt are better than multiarch-cross in wheezy and precise, but multiarch-cross is where all the current work is, and as of end 2012 it works for much of the base system in Debian unstable and Ubuntu raring. From Maverick and Jessie onwards multiarch cross-building is the recommended method.
Line 92: Line 111:
Packges do need to be uploaded to a bootstrap repository in order to satisfy build-dependecies (that's the whole point). However because packages of the same version get built, and thus uploaded, more than once we need a way to deal with this. One proposal is to append ~stageN+M to the package version automatically, where N is the stage number and M a continuously incremented (by the buildd) number. Or do binNMUs, which are already recognised by the tools and package management system very well, and almost all packages are (supposedly) binNMU safe. This part has not yet been implmented.

See 'Changed binary packages' below for a discussion of how profiles causing a binary package normally built to be omitted should be dealt with. How/if we choose to mark the profile info in control files affects whether a static analysis tool can determine in advance if the package set has a fully linearisable build-order or not.
Packages do need to be uploaded to a bootstrap repository in order to satisfy build-dependecies (that's the whole point). However because packages of the same version get built, and thus uploaded, more than once we need a way to deal with this. One proposal is to append ~stageN+M to the package version automatically, where N is the stage number and M a continuously incremented (by the buildd) number. Or do binNMUs, which are already recognised by the tools and package management system very well, and almost all packages are (supposedly) binNMU safe. This was awkward when combined with multiarch until the separate binNMU changelog feature was implemented.

See 'Changed binary packages' below for a discussion of how profiles causing a binary package normally built to be omitted should be dealt with. The meta-data in package Build-Deps allows static analysis tools to determine in advance if the package set has a fully linearisable build-order or not. The [[https://gitorious.org/debian-bootstrap/botch|botch]] tool can do this analysis.
Line 105: Line 124:
 * Most languages depend on themselves to build (gcc, openjdk, mono, haskell, perl, python, ada(gnat)).  * Most languages depend on themselves to build (gcc, openjdk, mono, haskell, python, ada(gnat)).
Line 107: Line 126:
    kerberos → ldap → kerberos<<BR>>
    qt → poppler → cups → qt
 * Documentation packages. Many packages need documentation tools (sgmltools, jade, tex, doxygen) which cannot be built until many other packages are built. This is largely only a problem for native-builds, as the doc-tools are generally available when cross-building.
   * kerberos → ldap → kerberos<<BR>>
   * qt → poppler → cups → qt
   * pulseaudio → bluez → gst-plugins-base0.10 → libtheora → libsdl1.2 → pulseaudio

 * Documentation packages. Many packages need documentation tools (sgmltools, jade, tex, doxygen) which cannot be built until many other packages are built. This is largely only a problem for native-builds, as the doc-tools are generally available when cross-building. A 'nodocs' profile is very useful here, and also when build hardware is slow or emulated.
Line 113: Line 133:
Encoding the build-profile dependencies in the control file makes it simple for automatic build-ordering and analysis tool.

The original spec was proposed here: https://wiki.ubuntu.com/Specs/M/ARMAutomatedBootstrap

This wiki page now comprises the spec developed out that spec and proposes some further ideas and change
s.
Encoding the build-profile dependencies in the control file makes it simple for automatic build-ordering and analysis tools.
Line 123: Line 139:
'Staged' builds are invoked by setting DEB_BUILD_OPTIONS to specify a staged build to dpkg-buildpackage. The original patch uses 'stage=N', but the dpkg team has suggested that using 'DEB_BUILD_PROFILES=<string>' is more versatile as arbitrary profiles could be useful for other purposes ('embedded' profile for example, and maybe <cross>). When no profile option is set then a normal build occurs. Some packages may need more than one staged build. We do not know what the maximum number of stages needed is: it is proably two, but to assume so would be foolish. We count up from Stage1, Stage2 to 'normal'. Hopefully this is reasonably clear to the average packager.

Any 'staged' package must be identified as such in the metadata so it is not accidentally uploaded as a 'real' package. The mechanism is an extra header inserted by dpkg-genchanges, to indicate the staged build. The existing patch sets 'Staged-Build=N' when DEB_BUILD_OPTIONS=stageN. The proposed profile mechanism would set Built-with-Profile=profile, i.e in the case of staged builds, Built-with-Profile=stage1 etc.
'Profile' builds are invoked by setting DEB_BUILD_PROFILES to specify which profile(s) to build to dpkg-buildpackage. When no profile option is set then a normal build occurs. Some packages may need more than one staged build. We do not know what the maximum number of stages needed is: it is proably two, but to assume so would be foolish. We count up from Stage1, Stage2 to 'normal'. Hopefully this is reasonably clear to the average packager.

Any profile-built package must be identified as such in the metadata so it is not accidentally uploaded as a 'real' package. The mechanism is an extra header inserted by dpkg-genchanges, to indicate which profiles it was built with. dpkg sets Built-with-Profiles=<profiles>, i.e in the case of a bootstrap builds, {{{Built-with-Profiles: stage1}}}
Line 129: Line 145:
Let's consider kerberos as a typical example of a library package involved in a circular dependency. krb5 needs libldap2-dev to build (from openldap). openldap need libkrb5-dev (from krb5) to build. To fix this we add a staged build to krb5 to miss out the generation of the krb5-ldap package. This is easy to do with a debhelper-based package by simply setting DH_OPTIONS="--no-package=krb5-ldap", and running configure with --without-ldap (when DEB_BUILD_OPTIONS=stage=1). Let's consider kerberos as a typical example of a library package involved in a circular dependency. krb5 needs libldap2-dev to build (from openldap). openldap need libkrb5-dev (from krb5) to build. To fix this we add a staged build to krb5 to miss out the generation of the krb5-ldap package. This is easy to do with a debhelper-based package by simply setting DH_OPTIONS="--no-package=krb5-ldap", and running configure with --without-ldap (when DEB_BUILD_PROFILES=stage1).
Line 139: Line 155:
This syntax allows the usual case of missing out a build-dep but also supports substitutions. It needs a dpkg patch to understand the <> syntax.  This syntax allows the usual case of missing out a build-dep but also supports substitutions. dpkg 1.17.2 onwards (i.e Jessie or later) understands this syntax.
Line 142: Line 158:
The earlier proposal was to use extra Build-depends lines:

Build-Depends-StageN simply lists all the build-dependencies again except changing or missing out some as required. This does need to be maintained along with the normal build-depends. This makes it very easy to implement as existing tools will not barf as no new syntax is used. However it is an ugly implementation inside dpkg with many extra fields defined and a limit (of 2) on how many stages could be supported. No other profiles types are possible without hard-defining more feilds inside dpkg.


So for krb5 we'd add:
So for krb5 we'd have:
Line 149: Line 160:
Build-Depends-Stage1:debhelper (>= 7), byacc | bison, comerr-dev, docbook-to-man, Build-Depends:debhelper (>= 7), byacc | bison, comerr-dev, docbook-to-man,
Line 151: Line 162:
 libncurses5-dev, libssl-dev, ss-dev, texinfo  libncurses5-dev, libssl-dev, ss-dev, ldap2-dev <!stage1> texinfo
Line 154: Line 165:
Setting DEB_BUILD_PROFILES=stage1 and building this package causes it to build without ldap support, and without needed ldap2-dev available.
Line 157: Line 169:
Setting DEB_BUILD_OPTIONS=profile=stage1 and building this package causes it to produce lang-bootstrap (which is normally not emitted). This is implemented by adding a new control stanza for lang-bootstrap and specifying --no-package=lang-bootstrap in debian/rules for normal builds, but not for the stage1 build (which will probably exclude a load of other stuff). Setting DEB_BUILD_PROFILES=stage1 and building this package causes it to produce lang-bootstrap (which is normally not emitted). This is implemented by adding a new control stanza for lang-bootstrap and specifying --no-package=lang-bootstrap in debian/rules for normal builds, but not for the stage1 build (which will probably exclude a load of other stuff).
Line 162: Line 174:
For documentation issues being able to specify {{{DEB_BUILD_OPTIONS=nodocs}}} would be simplest. Building with docs affects the dependencies, so it is not like other DEB_BUILD_OPTIONS, so perhaps this is not a good mechanism to use? Something generic is attractive if we can make it work. For documentation issues some packages already implement {{{DEB_BUILD_OPTIONS=nodocs}}}. Building without docs usually affects the build-dependencies, so it is not quite like other DEB_BUILD_OPTIONS, and using DEB_BUILD_PROFILES instead now makes more sense.
Line 168: Line 180:
Sometimes a profile/staged build will omit a binary package that is normally produced. e.g. if there is a foo-gui binary that brings in a lot of gtk build-deps, but foo is also a low-level package needed by other things. Often a profile/staged build will omit a binary package that is normally produced. e.g. if there is a foo-gui binary that brings in a lot of gtk build-deps, but foo is also a low-level package needed by other things (avahi is a good example of this).
Line 173: Line 185:
Builds-With-Profile: !stage1 (for packages not built in stage1)
Builds-With-Profile: stage2 (for packages built only in stage2)
X-Builds-With-Profile: !stage1 (for packages not built in stage1)
X-Builds-With-Profile: stage2 (for packages built only in stage2)
Line 180: Line 192:
It's a minor change to dpkg-gencontrol Implementing this is a minor change to dpkg-gencontrol

== History ==

This section collects some information about how this concept has developed over time in order to make the above page clearer.

The initial proposal was written down in 2010 as a result of a UDS (ubuntu Developer Summit) discussion: UbuntuWiki:Specs/M/ARMAutomatedBootstrap
Back then the concept was called 'Staged Builds', and used a simple extra line in the control file repeating and modifying the normal build-dependencies: Build-Depends-StageN
and builds controlled with DEB_BUILD_OPTIONS=profile=stage1

This made it very easy to implement as existing tools will not barf as no new syntax is used. However it is an ugly implementation inside dpkg with many extra fields defined and a limit (of 2) on how many stages could be supported. No other profiles types are possible without hard-defining more fields inside dpkg.
See DebianBug:661538 for how that developed.

3 successive years of GSOC projects by Gustavo Alkmim, P. J !McDermott, Johannes Schauer, and Thibaut Girka, progressed the practical and theoretical work, as well as heavy sponsorship of this work by Linaro and Canonical which enabled a lot of the supporting cross-building and multiarch work to get done.

It became clear from actually doing staged builds that it made more sense if cross-building, test-skipping, doc-skipping and language/tool extra dependencies (bootstrapping) changes were described independently rather than arbitrarily lumped together in the way a bootstrap on a particular arch might like it. So the newer profile modifier syntax was developed and implemented, whilst being tested on actual cross-builds and bootstraps.

Over 2013 an actual spec and implementation in dpkg was pinned down, agreed and implemented, leading to a profile-supporting dpkg upload at the end of 2013.
Line 193: Line 222:
    http://wiki.debian.org/SummerOfCode2011/AutomatedBootstrapping/GustavoAlkmim     SummerOfCode2011/AutomatedBootstrapping/GustavoAlkmim
Line 199: Line 228:
Thanks to Jonathan Austin, Steve McIntyre, Steve Lanagsek and Loic Minier for helping clarify the thoughts described above. Thanks to Jonathan Austin, [[SteveMcIntyre|Steve McIntyre]], Steve Langasek, Loic Minier, [[PJMcDermott|P. J. McDermott]], and Johannes Schauer for helping clarify the thoughts described above.
Line 203: Line 232:
CategoryEmdebian CategoryDebianDevelopment CategoryEmdebian | CategoryDebianDevelopment

Bootstrapping

This wiki page is intended to describe the issues of, and mechanisms for, bootstrapping Debian from zero to a full archive. Otherwise known as profile builds (and previously staged builds). It has developed somewhat over a couple of years as practical experience has grown, and is now technically useable, and making its way into the archive.

Overview

There is a real need to bootstrap Debian from sources when doing new ports or flavours (sub-arch builds). Every new architecture or optimisation flavour needs to do this at least once, and making it easier than the current 'really very hard' would be great. It is also very useful for cross-compiling to new or non-self-hosted architectures, and for a genuinely new arch at least part of the system (toolchains+build-essential) has to be cross-built until there is enough to become self-hosting.

New bootstraps are done every year. At the time of writing sh4, armhf, uclibc and avr32, and x32 have been done, and arm64 and mips64el are in progress. Subarch flavoured rebuilds (e.g. to optimise for a particular CPU) are particularly useful on ARM and MIPS architectures (Raspberry Pi is most popular recent example).

This is also helpful when bringing a lagged architecture up to date, especially considering documentation tools (that may be too old), optional dependencies (that may be too old or not exist) like php5, which depends on everything and the kitchen sink, etc. I wish this had been implemented when starting to work on m68k…

Currently people tend to use non-debian tools (such as Yocto/Gentoo/OpenEmbedded) to get a basic rootfs image of the target arch/ABI then do native building within that. This works but needs a great deal of manual loop-breaking and we really ought to be able to bootstrap our own OS.

Putting the necessary bootstrapping metadata and build rules into the packages themselves in an orderly fashion enables the info to be maintained easily. QA tests to report on breakage will help enormously here. It also makes for a repeatable and deterministic process.

This work needs build-system and policy changes, which are detailed on this page.

The goal is to be able to run regular automated bootstrap builds of the archive to check that it works, and allow porters to concentrate on actual porting work, rather than a lot of associated aggravation.

Status

The current (Jan 2014) status is:

Packaging/Design principles

An important principle in the design is that the packaging changes necessary for this to work are reasonably clear and transparent. A Debian packager should not have to understand this stuff (staged builds and cross-building) in loving detail to avoid breaking things whilst making maintenance changes.

All the metadata needed should be part of the packages, so it can be maintained over time. Any solution with external patches/metadata is doomed to bitrot.

These principles informed the design when deciding between different technically-satisfactory ways of achieving things.

Mechanisms

The concept is simple: add support for minimal/reduced/staged builds to packages involved in build-dependency loops, so that build loops are broken. Also ensure packages cross-build properly so that an initial minimal native-building system can be produced.

Working out which packages to modify, and how, is fundamentally a manual process, done by manually examining build-dep loops and choosing which packages are most easily and cleanly modified. Since the "Port bootstrap build-ordering tool" project in the GSoC 2012, the process of finding source packages which make sense to modify can be semi-automated. Once that is done, building bootstrap-able packages can be a fully automated process.

This spec caters for multiple stages of staged/bootstrap build, so that if necessary a package can have stage1, stage2, etc. before the final, normal, build. Almost all packages only need one stage other than the standard build. Only toolchain packages are known to have more than one stage at this time.

The reduced dependencies are specified in the control file, using a concept which is called build profiles. The build profile format was proposed by Guillem Jover together with other solutions he presented in bug#661538. Build profiles extend the Build-Depends format with a syntax similar to architecture restrictions but using < and > instead. The current Spec is detailed in BuildProfileSpec. Previous revisions were discussed on debian-devel and debian-dpkg.

Build-Depends: huge (>= 1.0) [i386 arm] <!cross !stage1>, tiny

The build dependency "huge" would not be required by the source package if it is built in the "cross" or "stage1" profile. This mechanism neatly allows for removed build-deps, replaced build-deps and added build-deps, and an arbitrary number of possible 'profiles'.

Besides bootstrapping, these build profiles can also be used for embedded builds, to allow for changed build-deps when cross-building, and build-dependencies only needed for running tests.

This scheme supersedes an earlier version, (referred-to as 'staged' builds), which used repeated Build-depends-StageN: lines. See the dpkg bug#661538 for the evolution of this. The profile labels are arbitrary but agreement on label usage is necessary. For bootstrap automation we have been using 'stage1', 'stage2', etc which fits with existing custom in packages which already have such internal mechanisms using DEB_STAGE (currently gcc, eglibc, libselinux, gcj, gnat, gdc, linux [9]) These seem like sensible names so we propose to stick with them. Other useful profiles are 'cross' and 'notest'.

An environment variable (DEB_BUILD_PROFILES) is used to control when packages are built in reduced staged/bootstrap mode, and at what stage.. debian/rules can check this variable and miss out some optional features to reduce the dependency tree (e.g. building kerberos without LDAP support, avahi without gtk and qt, etc). dpkg-buildpackage/dpkg-checkbuilddeps also checks the reduced/changed build-dependencies instead of the normal ones.

DEB_BUILD_PROFILES is used because it is preserved and passed on by all build tools, and this is a build option like other things it is used to control.

So setting DEB_BUILD_PROFILES=stage1 will cause dpkg-buildpackage to call dpkg-checkbuilddeps with -Pstage1 so that Stage1 dependencies are checked, rather than the normal set. Similarly other tools like apt and xdeb will use the stage1 dependencies for the top-level package.

Bootstrapped/Staged packages are marked as such (in the control file) with a new field called Built-for-profiles:. They are not uploaded to normal repositories. It is important to avoid accidentally mistaking a bootstrap/staged package for a 'real' (normally-built) package. As soon as possible a bootstrap package should be rebuilt as a full package, to avoid having to rebuild many packages against the full version once it is available. A version suffix is useful too, mostly to help humans, but is not enforced by the tools.

Terminology

This process is usually called 'profile' builds, but the terms 'staged' builds and 'bootstrap' builds have also been used as it developed.

Available Patches

These are the main patches that have implemented this, going from proof-of-concept to final archive uploads.

The dpkg bug gives a useful idea of how this spec has developed.

Here are some early patches for the tools: http://people.debian.org/~wookey/bootstrap/patches/profiles/tools/

Here are various patches for packages, adding support for profiled builds of packages: http://people.debian.org/~wookey/bootstrap/patches/profiles/packages/

Cross-building

Bootstrapping is closely related to support for cross-building Debian packages because at least part of the process must be done cross. Enough packages to make a bootable image need to be cross-buildable, because you cannot magic a system out of thin air. To move from cross to native building you need build-essential to be cross-buildable.

The number of build-loops that must be broken for cross-building is much smaller than the number that need to be broken for native building. This spec proposes that we start by fixing the loops that mean you can't even cross-build a base Debian image before going on to fix all the packages which have native build-dep loops.

Debian/Ubuntu cross-building is documented here: https://wiki.linaro.org/CrossBuilding

Patched sources with build-profile patches applied and extra multiarch and profile build support in package metadata are in http://people.debian.org/~wookey/bootstrap.html

For cross-building to be reliable cross-dependency metadata needs to be in packages, so that it is clear whether a build dependency should be satisfied by the build architecture or the host architecture. Multiarch information can be used to provide this information along with build-dependency decoration for the farily rare exceptions. Details are specified here: MultiarchCross

The current state of buildability using this technology is recorded here: http://people.linaro.org/~wookey/buildd/ and http://people.canonical.com/~cjwatson/cross/armhf/raring/

Prior to useful amounts of multiarch metadata is in packages, heurisitics must be used, as implemented in xdeb (and the now-deprecated apt-cross), or all dependencies must be installed for both host and native, as implemented in xapt. These are all ugly and horrid, but better than nothing. xdeb or xapt are better than multiarch-cross in wheezy and precise, but multiarch-cross is where all the current work is, and as of end 2012 it works for much of the base system in Debian unstable and Ubuntu raring. From Maverick and Jessie onwards multiarch cross-building is the recommended method.

Automated Bootstrapping

The full automated bootstrapping process needs to keep track of staged builds and rebuilding things as needed so that they don't hang around any longer than necessary. However any such tool could get out of sync with the current status, unless it is always determinable from the current package-set state. This spec attempts to define things such that it is always intrinsically stateful.

Packages do need to be uploaded to a bootstrap repository in order to satisfy build-dependecies (that's the whole point). However because packages of the same version get built, and thus uploaded, more than once we need a way to deal with this. One proposal is to append ~stageN+M to the package version automatically, where N is the stage number and M a continuously incremented (by the buildd) number. Or do binNMUs, which are already recognised by the tools and package management system very well, and almost all packages are (supposedly) binNMU safe. This was awkward when combined with multiarch until the separate binNMU changelog feature was implemented.

See 'Changed binary packages' below for a discussion of how profiles causing a binary package normally built to be omitted should be dealt with. The meta-data in package Build-Deps allows static analysis tools to determine in advance if the package set has a fully linearisable build-order or not. The botch tool can do this analysis.

Toolchain

The toolchain has a complex 3-stage bootstrapping process involving binutils, gcc, glibc and kernel-headers. It uses the DEB_STAGE variable name internally to control the build stages. The multiple stage sequencing is currently managed by external packages in Debian (buildcross) and Ubuntu (<arch>-cross-toolchain-base and gcc-cross-defaults), using binary -source packages.

Using multiarch to supply cross-libc actually removes some of the complexity of cross-toolchain building, and was the subject of a 2012 GSOC project. That was successful and cross-toolchains can now be built that way. Work is ongoing to get this in a state suitable for the archive, so allowing cross-toolchains to be built by buildds. This work is tracked on the MultiarchCrossToolchains page

Circular dependencies/staged builds

The main issue is circular build-dependencies. These fall into three main areas:

  • Most languages depend on themselves to build (gcc, openjdk, mono, haskell, python, ada(gnat)).
  • Libraries sometimes circularly depend:

    • kerberos → ldap → kerberos

    • qt → poppler → cups → qt
    • pulseaudio → bluez → gst-plugins-base0.10 → libtheora → libsdl1.2 → pulseaudio
  • Documentation packages. Many packages need documentation tools (sgmltools, jade, tex, doxygen) which cannot be built until many other packages are built. This is largely only a problem for native-builds, as the doc-tools are generally available when cross-building. A 'nodocs' profile is very useful here, and also when build hardware is slow or emulated.

The generic way to deal with all of these is 'profile builds', where a version of the package is built with lesser functionality and thus a smaller dependency tree. This allows the depending package to then be built, then for the 'stageN' package to be built normally.

Encoding the build-profile dependencies in the control file makes it simple for automatic build-ordering and analysis tools.

CircularBuildDependencies is a list of loops found in the initial (2011) analysis.

Specifying profiles/stages

'Profile' builds are invoked by setting DEB_BUILD_PROFILES to specify which profile(s) to build to dpkg-buildpackage. When no profile option is set then a normal build occurs. Some packages may need more than one staged build. We do not know what the maximum number of stages needed is: it is proably two, but to assume so would be foolish. We count up from Stage1, Stage2 to 'normal'. Hopefully this is reasonably clear to the average packager.

Any profile-built package must be identified as such in the metadata so it is not accidentally uploaded as a 'real' package. The mechanism is an extra header inserted by dpkg-genchanges, to indicate which profiles it was built with. dpkg sets Built-with-Profiles=<profiles>, i.e in the case of a bootstrap builds, Built-with-Profiles: stage1

It must be possible for the build-tools to identify what build-stages are available. By reading the profiles specified in the Build-Depends. The existence of that defines such a stage as being available.

Let's consider kerberos as a typical example of a library package involved in a circular dependency. krb5 needs libldap2-dev to build (from openldap). openldap need libkrb5-dev (from krb5) to build. To fix this we add a staged build to krb5 to miss out the generation of the krb5-ldap package. This is easy to do with a debhelper-based package by simply setting DH_OPTIONS="--no-package=krb5-ldap", and running configure with --without-ldap (when DEB_BUILD_PROFILES=stage1).

Dealing with changed build dependencies

The DEB_BUILD_PROFILES syntax is: Build_Depends: foo , bar <!stage1>, libfoo2 <!stage1 !stage2>, wibble <stage2>

Which is an implausibly complicated example which normally build-depnds on foo, bar, libfoo2. For a stage1 build it doesn't need bar or libfoo2. For stage2 it needs foo, bar and also needs wibble, but does not need libfoo2

This syntax allows the usual case of missing out a build-dep but also supports substitutions. dpkg 1.17.2 onwards (i.e Jessie or later) understands this syntax. Its major disadvantage is that packages using this syntax cannot be uploaded until the dpkg on the buildds understands this syntax. And the Wheezy dpkg does not understand this syntax.

So for krb5 we'd have:

Build-Depends:debhelper (>= 7), byacc | bison, comerr-dev, docbook-to-man,
 libkeyutils-dev [!kfreebsd-i386 !kfreebsd-amd64 !hurd-i386]
 libncurses5-dev, libssl-dev, ss-dev, ldap2-dev <!stage1> texinfo

Setting DEB_BUILD_PROFILES=stage1 and building this package causes it to build without ldap support, and without needed ldap2-dev available.

For packages which depend on themselves (usually languages), the Build-dependencies should be changed to depend on lang | lang-bootstrap. In a normal repository the (native version) lang-bootstrap will not be available so a lang will be used. In a bootstraping environment lang may well not be available in which case lang-bootstrap needs to be built. The bootstraping tool knows to do a staged build in this case.

Setting DEB_BUILD_PROFILES=stage1 and building this package causes it to produce lang-bootstrap (which is normally not emitted). This is implemented by adding a new control stanza for lang-bootstrap and specifying --no-package=lang-bootstrap in debian/rules for normal builds, but not for the stage1 build (which will probably exclude a load of other stuff).

Documentation loops

For documentation issues some packages already implement DEB_BUILD_OPTIONS=nodocs. Building without docs usually affects the build-dependencies, so it is not quite like other DEB_BUILD_OPTIONS, and using DEB_BUILD_PROFILES instead now makes more sense.

Documentation loops are primarily an issue for native building, although they do cause issues for cross-building too (gobject introspection, perl module docs).

Changed binary packages

Often a profile/staged build will omit a binary package that is normally produced. e.g. if there is a foo-gui binary that brings in a lot of gtk build-deps, but foo is also a low-level package needed by other things (avahi is a good example of this).

If these changed binary outputs are not recording in package metadata then a tool analysing the build-order cannot determine in advance whether a build-order will complete, because although it knows that a package has a build profile which needs reduced build-deps, it does not know if any of the package binaries will not be produced for this particular profile, and thus what could be built afterwards. For an iterative process where stuff that can be built next is built, then the new state examined for more things to build, this does not really matter, but it would be good if analysis tools like dose3 could determine from package files alone whether or not a bootstrap order exists and which order would be the most optimal in terms of least number of source packages which are profile built.

A suggested mechanism to allow this is to add a field to affected binary packages in debian/control saying X-Builds-With-Profile: !stage1 (for packages not built in stage1) X-Builds-With-Profile: stage2 (for packages built only in stage2)

Nearly all packages would not have any such notation and would thus always be produced by all profiles, as now. Current analysis suggests that there will not be more than 83 source packages which have to have this new field.

This feature may be deemed unnecessary complexity that is only actually needed by external analysis tools. Maintaining this data means that a bootstrap analysis can be known to be correct and if it says there is a build order then that order will actually work. Without this info the tool may produce a build order but it may not actually work in practice because some binary-package will not be built, but will actually be needed.

Implementing this is a minor change to dpkg-gencontrol

History

This section collects some information about how this concept has developed over time in order to make the above page clearer.

The initial proposal was written down in 2010 as a result of a UDS (ubuntu Developer Summit) discussion: Specs/M/ARMAutomatedBootstrap Back then the concept was called 'Staged Builds', and used a simple extra line in the control file repeating and modifying the normal build-dependencies: Build-Depends-StageN and builds controlled with DEB_BUILD_OPTIONS=profile=stage1

This made it very easy to implement as existing tools will not barf as no new syntax is used. However it is an ugly implementation inside dpkg with many extra fields defined and a limit (of 2) on how many stages could be supported. No other profiles types are possible without hard-defining more fields inside dpkg. See 661538 for how that developed.

3 successive years of GSOC projects by Gustavo Alkmim, P. J McDermott, Johannes Schauer, and Thibaut Girka, progressed the practical and theoretical work, as well as heavy sponsorship of this work by Linaro and Canonical which enabled a lot of the supporting cross-building and multiarch work to get done.

It became clear from actually doing staged builds that it made more sense if cross-building, test-skipping, doc-skipping and language/tool extra dependencies (bootstrapping) changes were described independently rather than arbitrarily lumped together in the way a bootstrap on a particular arch might like it. So the newer profile modifier syntax was developed and implemented, whilst being tested on actual cross-builds and bootstraps.

Over 2013 an actual spec and implementation in dpkg was pinned down, agreed and implemented, leading to a profile-supporting dpkg upload at the end of 2013.

These are some related activities and documents which have generated input for this one.

Credits

Thanks to Jonathan Austin, Steve McIntyre, Steve Langasek, Loic Minier, P. J. McDermott, and Johannes Schauer for helping clarify the thoughts described above.


CategoryEmdebian | CategoryDebianDevelopment