This page is not written in moin syntax
The plain text of this page is written to be an email to d-devel@l.d.o and thus will not render properly in moin. Please use the plain text view (by appending ?action=raw to the url) to read this page.
Hi,
this is the report for the Bootstrap Sprint [1] which was recently held in Paris, France from 2014-08-16T10:00:00+02 until 2014-08-19:T18:00:00+02. Six people attended the sprint:
- Wookey (cross toolchain, bootstrapping, event organizer)
- Guillem Jover (dpkg maintainer)
- Matthias Klose (gcc, binutils, build-essential, python maintainer)
- Aron Xu (mips64el porter)
- Helmut Grohne (rebootstrap)
- Johannes Schauer (botch)
We feel that our sprint was extremely productive and the next sections will give an overview about what was discussed, achieved and planned for the future.
[1] https://wiki.debian.org/Sprints/2014/BootstrapSprint
0. General notes
Throughout this email we will use the terms "build architecture" to mean the architecture that a package is built on, "host architecture" to mean the architecture a package is built for and "target architecture" to mean the architecture a compiler produces code for.
If not otherwise specified, the term "package" will mean both, "binary package" and "source package". And if not otherwise specified, the term "dependency" means both "build dependency" and "runtime dependency".
The term "Sources file" or "Sources" refers to the source package control file that can be retrieved from mirrors or be found in /var/lib/apt/lists/. The term "Packages file" or "Packages" refers to the binary package control file from the same locations.
The term FTCBFS is analogous to FTBFS and expands to "fails to cross build from source".
1. Problems with early bootstrapping
1.1. Acquiring the list of Essential:yes binary packages
When starting a new port, the host architecture binary packages are not known. The only input is the data from a Sources file. Thus, the set of binary packages which are Essential:yes are not known at this point, because that information is encoded in the Packages file which does not exist for the new architecture yet. It is important to know the set of Essential:yes packages so that the right source packages can be cross compiled for the new architecture to create a native build environment that at the very least includes all Essential:yes packages and build-essential. We decided that using the information of the Priority field for binary packages can be used to solve this problem. Priority:required packages are a small superset of Essential:yes packages and their information is encoded in the Packages-List field in the Sources file. This approach requires that the set of Priority:required continues to be a very conservative and small selection to not make the bootstrap problem harder.
1.2. Dependencies on build-essential are not spelled out by base packages
All binary packages implicitly depend on Essential:yes packages. All source packages implicitly depend on build-essential. This assumption helps reduce the number of needed dependencies in settings where all Essential:yes packages and build-essential are available. This is not the case during the initial bootstrap of an architecture where no packages of the host architecture exist yet. Some of the source and binary packages involved in creating the Essential:yes and build-essential packages must have additional dependencies on packages which are otherwise Essential:yes or build-essential so that cross-build dependencies in an initial bootstrap can be correctly resolved.
1.3. Source packages that build virtual packages are not known
Since the Packages file does not exist in an early bootstrap it is not possible to retrieve the information about which binary packages provide virtual packages. If either packages depend on virtual packages, then it is currently not possible to retrieve the source package that should be built to create the binary package that provides the virtual dependency. Here is a small overview of binary packages that were found to expose this problem in an early bootstrap:
[virtual] -> [real]
libz-dev -> zlib1g-dev
libpng-dev -> libpng12-dev
libgpmg1-dev -> libgpm-dev
libselinux-dev -> libselinux1-dev
libgmp10-dev -> libgmp-dev
libltdl3-dev -> libltdl-dev
libltdl7-dev -> libltdl-dev
libncurses-dev -> libncurses5-dev
libexpat-dev -> libexpat1-dev
libtiff-dev -> libtiff5-dev
We see three solutions to this problem which all carry their own flaws but can be used together to help mitigate each other's disadvantages.
1.3.1. Provides information in the Packages-List field
The information of which virtual packages are provided by a binary package could be encoded in the Packages-List field in a Sources file. This has some restrictions, though:
- the provided package must not vary between architectures (ocaml, haskell)
- the Provides relationship must not be versioned
- the Provides field must not contain substitution variables
1.3.2. Packages should depend on non-virtual packages
If packages depend on virtual packages, they must carry a non-virtual alternative as well to facilitate bootstrapping. We recognize that this solution makes transitions harder which would otherwise just be solved by a binNMU. Thus, this is probably not a solution that can be applied to all packages.
1.3.3. Switching virtual and real package names around
Consider a binary package of the form libfooN-dev which provides libfoo-dev. If the binary package does not bump API on each bump of N then the problem can be solved the following way. Switch the package content to a real binary package libfoo-dev and let it provide libfooN-dev. When there is a N+1 then there can be a real package libfooN-dev while the content of the N+1 goes into libfoo-dev and provides libfooN+1-dev. When the transition is over, the real package libfooN-dev can be dropped.
An example of a package which handles transitions like this even though it is not a shared library is automake.
1.4. Source packages create binary packages of different version
The information for which source package version corresponds to a given binary package resides in the Source field of a Packages file which does not exist in an early bootstrap. Thus, for now, versioned dependencies are ignored during a bootstrap. If a package depends on foo (>= 1.0) then the bootstrapper can only hope that if they build the source package building foo then the produced binary package does actually satisfy the versioned dependency. We have not yet encountered an example in practice where this limitation posed a problem.
1.5. Rebuild for Packages-List field
Where solutions require more information in the Packages-List to be present, the source packages in question must be rebuilt with a new enough dpkg version once all features are implemented in dpkg. So at some point in the future it might be necessary to do a rebuild of some seldom uploaded source packages.
2. Partial archives for different ISAs
TODO: aron
- partial archives for different ISAs?
- mips people want it, otherwise three new arches
- for 64bit ports (mips64/mips64el/mipsn32/mipsn32el), there would be MIPS III, MIPS64, and MIPS64rev2
- dpkg could have a binary package field specifiying HWcapabilities.
- not conflated with dependency resolution - all such packages are equivalent in dependency terms
- apt pinning could be told to examine that field
- dpkg could have a check in --audit to check if installed packages are runnable on current system
- repository layout is a separate issue. Simple version is separate repo, otherwise dak needs to understand to put in database. Could have separate tree, or filename suffix in order to go in shared file hierarchy
- Decide if the baseline (default set of caps) is empty field or shorthand?
3. Cross compilers in main
TODO: wookey, doko, helmut: please expand the notes which I sorted into sections into text. Feel free to reorganize the sections if necessary.
3.1. Cross compiler naming
TODO: wookey, doko, helmut: expand this
- - compiler package naming and compiler binary naming patterns (there are multiple ones atm)?
- using debian arch and triplet
- very confusing but historic reason
- so no changes and leave as is to not break things
- too many things rely on it (also out of the archive)
- there could be Provides of packages with Debian arch
3.2. Build/Host matrix and shared metadata
TODO: wookey, doko, helmut: expand this
- - there should be a limited number of host/build cross compiler combinations
- all cross compilers should work on amd64
and some other selected ones (armhf -> arm64)
3.3. Need for *-cross packages
TODO: wookey, helmut: expand this
- - do we need *-cross packages? (alternative multiarch)
- depends on multiarch cross build dependencies working
- are needed for some architectures
- - needed for architectures that are not in the archive
- bootstrapping
- different sysroot
- mingw64
- - needed for architectures that are not in the archive
3.4. Crossbuild checks
Many binary Multi-Arch:no packages still ship their headers in /usr/include. This can lead to the wrong version being picked up during cross or native building. To avoid this, two checks could be implemented.
The first check could be Lintian warning if a non-Multi-Arch:same package puts headers in /usr/include. If a binary package ships architecture independent headers in /usr/include then it should be marked Multi-Arch:same. If the headers differ between architectures, then they should be moved to /usr/include/<triplet>/.
The multiarch wiki page should be changed to encourage multiarching of *-dev packages.
3.5. Conflict with gcc-multilib
TODO: doko, helmut: expand this
- - cross compilers should conflict with gcc-multilib which provides the asm symlink
- creates an unclean environment otherwise
3.6. Source packages layout
TODO: wookey, helmut: expand this
- - what builds what? - right now: one source package per architecture
- one git repository generates the source packages
3.7. src:gcc-cross-defaults and src:gcc-cross-support
- TODO: wookey, helmut - where to put src:gcc-cross-defaults and src:gcc-cross-support
- src:gcc-defaults should be extended to include cross
- - but native and cross compilers do not build in sync - but new upload is only done once everything is built
- src:gcc-4.9 should be extended to include src:gcc-cross-support
- - helmut uploaded gcc-cross-support to experimental
- src:gcc-defaults should be extended to include cross
3.8. crossbuild-essential
TODO: wookey
- - what is left in crossbuild-essential? - build-essential needs to be
- m-a: same
- depend on gcc-for-host
- then crossbuild-essential can go the way of the dodo
3.9. Cross pkg-config
TODO: helmut
- - preliminary patch to teach pkg-config to bring the triplet symlink to cross-wrapper just by installing the right pkg-config. depends on dpkg fixing dpkg-maintscript-helper dir_to_symlink - where is the bug number with the patch?
3.10. Cross binutils
TODO: wookey
- - fix cross-binutils_0.6
3.11. config.site
TODO: wookey, helmut, doko
config.site currently lives in dpkg-cross for no good reason. It contains autoconf test results for various packages, but it makes more sense to move the test results to the package that is being tested. autoconf will examine /usr/share/config.site by default. It will source some files in /etc (local pre-configuration), then include /usr/lib/config.site/* and /usr/lib/<triplet>/config.site/* and afterwards source more files in /etc (local overrides). Files in the config.site directories are going to be shipped by binary packages and are named after the binary packages. For instance libc6-dev can contain scanf_cv_type_modifier=ms.
- this will be in cross-support
- Need to write corresponding policy
3.12. Coinstallable toolchains
TODO: helmut
- coinstallable toolchain
- - wrote a patch to pass --program-prefix=$(DEB_TARGET_GNU_TYPE)- and reverse symlinks to binutils' single build (preparation for coinstallable toolchains) and sent to doko and doko reviewed it
- - TODO helmut: needs to be rewritten to cover for binutils-multiarch
- - wrote a patch to pass --program-prefix=$(DEB_TARGET_GNU_TYPE)- and reverse symlinks to binutils' single build (preparation for coinstallable toolchains) and sent to doko and doko reviewed it
3.13. Archive support for arch qualifiers
TODO: wookey, helmut
- - is it dak that needs support? - which source packages need build dependencies on foo:arch?
3.14. Multiarch builds
These work nicely on a local machine (with apt-get --only-source build-dep <package>; apt-get --only-source --build -oDpkg::Build-options="-aarm64 -B" source <package>. See MultiarchCrossToolchainBuild for details). But this is no use for in-archive cross-toolchains as network access is not allowed.
Packages to do the build in an archive-compatible way exist at <http://anonscm.debian.org/cgit/crosstoolchain/cross-gcc.git> but to use those in the archive requires tools to understand what to do with cross-arch dependencies.
- We initially thought that a dpkg field to request such builds would be needed,
- but in fact sbuild can just infer it from the fact that a package has cross-arch buildd-deps (e.g libc6-dev:armhf)
- Wookey is working on sbuild support for this, which is straighforward, but it's
- not working yet.
- Other things in the archive will presumably get confused if they see such packages.
- Wanna-build may think they are unbuildable (although if it uses dose it may just get it right?), britney may never migrate them. Possibly other things could fail. It needs testing, which requires an upload to experimental, but we should check with ftp-masters first in order not to cause undue breakage.
- - we need to find out what is left to make these happen. - this needs to be in before jessie if we are to use it in jessie if any packages other
- than sbuild, dak, wanna-build and britney are affected. Needs ftp-master input to determine this.
3.15. Wookey stuff
TODO: wookey: please put the following stuff somewhere in the earlier subsections or create a new section
- wookey
- - cross toolchain feddling - make cross-gcc + cross-gcc-defaults uploadable. - fix that guile doesnt cross build for any arch (bug filed #758463, maintainer happy) - fix cross-binutils to use dpkg-vendor instead of lsb-release and complain if it's null so either get nothing or the right thing and added ppc64. Done, and uploaded as cross-binutils_0.6 - add a 'cross' suffix so that independent updates from binutils are possible.
4. Cross compile support in source packages
4.1. Build dependency translation when cross compiling
If a source package explicitly depends on a versioned build tool like gcc, g++ or cpp, then this build dependency might have to be translated to the appropriate cross compiler when cross compiling the source package. The wiki page at [1] gives an extensive list of how this could be accomplished. All existing solutions had considerable disadvantages so we came up and agreed upon a seventh solution.
[1] https://wiki.debian.org/CrossTranslatableBuildDeps
Consider the following example binary packages (it does not matter from which source package(s) they will build). The left column shows a cross compiler for mips as installed on a amd64 machine. The right column shows them installed on a mips machine but this time it's a cross compiler for amd64.
- amd64 | mips
+
- |
Package: gcc-for-build | Package: gcc-for-build Architecture: amd64 | Architecture: mips Multi-Arch: same | Multi-Arch: same Depends: gcc | Depends: gcc Contents: empty | Contents: empty
- |
Package: gcc-cross | Package: gcc-cross Architecture: mips | Architecture: amd64 Multi-Arch: same | Multi-Arch: same Depends: gcc-mips-linux-gnu | Depends: gcc-x86_64-linux-gnu Contents: empty | Contents: empty
- |
Package: gcc-mips-linux-gnu | Package: gcc-mips-linux-gnu Architecture: amd64 | Architecture: mips Multi-Arch: foreign | Multi-Arch: foreign Contents: /usr/bin/mips-linux-gnu-gcc | Depends: gcc
- |
Package: gcc-x86_64-linux-gnu | Package: gcc-x86_64-linux-gnu Architecture: amd64 | Architecture: mips Multi-Arch: foreign | Multi-Arch: foreign Depends: gcc | Contents: /usr/bin/x86_64-linux-gnu-gcc
With this binary package layout, source packages which need a versioned gcc build dependency can now build depend on gcc-for-build during native building and gcc-cross during cross building using the build profile "cross". Should the source package require execution of the native compiler during a cross build, then the source package can build depend on gcc-for-build in the "cross" profile as well. Packages depending on gcc-cross have to call the triplet version of gcc.
The mechanism works because the packages gcc-for-build and gcc-cross are Multi-Arch:same. Therefore, if they are build depended upon, their host architecture version will be chosen. Depending on its architecture, gcc-cross will depend on a different gcc-<triplet> binary package. The gcc-<triplet> binary packages in turn are Multi-Arch:foreign and either ship a cross compiler or depend on the native gcc depending on their architecture. If gcc-x86_64-linux-gnu:amd64 is installed, then one will get the native gcc compiler on amd64 through its dependency on gcc. If gcc-mips-linux-gnu:amd64 is installed then one will get the cross compiler for mips.
The system works because dependencies on Multi-Arch:foreign packages should be satisfied by the native architecture version of the package. Currently, apt resolves Multi-Arch:foreign dependencies in that way. On the other hand, the dependency of gcc-cross on gcc-mips-linux-gnu can also be satisfied by a foreign architecture package. This package would be the native version of the foreign architecture compiler. It would produce the code of the right architecture but would probably not be executable on the build architecture. Though this is a general problem of Multi-Arch:foreign packages which does not only affect this mechanism.
4.2. help2man issue
Source packages that build depend on help2man cannot be cross compiled because they require a build architecture binary to be executed for generating the man page. There exist different solutions to the problem:
- - one could build twice, first native then cross and run help2man on the
- native version of the binary
- so that the source package could build depend on the exact same version of the binary package it builds to generate the man page from it. This is filed as bug#751437
4.3 libtool
Build dependencies on libtool make cross build dependencies of a quarter (83) of the source packages in the transitively build-essential set unsatisfiable because the build as well as the host architecture version of the libtool binary package would be required but libtool cannot be co-installed and is not Multi-Arch:foreign either.
This bug was filed as bug#682045 and was taken care of by Matthias Klose. The libtool binary was split into libtool and libtool bin and an upload was done in accordance with the libtool maintainer. The next step is to do an archive rebuild with libtool not depending on libtool-bin, fixing packages that fail to compile and then dropping the build dependency.
5. Bootstrap and Crossbuild Quality Assurance
5.1. dose-builddebcheck for cross
dose-debcheck is run regularly for all binary packages at [1]. Ralf Treinen was contacted and agreed that dose-builddebcheck could be run in addition as well. We have to contact him and ask about the possibility to also include running dose-builddebcheck with a foreign host architecture for selected targets to spot FTCBFS problems early. Such a service would enhance the results that are already generated at [2].
[1] https://qa.debian.org/dose/debcheck/ [2] http://bootstrap.debian.net/cross.html
5.2. Archive Cross Rebuilds
Just as full archive rebuilds are regularly done natively, they could also be done regularly (for example on Amazon AWS). It should not be hard to implement such a service as sbuild has cross compilation support. The results of those rebuilds could be displayed on the pts or the new tracker platform. This item needs cross compilers in the main archive. It would be even better to schedule a crossbuild on every package upload to quickly identify problems right after they are introduced.
5.3. Cross-Builds:no field
When starting to cross build the archive regularly, then there should be a method mark source packages as not-cross buildable. Packages would be marked as such if their build system does not allow any cross compilation. This is for example the case if there exists no cross compiler (example: gobject-introspection). We already store a similar information in the Architecture field where we restrict the architecture to certain values if the software project can only be built for a subset of architectures. Having a Cross-Builds:yes field would invite trouble as it is not clear for which combination of build/host architecture the source package is supposed to cross compile. A Cross-Builds:no field would mark the source package as not cross compilable at all. If such a field existed, then an archive auto cross builder would not attempt to cross build those source packages and would also not warn in the pts or tracker about the failure. Until such a field is agreed upon, the information can also be stored local to the cross rebuild machinery or stored in a custom XS-Cross-Builds:no field. Another good heuristic might be to check the build dependencies of source packages before build. If they build depend on, for example gobject-introspection binary packages, then it is unlikely that they can be cross compiled. This heuristic would avoid having to flag tons of packages but would also not be always correct.
5.4. Bootstrap info in pts/tracker
The pts/tracker could show info about hard dependency cycles (self-cycles). There exist two patches for the pts (#745618 and #728298) which have to be rewritten for the new tracker. Additionally, the information should only be displayed once source packages with the new build profile syntax can be uploaded to the archive. Thus, any such info can only be available once Jessie is released. In addition it would be a good longterm goal to move the machinery behind bootstrap.debian.net onto Debian infrastructure to be more robust for such a QA task.
5.5. Display multiarch hints in pts/tracker
Downstream distributions of Debian like Ubuntu could be used to determine which packages in Debian lack multiarch annotations. Especially Ubuntu is useful as a source as multiarch cross building has been tested there for a while and thus more packages have been fixed.
5.6. Require an explicit multiarch value
Lintian could warn if a binary package was not explicitly marked with a multiarch value. The reason would be that this would force maintainers to make a clear decision about the multiarch status of their package instead of defaulting to Multi-Arch:no. The likely outcome though, would be that maintainers would tag their package Multi-Arch:no just to get rid of the Lintian warning instead of investing time to properly multiarch their package.
6. Build Profiles
6.1. New evaluation logic
The original build profile proposal defined an evaluation scheme of the restriction list which was very hard to understand, contained corner cases that would probably never be used and also limited expressiveness a lot. We agreed that if this new syntax is introduced it should allow for maximum expressiveness and be intuitive and easy to understand at the same time. We decided that letting profile restrictions form a disjunctive normal form would satisfy both requirements. The new evaluation scheme allows to specify more conditions and is at the same time easier to understand and implement.
Here an example:
- Build-Depends:
foo (>= 1.0) <stage1 cross> bar [amd64] <stage1> <cross>
The source package will depend on foo (>= 1.0) if both, stage1 and cross are active as a profile. The source package will depend on bar on amd64 if either stage1 or cross are active. Terms within <> "brackets" are AND-ed and form a conjunction. Multiple of those groups are OR-ed and form a conjunction. This also means that in contrast to the original proposal, the order of build profiles does not matter. With this solution, build profiles gain the same amount of expressiveness as Gentoo USE flags.
As a result of this change, we will also drop our efforts to make the archive understand the build proflie syntax before the Jessie release. This also let to a reassignment of bug#744246 to dpkg.
Patches for dpkg and apt to support this change have been written and are currently being tested.
6.2. No "profile" prefix
We decided to drop the profile prefix from the restriction syntax. The "profile" prefix was included in the original proposal to allow for future extensions of the syntax. But we cannot see how the build profile syntax as a selector of build dependencies can be extended for other use cases. All future additions that we came up with and that were suggested on discussions on debian-devel only imply a tagging of build dependencies but not their selection. But a tagging mechanism, if ever introduced, would not need to make use of the <> brackets as a meta character. With the current meaning of the syntax, any other prefix namespace can only ever take the role of a dependency selector and would thus fit in the "profile" namespace. Proponents of the need for "future extensibility" of the syntax were at no point able to give as a single concrete example for such a use case. We therefore think that having a prefix mechanism would noly introduce unnecessary complexity in the implementation and would make the restriction logic harder to read, write and understand for no conceivable practical gain.
6.3. Binary package contents
Binary packages built with a build profile must contain the exact same content as when they are built without a build profile (a full build). This requirement is necessary to not break the dependency system. Anybody depending on a binary package mast continue to be able to make the assumption that it ships a certain content, no matter how the binary package was built. If binary package contents change significantly due to a build with fewer dependencies and thus less features, then the binary package has to be split into two or more individual binary packages.
This requirement is relaxed for builds with the "stage1", "stage2" and "nodoc" profile. With these profiles, binary packages are allowed to leave out content which does not expose a functional interface. Such content would be documentation like man pages or locale information.
6.4. nodoc build profile
In addition to the existing build profiles "stage1", "stage2", "nocheck", "nobiarch" and "cross" we add an additional profile called "nodoc". Its purpose is to do builds without the build dependencies required for building documentation. There is no policy defined "nodoc" value for DEB_BUILD_OPTIONS. Current source packages use "nodoc" as well as "nodocs". We decided to go for the singular as "nocheck" is singular as well. In addition, *-doc packages have a "doc" prefix and not a "docs" prefix. We suggest to add "nodoc" as a DEB_BUILD_OPTIONS to policy.
6.5. Relationship to DEB_BUILD_OPTIONS
Some DEB_BUILD_OPTIONS like "nocheck" and "nodoc" are mapped to build profile names. Build profiles could take over the job of these DEB_BUILD_OPTIONS in the long term. For now, DEB_BUILD_OPTIONS and DEB_BUILD_PROFILES will coexist.
6.6. Profile name application
Especially the "stage1" and "stage2" build profiles with the meaning of "build the source package with less features" overlap with other profile names like "nocheck" and "nodoc". If in doubt, build dependencies should be flagged with the more specialized build profile. So instead of writing:
Build-Depends: texlive-latex-base <!nodoc> <!stage1> <!stage2>
It should be enough to write:
Build-Depends: texlive-latex-base <!nodoc>
In the same way, instead of writing:
- Package: foobar-doc
Build-Profiles: <!nodoc> <!stage1> <!stage2>
It should be enough to write:
- Package: foobar-doc
Build-Profiles: <!nodoc>
It is the task of the builder to figure out which profiles to activate to perform a build.
7. Rebootstrap
TODO: helmut
- rebootstrap achievements
- libgcrypt11 builds when setting ac_cv_sys_symbol_underscore=no
- multiarch guile-2.0 shall be uploaded soon (according to wookey)
- failing gcc stage2 multilib build (doesn't find crti.o). TODO helmut: try with an unstaged build
- explain jenkins problem
- maybe run rebootstrap on its own machine
8. Botch
8.1. Bootstrap from zero
Botch integrated a new script from Helmut Grohne which allows to create a build order without having access to the host architecture Packages file which does not exist at this point. Botch did not yet include such a facility because it catered for two other use cases: calculating a build order for rebootstrapping existing architectures and calculating and analyzing a dependency graph to find the best points to break it (finding a feedback arc set). With the new script botch will soon be able to also handle the case where not even the information about the host architecture binary packages exists in the beginning. This will then also allow botch to be used to schedule transitions as the resulting binary packages are not available in that scenario either. The challenges of bootstrapping from nothing have been explained in the first section.
8.2. Miscellaneous additions fixes
- - Man pages for all 43 commands that the botch binary package ships have been
- written
- output of wanna-build
9. Miscellaneous
9.1. Find outdated versions
Versioned dependencies are problematic for bootstrapping because versioned compiler dependencies have to be translated and the versions of binary packages is not known a priori during a bootstrap from zero. Many packages in the archive declare versioned dependencies with a minimum version which is even fulfilled by oldstable. We propose that lintian warns if a version constraint is obsolete because it would even be fulfilled by oldstable. This was filed as bug#758425 against Lintian.
9.2. Dpkg can set the target architecture
To have a common and unified interface, it was agreed with Guillem Jover that dpkg-buildpackage should have a --target-architecture switch added which sets the target architecture just as -a sets the host architecture. Perhaps the variables might only be output if the switch is passed, to avoid further confusing maintainers that do not usually need those variables.
9.3. Fixed bugs
TODO: all
The following bugs have been fixed during the sprint in addition to the ones mentioned earlier:
- - bug#758408 - bug#750478 - bug#746523
9.4. Filed bugs
TODO: all
The following bugs have been filed during the sprint in addition to the ones mentioned earlier:
- - bug#758463
9.5. Fix dpkg-maintscript-helper
pkg-config required a dir-to-symlink switch (TODO: helmut, expand), and to comply with policy the symlink target pathname should be relative as specified by the Debian policy. But dpkg-maintscript-helper was not handling those correctly. Helmut prepared a patch to fix the dir_to_symlink command, and Guillem merged and added a check to the symlink_to_dir command so that it ensured only absolute paths are being passed on. These were included in the dpkg 1.17.13 upload.
9.6. Arch:all sometimes implies M-A:Foreign for dpkg-checkbuilddeps
TODO: helmut, guillem Once you have a bug filed for this you can add it to section 8.4. and delete this section
10. Epilogue
We would like to thank Roberto di Cosmo for kindly hosting the Bootstrap Sprint at IRILL in Paris, France. We'd also like to thank Debian and it's donors who covered our travel expenses and without who this sprint could not have happened.
+
+ | The original sprint notes start here | +
+
- - Looked at script for rebootstrapping into main archive from ports.
- botch-main-archive-bootstrap-what-next - problem: only considers one source package at a time - needs global view (botch)
- Useful for understanding what to upload next to unblock buildds.
- multi-line output as well as build-needed single-line style.
Used for: https://people.debian.org/~wookey/bootstrap/blocked-deps-list
- using debian arch and triplet
- very confusing but historic reason
- so no changes and leave as is to not break things
- too many things rely on it (also out of the archive)
- there could be Provides of packages with Debian arch
- depends on multiarch cross build dependencies working
- are needed for some architectures
- - needed for architectures that are not in the archive
- bootstrapping
- different sysroot
- mingw64
- - needed for architectures that are not in the archive
- all cross compilers should work on amd64
and some other selected ones (armhf -> arm64)
- - two checks would be nice:
- before crossing, a script should check if /usr/include contains files from packages that are not m-a:same
lintian should warn if a non m-a:same package puts headers in /usr/include (if a package shares files in /usr/include then it should be m-a:same and if it is not m-a:same then those headers should be moved to /usr/include/<triplet>/)
- this warns when cross builds pick up header files installed for the build system
- this warns when native builds pick up header files installed for the host architecture
- multiarch wiki should be changed to encourage multiarching *-dev packages
- TODO: wookey fixes this
- creates an unclean environment otherwise
- one git repository generates the source packages
- src:gcc-defaults should be extended to include cross
- - but native and cross compilers do not build in sync - but new upload is only done once everything is built
- src:gcc-4.9 should be extended to include src:gcc-cross-support
- - helmut uploaded gcc-cross-support to experimental
- Discuss toolchain dependency translation (e.g. gcc-${host})
see also: CrossTranslatableBuildDeps and 695287
affected packages: http://bootstrap.debian.net/cross.html in the "conflict" table
- solved by src:gcc-cross-support
- - build-essential needs to be
- m-a: same
- depend on gcc-for-host
- then crossbuild-essential can go the way of the dodo
- - when building build-essential, build-essential is not available - packages which are transitively build-essential should spell out dependencies - can be put in policy
- example: glibc has different library names for different archs
- also versions cannot be predicted
- either: essential packages must list non-virtual dependencies as first alternatives in depends AND build-depends
- - but breaks transitions
- or: extend Packages-List field with Provides information
- - must not be architecture independent (haskell, ocaml) - must not be versioned - must not contain substvars curly braces - must not contain arch-restrict. bracket
- TODO: bring it up with affected maintainers (tiff, png, jpeg)
- Problems with virtual packages within essential:
- When Build-Depends or Depends list virtual packages, it is non-obvious to figure out which source package needs to be build for satifying such a dependency, because in the bootstrap case, binary packages for the host architecture are missing. Proposed exclusive solutions, which have to be implemented for every package the same way:
- Require transitively essential packages to list real packages in all of their Depends and Build-Depends as first alternatives. For example, this means depending on zlib1g-dev instead of libz-dev or shipping a real libz-dev meta package.
- Require transitively essential packages to not vary their Provides among different architectures (Haskell and Ocaml do this) and have them list Provides in the Package-List field of their .dsc files. In this case, dpkg needs to allow for packages to add new entries to Package-List (in .dsc) and src:zlib1g would need to append provides=libz-dev to the Package-List entry for bin:zlib1g-dev.
- Transitively build essential packages affected are:
[virtual] -> [real]
libz-dev -> zlib1g-dev
libpng-dev -> libpng12-dev
libgpmg1-dev -> libgpm-dev
libselinux-dev -> libselinux1-dev
libgmp10-dev -> libgmp-dev
libltdl3-dev -> libltdl-dev
libltdl7-dev -> libltdl-dev
libncurses-dev -> libncurses5-dev
libexpat-dev -> libexpat1-dev
libtiff-dev -> libtiff5-dev
- When Build-Depends or Depends list virtual packages, it is non-obvious to figure out which source package needs to be build for satifying such a dependency, because in the bootstrap case, binary packages for the host architecture are missing. Proposed exclusive solutions, which have to be implemented for every package the same way:
- standard way of defining target architecture in as environment variable
- dpkg-buildpackage --target-architecture can set that
- only outputting variables if option is specified
2014-08-17
config.site currently lives in dpkg-cross for no good reason. It contains autoconf test results for various packages, but it makes more sense to move the test results to the package that is being tested. autoconf will examine /usr/share/config.site by default. It will source some files in /etc (local pre-configuration), then include /usr/lib/config.site/* and /usr/lib/<triplet>/config.site/* and afterwards source more files in /etc (local overrides). Files in the config.site directories are going to be shipped by binary packages and are named after the binary packages. For instance libc6-dev can contain scanf_cv_type_modifier=ms.
- this will be in cross-support
- Need to write corresponding policy
* josch
- - write documentation for botch - integrating tobuild.py - make it python3 - find old versions in lintian - write patches for dpkg and apt for new build profile support
- doko
- - libtool split (as agreed with maintainer)
- uploaded
- next:
- archive test rebuild with libtool not depending on libtool-bin
- fix failing packages
- actually drop the dependency
- - libtool split (as agreed with maintainer)
- helmut
- - preliminary patch to teach pkg-config to bring the triplet symlink to cross-wrapper just by installing the right pkg-config. depends on dpkg fixing dpkg-maintscript-helper dir_to_symlink
- TODO: helmut (file bug with patch and put bugnumber)
- - preliminary patch to teach pkg-config to bring the triplet symlink to cross-wrapper just by installing the right pkg-config. depends on dpkg fixing dpkg-maintscript-helper dir_to_symlink
- coinstallable toolchain
- - wrote a patch to pass --program-prefix=$(DEB_TARGET_GNU_TYPE)- and reverse symlinks to binutils' single build (preparation for coinstallable toolchains) and sent to doko and doko reviewed it
- - TODO helmut: needs to be rewritten to cover for binutils-multiarch
- - wrote a patch to pass --program-prefix=$(DEB_TARGET_GNU_TYPE)- and reverse symlinks to binutils' single build (preparation for coinstallable toolchains) and sent to doko and doko reviewed it
- wookey
- - cross toolchain feddling - make cross-gcc + cross-gcc-defaults uploadable. - fix that guile doesnt cross build for any arch (bug filed #758463, maintainer happy) - fix cross-binutils to use dpkg-vendor instead of lsb-release and complain if it's null so either get nothing or the right thing and added ppc64. Done, and uploaded as cross-binutils_0.6 - add a 'cross' suffix so that independent updates from binutils are possible.
- aron
- - fixed gmp symbols #750478 #746523
2014-08-18
- helmut
- - write preliminary patch fixing dpkg-maintscript-helper dir_to_symlink to support relative targets (needed for pkg-config package split) and sent to guillem and locally committed
- propose a new solution for CrossTranslatableBuildDeps and write a rough implementation
- - write preliminary patch fixing dpkg-maintscript-helper dir_to_symlink to support relative targets (needed for pkg-config package split) and sent to guillem and locally committed
What do we want to have in Jessie
- splitting libtool? (into libtool-bin etc) 682045
- archive rebuild required to find bugs
- libtool architecture dependent, libtoolize (used by autoconf) is arch indep
- Multiarch builds
- Specify additional arches in header?
- - but information can be extracted from Build-Depends instead
- Anything else needed? dak? dpkg?
- - they don't care if no new field
- wanna-build, britney, dak need to understand this
- whole bootstrap in one package, outputs glibc packages
- - TODO: doko: revisit the build of the libgcc1 package
- cross-binutils are separate
- needs fixing for Jessie
- Specify additional arches in header?
Agenda
- - a machine which tests crossing packages to see if they cross compile or not
- full archive cross builds
- display results in the pts
- current amazon based solution takes too long between upload and check
How to ensure that build profiles work in Jessie? See 744246
- Push into archive? (trying atm)
- Set up copy of archive for testing? (painful)
- we are just gonna wait for jessie
- Other cross-build pieces
- What should be in cross-build-essential (or finer-grained mechanism?)
- nothing - it turns out.
- autofoo cache data (, cmake toolchain files etc. Move out of dpkg-cross? yes.
- Shared metadata for Host/Target matrix
- good idea to have that but not important
- also distribution specific
- What should be in cross-build-essential (or finer-grained mechanism?)
- rebootstrap achievements
- libgcrypt11 builds when setting ac_cv_sys_symbol_underscore=no
- multiarch guile-2.0 shall be uploaded soon (according to wookey)
- failing gcc stage2 multilib build (doesn't find crti.o). TODO helmut: try with an unstaged build
discuss need for Cross-Builds: no field (or how to otherwise store that information)
- it would be useful to know about package that can never be cross compiled because there does not exist a cross compiler (for example for gobject-introspection) or because of cases like 753047 - we can use an X-* header - we can store the knowledge in build tools
- Crossbuild QA feedback in PTS
- needs crosstoolchains
- libtool bug
- missing multiarch by comparing with Ubuntu Multiarched-or-Not feedback in PTS
- simple static list of whether library is multiarched on not - put in PTS
- - should all binary packages should have a multi-arch value? - explicitly set m-a: no? - people might just put m-a:no and be done with it
discuss how intersecting build profiles should be applied. Either doc-generator <!profile.nodoc>, testsuite-runner <!profile.notest> or doc-generator <!profile.nodoc> <!profile.stage1>, testsuite-runner <!profile.notest> <!profile.stage1>
- - individual profiles (nocheck, nodoc) are preferred over making everything stage1
- discuss whether more build profiles should be introduced, how about mapping some DEB_BUILD_OPTIONS to build profiles? (nodoc, nocheck, ...)
- - yes: nodoc
- should some build profiles replace DEB_BUILD_OPTIONS?
- - coexist
discuss status of ReleaseGoals/CrossBuildableBase what is missing? Where are blockers? How to identify the set that must be cross-able for a minimal base system?
- bootstrap info at qa.debian.org
example: https://packages.qa.debian.org/~pabs/g/ghc.html https://packages.qa.debian.org/~pabs/a/avahi.html
cross: 745618 (problem: it must be manually decided in which package the bug is)
cycles: 728298 (problem: all but self-cycles are not hard requirements)
- TODO: josch change patches for new tracker
- TODO: helmut file a bug
Guillem topics
- discuss handling of mixed positive and negative restrictions
- make similar to gentoo use flags? yes, disjunctive normal-form
<!profile.stage1 !profile.nobiarch> <!profile.nocheck>
=> means that dependency applies if ((stage1 AND nobiarch) OR nocheck )
- what should happen when an unknown namespace is encountered? unknown namespaces are interpreted as false
- the current way is complicated and it is hard to see how this makes future extensions possible (solved by disjunctive normal-form)
- super tiny issue of Build-Profiles field in binary package stanzas and Packages-List representation (whoever changes that gets to fix the bootstrap patches)
- make similar to gentoo use flags? yes, disjunctive normal-form
- discuss the requirement that binary packages built with a build profile must be equal in functionality to those normally built (or be split or renamed) and how to enforce this (reproducible builds?)
- when profile stage1 is active (with or without other profiles active at the same time ):
- only things that expose an interface must remain the same
- man pages and docs should be safely removable
- for other profiles like cross, the created packages should contain everything
- when profile stage1 is active (with or without other profiles active at the same time ):
- help2man is used during build by coreutils, bison, flex. all of them therefore fail to build
discuss substitution variables in Build-Depends 751437 -> just use profile.nodoc
- Guillem's 'fake' cross-toolchains idea
- wookey will reuse the ideas for the cross-toolchains pending to be uploaded
- partial archives for different ISAs?
- mips people want it, otherwise three new arches
- for 64bit ports (mips64/mips64el/mipsn32/mipsn32el), there would be MIPS III, MIPS64, and MIPS64rev2
- dpkg could have a binary package field specifiying HWcapabilities.
- not conflated with dependency resolution - all such packages are equivalent in dependency terms
- apt pinning could be told to examine that field
- dpkg could have a check in --audit to check if installed packages are runnable on current system
- repository layout is a separate issue. Simple version is separate repo, otherwise dak needs to understand to put in database. Could have separate tree, or filename suffix in order to go in shared file hierarchy
- Decide if the baseline (default set of caps) is empty field or shorthand?
- stopped tech-ctte bug
- Multi-Arch interpreter