Differences between revisions 1 and 47 (spanning 46 versions)
Revision 1 as of 2013-08-12 16:30:41
Size: 2940
Editor: josch
Comment: initial version
Revision 47 as of 2014-06-17 21:03:54
Size: 11047
Editor: josch
Comment: add libconfig-model-dpkg-perl
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== Document status ==

Preliminary support of this spec was implemented in

|| '''package''' || '''version''' || '''bug''' || '''vcs commit''' ||
|| dpkg || 1.17.2 || DebianBug:661538 || [[http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commit;h=7662e0937bb064a0754d12605d80a96a17e2aadf|git:7662e093]] [[http://anonscm.debian.org/gitweb/?p=dpkg/dpkg.git;a=commit;h=c87941de954281a8aa66d032c85657f64bfe4dce|git:c87941de]]||
|| sbuild || || DebianBug:731798 || ||
|| apt || 0.9.16.1 || DebianBug:661537 || [[http://anonscm.debian.org/gitweb/?p=apt/apt.git;a=commit;h=565ded7b65240b25ad8551789ac388c8ce72b1f4|git:565ded7b]] [[http://anonscm.debian.org/gitweb/?p=apt/apt.git;a=commit;h=ce7f128c020e1347f91c6074238fc5da58c5df71|git:ce7f128c]] ||
|| python-apt || 0.9.3.6 || DebianBug:744243 || [[http://anonscm.debian.org/gitweb/?p=apt/python-apt.git;a=commit;h=c793c367d32d5a5adc220268ac4a0144419ac4d0|git:c793c367]]||
|| debhelper || 9.20140227 || || [[http://anonscm.debian.org/gitweb/?p=debhelper/debhelper.git;a=commit;h=f16d0915b10ff9ef0b0826ae896aaccd6f3ad84f|git:f16d0915b]] ||
|| pbuilder || || DebianBug:740577 || ||
|| lintian || 2.5.22 || DebianBug:740607 || [[http://anonscm.debian.org/gitweb/?p=lintian/lintian.git;a=commit;h=1f8821b8b62205aa36eadbd799faad443cfa8b85|git:1f8821b8]] ||
|| dose3 || 3.1 (partial) || || ||
|| wanna-build || || || ||
|| dak || || DebianBug:744246 || ||
|| devscripts (mk-build-deps) |||||| Uses libdpkg-perl so supports build profiles automatically ||
|| python-debian || || || ||
|| libconfig-model-dpkg-perl || || || ||
Line 3: Line 22:
For some compilation scenarios it is required to build-depend on a different set of binary packages than specified in the Build-Depends line. The two most important scenarios are: For some compilation scenarios it is required to build-depend on a different
set of binary packages than specified in the Build-Depends line. The two most
important scenarios are:
Line 5: Line 26:
 * Bootstrapping (for breaking build dependency cycles) and  * [[https://wiki.debian.org/DebianBootstrap|Bootstrapping]] (for breaking build dependency cycles) and
Line 8: Line 29:
We propose an extension of the Build-Depends field syntax. This extension allows to mark build dependencies as being needed or not needed when a specific profile is activated. We also propose the introduction of a new field called "Profile" which aids in marking binary packages as being built or not built while a certain profile is activated or having been built with a certain set of profiles activated. This specification describes two possible extensions of the Build-Depends field
syntax. These extensions allow the marking of build dependencies as being needed
or not needed when a specific build profile is activated. It also defines a new
field called "Build-Profiles" which aids in marking binary packages as being
built or not built whilst a certain build profile is activated or having been
built with a certain set of build profiles activated.

Build profiles can be activated by setting the environment variable
`DEB_BUILD_PROFILES` or by using the -P option with dpkg-buildpackage (or -o
Apt::Build-Profiles for apt, or --profiles in sbuild). More than one build
profile can be activated at a time. Multiple profiles are specified by
separating them with commas in commandline arguments and by separating them
with spaces in the `DEB_BUILD_PROFILES` environment variable. The initial
profile names are `"stage1"`, `"stage2"`, `"nocheck"`, `"nobiarch"` and
`"cross"`. Other possibilities are `"nodoc"` or `"embedded"`.
Line 12: Line 47:
We propose a syntax which in practice looks as follows: An example demonstrating the build profile syntax:
Line 15: Line 50:
Build-Depends: foo (>= 1.0) [i386 arm] [!profile.stage1], bar Build-Depends: foo (>= 1.0) [i386 arm] <!profile.stage1 !profile.cross>, bar
Line 18: Line 53:
Our proposal extends the architecture restriction syntax in square brackets from a single disjunctive list of architectures to a conjunctive list of logical disjunctions (a conjunctive normal form expression). Each clause of the conjunction is enclosed in square brackets. Every literal inside square brackets is a logical disjunction. The above example would therefore make the source package build depend on foo if the host architecture is either i386 or amd64 and if the profile named "stage1" is not active. This specification introduces a new pair of brackets (using `<` as the opening
and `>` as the closing bracket) to be used after the architecture qualification
list. Just as in the architecture qualification list the `<` and `>` brackets
enclose a space separated list of terms called a restriction list.
Line 20: Line 58:
Every literal in a disjunction follows the following regular expression: Every term in the restriction list follows the following regular expression:
Line 23: Line 61:
([a-z][a-z0-9]*)\.([a-z][a-z0-9]*) ([a-z][a-z0-9-]*)\.([a-z][a-z0-9-]*)
Line 26: Line 64:
The first group is called a namespace and we propose the namespace "profile" to support build profiles. The second group is called a label and for the "profile" namespace we propose as initial labels "stage1", "stage2" and "cross". The first group is called a namespace. The namespace `"profile"` is used to
support build profiles and is the only allowed namespace for now. By
introducing additional namespaces, this syntax can later be extended for other
uses besides build profiles (for example an `"arch"` namespace can be
introduced). No other namespace besides "profile" exists yet but it is
forbidden to mix different namespaces within one restriction list. The second
group is called a label and specifies the profile name.
Line 28: Line 72:
The other already existing namespace would be "arch" for architectures. For backwards compatibility it is allowed to specify architecture labels without the "arch" namespace in front. The following is therefore equivalent: Terms can be negated by using an exclamation mark as a prefix. In contrast to
the architecture qualification list, positive and negative terms are allowed to
be mixed. The semantics of a restriction list are computed as follows:

 * one or more profiles can be activated at the same time (by commandline argument or environment variable)
 * for each dependency, the restriction list is processed from left to right
 * if a negated term is encountered and the specified profile is set, the build dependency is dropped and processing of the list stops
 * if a positive term is encountered and the specified profile is set, then the build dependency is kept and processing of the list stops
 * if no profile is set for any term in the restriction list and at least one term in the restriction list is negated then keep the build dependency, otherwise drop the build dependency

Above rules express the same semantics as architecture restrictions if positive
and negative terms are not mixed. If positive and negative terms are mixed (and
only then) then the order of the terms in the restriction list matters. The
following table illustrates the implication of the above rules. Each cell
indicates whether or not the dependency foo is dropped with a certain value of
DEB_BUILD_PROFILES.

|| || `""` || `"stage1"` || `"nocheck"` || `"stage1 nocheck"` ||
|| `foo <!profile.stage1>` ||<#00FF00> keep ||<#FF0000> drop ||<#00FF00> keep ||<#FF0000> drop ||
|| `foo <profile.stage1>` ||<#FF0000> drop ||<#00FF00> keep ||<#FF0000> drop ||<#00FF00> keep ||
|| `foo <!profile.stage1 !profile.nocheck>` ||<#00FF00> keep ||<#FF0000> drop ||<#FF0000> drop ||<#FF0000> drop ||
|| `foo <profile.stage1 profile.nocheck>` ||<#FF0000> drop ||<#00FF00> keep ||<#00FF00> keep ||<#00FF00> keep ||
|| `foo <!profile.stage1 profile.nocheck>` ||<#00FF00> keep ||<#FF0000> drop ||<#00FF00> keep ||<#FF0000> drop ||
|| `foo <profile.nocheck !profile.stage1>` ||<#00FF00> keep ||<#00FF00> keep ||<#FF0000> drop ||<#00FF00> keep ||

=== Future extensions ===

 * The syntax could be extended to allow more than one <> block
 * It could also be allowed to allow architecture names under the "arch" namespace within a <> block.
 * It could be allowed to mix different namespaces within a <> block

== The Build-Profiles field ==

In debian/control binary package stanzas, the content of the Build-Profiles
field specifies the (unordered) list of build profiles for which that binary
package does or does not build. This list can either be all positive or all
negative. Entries are negated by using an exclamation mark as a prefix.
Line 31: Line 111:
Build-Depends: foo [i386 arm64]
Build-Depends: foo [arch.i385 arch.arm64]
Build-Profiles: !cross !stage1
Line 35: Line 114:
These literals can be negated by using an exclamation mark as a prefix. We propose that literals inside a disjunction can individually be negated or not. To provide backwards compatibility, a disjunction which only contains architecture lables without their "arch" namespace must either be all negated or all positive. The following statements would therefore be legal: If a binary package stanza in a debian/control file does not contain a
Build-Profiles field, then it implicitly means that it builds with all build
profiles.

== The Built-For-Profiles field ==

In *.changes and Packages files, the content of the Built-For-Profiles field
specifies the list of build profiles for which that binary or source package
was built.

== The DEB_BUILD_PROFILES environment variable ==

The DEB_BUILD_PROFILES environment variable contains a space separated
unordered list of activated profiles. The content of this variable must be
honored by all tools involved in package compilation. Here an example for
debian/rules (enabling sql for any build except stage1 profile - notice the
negated test):
Line 38: Line 133:
Build-Depends: foo [profile.stage1 !profile.cross]
}}}
        
But the following would be illegal as it was before:

{{{
Build-Depends: foo [i386 !amd64]
}}}
        
We propose that literals of different namespaces can be mixed within a disjunction. Therefore, the following would be legal:

{{{
Build-Depends foo [arch.i386 !profile.cross]
ifneq ($(filter stage1,$(DEB_BUILD_PROFILES)),)
    DH_OPTIONS += -Nlibdb5.1-sql # not needed with debhelper (>= 9.20140227)
    CONFIGURE_SWITCHES += --disable-sql
else
    CONFIGURE_SWITCHES += --enable-sql
endif
Line 53: Line 141:
== The Profile field == == Profile built binary packages ==

A binary package must offer the exact same functionality for all profiles with
which it builds including no activated profile at all (if it builds in that
case). Otherwise a package depending on that binary package might not find the
functionality it expects it to provide. This means that if necessary binary
packages have to be split or that a source package has to be built in two
stages. This requirement can not technically be checked before
ReproducibleBuilds happen.

=== Example ===

Imagine `src:foo` depends on `bar` which is needed to build a plugin that is
shipped together with other plugins in the package `foo-plugins`. When `bar` is
made optional as a build dependency of `src:foo` in a `stage1` build, then the
resulting `stage1` binary package `foo-plugins` would differ from the
`foo-plugins` package that is created during a full build. Thus the creation of
the `foo-plugins` binary package must be disabled by using the `Build-Profiles`
field in the `foo-plugins` binary package stanza and by adding the necessary
bits to `debian/rules` for packages without debhelper or when build depending
on debhelper (>> 9.20140227) is not desired.

If `foo-plugins` has reverse dependencies which are part of a cycle with `src:foo` itself, then not building `foo-plugins` would not solve the bootstrap problem. In that case it is necessary to either:

 * split the `foo-plugins` package into `foo-plugins-nobar` and `foo-plugins-bar` and adjust *all* reverse dependencies of `foo-plugins` to correctly depend on `foo-plugins-nobar` and `foo-plugins-bar`
 * or create a package `foo-plugins-nobar` during a `stage1` build and modify only the reverse dependencies that are in a cycle with `src:foo` to build depend on `foo-plugins-nobar` in their own `stage1` build. When `src:foo` is built fully, it will not produce `foo-plugins-nobar` anymore but the full `foo-plugins` package instead.

== Other uses for the restriction syntax ==

=== Mark build dependencies that need to be translated when cross building ===

See CrossTranslatableBuildDeps#A3._Use_restriction_syntax_with_new_namespace

=== Replace Build-Depends-Indep and Build-Depends-Arch ===

Instead of moving build dependencies needed to build arch:all or arch:any
packages into `Build-Depends-Indep` and `Build-Depends-Arch`, respectively,
mark them directly in the `Build-Depends` field using `<debbuildopt.archindep>`
and `<debbuildopt.archdep>`, respectively.

=== Map DEB_BUILD_OPTIONS to namespaces ===

Some `DEB_BUILD_OPTIONS` like `nocheck` or `noopt` could be mapped to
`<debbuildopt.nocheck>` or `<debbuildopt.noopt>`. This would allow not needing
to install certain dependencies if certain `DEB_BUILD_OPTIONS` are active.

== Discussion ==

Document status

Preliminary support of this spec was implemented in

package

version

bug

vcs commit

dpkg

1.17.2

661538

git:7662e093 git:c87941de

sbuild

731798

apt

0.9.16.1

661537

git:565ded7b git:ce7f128c

python-apt

0.9.3.6

744243

git:c793c367

debhelper

9.20140227

git:f16d0915b

pbuilder

740577

lintian

2.5.22

740607

git:1f8821b8

dose3

3.1 (partial)

wanna-build

dak

744246

devscripts (mk-build-deps)

Uses libdpkg-perl so supports build profiles automatically

python-debian

libconfig-model-dpkg-perl

Problem description

For some compilation scenarios it is required to build-depend on a different set of binary packages than specified in the Build-Depends line. The two most important scenarios are:

  • Bootstrapping (for breaking build dependency cycles) and

  • Cross building (for source packages having different build dependencies during cross building than during native building)

This specification describes two possible extensions of the Build-Depends field syntax. These extensions allow the marking of build dependencies as being needed or not needed when a specific build profile is activated. It also defines a new field called "Build-Profiles" which aids in marking binary packages as being built or not built whilst a certain build profile is activated or having been built with a certain set of build profiles activated.

Build profiles can be activated by setting the environment variable DEB_BUILD_PROFILES or by using the -P option with dpkg-buildpackage (or -o Apt::Build-Profiles for apt, or --profiles in sbuild). More than one build profile can be activated at a time. Multiple profiles are specified by separating them with commas in commandline arguments and by separating them with spaces in the DEB_BUILD_PROFILES environment variable. The initial profile names are "stage1", "stage2", "nocheck", "nobiarch" and "cross". Other possibilities are "nodoc" or "embedded".

Build-Depends syntax extension

An example demonstrating the build profile syntax:

Build-Depends: foo (>= 1.0) [i386 arm] <!profile.stage1 !profile.cross>, bar

This specification introduces a new pair of brackets (using < as the opening and > as the closing bracket) to be used after the architecture qualification list. Just as in the architecture qualification list the < and > brackets enclose a space separated list of terms called a restriction list.

Every term in the restriction list follows the following regular expression:

([a-z][a-z0-9-]*)\.([a-z][a-z0-9-]*)

The first group is called a namespace. The namespace "profile" is used to support build profiles and is the only allowed namespace for now. By introducing additional namespaces, this syntax can later be extended for other uses besides build profiles (for example an "arch" namespace can be introduced). No other namespace besides "profile" exists yet but it is forbidden to mix different namespaces within one restriction list. The second group is called a label and specifies the profile name.

Terms can be negated by using an exclamation mark as a prefix. In contrast to the architecture qualification list, positive and negative terms are allowed to be mixed. The semantics of a restriction list are computed as follows:

  • one or more profiles can be activated at the same time (by commandline argument or environment variable)
  • for each dependency, the restriction list is processed from left to right
  • if a negated term is encountered and the specified profile is set, the build dependency is dropped and processing of the list stops
  • if a positive term is encountered and the specified profile is set, then the build dependency is kept and processing of the list stops
  • if no profile is set for any term in the restriction list and at least one term in the restriction list is negated then keep the build dependency, otherwise drop the build dependency

Above rules express the same semantics as architecture restrictions if positive and negative terms are not mixed. If positive and negative terms are mixed (and only then) then the order of the terms in the restriction list matters. The following table illustrates the implication of the above rules. Each cell indicates whether or not the dependency foo is dropped with a certain value of DEB_BUILD_PROFILES.

""

"stage1"

"nocheck"

"stage1 nocheck"

foo <!profile.stage1>

keep

drop

keep

drop

foo <profile.stage1>

drop

keep

drop

keep

foo <!profile.stage1 !profile.nocheck>

keep

drop

drop

drop

foo <profile.stage1 profile.nocheck>

drop

keep

keep

keep

foo <!profile.stage1 profile.nocheck>

keep

drop

keep

drop

foo <profile.nocheck !profile.stage1>

keep

keep

drop

keep

Future extensions

  • The syntax could be extended to allow more than one <> block

  • It could also be allowed to allow architecture names under the "arch" namespace within a <> block.

  • It could be allowed to mix different namespaces within a <> block

The Build-Profiles field

In debian/control binary package stanzas, the content of the Build-Profiles field specifies the (unordered) list of build profiles for which that binary package does or does not build. This list can either be all positive or all negative. Entries are negated by using an exclamation mark as a prefix.

Build-Profiles: !cross !stage1

If a binary package stanza in a debian/control file does not contain a Build-Profiles field, then it implicitly means that it builds with all build profiles.

The Built-For-Profiles field

In *.changes and Packages files, the content of the Built-For-Profiles field specifies the list of build profiles for which that binary or source package was built.

The DEB_BUILD_PROFILES environment variable

The DEB_BUILD_PROFILES environment variable contains a space separated unordered list of activated profiles. The content of this variable must be honored by all tools involved in package compilation. Here an example for debian/rules (enabling sql for any build except stage1 profile - notice the negated test):

ifneq ($(filter stage1,$(DEB_BUILD_PROFILES)),)
    DH_OPTIONS += -Nlibdb5.1-sql # not needed with debhelper (>= 9.20140227)
    CONFIGURE_SWITCHES += --disable-sql
else
    CONFIGURE_SWITCHES += --enable-sql
endif

Profile built binary packages

A binary package must offer the exact same functionality for all profiles with which it builds including no activated profile at all (if it builds in that case). Otherwise a package depending on that binary package might not find the functionality it expects it to provide. This means that if necessary binary packages have to be split or that a source package has to be built in two stages. This requirement can not technically be checked before ReproducibleBuilds happen.

Example

Imagine src:foo depends on bar which is needed to build a plugin that is shipped together with other plugins in the package foo-plugins. When bar is made optional as a build dependency of src:foo in a stage1 build, then the resulting stage1 binary package foo-plugins would differ from the foo-plugins package that is created during a full build. Thus the creation of the foo-plugins binary package must be disabled by using the Build-Profiles field in the foo-plugins binary package stanza and by adding the necessary bits to debian/rules for packages without debhelper or when build depending on debhelper (>> 9.20140227) is not desired.

If foo-plugins has reverse dependencies which are part of a cycle with src:foo itself, then not building foo-plugins would not solve the bootstrap problem. In that case it is necessary to either:

  • split the foo-plugins package into foo-plugins-nobar and foo-plugins-bar and adjust *all* reverse dependencies of foo-plugins to correctly depend on foo-plugins-nobar and foo-plugins-bar

  • or create a package foo-plugins-nobar during a stage1 build and modify only the reverse dependencies that are in a cycle with src:foo to build depend on foo-plugins-nobar in their own stage1 build. When src:foo is built fully, it will not produce foo-plugins-nobar anymore but the full foo-plugins package instead.

Other uses for the restriction syntax

Mark build dependencies that need to be translated when cross building

See CrossTranslatableBuildDeps#A3._Use_restriction_syntax_with_new_namespace

Replace Build-Depends-Indep and Build-Depends-Arch

Instead of moving build dependencies needed to build arch:all or arch:any packages into Build-Depends-Indep and Build-Depends-Arch, respectively, mark them directly in the Build-Depends field using <debbuildopt.archindep> and <debbuildopt.archdep>, respectively.

Map DEB_BUILD_OPTIONS to namespaces

Some DEB_BUILD_OPTIONS like nocheck or noopt could be mapped to <debbuildopt.nocheck> or <debbuildopt.noopt>. This would allow not needing to install certain dependencies if certain DEB_BUILD_OPTIONS are active.

Discussion