Differences between revisions 2 and 26 (spanning 24 versions)
Revision 2 as of 2014-10-02 21:46:22
Size: 8545
Editor: Lunar
Comment: link to bug against ftp.debian.org
Revision 26 as of 2016-10-10 16:19:56
Size: 12580
Editor: Infinity0
Comment: minor section tweaks
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Information on a build will be recorded in a [[https://www.debian.org/doc/debian-policy/ch-controlfields.html|new control file]] with extension `.buildinfo`.

Discussion with ''ftpmasters'' is happening in Bug:763822.
Information on the system used during a build will be recorded in a [[https://www.debian.org/doc/debian-policy/ch-controlfields.html|new control file]], a “buildinfo” file with suffix `.buildinfo`.

See also [[ReproducibleBuilds/BuildinfoInfrastructure]] for higher-level discussion about how these files are used to further the goal of reproducible builds.
Line 7: Line 7:
= .buildinfo example =

The following file would be named `fweb_1.62-12_i386.buildinfo`:
== Definition ==

The `.buildinfo` file has several goals which are related to each other:

 * It records information about the system environment used during a particular build -- packages installed (toolchain, etc), system architecture, etc. This can be useful for forensics/debugging.
 * It can also be used to try to recreate (partially or in full) the system environment when trying to reproduce a particular build.

We want a debian user (or derivative) to be able to find and fetch all .buildinfo
files that were known to produce a given binary package so that they can try to reproduce the package themselves.

In the future, there may be more uses for .buildinfo files (or for collections of .buildinfo files related to a single binary artifact). For example:

 * A debian user could elect to only install binary packages that have been successfully built by multiple builders
 * A debian derivative could attest to the packages it has deterministically rebuilt
 * A triage process could identify toolchain changes that have an effect on large numbers of binary packages.
 * A cross-building process could demonstrate the correctness of the cross-compiling toolchain by reproducing the exact binary artifacts.

== Example ==

a `.buildinfo` file is a UTF-8-encoded textfile, usually clearsigned with OpenPGP.

The following file would be named `fweb_1.62-12+b2_brahms-20120530114812.buildinfo`:
Line 12: Line 31:
Format: 1.9
Build-Archictecture: i386
Source: fweb
Version:
1.62-12
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Build-Architecture: i386
Source: fweb (1.62-12)
Line 18: Line 39:
Version: 1.62-12+b2
Binary-Only-Changes:
 fweb (1.62-12+b2) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for amd64; no source changes.
   * Rebuild for multiarch sync
 .
  -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms@buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000
Line 21: Line 50:
 a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12_i386.deb  a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12+b2_i386.deb
Build-Path: /usr/src/debian/fweb-1.62-12+b2
Line 174: Line 204:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQJ8BAEBCgBmBQJWYNoZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
Dx1xK9OgUkDt+gwh9WK/QrvV7IOjAg/pl6j7px5u6MNKHWPW0tC9M5123Q2KmaGT
-----END PGP SIGNATURE-----
Line 176: Line 212:
= .buildinfo specification =

== .buildinfo field descriptions ==

 Format:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Format|Same as in .changes]].
 Build-Architecture:: The [[https://www.debian.org/doc/debian-policy/ch-customized-programs.html#s-arch-spec|Debian machine architecture]] that was used to perform the build.
 Source:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|Same as in .changes]].
 Version:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version|Same as in .changes]].
 Binary:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Binary|Same as in .changes]].
 Architecture:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Architecture|Same as in .changes]] except `source` should not be specified: only concrete architectures, no wildcards or `any`.
 Checksums-Sha256:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Checksums|Same format as other control files]]. Must list the `.dsc` file and all [[https://www.debian.org/doc/debian-policy/ch-source.html#s-debianfiles|files listed in `debian/files`]].
== Specification ==

=== File name ===

buildinfo files should be named `${SOURCE}_${DEBIAN_VERSION}_${STRING}.buildinfo`.

 * ''SOURCE'' is the source package name.
 * ''DEBIAN_VERSION'' is the full version of the Debian package without the epoch. This is the same as the Version: field in the corresponding `.changes` file.
 * ''STRING'' is an arbitrary string, unique within all ${SOURCE},${DEBIAN_VERSION} builds. The string has the same [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|restrictions as package names]]. It's likely that debian buildd's would create this string as `${HOSTNAME}-${TIMESTAMP}`, where `TIMESTAMP` is an ISO-8601-formatted numeric string. Debian developers may choose arbitrary strings here. The only requirement from the archive's perspective is that the `STRING` be unique for source package version, so that multiple `.buildinfo` files may exist for a given version of a package.

=== Field descriptions ===

 Format:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Format|Same as in `*.changes`]].
 Build-Architecture:: The [[https://www.debian.org/doc/debian-policy/ch-customized-programs.html#s-arch-spec|Debian machine architecture]] that was used to perform the build. <<FootNote(This is, the build architecture in GNU terminology. See `dpkg-architecture(1)` for a definition. The target architecture for cross compilers is usually encoded in the package name. The host architecture is the binary package architecture (in the `Architecture` field and file name). Thus, target and host architecture do not need to be encoded here.)>>
 Source:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|Same as in `*.changes`]]. If the source and binary versions differ (e.g. [[binNMU]]s), the source version is added between parenthesis.
 Binary:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Binary|Same as in `*.changes`]].
 Architecture:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Architecture|Same as in `*.changes`]] except `source` should not be specified: only concrete architectures, no wildcards or `any`.
 Version:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version|Same as in `*.changes`]].
 Binary-Only-Changes:: Only present for [[binNMU|binary only uploads]]: contains the full extra changelog entries.
 Checksums-Sha256:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Checksums|Same format as other control files]]. Must list the `*.dsc` file and all [[https://www.debian.org/doc/debian-policy/ch-source.html#s-debianfiles|files listed in `debian/files`]].
 Build-Path:: (optional) Absolute path of the directory in which the package has been built. See [[ReproducibleBuilds/History#Giving_up_on_build_paths]] for a rationale.
Line 189: Line 235:
The content of the `Build-Environment` is close to what DebianPts:dh-buildinfo currently produces.

== .buildinfo signatures ==

As `.buildinfo` are meant to enable to reproduce a given build, multiple
parties should be able to assess their content. `.buildinfo` files thus
are signed using detached signatures, with the full fingerprint of the
key in the filename.

Example file list:

{{{
hello_2.8-1_amd64.buildinfo
hello_2.8-1_amd64.buildinfo.0603CCFD91865C17E88D4C798382C95C29023DF9.asc
hello_2.8-1_amd64.buildinfo.0EE5BE979282D80B9F7540F1CCD2ED94D21739E9.asc
}}}

== Inclusion of .buildinfo in the archive ==

`.buildinfo` files will be referenced by their assessers through a
`Build-Signed-Off-By` field in the [[https://wiki.debian.org/RepositoryFormat#A.22Packages.22_Indices|Packages` index]].

Example:

{{{
Package: hello
Version: 2.9-1
Installed-Size: 140
Maintainer: Santiago Vila <sanvila@debian.org>
Architecture: i386
[…]
Build-Signed-Off-By:
 0603CCFD91865C17E88D4C798382C95C29023DF9 Jérémy Bobbio <lunar@debian.org>,
 D54C3BFAFFB042DE382DA5D741CE7F0B9F1B8B32 Santiago Vila <sanvila@debian.org>
}}}

= Previous ideas =
=== Signatures ===

`.buildinfo` files describe the state of a particular build. The operator having made the build may sign the file by wrapping the entire file in a cleartext OpenPGP signature.

== Possible future extensions ==

We are not currently including the following things, mostly because it is quite hard to do them right now. But it would be nice to, in the future:

 * order in which the system packages (essential, build-essential, build-deps) were installed.
 * the cryptographic digest of the system packages themselves
 * the digest of the source code of the system packages

=== Inclusion of *.buildinfo in the archive ===

See [[ReproducibleBuilds/BuildinfoInfrastructure]]

== Previous ideas ==
Line 228: Line 254:
files looked like a good place to record the environment as theylist the files looked like a good place to record the environment as they list the
Line 232: Line 258:
But the meaning of `.changes` files is pretty clear: they describe a But the meaning of `*.changes` files is pretty clear: they describe a
Line 234: Line 260:
the archive: they are equivalent of a log entry. The name of `.changes` the archive: they are equivalent of a log entry. The name of `*.changes`
Line 238: Line 264:

We thought that .buildinfo files would be all the data ''required'' to do a rebuild, as opposed to a description of the state of the build system. However, it's likely that we don't know the actual requirements, and it's likely that the description will be more detailed than is necessary in some cases. This means that two different buildinfo files could attest to the same exact binary artifact.

We thought at some point that the .buildinfo could be referenced in the `Packages` index, but this does not seem necessary, and might be overkill, since only some users may actually have varied systems.

We do not need to specify the ordering of fields?

Information on the system used during a build will be recorded in a new control file, a “buildinfo” file with suffix .buildinfo.

See also ReproducibleBuilds/BuildinfoInfrastructure for higher-level discussion about how these files are used to further the goal of reproducible builds.

Definition

The .buildinfo file has several goals which are related to each other:

  • It records information about the system environment used during a particular build -- packages installed (toolchain, etc), system architecture, etc. This can be useful for forensics/debugging.
  • It can also be used to try to recreate (partially or in full) the system environment when trying to reproduce a particular build.

We want a debian user (or derivative) to be able to find and fetch all .buildinfo files that were known to produce a given binary package so that they can try to reproduce the package themselves.

In the future, there may be more uses for .buildinfo files (or for collections of .buildinfo files related to a single binary artifact). For example:

  • A debian user could elect to only install binary packages that have been successfully built by multiple builders
  • A debian derivative could attest to the packages it has deterministically rebuilt
  • A triage process could identify toolchain changes that have an effect on large numbers of binary packages.
  • A cross-building process could demonstrate the correctness of the cross-compiling toolchain by reproducing the exact binary artifacts.

Example

a .buildinfo file is a UTF-8-encoded textfile, usually clearsigned with OpenPGP.

The following file would be named fweb_1.62-12+b2_brahms-20120530114812.buildinfo:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Build-Architecture: i386
Source: fweb (1.62-12)
Binary: fweb fweb-doc
Architecture: all i386
Version: 1.62-12+b2
Binary-Only-Changes:
 fweb (1.62-12+b2) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for amd64; no source changes.
   * Rebuild for multiarch sync
 .
  -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms@buildd.debian.org>  Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000
Checksums-Sha256:
 9921500c4c6159c0019d4b8b600d2d06eef6b1da056abd2f78e66a9f0c3843b9 879 fweb_1.62-12.dsc
 3a7492c2013fbeebff08bee0514481ec0f56d2c4d138188d1ef85156d08ded00 436982 fweb-doc_1.62-12_all.deb
 a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12+b2_i386.deb
Build-Path: /usr/src/debian/fweb-1.62-12+b2
Build-Environment:
 acl (= 2.2.52-1),
 adduser (= 3.113+nmu3),
 base-files (= 7.5),
 base-passwd (= 3.5.33),
 bash (= 4.3-9),
 binutils (= 2.24.51.20140818-1),
 bsdmainutils (= 9.0.5),
 bsdutils (= 1:2.20.1-5.8),
 build-essential (= 11.7),
 bzip2 (= 1.0.6-7),
 coreutils (= 8.21-1.2),
 cpp (= 4:4.9.1-3),
 cpp-4.9 (= 4.9.1-9),
 dash (= 0.5.7-4),
 debconf (= 1.5.53),
 debhelper (= 9.20140817),
 debianutils (= 4.4),
 dh-buildinfo (= 0.11),
 diffutils (= 1:3.3-1),
 dmsetup (= 2:1.02.88-1),
 dpkg (= 1.17.13),
 dpkg-dev (= 1.17.13),
 e2fslibs (= 1.42.11-2),
 e2fsprogs (= 1.42.11-2),
 file (= 1:5.19-1),
 findutils (= 4.4.2-9),
 g++ (= 4:4.9.1-3),
 g++-4.9 (= 4.9.1-9),
 gcc (= 4:4.9.1-3),
 gcc-4.9 (= 4.9.1-9),
 gcc-4.9-base (= 4.9.1-9),
 gettext (= 0.19.2-1),
 gettext-base (= 0.19.2-1),
 grep (= 2.20-2),
 groff-base (= 1.22.2-6),
 gzip (= 1.6-3),
 hostname (= 3.15),
 init (= 1.21),
 initscripts (= 2.88dsf-53.4),
 insserv (= 1.14.0-5),
 intltool-debian (= 0.35.0+20060710.1),
 libacl1 (= 2.2.52-1),
 libasan1 (= 4.9.1-9),
 libasprintf0c2 (= 0.19.2-1),
 libatomic1 (= 4.9.1-9),
 libattr1 (= 1:2.4.47-1),
 libaudit1 (= 1:2.3.7-1),
 libaudit-common (= 1:2.3.7-1),
 libblkid1 (= 2.20.1-5.8),
 libbz2-1.0 (= 1.0.6-7),
 libc6 (= 2.19-10),
 libc6-dev (= 2.19-10),
 libcap2 (= 1:2.24-4),
 libcap2-bin (= 1:2.24-4),
 libc-bin (= 2.19-10),
 libc-dev-bin (= 2.19-10),
 libcilkrts5 (= 4.9.1-9),
 libcloog-isl4 (= 0.18.2-1),
 libcomerr2 (= 1.42.11-2),
 libcroco3 (= 0.6.8-3),
 libcryptsetup4 (= 2:1.6.6-1),
 libdb5.3 (= 5.3.28-6),
 libdbus-1-3 (= 1.8.6-2),
 libdebconfclient0 (= 0.191),
 libdevmapper1.02.1 (= 2:1.02.88-1),
 libdpkg-perl (= 1.17.13),
 libffi6 (= 3.1-2),
 libgcc1 (= 1:4.9.1-9),
 libgcc-4.9-dev (= 4.9.1-9),
 libgcrypt11 (= 1.5.4-2),
 libgcrypt20 (= 1.6.2-2),
 libgdbm3 (= 1.8.3-13),
 libglib2.0-0 (= 2.40.0-4),
 libgmp10 (= 2:6.0.0+dfsg-6),
 libgomp1 (= 4.9.1-9),
 libgpg-error0 (= 1.13-3),
 libintl-perl (= 1.23-1),
 libisl10 (= 0.12.2-2),
 libitm1 (= 4.9.1-9),
 libkmod2 (= 18-1),
 liblzma5 (= 5.1.1alpha+20120614-2),
 libmagic1 (= 1:5.19-1),
 libmount1 (= 2.20.1-5.8),
 libmpc3 (= 1.0.2-1),
 libmpfr4 (= 3.1.2-1),
 libncurses5 (= 5.9+20140712-2),
 libncurses5-dev (= 5.9+20140712-2),
 libncursesw5 (= 5.9+20140712-2),
 libpam0g (= 1.1.8-3.1),
 libpam-modules (= 1.1.8-3.1),
 libpam-modules-bin (= 1.1.8-3.1),
 libpam-runtime (= 1.1.8-3.1),
 libpcre3 (= 1:8.35-3),
 libpipeline1 (= 1.3.0-1),
 libprocps3 (= 1:3.3.9-7),
 libquadmath0 (= 4.9.1-9),
 libselinux1 (= 2.3-1),
 libsemanage1 (= 2.3-1),
 libsemanage-common (= 2.3-1),
 libsepol1 (= 2.3-1),
 libss2 (= 1.42.11-2),
 libstdc++-4.9-dev (= 4.9.1-9),
 libstdc++6 (= 4.9.1-9),
 libsystemd-journal0 (= 208-8),
 libsystemd-login0 (= 208-8),
 libtext-unidecode-perl (= 0.04-2),
 libtimedate-perl (= 2.3000-2),
 libtinfo5 (= 5.9+20140712-2),
 libtinfo-dev (= 5.9+20140712-2),
 libubsan0 (= 4.9.1-9),
 libudev1 (= 208-8),
 libunistring0 (= 0.9.3-5.2),
 libustr-1.0-1 (= 1.0.4-3),
 libuuid1 (= 2.20.1-5.8),
 libwrap0 (= 7.6.q-25),
 libxml2 (= 2.9.1+dfsg1-4),
 libxml-libxml-perl (= 2.0116+dfsg-1+b1),
 libxml-namespacesupport-perl (= 1.11-1),
 libxml-sax-base-perl (= 1.07-1),
 libxml-sax-perl (= 0.99+dfsg-2),
 linux-libc-dev (= 3.14.15-2),
 login (= 1:4.2-2+b1),
 lsb-base (= 4.1+Debian13),
 make (= 4.0-8),
 man-db (= 2.6.7.1-1),
 mawk (= 1.3.3-17),
 mount (= 2.20.1-5.8),
 ncurses-base (= 5.9+20140712-2),
 ncurses-bin (= 5.9+20140712-2),
 passwd (= 1:4.2-2+b1),
 patch (= 2.7.1-6),
 perl (= 5.20.0-4),
 perl-base (= 5.20.0-4),
 perl-modules (= 5.20.0-4),
 po-debconf (= 1.0.16+nmu3),
 procps (= 1:3.3.9-7),
 sed (= 4.2.2-4),
 sensible-utils (= 0.0.9),
 startpar (= 0.59-3),
 systemd (= 208-8),
 systemd-sysv (= 208-8),
 sysvinit-utils (= 2.88dsf-53.4),
 sysv-rc (= 2.88dsf-53.4),
 tar (= 1.27.1-2),
 texinfo (= 5.2.0.dfsg.1-4),
 tzdata (= 2014f-1),
 ucf (= 3.0030),
 udev (= 208-8),
 util-linux (= 2.20.1-5.8),
 xz-utils (= 5.1.1alpha+20120614-2),
 zlib1g (= 1:1.2.8.dfsg-2)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQJ8BAEBCgBmBQJWYNoZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
Dx1xK9OgUkDt+gwh9WK/QrvV7IOjAg/pl6j7px5u6MNKHWPW0tC9M5123Q2KmaGT
-----END PGP SIGNATURE-----

Specification

File name

buildinfo files should be named ${SOURCE}_${DEBIAN_VERSION}_${STRING}.buildinfo.

  • SOURCE is the source package name.

  • DEBIAN_VERSION is the full version of the Debian package without the epoch. This is the same as the Version: field in the corresponding .changes file.

  • STRING is an arbitrary string, unique within all ${SOURCE},${DEBIAN_VERSION} builds. The string has the same restrictions as package names. It's likely that debian buildd's would create this string as ${HOSTNAME}-${TIMESTAMP}, where TIMESTAMP is an ISO-8601-formatted numeric string. Debian developers may choose arbitrary strings here. The only requirement from the archive's perspective is that the STRING be unique for source package version, so that multiple .buildinfo files may exist for a given version of a package.

Field descriptions

Format

Same as in `*.changes`.

Build-Architecture

The Debian machine architecture that was used to perform the build. 1

Source

Same as in `*.changes`. If the source and binary versions differ (e.g. binNMUs), the source version is added between parenthesis.

Binary

Same as in `*.changes`.

Architecture

Same as in `*.changes` except source should not be specified: only concrete architectures, no wildcards or any.

Version

Same as in `*.changes`.

Binary-Only-Changes

Only present for binary only uploads: contains the full extra changelog entries.

Checksums-Sha256

Same format as other control files. Must list the *.dsc file and all files listed in `debian/files`.

Build-Path

(optional) Absolute path of the directory in which the package has been built. See ReproducibleBuilds/History#Giving_up_on_build_paths for a rationale.

Build-Environment

List of all packages forming the build environment, their architecture if different from build architecture, and their version. This includes Essential packages, build-essential, and Build-Depends and Build-Depends-Indep. For each packages, their dependencies should be recursively listed. The format is the same as Built-Using.

Signatures

.buildinfo files describe the state of a particular build. The operator having made the build may sign the file by wrapping the entire file in a cleartext OpenPGP signature.

Possible future extensions

We are not currently including the following things, mostly because it is quite hard to do them right now. But it would be nice to, in the future:

  • order in which the system packages (essential, build-essential, build-deps) were installed.
  • the cryptographic digest of the system packages themselves
  • the digest of the source code of the system packages

Inclusion of *.buildinfo in the archive

See ReproducibleBuilds/BuildinfoInfrastructure

Previous ideas

.changes files looked like a good place to record the environment as they list the checksums of the build products and are signed by either the maintainer or the buildd operator.

But the meaning of *.changes files is pretty clear: they describe a transactional change operation on the archive. They are not saved directly in the archive: they are equivalent of a log entry. The name of *.changes file is also not specified and multiple operations can have the same name.

(See also 719854 for the first attempt which tried using XC- field in debian/control.)

We thought that .buildinfo files would be all the data required to do a rebuild, as opposed to a description of the state of the build system. However, it's likely that we don't know the actual requirements, and it's likely that the description will be more detailed than is necessary in some cases. This means that two different buildinfo files could attest to the same exact binary artifact.

We thought at some point that the .buildinfo could be referenced in the Packages index, but this does not seem necessary, and might be overkill, since only some users may actually have varied systems.

We do not need to specify the ordering of fields?

  1. This is, the build architecture in GNU terminology. See dpkg-architecture(1) for a definition. The target architecture for cross compilers is usually encoded in the package name. The host architecture is the binary package architecture (in the Architecture field and file name). Thus, target and host architecture do not need to be encoded here. (1)