Differences between revisions 2 and 22 (spanning 20 versions)
Revision 2 as of 2014-10-02 21:46:22
Size: 8545
Editor: Lunar
Comment: link to bug against ftp.debian.org
Revision 22 as of 2016-01-30 13:29:46
Size: 12733
Editor: Lunar
Comment: Rename Changes field to the more meaningful Binary-Only-Changes ; also the content is different than the one in .changes file (name suggested by Guillem)
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Information on a build will be recorded in a [[https://www.debian.org/doc/debian-policy/ch-controlfields.html|new control file]] with extension `.buildinfo`. Information on the system used during a build will be recorded in a [[https://www.debian.org/doc/debian-policy/ch-controlfields.html|new control file]], a “buildinfo” file with suffix `.buildinfo`.
Line 7: Line 7:
= .buildinfo example =

The following file would be named `fweb_1.62-12_i386.buildinfo`:
= uses =

The `.buildinfo` file has several goals which are related to each other:

 * It records information about the system environment used during a particular build -- packages installed (toolchain, etc), system architecture, etc. This can be useful for forensics/debugging.
 * It can also be used to try to recreate (partially or in full) the system environment when trying to reproduce a particular build.

We want a debian user (or derivative) to be able to find and fetch all .buildinfo
files that were known to produce a given binary package so that they can try to reproduce the package themselves.

In the future, there may be more uses for .buildinfo files (or for collections of .buildinfo files related to a single binary artifact). For example:

 * A debian user could elect to only install binary packages that have been successfully built by multiple builders
 * A debian derivative could attest to the packages it has deterministically rebuilt
 * A triage process could identify toolchain changes that have an effect on large numbers of binary packages.
 * A cross-building process could demonstrate the correctness of the cross-compiling toolchain by reproducing the exact binary artifacts.

= buildinfo example =

a `.buildinfo` file is a UTF-8-encoded textfile, usually clearsigned with OpenPGP.

The following file would be named `fweb_1.62-12+b2_brahms-20120530114812.buildinfo`:
Line 12: Line 31:
Format: 1.9
Build-Archictecture: i386
Source: fweb
Version:
1.62-12
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Build-Architecture: i386
Source: fweb (1.62-12)
Line 18: Line 39:
Version: 1.62-12+b2
Binary-Only-Changes:
 fweb (1.62-12+b2) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for amd64; no source changes.
   * Rebuild for multiarch sync
 .
  -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms@buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000
Line 21: Line 50:
 a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12_i386.deb  a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12+b2_i386.deb
Build-Path: /usr/src/debian/fweb-1.62-12+b2
Line 174: Line 204:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQJ8BAEBCgBmBQJWYNoZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
Dx1xK9OgUkDt+gwh9WK/QrvV7IOjAg/pl6j7px5u6MNKHWPW0tC9M5123Q2KmaGT
-----END PGP SIGNATURE-----
Line 176: Line 212:
= .buildinfo specification =

== .buildinfo field descriptions ==

 Format:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Format|Same as in .changes]].
 Build-Architecture:: The [[https://www.debian.org/doc/debian-policy/ch-customized-programs.html#s-arch-spec|Debian machine architecture]] that was used to perform the build.
 Source:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|Same as in .changes]].
 Version:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version|Same as in .changes]].
 Binary:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Binary|Same as in .changes]].
 Architecture:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Architecture|Same as in .changes]] except `source` should not be specified: only concrete architectures, no wildcards or `any`.
 Checksums-Sha256:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Checksums|Same format as other control files]]. Must list the `.dsc` file and all [[https://www.debian.org/doc/debian-policy/ch-source.html#s-debianfiles|files listed in `debian/files`]].
= buildinfo specification =

== File name ==

buildinfo files should be named `${SOURCE}_${DEBIAN_VERSION}_${STRING}.buildinfo`.

 * ''SOURCE'' is the source package name.
 * ''DEBIAN_VERSION'' is the full version of the Debian package without the epoch. This is the same as the Version: field in the corresponding `.changes` file.
 * ''STRING'' is an arbitrary string, unique within all ${SOURCE},${DEBIAN_VERSION} builds. The string has the [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|restrictions as package names]]. It's likely that debian buildd's would create this string as `${HOSTNAME}-${TIMESTAMP}`, where `TIMESTAMP` is an ISO-8601-formatted numeric string. Debian developers may choose arbitrary strings here. The only requirement from the archive's perspective is that the `STRING` be unique for source package version, so that multiple `.buildinfo` files may exist for a given version of a package.

== buildinfo field descriptions ==

 Format:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Format|Same as in `*.changes`]].
 Build-Architecture:: The [[https://www.debian.org/doc/debian-policy/ch-customized-programs.html#s-arch-spec|Debian machine architecture]] that was used to perform the build. <<FootNote(This is, the build architecture in GNU terminology. See `dpkg-architecture(1)` for a definition. The target architecture for cross compilers is usually encoded in the package name. The host architecture is the binary package architecture (in the `Architecture` field and file name). Thus, target and host architecture do not need to be encoded here.)>>
 Source:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source|Same as in `*.changes`]]. If the source and binary versions differ (e.g. [[binNMU]]s), the source version is added between parenthesis.
 Binary:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Binary|Same as in `*.changes`]].
 Architecture:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Architecture|Same as in `*.changes`]] except `source` should not be specified: only concrete architectures, no wildcards or `any`.
 Version:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version|Same as in `*.changes`]].
 Binary-Only-Changes:: Only present for [[binNMU|binary only uploads]]: contains the full extra changelog entries.
 Checksums-Sha256:: [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Checksums|Same format as other control files]]. Must list the `*.dsc` file and all [[https://www.debian.org/doc/debian-policy/ch-source.html#s-debianfiles|files listed in `debian/files`]].
 Build-Path:: (optional) Absolute path of the directory in which the package has been built. See [[ReproducibleBuilds/History#Giving_up_on_build_paths]] for a rationale.
Line 189: Line 235:
The content of the `Build-Environment` is close to what DebianPts:dh-buildinfo currently produces.

== .buildinfo signatures ==

As `.buildinfo` are meant to enable to reproduce a given build, multiple
parties should be able to assess their content. `.buildinfo` files thus
are signed using detached signatures, with the full fingerprint of the
key in the filename.

Example file list:

{{{
hello_2.8-1_amd64.buildinfo
hello_2.8-1_amd64.buildinfo.0603CCFD91865C17E88D4C798382C95C29023DF9.asc
hello_2.8-1_amd64.buildinfo.0EE5BE979282D80B9F7540F1CCD2ED94D21739E9.asc
}}}

== Inclusion of .buildinfo in the archive ==

`.buildinfo` files will be referenced by their assessers through a
`Build-Signed-Off-By` field in the [[https://wiki.debian.org/RepositoryFormat#A.22Packages.22_Indices|Packages` index]].

Example:

{{{
Package: hello
Version: 2.9-1
Installed-Size: 140
Maintainer: Santiago Vila <sanvila@debian.org>
Architecture: i386
[…]
Build-Signed-Off-By:
 0603CCFD91865C17E88D4C798382C95C29023DF9 Jérémy Bobbio <lunar@debian.org>,
 D54C3BFAFFB042DE382DA5D741CE7F0B9F1B8B32 Santiago Vila <sanvila@debian.org>
}}}
== buildinfo signatures ==

`.buildinfo` files describe the state of a particular build. The operator having made the build may sign the file by wrapping the entire file in a cleartext OpenPGP signature.

== Inclusion of *.buildinfo in the archive ==

We want an interested user (an individual, organization, or derivative) to be able to find the relevant `.buildinfo` files from the archive they already have configured. There are several ways this could be done, but we imagine that a relatively easy way is to ship all the `.buildinfo` files for a given architecture for a suite in a `Buildinfos.tgz` archive file placed next to `Packages.gz`. See Bug:763822 for more details.
Line 228: Line 246:
files looked like a good place to record the environment as theylist the files looked like a good place to record the environment as they list the
Line 232: Line 250:
But the meaning of `.changes` files is pretty clear: they describe a But the meaning of `*.changes` files is pretty clear: they describe a
Line 234: Line 252:
the archive: they are equivalent of a log entry. The name of `.changes` the archive: they are equivalent of a log entry. The name of `*.changes`
Line 238: Line 256:


We thought that .buildinfo files would be all the data ''required'' to do a rebuild, as opposed to a description of the state of the build system. However, it's likely that we don't know the actual requirements, and it's likely that the description will be more detailed than is necessary in some cases. This means that two different buildinfo files could attest to the same exact binary artifact.

We thought at some point that the .buildinfo could be referenced in the `Packages` index, but this does not seem necessary, and might be overkill, since only some users may actually have varied systems.

We do not need to specify the ordering of fields?

= things we are not currently including =

 * order in which the system packages (essential, build-essential, build-deps) were installed.
 * the cryptographic digest of the system packages themselves
 * the digest of the source code of the system packages

Information on the system used during a build will be recorded in a new control file, a “buildinfo” file with suffix .buildinfo.

Discussion with ftpmasters is happening in 763822.

uses

The .buildinfo file has several goals which are related to each other:

  • It records information about the system environment used during a particular build -- packages installed (toolchain, etc), system architecture, etc. This can be useful for forensics/debugging.
  • It can also be used to try to recreate (partially or in full) the system environment when trying to reproduce a particular build.

We want a debian user (or derivative) to be able to find and fetch all .buildinfo files that were known to produce a given binary package so that they can try to reproduce the package themselves.

In the future, there may be more uses for .buildinfo files (or for collections of .buildinfo files related to a single binary artifact). For example:

  • A debian user could elect to only install binary packages that have been successfully built by multiple builders
  • A debian derivative could attest to the packages it has deterministically rebuilt
  • A triage process could identify toolchain changes that have an effect on large numbers of binary packages.
  • A cross-building process could demonstrate the correctness of the cross-compiling toolchain by reproducing the exact binary artifacts.

buildinfo example

a .buildinfo file is a UTF-8-encoded textfile, usually clearsigned with OpenPGP.

The following file would be named fweb_1.62-12+b2_brahms-20120530114812.buildinfo:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Build-Architecture: i386
Source: fweb (1.62-12)
Binary: fweb fweb-doc
Architecture: all i386
Version: 1.62-12+b2
Binary-Only-Changes:
 fweb (1.62-12+b2) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for amd64; no source changes.
   * Rebuild for multiarch sync
 .
  -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms@buildd.debian.org>  Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000
Checksums-Sha256:
 9921500c4c6159c0019d4b8b600d2d06eef6b1da056abd2f78e66a9f0c3843b9 879 fweb_1.62-12.dsc
 3a7492c2013fbeebff08bee0514481ec0f56d2c4d138188d1ef85156d08ded00 436982 fweb-doc_1.62-12_all.deb
 a916dbb1c63707eaf52a5cdd10769871d2f621848176dc8f7ab4f0dcd999af85 229990 fweb_1.62-12+b2_i386.deb
Build-Path: /usr/src/debian/fweb-1.62-12+b2
Build-Environment:
 acl (= 2.2.52-1),
 adduser (= 3.113+nmu3),
 base-files (= 7.5),
 base-passwd (= 3.5.33),
 bash (= 4.3-9),
 binutils (= 2.24.51.20140818-1),
 bsdmainutils (= 9.0.5),
 bsdutils (= 1:2.20.1-5.8),
 build-essential (= 11.7),
 bzip2 (= 1.0.6-7),
 coreutils (= 8.21-1.2),
 cpp (= 4:4.9.1-3),
 cpp-4.9 (= 4.9.1-9),
 dash (= 0.5.7-4),
 debconf (= 1.5.53),
 debhelper (= 9.20140817),
 debianutils (= 4.4),
 dh-buildinfo (= 0.11),
 diffutils (= 1:3.3-1),
 dmsetup (= 2:1.02.88-1),
 dpkg (= 1.17.13),
 dpkg-dev (= 1.17.13),
 e2fslibs (= 1.42.11-2),
 e2fsprogs (= 1.42.11-2),
 file (= 1:5.19-1),
 findutils (= 4.4.2-9),
 g++ (= 4:4.9.1-3),
 g++-4.9 (= 4.9.1-9),
 gcc (= 4:4.9.1-3),
 gcc-4.9 (= 4.9.1-9),
 gcc-4.9-base (= 4.9.1-9),
 gettext (= 0.19.2-1),
 gettext-base (= 0.19.2-1),
 grep (= 2.20-2),
 groff-base (= 1.22.2-6),
 gzip (= 1.6-3),
 hostname (= 3.15),
 init (= 1.21),
 initscripts (= 2.88dsf-53.4),
 insserv (= 1.14.0-5),
 intltool-debian (= 0.35.0+20060710.1),
 libacl1 (= 2.2.52-1),
 libasan1 (= 4.9.1-9),
 libasprintf0c2 (= 0.19.2-1),
 libatomic1 (= 4.9.1-9),
 libattr1 (= 1:2.4.47-1),
 libaudit1 (= 1:2.3.7-1),
 libaudit-common (= 1:2.3.7-1),
 libblkid1 (= 2.20.1-5.8),
 libbz2-1.0 (= 1.0.6-7),
 libc6 (= 2.19-10),
 libc6-dev (= 2.19-10),
 libcap2 (= 1:2.24-4),
 libcap2-bin (= 1:2.24-4),
 libc-bin (= 2.19-10),
 libc-dev-bin (= 2.19-10),
 libcilkrts5 (= 4.9.1-9),
 libcloog-isl4 (= 0.18.2-1),
 libcomerr2 (= 1.42.11-2),
 libcroco3 (= 0.6.8-3),
 libcryptsetup4 (= 2:1.6.6-1),
 libdb5.3 (= 5.3.28-6),
 libdbus-1-3 (= 1.8.6-2),
 libdebconfclient0 (= 0.191),
 libdevmapper1.02.1 (= 2:1.02.88-1),
 libdpkg-perl (= 1.17.13),
 libffi6 (= 3.1-2),
 libgcc1 (= 1:4.9.1-9),
 libgcc-4.9-dev (= 4.9.1-9),
 libgcrypt11 (= 1.5.4-2),
 libgcrypt20 (= 1.6.2-2),
 libgdbm3 (= 1.8.3-13),
 libglib2.0-0 (= 2.40.0-4),
 libgmp10 (= 2:6.0.0+dfsg-6),
 libgomp1 (= 4.9.1-9),
 libgpg-error0 (= 1.13-3),
 libintl-perl (= 1.23-1),
 libisl10 (= 0.12.2-2),
 libitm1 (= 4.9.1-9),
 libkmod2 (= 18-1),
 liblzma5 (= 5.1.1alpha+20120614-2),
 libmagic1 (= 1:5.19-1),
 libmount1 (= 2.20.1-5.8),
 libmpc3 (= 1.0.2-1),
 libmpfr4 (= 3.1.2-1),
 libncurses5 (= 5.9+20140712-2),
 libncurses5-dev (= 5.9+20140712-2),
 libncursesw5 (= 5.9+20140712-2),
 libpam0g (= 1.1.8-3.1),
 libpam-modules (= 1.1.8-3.1),
 libpam-modules-bin (= 1.1.8-3.1),
 libpam-runtime (= 1.1.8-3.1),
 libpcre3 (= 1:8.35-3),
 libpipeline1 (= 1.3.0-1),
 libprocps3 (= 1:3.3.9-7),
 libquadmath0 (= 4.9.1-9),
 libselinux1 (= 2.3-1),
 libsemanage1 (= 2.3-1),
 libsemanage-common (= 2.3-1),
 libsepol1 (= 2.3-1),
 libss2 (= 1.42.11-2),
 libstdc++-4.9-dev (= 4.9.1-9),
 libstdc++6 (= 4.9.1-9),
 libsystemd-journal0 (= 208-8),
 libsystemd-login0 (= 208-8),
 libtext-unidecode-perl (= 0.04-2),
 libtimedate-perl (= 2.3000-2),
 libtinfo5 (= 5.9+20140712-2),
 libtinfo-dev (= 5.9+20140712-2),
 libubsan0 (= 4.9.1-9),
 libudev1 (= 208-8),
 libunistring0 (= 0.9.3-5.2),
 libustr-1.0-1 (= 1.0.4-3),
 libuuid1 (= 2.20.1-5.8),
 libwrap0 (= 7.6.q-25),
 libxml2 (= 2.9.1+dfsg1-4),
 libxml-libxml-perl (= 2.0116+dfsg-1+b1),
 libxml-namespacesupport-perl (= 1.11-1),
 libxml-sax-base-perl (= 1.07-1),
 libxml-sax-perl (= 0.99+dfsg-2),
 linux-libc-dev (= 3.14.15-2),
 login (= 1:4.2-2+b1),
 lsb-base (= 4.1+Debian13),
 make (= 4.0-8),
 man-db (= 2.6.7.1-1),
 mawk (= 1.3.3-17),
 mount (= 2.20.1-5.8),
 ncurses-base (= 5.9+20140712-2),
 ncurses-bin (= 5.9+20140712-2),
 passwd (= 1:4.2-2+b1),
 patch (= 2.7.1-6),
 perl (= 5.20.0-4),
 perl-base (= 5.20.0-4),
 perl-modules (= 5.20.0-4),
 po-debconf (= 1.0.16+nmu3),
 procps (= 1:3.3.9-7),
 sed (= 4.2.2-4),
 sensible-utils (= 0.0.9),
 startpar (= 0.59-3),
 systemd (= 208-8),
 systemd-sysv (= 208-8),
 sysvinit-utils (= 2.88dsf-53.4),
 sysv-rc (= 2.88dsf-53.4),
 tar (= 1.27.1-2),
 texinfo (= 5.2.0.dfsg.1-4),
 tzdata (= 2014f-1),
 ucf (= 3.0030),
 udev (= 208-8),
 util-linux (= 2.20.1-5.8),
 xz-utils (= 5.1.1alpha+20120614-2),
 zlib1g (= 1:1.2.8.dfsg-2)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQJ8BAEBCgBmBQJWYNoZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
Dx1xK9OgUkDt+gwh9WK/QrvV7IOjAg/pl6j7px5u6MNKHWPW0tC9M5123Q2KmaGT
-----END PGP SIGNATURE-----

buildinfo specification

File name

buildinfo files should be named ${SOURCE}_${DEBIAN_VERSION}_${STRING}.buildinfo.

  • SOURCE is the source package name.

  • DEBIAN_VERSION is the full version of the Debian package without the epoch. This is the same as the Version: field in the corresponding .changes file.

  • STRING is an arbitrary string, unique within all ${SOURCE},${DEBIAN_VERSION} builds. The string has the restrictions as package names. It's likely that debian buildd's would create this string as ${HOSTNAME}-${TIMESTAMP}, where TIMESTAMP is an ISO-8601-formatted numeric string. Debian developers may choose arbitrary strings here. The only requirement from the archive's perspective is that the STRING be unique for source package version, so that multiple .buildinfo files may exist for a given version of a package.

buildinfo field descriptions

Format

Same as in `*.changes`.

Build-Architecture

The Debian machine architecture that was used to perform the build. 1

Source

Same as in `*.changes`. If the source and binary versions differ (e.g. binNMUs), the source version is added between parenthesis.

Binary

Same as in `*.changes`.

Architecture

Same as in `*.changes` except source should not be specified: only concrete architectures, no wildcards or any.

Version

Same as in `*.changes`.

Binary-Only-Changes

Only present for binary only uploads: contains the full extra changelog entries.

Checksums-Sha256

Same format as other control files. Must list the *.dsc file and all files listed in `debian/files`.

Build-Path

(optional) Absolute path of the directory in which the package has been built. See ReproducibleBuilds/History#Giving_up_on_build_paths for a rationale.

Build-Environment

List of all packages forming the build environment, their architecture if different from build architecture, and their version. This includes Essential packages, build-essential, and Build-Depends and Build-Depends-Indep. For each packages, their dependencies should be recursively listed. The format is the same as Built-Using.

buildinfo signatures

.buildinfo files describe the state of a particular build. The operator having made the build may sign the file by wrapping the entire file in a cleartext OpenPGP signature.

Inclusion of *.buildinfo in the archive

We want an interested user (an individual, organization, or derivative) to be able to find the relevant .buildinfo files from the archive they already have configured. There are several ways this could be done, but we imagine that a relatively easy way is to ship all the .buildinfo files for a given architecture for a suite in a Buildinfos.tgz archive file placed next to Packages.gz. See 763822 for more details.

Previous ideas

.changes files looked like a good place to record the environment as they list the checksums of the build products and are signed by either the maintainer or the buildd operator.

But the meaning of *.changes files is pretty clear: they describe a transactional change operation on the archive. They are not saved directly in the archive: they are equivalent of a log entry. The name of *.changes file is also not specified and multiple operations can have the same name.

(See also 719854 for the first attempt which tried using XC- field in debian/control.)

We thought that .buildinfo files would be all the data required to do a rebuild, as opposed to a description of the state of the build system. However, it's likely that we don't know the actual requirements, and it's likely that the description will be more detailed than is necessary in some cases. This means that two different buildinfo files could attest to the same exact binary artifact.

We thought at some point that the .buildinfo could be referenced in the Packages index, but this does not seem necessary, and might be overkill, since only some users may actually have varied systems.

We do not need to specify the ordering of fields?

things we are not currently including

  • order in which the system packages (essential, build-essential, build-deps) were installed.
  • the cryptographic digest of the system packages themselves
  • the digest of the source code of the system packages
  1. This is, the build architecture in GNU terminology. See dpkg-architecture(1) for a definition. The target architecture for cross compilers is usually encoded in the package name. The host architecture is the binary package architecture (in the Architecture field and file name). Thus, target and host architecture do not need to be encoded here. (1)