Differences between revisions 1 and 339 (spanning 338 versions)
Revision 1 as of 2013-08-15 23:40:25
Size: 3244
Comment: initial page
Revision 339 as of 2016-11-15 13:12:09
Size: 5119
Editor: HolgerLevsen
Comment: update status
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== The goal == It should be possible to reproduce, byte for byte, every build of every package in Debian. More information about reproducible builds in general are available at [[https://reproducible-builds.org|reproducible-builds.org]].
Line 3: Line 3:
It should be possible to reproduce, byte for byte, every build of every package in Debian. ||||||||<tablestyle="width: 800px;font-size: 1.2em; vertical-align: top">||
||<style="width: 25%;vertical-align: top;text-align: center">[[ReproducibleBuilds/About|{{attachment:ReproducibleBuilds/rb-about.png|About}}]] <<BR>> [[ReproducibleBuilds/About|About]] ||<style="width: 25%;vertical-align: top;text-align: center">[[ReproducibleBuilds/Howto|{{attachment:ReproducibleBuilds/rb-howto.png|Howto}}]] <<BR>> [[ReproducibleBuilds/Howto|Make a package reproducible]] ||<style="width: 25%;vertical-align: top;text-align: center">[[ReproducibleBuilds/Contribute|{{attachment:ReproducibleBuilds/rb-contribute.png|Contribute}}]] <<BR>> [[ReproducibleBuilds/Contribute|How to help]] ||<style="width: 25%;vertical-align: top;text-align: center">[[ReproducibleBuilds/ExperimentalToolchain|{{attachment:ReproducibleBuilds/rb-toolchain.png|Toolchain}}]] <<BR>> [[ReproducibleBuilds/ExperimentalToolchain|Experimental toolchain]] ||
||<style="width: 25%;vertical-align: top;text-align: center">[[ReproducibleBuilds/History|{{attachment:ReproducibleBuilds/rb-history.png|History}}]] <<BR>> [[ReproducibleBuilds/History|Project history]] ||<style="width: 25%;vertical-align: top;text-align: center">[[https://alioth.debian.org/projects/reproducible/|{{attachment:ReproducibleBuilds/rb-alioth.png|Alioth}}|class=]] <<BR>> [[https://alioth.debian.org/projects/reproducible/|Alioth project|class=]] ||<style="width: 25%;vertical-align: top;text-align: center">[[https://bugs.debian.org/cgi-bin/pkgreport.cgi?usertag=reproducible-builds@lists.alioth.debian.org|{{attachment:ReproducibleBuilds/rb-bugs.png|Bugs}}|class=]] <<BR>> [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?usertag=reproducible-builds@lists.alioth.debian.org|Bug reports|class=]] ||<style="width: 25%;vertical-align: top;text-align: center">[[https://reproducible.debian.net/|{{attachment:ReproducibleBuilds/rb-jenkins.png|Jenkins}}|class=]] <<BR>> [[https://reproducible.debian.net/|Continuous integration|class=]] ||
Line 5: Line 7:
For now, we will start with a few maintainers who want to opt in to this
goal as we flesh out the details of what will make it possible. This page
tracks our progress.
= Status =
Line 9: Line 9:
== Status == {{{#!wiki important
Reproducible builds of Debian as a whole are still not a reality, though individual reproducible builds of packages are possible. So while we are making very good progress, it is a ''stretch'' to say that Debian is reproducible.
}}}
Line 11: Line 13:
 * hello package: Contents of data.tar.gz and control.tar.gz can be made reproducible when 'gzip' replaced by 'gzip -n' in debian/rules. (#xyz)  * Our patches for dpkg finally landed in Debian unstable with [[https://tracker.debian.org/news/812924/|dpkg 1.18.1]] so the next big step is to make `dak` process `*.buildinfo` files, see [[https://bugs.debian.org/763822|#763822 ftp.debian.org: please include .buildinfo file in the archive]] for the relevant bug report.
 * We have a tentative [[ReproducibleBuilds/BuildinfoSpecification|specification]] for a new control file `*.buildinfo` that records the build environment. https://sources.debian.net/src/dpkg/1.18.13/man/deb-buildinfo.man/ is the real reference though.
 * We have an [[ReproducibleBuilds/ExperimentalToolchain|experimental toolchain]] that creates `*.buildinfo` files and allows a good amount of source packages to be reproducible.
 * We have a addendum to DebianPts:sbuild that can [[ReproducibleBuilds/About#Reproduce_the_build_environment|rebuild a package after recreating the recorded enviroment]].
 * We have a [[https://reproducible.debian.net/|continuous integration]] platform that builds and immediately rebuilds packages. With this we can detect problems related to timestamps, file ordering, CPU usage, (pseudo-)randomness and [[https://reproducible.debian.net/index_variations.html|other things]].
 * We are [[https://reproducible.debian.net/unstable/amd64/index_notes.html|examining packages]] and sorting out [[https://reproducible.debian.net/index_issues.html|common problems]].
 * Many patches have already been [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?usertag=reproducible-builds@lists.alioth.debian.org|submitted]], and we are continuously writing new ones.
 * You can check which packages installed on your system are still unreproducible by using the [[https://anonscm.debian.org/cgit/reproducible/misc.git/tree/unreproducible-installed|unreproducible-installed]] script.
Line 13: Line 22:
 * Waiting on a few dpkg bugs for avoiding timestamps and file order inconsistency in {data,control}.tar.gz (or .xz) [[https://reproducible.debian.net/userContent/unstable/amd64/stats_pkg_state.png|{{https://reproducible.debian.net/userContent/unstable/amd64/stats_pkg_state.png|Statistics from the continuous integration platform|width=100%}}|class=]]
Line 15: Line 24:
 * 5 packages from 5 maintainers are interested, of which 0 so far have reproducible contents of {data,control}.tar.gz = Next =
Line 17: Line 26:
 * You can use a script to rebuild a package, with the same build-depends that were used by the build daemons. See "How to reproduce a build" below.  * Identify more common problems.
 * Get toolchain changes integrated.
 * Start a campaign to get developers to fix their packages.
 * Get `.buildinfo` files [[Bug:763822|in the archive]].
 * Get `dpkg` supporting reproducible builds upload to unstable.
 * Require matching binary packages from the developer and a buildd before accepting the package in the archive. This could initially be opt-in.
Line 19: Line 33:
 * Things that need further investigation (by e.g. you!)
   * Document how to use Lunar's script to reproduce a build.
   * Find out if {control,data}.tar.gz files created by dpkg 1.17.1+ have a timestamp embedded.
For more concrete tasks to be done, look at [[ReproducibleBuilds/Contribute|how to contribute]].
Line 23: Line 35:
== Use cases == = Drivers =
Line 25: Line 37:
 * If the Debian build daemons are compromised, end users can assure themselves that their binaries are OK if they can regenerate them (and their build dependencies). (You could use a more complicated equivalence test than "do the hashes match?" but if the hashes do match, this is simple.)

== Detailed package status list ==

 * alpine (Asheesh Laroia)
   * Status: Untested
 * haveged (Lunar)
   * Status: Unknown
 * iotop (pabs)
   * Status: Unknown
 * debhelper (joeyh)
   * Status: Unknown
 * magit (lindi)
   * Status: Unknown

== How to reproduce a build ==

* Someone needs to document Lunar's script here: http://people.debian.org/~paulproteus/lunar-verify-script.rb

== Known bugs we are waiting on ==

 * dpkg: some bug #xxx about gzip timestamps
 * dpkg: some other bug #xxx tar directory order

== Different problems, and their solutions ==

=== Non-problems ===

 * You might think ELF binaries (e.g. /usr/bin/hello in the hello package) have embedded timestamps. Luckily, they don't!

=== Data files in data.tar.gz have timestamps ===

 * Recommended solution:
   * Use the timestamp of the of the last debian/changelog entry as reference.
   * touch all files to the reference timestamp before building the binary packages.
   * gzip -n when gzipping anything
   * get rid of non-determinisim (yup...)
  * Alternate solutions:
   * (or) libfaketime (probably breaks some things) (sudo apt-get install faketime)

=== {data,control}.tar.{gz,xz,bz2} may have timestamps ===

 * dpkg 1.17.1 might or might not store a timestamp for the .gz versions of these files.
 * *.xz and *.bz2 seem to provide no ability to store a timestamp.

=== {data,control}.tar.{gz,xz,bz2} will store files in readdir order ===

This is dependent on an accident of filesystem layout at build time, so it would
sometimes not be reproducible.

We should probably fix this in dpkg by sorting the contents of the tar files.

== References ==

* Mike Perry's discussion of how it took him eight weeks to make the Tor Browser Bundle have this feature: http://people.debian.org/~paulproteus/mike-perry-reproducible-tbb.txt
 * h01ger
 * lamby
 * infinity0

It should be possible to reproduce, byte for byte, every build of every package in Debian. More information about reproducible builds in general are available at reproducible-builds.org.

About
About

Howto
Make a package reproducible

?Contribute
?How to help

Toolchain
Experimental toolchain

History
Project history

Alioth
Alioth project

Bugs
Bug reports

Jenkins
Continuous integration

Status

Reproducible builds of Debian as a whole are still not a reality, though individual reproducible builds of packages are possible. So while we are making very good progress, it is a stretch to say that Debian is reproducible.

Statistics from the continuous integration platform

Next

  • Identify more common problems.
  • Get toolchain changes integrated.
  • Start a campaign to get developers to fix their packages.
  • Get .buildinfo files in the archive.

  • Get dpkg supporting reproducible builds upload to unstable.

  • Require matching binary packages from the developer and a buildd before accepting the package in the archive. This could initially be opt-in.

For more concrete tasks to be done, look at ?how to contribute.

Drivers

  • h01ger
  • lamby
  • infinity0