Differences between revisions 66 and 74 (spanning 8 versions)
Revision 66 as of 2016-08-19 12:55:24
Size: 15840
Editor: HolgerLevsen
Comment: we could also make this a comment in the wiki source…
Revision 74 as of 2021-07-31 13:22:47
Size: 140
Editor: lamby
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
Please keep in mind that [[http://orwell.ru/library/articles/As_I_Please/english/eaip_01#Feb0444|history is written by the winners]]. Let's just hope for not too much betrayal. Another way to say this is an African proverb: until the lion learns how to write, every story will glorify the hunter.


= Tell the tale =

A history of ''reproducible builds'' in Debian and around.

== An old idea ==

The idea of ''reproducible builds'' is not very new. In Debian world, it was mentioned first in [[https://lists.debian.org/debian-devel/2000/11/msg01758.html|2000]], and then more explicitly in [[https://lists.debian.org/debian-devel/2007/09/msg00746.html|2007]] on DebianList:debian-devel: “''I think it would be really cool if the Debian policy required that packages could be rebuild bit-identical from source.”'' The reactions were unfortunately not really enthusiastic both times.

== Private property + Snowden effect ==

The interest on reproducible builds picked up again with WikiPedia:Bitcoin. Users of bitcoins needed a way to trust that they were not downloading corrupted software. Initial versions of [[http://gitian.org/|Gitian]] were written in 2011 to solve the problem. It drives builds using virtual machines and Git.

The [[WikiPedia:Global_surveillance_disclosures_%282013%E2%80%93present%29|global surveillance disclosures]] in 2013 raised the interest even further. Mike Perry worked on [[https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details|making the Tor Browser build reproducibly]] in [[https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise|fear]] of a “''malware that attacks the software development and build processes themselves to distribute copies of itself to tens or even hundreds of millions of machines in a single, officially signed, instantaneous update''”.

== Kick-off ==

The success of making such a large piece of software build reproducibly proved that it was feasible for other projects. This prompted Lunar to organize a [[http://penta.debconf.org/dc13_schedule/events/1063.en.html|discussion at DebConf13]] happening July 2013. Even scheduled at the last minute, there was still about thirty attendees who were very much interested, amongst them members of the [[https://www.debian.org/devel/tech-ctte|technical committee]] and a few other core teams. [[attachment:dc13-bof-reproducible-builds.txt|Minutes]] are available.

After some more research during the conference, a [[https://wiki.debian.org/ReproducibleBuilds?action=recall&rev=1|wiki page]] was created. The initial approach was to get Debian to “buy-in” on the idea by making five packages from different maintainers build reproducibly. However, it quickly appeared that before fixing issues in the toolchain, it would not be possible to even get a single package to be reproducible.

== First mass-rebuilds ==

Lunar came up with the first patches for DebianPts:dpkg at the August 2013. This enabled DebianPts:hello to build reproducibly. The [[ReproducibleBuilds/Rebuild20130907|first large scale rebuild]] was performed soon after by David Suárez, with variations on time and build path. 24% of 5240 source packages were identified as reproducible. The [[https://wiki.debian.org/ReproducibleBuilds?action=recall&rev=83#Crude_bash_script_to_compare_two_binary_packages|first version of a “smart” comparison script]] was written to help reviewing differences.

A [[ReproducibleBuilds/Rebuild20140126|second mass rebuild]] was made before the [[https://fosdem.org/2014/schedule/event/reproducibledebian/|presentation in the distro devroom at FOSDEM’14]]. It used a slightly different approach regarding build paths and had DebianPts:binutils built in deterministic mode. 67% of 6887 source packages were found reproducible. A result applauded by the FOSDEM crowd.

The presentation sparked interest and woke up the [[https://lists.alioth.debian.org/mailman/listinfo/reproducible-builds|mailing-list]] created some months ago. Tomasz Buchert wrote a [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140210/000038.html|lintian check for gzip files]]. Stéphane Glondu worked on sorting logs and [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140217/000053.html|experimenting with alternatives for build path issues]].

== .buildinfo control files ==

In parallel, several approaches on where and how to record the build environment were considered. The first idea was to use the [[https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles|.changes control file]] through a substitution variable (Bug:719854). Instead, Guillem Jover suggested to add new fields by passing `--changes-option="-DBuild-Env=…` to `dpkg-buildpackage`. As for the value, we discovered DebianPkg:dh-buildinfo written by Yann Dirson, described as a “debhelper addon to track package versions used to build a package”. Fit for reproducible builds!

[[https://summit.debconf.org/debconf14/meeting/78/reproducible-builds-for-debian/|What happened for a year]] was presented at [[http://debconf14.debconf.org/|DebConf14]]. The reception was unexpectedly good and the follow-up BoF truly productive. For one thing, a suitable way to record the build environment was sketched out.

One issue about using `.changes` files is that they are not kept in the archive. So to be used as a way to record the environment, they would need to be distributed with the archive. But this would be a misunderstanding of their purpose. As their name implies, `.changes` control files represent ''changes'' to archive. They were inherently designed to be transient.

So instead, we had the idea of a new `.buildinfo` control file which would be added to the archive alongside binary packges — and be uploaded by referencing them in `.changes`. We quickly drafted a [[https://wiki.debian.org/ReproducibleBuilds/BuildinfoSpecification|specification]], and a couple of days later Niko Tyni came up with [[https://anonscm.debian.org/cgit/reproducible/debhelper.git/tree/dh_genbuildinfo?h=pu/reproducible_builds_2014&id=1543ea2535160bf9578149c681eb7ff324901471|an addition to debhelper]] which created a `.buildinfo` using the output of the aforementioned `dh-buildinfo`.

== strip-nondeterminism ==

Before DebConf14, an explicit ''timestamp'' was given during rebuilds, extracted from the `.changes` file. However, during the discussions, there was a consensus that the date of the latest entry in the [[https://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog|debian/changelog]] file could be used as the reference timestamp when needed.

This helped another idea: a generic tool that would post-process different file formats to remove timestamps or other source of non-determinism. Andrew Ayer further took the task of creating DebianPkg:strip-nondeterminism. The first released version handled files created by gzip, Zip, Jar, Javadoc, and `.a` files.

== Giving up on build paths ==

Initially we though that variations happening when building the package from different build path should be eliminated. This has proven difficult. The main problem that has been identified is that full path to source files are written in debug symbols of ELF files.

First attempt used the [[https://gcc.gnu.org/onlinedocs/gcc-4.9.2/gcc/Debugging-Options.html|-fdebug-prefix-map]] option which allows to map the current directory to a canonical one in what gets recorded. But compiler options get written to debug file as well. So it has to be doubled with `-gno-record-gcc-switches` to be used for reproducibility. The [[ReproducibleBuilds/Rebuild20130907|first large scale rebuild]] has proven that it was also hard to determine what the actual build path has been accurately.

Second attempt used DebianPkg:debugedit which is used by Fedora and other to change the source paths to a canonical location after the build. Unfortunately, `gcc` write debug strings in a ''hashtable''. `debugedit` will not reorder the table after patching the strings, so the result is still unreproducible. Adding this feature to `debugedit` looked difficult. We can still make the approach work by passing `-fno-merge-debug-strings` but this is space expensive. The [[ReproducibleBuilds/Rebuild20140126|second large scale rebuild]] used the latter approach. It was still difficult to guess the initial build path properly. Stéphane Glondu was the first to [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140217/000065.html|suggest to using a canonical build path]] to solve the issue.

During [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140901/000198.html|discussions at DebConf14]], we revisited the idea, and felt it was indeed appropriate to decide on a canonical build path. It has an added benefit of making it easier to use debug packages: one simply has to unpack the source in the right place, no extra configuration required.

Finally, it was [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20141117/000560.html|agreed]] to add a `Build-Path` field to `.buildinfo` as it made it easier to reproduce the initial build if the canonical build location would change.

== Improved comparison tool ==

After DebianPkg:strip-nondeterminism initial upload and integrating some more changes discussed during DebConf14 in DebianPts:dpkg and DebianPts:debhelper, Lunar [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140915/000441.html|experimented with 172 core packages]]. 30% were reproduced without further modifications.

As the current tools to understand differences between builds were slow and hard to read, Lunar wrote DebianPkg:debbindiff. It [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20140929/000492.html|replaced]] inefficient shell scripts by structured Python with a HTML output.

== Continuous integration ==

At the end of September 2014, Holger Levsen started to work on extending [[Services/jenkins.debian.net|jenkins.debian.net]] to perform continuous integration for build reproducibility. Packages from ''sid'' started to be built and rebuild. This initially introduced variations for time and file ordering, and was extended later on to also use different users, groups, hostnames, and locales.

The results were visible through a new [[https://reproducible.debian.net/|reproducible.debian.net]] website. The process of analyzing reproducibility failures could now be more easily shared. New contributors indeed showed up and started submitting sorting out common issues and providing patches.

== dpkg-genbuildinfo ==

The turn of 2015 saw the replacement of the prototype `.buildinfo` generator by a new implementation suitable for proper inclusion in `dpkg`. Previously, only packages using `dh` could generate `.buildinfo` and could thus be considered reproducible. After updating the experimental toolchain, the change allowed to reach the mark of [[https://people.debian.org/~lunar/blog/posts/eighty_percent/|80% source packages reproducible]].

== FOSDEM 2015 and aftermath ==

The presentation [[https://fosdem.org/2015/schedule/event/stretching_out_for_trustworthy_reproducible_builds/|Stretching out for trustworthy reproducible builds]] was well received at FOSDEM 2015 and was followed up by
 * [[https://tracker.debian.org]] inclusion, see [[DebianBug:739497]]
 * [[https://qa.debian.org/developer.php|Debian Developer's Packages Overview]] (DDPO) inclusion
 * debbindiff gained .rpm support
 * [[https://udd.debian.org/dmd/|Debian Maintainer Dashbord]] inclusion

Finally, for now, not even two weeks after FOSDEM 2015 a mail with the subject "[[https://lists.debian.org/debian-devel-announce/2015/02/msg00007.html|Reproducible Builds — proof of concept successful for 83% of all sources in main]]" was send to [[https://lists.debian.org/debian-devel-announce/|debian-devel-announce@lists.ddebian.org]] officially anouncing the project to the Debian developer community at large.

== To be sorted out ==

 * 2015-03-26: [[https://tracker.debian.org/news/675691|binutils 2.25-6 is built with --enable-deterministic-archives]]
 * testing ''testing'' and ''experimental'' now, pkg sets available too.
 * 2015-05-27: [[https://tracker.debian.org/news/687110|iceweasel 38.0.1-5]] is reproducible.

== Google Summer of Code 2015 ==

During the summer of 2015 akira and Dhole will be working on moving forward reproducible builds as a Google Summer of Code project. Follow the links to check the accepted [[SummerOfCode2015/StudentApplications/MariaValentinaMarinRodrigues|akira's application]] and [[SummerOfCode2015/StudentApplications/EduardSanou|Dhole's application]]. Dhole also made a [[https://dhole.github.io/post/reproducible_builds_debian_gsoc2015/|blog post about how Dhole got into GSoC 2015]].

== CCCamp 2015 ==

Short mention of Lunar's talk to be written here. Add links.

== DebConf15 ==

To be written: the first real life meeting of the Debian team. Talk given, roundstable discussion, hacking session. Mentioned in several talks, incl DPL key note. SOURCE_DATE_EPOCH was invented around this time too.

== Continous tests for Coreboot, OpenWrt, NetBSD, FreeBSD, Archlinux and Fedora ==

to be written: tests for these six projects have been added between Juni and December 2015…

== Reproducible World Summit, December 1-3, 2015, Athens, Greece

to be written, maybe some photos to be shared, pointers to reports, new mailinglists, new irc channel, an even wider community has started to grow, [[https://reproducible-builds.org/events/athens2015/|website]].

{{{#!wiki note
Do you want to make the rest of the story? [[ReproducibleBuilds/Contribute|Join us!]]

= Archive wide rebuilds =

 * [[ReproducibleBuilds/Rebuild20130907|2013-09-07]] by David Suárez. 24% of 5240 source packages reproducible. Variations: time, build path.
 * [[ReproducibleBuilds/Rebuild20140126|2014-01-26]] by David Suárez. 67% of 6887 source packages reproducible. Variations: time, build path.
 * [[ReproducibleBuilds/RebuildCore20140919|2014-09-19]] by Lunar, 30% of 172 source core packages reproducible. Variations: time, file order.
 * [[https://reproducible.debian.net/userContent/reproducible.html|Updated daily since 2014-09-28]] by jenkins.debian.net. On 2014-11-11, 13213 (61.4%) out of 21448 packages are reproducible.

= Presentations =

<<Include(ReproducibleBuilds/About, , from="^= Presentations =",to="^= ")>>

= Publicity =

<<Include(ReproducibleBuilds/About, , from="^= Publicity =",to="^= ")>>

= Weekly reports =

<<Include(ReproducibleBuilds/About, , from="^= Weekly reports =",to="^= ")>>

= Contributors =

 * akira (Maria Valentina Marin)
 * Alexis Bienvenüe
 * Andrew Ayer
 * Asheesh Laroia
 * Ceridwen
 * Chris Lamb
 * Chris West
 * Christoph Berg
 * Daniel Kahn Gillmor
 * Daniel Shahaf
 * David Suarez
 * Dhole
 * Dmitry Bogatov
 * Drew Fisher
 * Esa Peuha
 * Fabian Wolff
 * Guillem Jover
 * Hans-Christoph Steiner
 * Helmut Grohne
 * Holger Levsen
 * HW42
 * Intrigeri
 * Jelmer Vernooij
 * josch (Johannes Schauer)
 * Juan Picca
 * Lunar (Jérémy Bobbio)
 * Mathieu Bridon
 * Mattia Rizzolo
 * Nicolas Boulenguez
 * Niels Thykier
 * Niko Tyni
 * Paul Gevers
 * Paul Wise
 * Peter De Wachter
 * Philip Rinn
 * Reiner Herrmann
 * hefee (Sandro Knauß)
 * Sascha Steinbiss
 * Satyam Zode
 * Scarlett Clark
 * Santiago Vila
 * Stefano Rivera
 * Stéphane Glondu
 * Steven Chamberlain
 * Tom Fitzhenry
 * Valerie Young
 * Valentin Lorentz
 * Wookey
 * Ximin Luo
This page has moved to: [[https://reproducible-builds.org/docs/history/|reproducible-builds.org/docs/history/]]…}}}

This page has moved to: reproducible-builds.org/docs/history/