With free software, anyone can inspect the source code for malicious flaws. But Debian provide binary packages to its users. The idea of “deterministic” or “reproducible” builds is to empower anyone to verify that no flaws have been introduced during the build process by reproducing byte-for-byte identical binary packages from a given source.
Contents
Why do we want reproducible builds?
- Allow independent verifications that a binary matches what the source intended to produce.
- Should reproducible uploads become mandatory, then the incentive of an attacker to compromise the system of a developer with upload rights is lowered because it is not anymore possible for the developer to upload a binary that does not match the uploaded sources.
- Additionally, the incentive for this kind of attack is further lowered because an attacker now has to compromise all machines that can check the reproducibility of the uploaded source.
- Finally, with a sufficiently large body of independent (geographically and administratively) machines, reproducible builds can help find systems which are compromised in a way to produce binaries with altered functionality.
Help Multi-Arch: same packages co-installation (as they need every matching file to be byte identical).
- Be able to generate debug symbols for packages which do not have a “debug package”.
- Ensure packages can be built from source. The archive could be made to only accept reproducible uploads: the maintainer would stop uploading .deb files but keep them referenced in the .changes. A buildd would then build the source. Only if the hash matches the upload gets accepted.
- Allow file-level deduplication on Debian mirror sites, or maybe snapshots.d.o, of .deb files whose contents didn't really change between versions.
- Allow .deb deltas to be smaller.
Packages with build profiles must offer the exact same functionality for all profiles. Reproducible builds could be use to verify that it is the case.
Making sure that Architecture:all packages are build identically on different build architectures.
Validate cross-builds against native builds.
Reproducing builds
There are two sides to the problem: the build environment needs to be recorded during the initial build, and the same environment needs to be reproduced for later rebuilds.
Recording the environment
Information on a build will be recorded in a new control file with extension `.buildinfo`.
Reproduce the build environment
The srebuild program is a sbuild wrapper which finds a timestamp from snapshot.debian.org containing all versions of the binary packages in a .buildinfo file and then carries out the build with the right versions installed.
See srebuild.
References
Gitian: a secure software distribution method
Bazel: build tool that specifies all direct build dependencies
http://rbm.boklm.eu/: the Reproducible Build Manager
- Deterministic virtual machines:
Reflections on Trusting Trust, by Ken Thompson
Fully Countering Trusting Trust through Diverse Double-Compiling (DDC) - a PhD dissertation on how to use reproducible builds to counter the "trusting trust" attack on compilers
Is that really the source code for this software? by Jos van den Oever on blogs.kde.org (2013-06-19). Compare reproducing tar from the Debian, Fedora and OpenSUSE packages.
Deterministic Builds Part Two: Technical Details by Mike Perry
Verifying the source code for binaries by Jake Edge in Linux Weekly News.
Colin Watson's answer on ubuntu-devel to “Will Ubuntu use "reproducible builds" as debian is planning to do?”
guardianproject wiki:
Why and How of Reproducible Builds: Distrusting Our Own Infrastructure for Safer Software Releases, Seth Schoen and Mike Perry at Mozilla San Francisco, 2014-11-05
Challenges and implications of verifiable builds for security-critical open-source software by Xavier de Carné de Carnavalet and Mohammad Mannan, in ACSAC '14
The CIA Campaign to Steal Apple’s Secrets by Jeremy Scahill and Josh Begley, in The Intercept. Specifically (S//NF) Strawhorse: Attacking the MacOS and iOS Software Development Kit
Spy agencies target mobile phones, app stores to implant spyware by Amber Hildebrandt and Dave Seglins
Dave Cheney on Reproducible Builds in Go at GDG Berlin Golang, 20 April 2015
Triangle of Secure Code Delivery, Taylor Hornby, July 24, 2014
- Misc. upstream discussions:
Octave: bug report and mailing list thread
groff: mailing list thread
GHC (Glasgow Haskell Compiler): #4012
Groovy: GROOVY-6308
coreboot: mailing list thread and patches
libreboot: Feature #16: Reproducible Builds
mono and dotnet: https://github.com/dotnet/roslyn/commit/04462c44e30dfa91267581abdb029f3102796486, https://github.com/mono/ikvm-fork/commit/cd4bed9dd6540c380177c5b9c72f4d020f1b138f and https://github.com/mono/mono/commit/a803d17038c0fcc8b40b12744801a87ceddb15ba
OpenWrt: mailing list thread
NetBSD has a MKREPRO build switch (if you have more infos about this, please add it here, eg link to documentation about it). Also see http://gnats.netbsd.org/50116, http://gnats.netbsd.org/50120 and http://gnats.netbsd.org/50121.
Mozilla projects: Firefox/Iceweasel Thunderbird/Icedove
More in the publicity and presentations sections.
Presentations
Reproducible Builds for Debian, Distributions devroom, FOSDEM’14, Video, Slides (Sources)
Reproducible Builds, a year later, DebConf14, Video, Slides (Sources)
Reproducible Builds, Moving Beyond Single Points of Failure for Software Distribution, 31st Chaos Communication Congress, Video, Slides
Stretching out for trustworthy reproducible builds, FOSDEM’15, Slides (Sources), Interview
Stretching out for trustworthy reproducible builds, Datengarten 52, CCC Berlin, Recordings, Slides (Sources)
Stretching out for trustworthy reproducible builds, Gulaschprogrammiernacht 15, Karlsruhe, Germany, Recordings, Slides (Sources)
Compilations reproductibles : permettre le lien entre un binaire et sa source, Pas Sage en Seine 2015, Paris, France, Video, Slides (Sources)
Reproducible builds in Debian and everywhere, Libre Software Meeting 2015, Beauvais, France, Video, Slides (Sources)
How to make your software build reproducibly, Chaos Communication Camp 2015, Mildenberg, Germany, Recordings, Slides, Notes for PDF Presenter Console, Slides+Script, Sources
Publicity
This section lists URLs, people, and dates for when other people have publicly expressed interest, or shared information about, the project.
Mike Perry, 2013-08-20: Deterministic Builds Part One: Cyberwar and Global Compromise
Jake Edge, 2013-08-21: Security software verifiability
Holger Levsen, 2014-09-26: Reproducible builds? I never did any - manually
Lunar, 2014-12-29: Reproducible builds against RC bugs
Lunar, 2015-01-15: 80%
Jake Edge, 2015-01-21: Lots of progress for Debian's reproducible builds (discussion on LWN, HN, reddit)
Kristian Kißling, 2015-01-27: Debian bringt Reproducible Builds voran in linux-magazin.de (Note: the article contains several misunderstandings.)
Hanno Böck, 2015-02-02: "Vertrauen durch reproduzierbare Build-Prozesse" pages 1, 2 in Golem.de (German)
Hans-Joachim Baader, 2015-02-16: Reproduzierbare Builds in Debian nähern sich in pro-linux.de (German)
2015-02-16: Debian Project Reaches 83% Reproducible Builds for Source Packages in softpedia.com.
Tannhausser, 2015-02-17: Debian mejora su seguridad con los reproducible builds in La Mirada del Replicante (Spanish)
Darren Pauli, 2015-02-23: Debian on track to prove binaries' origins in The Register
Arun, 2015-02-24: Debian working on reproducible builds in ?TuxDiary
2015-02-24: Debian está trabajando en compilaciones reproducibles in Detrás del pingüino (Spanish)
2015-02-27: Debian security initiative for reproducible builds reaches milestone in ?TechRepublic by James Sanders
2015-05-03: Reproducible builds: first week in Stretch cycle by Lunar
2015-05-06: http://www.linux-magazin.de/Ausgaben/2015/06/Reproducible-Builds in Linux-Magazin 06/2015 by Daniel Stender (in German)
2015-05-10: Reproducible builds on Debian for GSoC 2015 by dhole
2015-05-11: Reproducible builds: week 2 in Stretch cycle by Lunar
2015-05-17: Reproducible builds: week 3 in Stretch cycle by Lunar
2015-05-25: Reproducible builds: week 4 in Stretch cycle by Lunar
2015-06-01: Reproducible builds: week 5 in Stretch cycle by Lunar
2015-06-07: GSoC 2015 Week 2: Move forward reproducible builds by akira
2015-06-07: GSoC 2015 Week 2: Move forward reproducible builds by dhole
2015-06-08: Reproducible builds: week 6 in Stretch cycle by Lunar
2015-06-14: GSoC 2015 Week 3: Move forward reproducible builds by dhole
2015-06-15: GSoC 2015 Week 3: Move forward reproducible builds by akira
2015-06-15: Reproducible builds: week 7 in Stretch cycle by Lunar
2015-06-21: GSoC 2015 Week 4: Move forward reproducible builds by dhole
2015-06-21: GSoC 2015 Week 4: Move forward reproducible builds by akira
2015-06-22: Reproducible builds: week 8 in Stretch cycle by Lunar
2015-06-28: GSoC 2015 Week 5: Move forward reproducible builds by dhole
2015-06-29: GSoC 2015 Week 5: Move forward reproducible builds by akira
2015-06-29: Reproducible builds: week 9 in Stretch cycle by Lunar
2015-07-06: GSoC 2015 Week 6: Move forward reproducible builds by dhole
2015-07-06: GSoC 2015 Week 6: Move forward reproducible builds by akira
2015-07-07: Reproducible builds: week 10 in Stretch cycle by Lunar
2015-07-12: Reproducible builds: week 11 in Stretch cycle by Lunar
2015-07-13: GSoC 2015 Week 7: Move forward reproducible builds by dhole
2015-07-13: GSoC 2015 Week 7: Move forward reproducible builds by akira
2015-07-20: Reproducible builds: week 12 in Stretch cycle by Lunar
2015-07-20: GSoC 2015 Week 8: Move forward reproducible builds by dhole
2015-07-20: GSoC 2015 Week 8: Move forward reproducible builds by akira
2015-07-25: GSoC 2015 Week 9: Move forward reproducible builds by dhole
2015-07-26: Reproducible builds: week 13 in Stretch cycle by Lunar
2015-07-27: GSoC 2015 Week 9: Move forward reproducible builds by akira
2015-07-31: GSoC 2015 Week 10: Move forward reproducible builds by dhole
2015-08-03: GSoC 2015 Week 10: Move forward reproducible builds by akira
2015-08-03: Reproducible builds: week 14 in Stretch cycle by Lunar
2015-08-07: GSoC 2015 Week 11: Move forward reproducible builds by dhole
2015-08-03: GSoC 2015 Week 11: Move forward reproducible builds by akira
2015-08-10: Reproducible builds: week 15 in Stretch cycle by Lunar
2015-08-16: GSoC 2015 Week 12: Move forward reproducible builds by dhole
2015-08-16: Reproducible builds: week 16 in Stretch cycle by Lunar
2015-08-19: GSoC 2015 Week 12: Move forward reproducible builds by akira
Related projects
CARE monitors the execution of the specified command to create an archive that contains all the material required to re-execute it in the same context.
Further work
Having reproducible builds allows us to trust binary packages better, because it becomes easier to have:
- diversity of buildd location and jurisdiction - build packages in more than one location, including the developer's
- diversity of buildd hardware, in case of hardware bugs, or malicious implants - a mix of VMs, some real hardware, different CPU manufacturers, different date of manufacture and supplier
- diversity of people - multiple signatures on a .changes file
- diversity of kernels, explained below
Kernel packages
Special features of kernel packages (including bootloaders and hypervisors) - GRUB2, Xen, linux, kfreebsd...
- we put huge trust in them - kernels are the ultimate target of any rootkit, able to completely hide from userland
- a kernel image built for amd64, if the build system is portable and reproducible enough, will be the same whether built from linux-amd64 or kfreebsd-amd64
- or maybe from different kernel versions - for example, a jessie build chroot on a wheezy host system
Then we would be better protected from something that could affect many systems at once, such as a kernel vulnerability; or widespread infection by a rootkit, which now must be compatible with more than one type of kernel to go unnoticed.