It should be possible to reproduce, byte for byte, every build of every package in Debian.

This is an on-going project. To participate, we recommend you create an account on this wiki, subscribe to this page, and join the reproducible-builds@lists.alioth.debian.org mailing list.

Alioth project

Drivers

Why do we want reproducible builds?


Status

Useful things you (yes, you!) can do

If you want to help with this, feel free to ping the mailing list or edit this wiki page.

Reported bugs

All bugs relevant to the reproducible builds project should use usertags with user reproducible-builds@lists.alioth.debian.org.

Current usertags in use:

toolchain
affects a tool used by other package build systems
infrastructure
affects the whole Debian infrastructure or policies
timestamps
time of build in recorded during the build process
fileordering
build output varies with readdir() order
buildpath
path of sources is recorded during the build process

Archive wide rebuilds


Reproducing builds

There are two sides to the problem: first we need to record the initial build environment, and then we need a way to set up the same environment.

Recording the environment

The right place to record the build environment is the .changes file. Rationale: it lists the checksums of the build products and is signed by either the maintainer or the buildd operator.

To add a field to the .changes file, we need to call dpkg-buildpackage using something like:

dpkg-buildpackage --changes-option="-DBuild-Environment=$(
COLUMNS=999 | dpkg -l | awk '
            /^ii/ { ORS=", "; print $2 " (= " $3 ")" }' |
        sed -e 's/, $//'
)"

The idea is not new, see 138409. The above could eventually be integrated in dpkg proper if our experiments turn successful.

The dh-buildinfo script already captures the build environment quite nicely. It currently encode the information about the build environment in a buildinfo_$ARCH.gz in the doc directory. For our goal, we believe that such information should not be recorded in the binary package itself.

Lunar wrote on 2013-09-08 to Yann Dirson to ask his opinion. Yann agrees that what buildinfo produces should not be in the .deb file but rather a separate file. He's advocating for having an additional file as part as an upload rather than adding fields in .changes. He also agree that what dh-buildinfo produces should actually be done at the dpkg level.

In any cases, we should simply re-use or adapt dh-buildinfo, and turn it into dpkg-buildinfo. Either it should produce an extra file that is bundled together with a .changes or add some extra fields directly in .changes.

(See 719854 for the first attempt which tried using XC- field in debian/control.)

Reproduce the build environment

Actions:

Ruby script that generates URL to .deb on snapshot.debian.org from a list of binary packages and their respective version: http://people.debian.org/~paulproteus/lunar-verify-script.rb

Here's another potential piece of the puzzle. The following script will convert a RFC822 date (as found in a .changes) to the URL of the last known archive state recorded by snapshot.debian.org. This might be useful to debootstrap the proper chroot before installing packages…

require 'date'
require 'uri'
require 'net/http'
require 'nokogiri'

changes_date = 'Mon, 30 Jan 2012 12:52:28 +0100'

build_date = DateTime.rfc822(changes_date)
url = "http://snapshot.debian.org/archive/debian/?year=#{build_date.year}&month=#{build_date.month}"
response = Net::HTTP.get_response(URI.parse(url))

run = nil
doc = Nokogiri::HTML(response.body)
doc.css('p a').each do |link|
  date = DateTime.parse(link.content)
  break if date >= build_date
  run = link['href']
end
puts "http://snapshot.debian.org/archive/debian/#{run}"

Note : it would probably be a lot better of adding a new query to the machine interface of snapshot.d.o instead of parsing HTML.


Different problems, and their solutions

Build systems tend to capture information about the environment that makes them produce different results accross different systems, despite having the same architecture and software installed.

Ideally, such variations should be fixed in the build system itself, but it might sometimes not be possible.

Non-problems

Files in data.tar.gz contains build paths

These should really be patched out in one way or another. This is not useful information and can actually hide real bugs.

DWARF data

The build path is embedded in DWARF sections of ELF files. Like Fedora, we should replace actual build paths to a canonical location in /usr/src.

One option is to do so by adding two CFLAGS for GCC:

  1. -fdebug-prefix-map to replace the build path by a predetermined path.

  2. -gno-record-gcc-switches to prevent the previous option to be recorded in the debug file (as it changes with the build path).

Both are documented in GCC's manual.

We can pass both options using the dpkg-buildflags mechanism. Lunar's branch has a patch for that.

Problems with that approach:

Another option is to use debugedit. Currently, it needs to be used together with -fno-merge-debug-strings because the hashtable used when strings are merged will output strings in a different order depending on the initial build path.

Files in data.tar.gz depends on readdir order

The build system needs to be patched to sort directory listings.

Epydoc

It looks like python-epydoc will produce links in an order that depends on the readdir order. This needs to be investigated.

Files in data.tar.gz varies with the locale

Builds should be made with LC_ALL=C.UTF-8.

It's quite unpractical to force such value in debian/rules and there is actually no reason this should not be the default.

Actions:

Files in data.tar.gz contains hostname, uname output, username

Actions:

Files in data.tar.gz contains timestamps

For the worse cases, we could record the calls to gettimeofday() on the first build and have something like libfaketime replay them on rebuilds.

.a files

.a files are ar archive. GNU ar has a deterministic mode which will use zero for UIDs, GIDs, timestamps, and use consistent file modes for all files.

binutils can be built with --enable-deterministic-archives to make it the default.

jar files

See ?ReproducibleBuilds/TimestampInJarFiles.

Epydoc

python-epydoc will add timestamps to the HTML file it produces. This needs to be fixed.

javadoc

javadoc will add timestamps to the HTML file it produces. This needs to be fixed.

PHP PEAR registry files

See ?ReproducibleBuilds/TimestampInPhpRegistryFiles.

Manpages

(TODO)

PDF files

(TODO)

Random variation

dh_ocaml md5sums files

See ReproducibleBuilds/RandomOrderInOcamlMd5sums.

Members of control.tar have varying mtime

We can fix this by giving tar the --mtime= option with the date of the last debian/changelog entry or a similar fixed point in time. Change to be done in dpkg-deb/build.c:do_build() around line 462.

Lunar's branch use a single timestamp for all mtimes of tar members and allow to preset it during rebuilds, see below. Except this uncovered a new issue coming straight from tar itself.

For more consistency in tarfiles, the owner and group can be set using --owner=1000 --group=1000 --numeric-owner.

{data,control}.tar.{gz,xz,bz2} does not have timestamps

{data,control}.tar.{gz,xz,bz2} will store files in readdir order

This is dependent on an accident of filesystem layout at build time, so it would sometimes not be reproducible.

We should probably fix this in dpkg by sorting the contents of the tar files.

Changes are discussed in 719845. Test case patch for pkg-tests. Patches that fork `sort` to get a stable order for files in control and data archives.

.deb ar-archive header contains a timestamp

.deb are ar-archives. The header currently contains the “current time”. It is written by dpkg at line line 103 of lib/dpkg/ar.c.

Guillem said he would rather keep this.

Lunar's branch use a single timestamp for all ar headers and allow to preset it during rebuilds, see below.

dh-buildinfo

dh-buildinfo used to encode a timestamp in the gzip file it produces, and also did not output the package list in the same order on every run. A patch had been submitted in 722186 and the packages should be good with version 0.10.

building the kernel

See dedicated page: SameKernel.


Custom build environment

We maintain a set of modifications to the toolchain to perform our experiments.

debhelper

The pu/reproducible_builds debhelper branch in the reproducible repository makes dh_strip call debugedit to adjust the source path embedded in debug objects.

dpkg

The pu/reproducible_builds dpkg branch in the reproducible repository makes:

  1. file order deterministic in control and data part of the .deb,
  2. uses a single timestamp for .deb mtimes and allows to preset the timestamp,
  3. adjust dpkg-buildflags to pass -fno-merge-debug-strings in CFLAGS and CXXFLAGS

binutils

binutils has been rebuilt with the --enable-deterministic-archives flag passed to ./configure.

Usage example

$ apt-get source hello
$ cd hello-2.8
$ dpkg-buildpackage
[…]
$ cp ../hello_2.8-4_amd64.deb ../hello_2.8-4_amd64.deb.orig
$ DEB_BUILD_TIMESTAMP=$(date +%s -d"$(sed -n -e 's/^Date: //p' ../hello_2.8-4_amd64.changes)") dpkg-buildpackage
[…]
$ sha256sum ../hello_2.8-4_amd64.deb ../hello_2.8-4_amd64.deb.orig
1e944abfceac7e593f6706da971e0444e5cee9aab680de5292d52661940ee9c4  ../hello_2.8-4_amd64.deb
1e944abfceac7e593f6706da971e0444e5cee9aab680de5292d52661940ee9c4  ../hello_2.8-4_amd64.deb.orig

Success!


bash script to compare two package builds

Usage: ./diffp r1/hello_2.8-4_amd64.changes r2/hello_2.8-4_amd64

The script is available in the misc.git repository.


How to build a deb using faketime

sudo apt-get install faketime
echo > /tmp/fakeroot-faketime << EOF
faketime "2013-08-15T11:02:00" fakeroot "$@"
EOF
chmod a+x /tmp/fakeroot-faketime
dpkg-buildpackage -r/tmp/fakeroot-faketime

Note that this retians *one* timestamp, which is the timestamp of the 'ar' container of the *.deb. To erase that, somehow regenerate the package within the fakeroot-faketime environment by using dpkg-deb to unpack it, then dpkg-deb to repack it.

Note also that this is a total hack and not something I (AsheeshLaroia) think it makes sense to do on the Debian build daemons. In particular, some programs (e.g., gpg) hang forever when time does not advance.

Upstream changes may solve the problems we face with faketime 0.9.1. (rbalint) Faketime upstream has been improved to advance time linearly at a preset pace per each time() call and save/load timstamps. We could try rebuilding many packages saving timestamps in the first build and replaying them in successive builds. For example gnupg 1.4.14-1 builds fine:

NO_FAKE_STAT=1  ~/projects/libfaketime.git/src/faketime -f '+0 i0.01' dpkg-buildpackage -rfakeroot -us -uc


Further work

Having reproducible builds allows us to trust binary packages better, because it becomes easier to have:

Kernel packages

Special features of kernel packages (including bootloaders and hypervisors) - GRUB2, Xen, linux, kfreebsd...

Then we would be better protected from something that could affect many systems at once, such as a kernel vulnerability; or widespread infection by a rootkit, which now must be compatible with more than one type of kernel to go unnoticed.


References

Presentations

Publicity

This section lists URLs, people, and dates for when other people have publicly expressed interest, or shared information about, the project.