It should be possible to reproduce, byte for byte, every build of every package in Debian.

This is an on-going project. To participate, we recommend you create an account on this wiki, subscribe to this page, and join the mailing list. IRC channel is #debian-reproducible on OFTC.

Other resources:


Why do we want reproducible builds?


Useful things you (yes, you!) can do

If you want to help with this, feel free to ping the mailing list or edit this wiki page.

Reported bugs

All bugs relevant to the reproducible builds project should use usertags with user

Current usertags in use:

affects a tool used by other package build systems
affects the whole Debian infrastructure or policies
time of build in recorded during the build process
build output varies with readdir() order
path of sources is recorded during the build process
username is recorded during the build process
hostname is recorded during the build process
uname output is recorded during the build process
some build aspects are dependent on (pseudo-)randomness
some build aspects are dependent on CPU features or computation speed
issues related to .buildinfo control files

Control commands to update the view on the BTS.

Lintian tags

Here's a list of relevant Lintian tags:

Archive wide rebuilds

UDD query to get a list of core packages (172 as of 2014-09-19):

  FROM packages
 WHERE release = 'sid'
   AND section != 'debian-installer'
   AND (   essential = 'yes'
        OR build_essential = 'yes'
        OR priority IN ('required', 'important', 'standard')
 ORDER BY source;

Reproducing builds

There are two sides to the problem: first we need to record the initial build environment, and then we need a way to set up the same environment.

Recording the environment

Information on a build will be recorded in a new control file with extension `.buildinfo`.

Reproduce the build environment

The srebuild program is a sbuild wrapper which finds a timestamp from containing all versions of the binary packages in a .buildinfo file and then carries out the build with the right versions installed.

See srebuild.

Reproducible builds automated on jenkins.d.n

Several jobs have been created to regularily test packages (from sid main) on jenkins.d.n. As a result there is the reproducible build overview of packages, which eventually will have results for all >21k sources packages in Debian.

The setup is explained in this blog post only, but this post is somewhat outdated by now and needs to be amended.

The basics for making packages build reproducible

Currently a plain sid environement is not enough to build packages reproducibly easily. Instead a few packages needs to be taken from the reproducible apt repository as explained above (at least debhelper >= 9.20141004~reproducible1 and dpkg >= 1.17.17~reproducible1 are needed). Besides this, these are the bascis for different types of packaging:

  1. use dh with compat=9

  2. use debhelper version 7 style packaging, add strip-nondeterminism to build-depends and make these modifications to debian/rules:

    1. add dh_strip_nondeterminism before dh_compress

    2. add dh_fixmtimes after dh_md5sums, right before dh_builddeb

    3. add dh_genbuildinfo at the end of the dh_ sequence, so after dh_builddeb

  3. use cdbs (needs cdbs >= 0.4.127~reproducible1)

Other types of packaging should be avoided and really be converted to dh.

With this, the basics should be covered and simple packages should build reproducible. See the next chapter for a discussion of common reproducibility issues and their solutions.

Identified problems, and possible solutions

Build systems tend to capture information about the environment that makes them produce different results accross different systems, despite having the same architecture and software installed.

Ideally, such variations should be fixed in the build system itself, but it might sometimes not be possible.

Files in data.tar.gz contains build paths

The build path is embedded in DWARF sections of ELF files among other types of file generated during builds. This has proven a real headache to fix after the path have been captured.

We are thus going to make mandatory to build package in a directory named like /usr/src/debian/hello-2.8-1.

As a bonus, this means that it will be easier to unpack packages in this canonical location for use with tools looking at the source code like gdb.

Files in data.tar.gz depends on readdir order

The build system needs to be patched to sort directory listings.


It looks like python-epydoc will produce links in an order that depends on the readdir order. This needs to be investigated.

Files in data.tar.gz varies with the locale

Builds should be made with LC_ALL=C.UTF-8.

It's quite unpractical to force such value in debian/rules and there is actually no reason this should not be the default.


Files in data.tar.gz contains hostname, uname output, username

We could write a LD_PRELOAD library that could answers consistent results for several system calls on the same model as libfaketime. Bdale suggested we call it liblietome.

But we can also consider that no build systems should capture or produce different builds depending on such information and fix them.

Files in data.tar contains timestamps

Recommended solutions in order of preference:

Specific issues:

Now fixed:

POSIX says that the file mode on symlinks is undefined, so this ends up being system dependent behavior. On Linux the umask is ignored when creating symlinks, but on other systems such as kFreeBSD or Hurd the umask is honored, which can produce varying file modes. This is at least a problem for architecture independent packages which are built on different operating systems.

A way to always get the same file mode for symlinks is to set umask to 0 before creating them, and restore the previous umask afterwards, but this might be unfeasible in general.

Generation of files in data.tar depends on (pseudo-)randomness

Now fixed:

Members of control.tar and data.tar have varying mtimes

dpkg-deb will record the mtime of files it packs in control.tar and data.tar. This is bad as most of these files are generated during the build process and will thus change with each build.

759886 contains a patch against debhelper that adds a new dh_fixmtimes helper that will ensure that the mtime of any file created after the date of latest changelog entry will be set to the date of the latest changelog entry.

{data,control}.tar.{gz,xz,bz2} will store files in readdir order

This is dependent on an accident of filesystem layout at build time, so it would sometimes not be reproducible.

We should probably fix this in dpkg by sorting the contents of the tar files.

Changes are discussed in 719845. Test case patch for pkg-tests. Patches that fork `sort` to get a stable order for files in control and data archives.

Files generated by debhelper depend on readdir order

At least the shlibs files generated by dh_makeshlibs depends on the order returned by find(1). There might be other debhelper programs with the same issue.

Randomness in control file

Now fixed:

.deb ar-archive header contains a timestamp

.deb are ar-archives. The header currently contains the “current time”.

759999 contains patches against dpkg that will preset the timestamp to the time of the latest entry of debian/changelog when a package is built using dpkg-buildpackage.

building the kernel

Uncoordinated experiments, see SameKernel.

XSLT generate-id() is non-deterministic

XSLT's generate-id() function is explicitly allowed by the XSLT spec to be non-deterministic, and is frequently implemented using memory addresses of XML nodes, which are of course non-deterministic thanks to ASLR. Consequentially, files that are generated by XSLT (typically documentation) that include the result of generate-id() in their output do not build deterministically.

piuparts, which uses xmlto to generate documentation, is affected by this.

PNG files contain timestamps

If the PNG is being generated by ?ImageMagick, some options can be passed to convert to inhibit inclusion of the timestamps. Please note that the options have to be passed just before the output filename and composite only supports the time chunk removal, for example:

convert -scale 32x32 input.png +set date:create +set date:modify -define png:exclude-chunk=time output.png
composite -compose CopyOpacity mask.png input.png -define png:exclude-chunk=time output.png

Custom build environment

We maintain a set of modifications to the toolchain to perform our experiments. Commit notifications are sent to a dedicated mailing list.

Our modified packages can be found in the following APT archive, which is signed by 49B6 5747 36D0 B637 CC37 01EA 5DB7 CA67 EA59 A31F:

deb ./
deb-src ./


The pu/reproducible_builds debhelper branch in the reproducible project contains several fixes and calls dh_strip_nondeterminism (see below) will be called before dh_compress in dh. See the changelog for details.


The pu/reproducible_builds dpkg branch in the reproducible repository makes:

  1. file order deterministic in control and data part of the .deb,
  2. uses a single timestamp for .deb ar members
  3. preset the aforementioned timestamp to the latest changelog entry
  4. add -Wdate-time as part of CPPFLAGS in dpkg-buildflags

  5. add support for .buildinfo files


strip-nondeterminism is a post-processing tool that will normalize various file types. dh_strip_nondeterminism will be run by debhelper at the end of the build process.


The pu/reproducible_builds cdbs branch in the reproducible project contains a fix for 764478 which makes cdbs call the newly introduced dh_strip_nondeterminism commands.


The pu/reproducible_builds javatools branch in the reproducible repository changes javahelper to call jh_installlibs after dh_install. See 764988.


discount needs a patch to produce stable output of email addresses. See 762622 and the pu/reproducible_builds branch.


The pu/reproducible_builds ghc branch in the reproducible repository changes ghc to not include timestamps in interface hashes. See 769893


python-setuptools needs a patch to write names top_level.txt in a stable order. See 773969


r-base needs a patch to remove build time and username written while building packages. See 774031


fontforge needs a patch to propagate creation and modification times from source file. See 774148 and the pu/reproducible_builds branch.


python-qt4 needs a patch to remove timestamps from generated files. See 774509


pyqt5 needs a patch to remove timestamps from generated files. See 774510


libxslt needs a patch to make generate-id() return identifiers in a deterministic way. See the pu/reproducible_builds branch.

Usage example

If you have a pbuilder already setup, it's fairly easy to setup an environment with the custom toolchain:

sudo cp /var/cache/pbuilder/base.tgz /var/cache/pbuilder/base-reproducible.tgz
sudo pbuilder --login --save-after-exec --basetgz /var/cache/pbuilder/base-reproducible.tgz
echo 'deb ./' > /etc/apt/sources.list.d/reproducible.list
apt-get install busybox
busybox wget -O- | apt-key add -
apt-get purge busybox
apt-key fingerprint | grep '49B6 5747 36D0 B637 CC37  01EA 5DB7 CA67 EA59 A31F' || echo 'Something is wrong'
apt-get update
apt-get upgrade
exit 0

Once that's done, one can use the prebuilder script or manually test a package through:

apt-get source --download-only acl
sudo DEB_BUILD_OPTIONS=nocheck pbuilder --build --debbuildopts '-b' --basetgz /var/cache/pbuilder/base-reproducible.tgz acl_*.dsc
mkdir b1 b2
dcmd cp /var/cache/pbuilder/result/acl_*.changes b1
sudo dcmd rm /var/cache/pbuilder/result/acl_*.changes
sudo DEB_BUILD_OPTIONS=nocheck pbuilder --build --debbuildopts '-b' --basetgz /var/cache/pbuilder/base-reproducible.tgz acl_*.dsc
dcmd cp /var/cache/pbuilder/result/acl_*.changes b2
sudo dcmd rm /var/cache/pbuilder/result/acl_*.changes

debbindiff (available in Debian main) is useful to check the result:

debbindiff --html $output_file b1/*.changes b2/*.changes

Adding a package to the APT archive


  1. Import the private signing key to your keyring, if you haven't already: gpg --import /home/groups/reproducible/private/reproducible-private.gpg

  2. Place the package files in /home/groups/reproducible/htdocs/debian/

  3. Run make from that directory.

Countering "trusting trust" issues by building Perl without Perl

Ken Thompson's classic "trusting trust" essay outlines an attack vector for embedding code in a compiler that will then be automatically added to anything that the compiler compiles. Since the main approach for making reproducible builds, using debhelper tools written in Perl, relies heavily on Perl, Debian's Perl package should be reproducibly buildable without any Perl at all. That will ensure the "trusting trust" cycle is broken. The current Perl packaging does not use debhelper, so that's a good start. Also, it looks like the Perl build system is written in make.

bash script to compare two package builds

Usage: ./diffp r1/hello_2.8-4_amd64.changes r2/hello_2.8-4_amd64

The script is available in the misc.git repository.

bash script to analyze images

Deterministic images (raw images, qcow2 images, iso's) are the next logical evolution. There is a analyze_image bash script that creates sha512 hashes of all files included within an image, access rights, symlinks, parition table, bootloader and more. Doing this with two images that should match and comparing the reports the script creates can help to identify sources of non-determinism in images.

See also:

Does not have iso support yet. The autor (Patrick Schleizer) is interested to generalize the script for more generic, Debian use cases.

Further work

Having reproducible builds allows us to trust binary packages better, because it becomes easier to have:

Kernel packages

Special features of kernel packages (including bootloaders and hypervisors) - GRUB2, Xen, linux, kfreebsd...

Then we would be better protected from something that could affect many systems at once, such as a kernel vulnerability; or widespread infection by a rootkit, which now must be compatible with more than one type of kernel to go unnoticed.




This section lists URLs, people, and dates for when other people have publicly expressed interest, or shared information about, the project.

Related projects