Got a spare moment? Please migrate this to our new webpages

Introduction

A source package should build byte-for-byte identical products when rebuilt in the same environment. The environment is defined as:

We have seen variations related to the time of the build, the order of files on the filesystem, the current user, the system hostname, the uname output, (pseudo-)-randomness, and the CPU features or load. Such aspects must not be captured by the build process to make a package reproducible.

Testing procedure

Newer method

  1. Set up a schroot as described in sbuild#Create_the_chroot

  2. Avoid sudo password prompts. (For more information see Notes on variations and Avoid sudo(1) password prompts in reprotest README)

    • $ reprotest --print-sudoers \
          --variations=user_group.available+=guest-builder,domain_host.use_sudo=1 \
          | sudo EDITOR=tee visudo -f /etc/sudoers.d/local-reprotest
      $ echo "/etc/sudoers.d/local-reprotest /etc/sudoers.d/local-reprotest none bind 0 0" \
          | sudo tee -a /srv/chroot/unstable-amd64-sbuild/etc/fstab
  3. Check package reproducibility
    • $ sudo reprotest --vary=-build_path,domain_host.use_sudo=1 --auto-build XXX.dsc -- schroot unstable-amd64-sbuild

      (or path/to/unpacked/pkgdir instead of XXX.dsc)

    • Note: for the time being this omits certain variations compared to our test infrastructure, but we're working on it. Specifically, these are: domain/host, bash-vs-dash, num_cpus, linux-namespace. If you need to test these, then either wait or use the "older method" below.
  4. If your package is unreproducible (after 2 builds), --auto-build will run further builds with different variations to try to identify which ones are causing the non-determinism.

Older method, deprecated

The easiest way to get a test environment for now is to use pbuilder. Setup a build chroot with the experimental toolchain. (We use the plain Debian toolchain nowadays, since 2016 probably.)

Then clone the reproducible-misc.git repository.)--

Hacking packages then looks like:

  1. apt-get source foo

  2. cd foo*

  3. dch -v "$(dpkg-parsechangelog -S Version).0~reproducible1"

  4. Fix reproducibility issues.
  5. dpkg-buildpackage -S

  6. cd ../../misc/prebuilder

  7. dcmd cp ../foo*reproducible1*.dsc .

  8. ./rebuild.sh foo

Look for a fat big “reproducible” banner if test is successful. Otherwise, look for logs and diffoscope output in the logs directory.

First steps to make a package reproducible

Currently a plain sid environment is not enough to build packages reproducibly. An experimental toolchain is available while changes are waiting to be integrated in the main archive.

The first steps needed to make a package build reproducibly depends on the packaging style. With this, the basics should be covered and simple packages should build reproducible. See the next chapter for a discussion of common reproducibility issues and their solutions.

dh

No changes required.

cdbs

No changes required.

Explicit calls to dh_*

Please consider migrating the package to dh.

If the packages is not reproducible, start by adding dh_strip_nondeterminism before dh_compress.

Do It Yourself

Please consider migrating the package to dh.

Usual fixes are:

Identified problems, and possible solutions

Once applied the basic recommendations, a package might still not build reproducibly.

diffoscope has been written to help one understanding the sources of unreproducibility. It compares two files and output their differences, recursively unpacking archives or using transformation tool to generates meaningful diffs. In the Debian context, it's usually called on the .changes files of two different builds.

What follows is a list of common issues and known solutions. For reference, we keep documentation on old issues that should not happen with the current experimental toolchain.

Files in data.tar depends on filesystem order

The build system needs to be patched to sort directory listings.

Example patch

Specific issues:

Files in data.tar vary with the locale

The build system should be patched to add LC_ALL=C.UTF-8 to the environment.

Files in data.tar vary with the timezone

The timezone used while building can also have an effect on reproducibility.

Specific issues:

Files in data.tar contains hostname, uname output, username

Remove places where the information is recorded. If that's not possible, use dummy values. Example

Specific issues:

Files in data.tar contain timestamps

Recommended solutions in order of preference:

Specific issues:

Fixed in Debian, needs to be upstreamed:

Files in data.tar depends on (pseudo-)randomness

Permissions in data.tar depends on umask

The permissions of files shipped by Debian packages should not change with the value of the file creation mask (umask).

Example diffoscope

Easy solution: use dh or dh_fixperms. Otherwise, care to fix the permissions manually.

POSIX says that the file mode on symlinks is undefined, so this ends up being system dependent behavior. On Linux the umask is ignored when creating symlinks, but on other systems such as kFreeBSD or Hurd the umask is honored, which can produce varying file modes. This is at least a problem for architecture independent packages which are built on different operating systems.

A way to always get the same file mode for symlinks is to set umask to 0 before creating them, and restore the previous umask afterwards, but this might be unfeasible in general.

Members of control.tar and data.tar have varying mtimes

dpkg-deb will record the mtime of files it packs in control.tar and data.tar. Files generated during the build process will get a different mtime with each build. Best solution is to change their mtime to the date of the latest debian/changelog entry.

debhelper will take care of adjusting mtimes (759886).

Example patch for custom packaging style. Known affected packages.

Installed-Size differs

Installed-Size can be different depending See 650077.

Guillem Jover wrote a patch for dpkg solving the issue.

RPATH tags differ

RPATH is an optional tag in an ELF file to indicate additional paths for the linker to use. If an absolute path is used, the build will be non-reproducible. (The tags live in the .dynstr section of the ELF file; the section contents can be dumped with objdump -j .dynstr -s ELFFILE). The chrpath utility can be used to strip out the RPATH tags. This works, but the tag text remains in the ELF file, so the build remains non-reproducible. Editing ELF files after they have been created is difficult, so it is recommended to either

Lintian tags

Some Lintian tags can help identify issues that could prevent a package from being reproducible: