Differences between revisions 57 and 58
Revision 57 as of 2016-11-16 16:37:03
Size: 11874
Editor: lamby
Revision 58 as of 2017-11-09 12:35:14
Size: 124
Editor: lamby
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:

= Stay in touch =

 * Subscribe to the [[https://lists.alioth.debian.org/mailman/listinfo/reproducible-builds|reproducible-builds@lists.alioth.debian.org mailing list]]
  * and/or other [[https://lists.reproducible-builds.org/|reproducible builds]] oriented lists.
 * Join the [[https://webchat.oftc.net/?channels=#debian-reproducible|#debian-reproducible IRC channel on OFTC]]
  * and/or #reproducible-builds too.
 * Add `ReproducibleBuilds.*` to your [[https://wiki.debian.org/ReproducibleBuilds?action=userprefs&sub=notification|wiki notifications]].
 * Subscribe to [[https://lists.alioth.debian.org/mailman/listinfo/reproducible-commits|commit notifications]].

= Task suggestions =

 * If you maintain a package for Debian, you can make sure that your package uses a [[http://anonscm.debian.org/cgit/debhelper/debhelper.git/tree/dh#n77|modern debhelper style]] (e.g. one-liner `debian/rules` with overrides as needed). We aim to fix many causes of non-deterministic builds in the debhelper suite directly, so packages that use debhelper will be much easier to make reproducible with just an upgrade of the toolchain.
 * [[#Inventorying_issues|Inventory issues]] found by the continuous integration platform.
 * [[#Fixing_issues|Fix known reproducibility issues]]. See the [[https://reproducible.debian.net/index_issues.html|inventory of identified issues]].
 * Improve our common tools: DebianPts:diffoscope, DebianPts:strip-nondeterminism, DebianPts:disorderfs.
 * Redesign [[https://reproducible.debian.net/|reproducible.debian.net]] status pages using a CSS toolkit like Bootstrap.
 * Enhance DebianPts:dak [[Bug:763822|support for .buildinfo]].
 * Research how to run rebuilds on ''buildd''s.
 * Research on how change dak to only accept packages after multiple matching builds.
 * Hack binNMU infrastructure (dak?) so .dsc for binNMUs are kept in the archive instead of being thrown away.
 * We need a [[ReproducibleBuilds/Logo|logo]]! Please send suggestions and/or proposals to the mailing list.

To get help, feel free to ask on the IRC channel or the mailing list. We want to be friendly, supportive, and have fun experimenting together.

= How to report bugs =

{{{#!wiki note
[[http://bugs.debian.org/cgi-bin/pkgreport.cgi?usertag=reproducible-builds@lists.alioth.debian.org|Overview of all bug reports concerning reproducible builds]]

All bugs relevant to the reproducible builds project should use [[bugs.debian.org/usertags|usertags]] with user `reproducible-builds@lists.alioth.debian.org`. Also use `X-Debbugs-Cc` to notify the list, but please use our reproducible-bugs@lists.alioth.debian.org list for this header.

To usertag a bug after it has been submitted use:

bts user reproducible-builds@lists.alioth.debian.org . usertag XXXXXX + timestamps toolchain

Current usertags in use:

 toolchain:: affects a tool used by other package build systems
 infrastructure:: affects the whole Debian infrastructure or policies
 timestamps:: time of build in recorded during the build process
 fileordering:: build output varies with readdir() order
 buildpath:: path of sources is recorded during the build process
 username:: username is recorded during the build process
 hostname:: hostname is recorded during the build process
 uname:: uname output is recorded during the build process
 environment:: environment variables are recorded during the build process
 randomness:: some build aspects are dependent on (pseudo-)randomness
 cpu:: some build aspects are dependent on CPU features or computation speed
 signatures:: uses a cryptographic signatures as part of the build process
 umask:: permissions depend on current umask
 buildinfo:: issues related to .buildinfo control files
 ftbfs:: fails to build from source
 locale:: varying locales lead to differing behavior (e.g. sorting)

''[[../UserCategory|Control commands to update the view on the BTS]].''

Example email to submit a patch:

From: J. Random Hacker <jrhacker@example.org>
To: submit@bugs.debian.org
Subject: <PACKAGE>: please make the build reproducible (timestamps, fileordering)

Source: <PACKAGE>
Version: <VERSION>
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps fileordering
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org


While working on the “reproducible builds” effort [1], we have noticed
that <PACKAGE> could not be built reproducibly.

The attached patch removes extra timestamps from the build system and
ensure a stable file order when creating the source archive. Once applied,
<PACKAGE> can be built reproducibly in our current experimental framework.

 [1]: https://wiki.debian.org/ReproducibleBuilds

= Inventorying issues =

The easiest way to find issues is to examine the list of [[https://reproducible.debian.net/index_FTBR.html|packages failing to build reproducibly]] as found by continuous integration. The first packages in the list are the one who have been tried most recently.

Notes about packages are kept in the [[https://anonscm.debian.org/cgit/reproducible/notes.git|notes]] Git repository in `packages.yml`.
The list of [[https://reproducible.debian.net/index_issues.html|known common issues]] is kept in the `issues.yml` file.

The page for a given package should open on the DebianPts:diffoscope output. Read the list of known issues to get an idea of what you may found. Here are some more advices:

 * When a binary has mismatching mtimes for files in `control.tar.gz`, it means that they are [[https://reproducible.debian.net/issues/not_using_dh_builddeb_issue.html|not adjusted before creating the binary package]].
 * [[https://reproducible.debian.net/issues/timestamps_in_gzip_headers_issue.html|Timestamps in gzip headers]] are a no-brainer.
 * When there's a mismatching ''Build ID'' in an executable, it means a variation happens during the compilation. Investigation can be done using [[https://sources.debian.net/|sources.debian.net]] (see link at the top). First step should be a search for the `__(DATE|TIME|TIMESTAMP)__` [[https://reproducible.debian.net/issues/timestamps_from_cpp_macros_issue.html|macros]] using [[https://codesearch.debian.net/|codesearch]]. Otherwise, try to locate calls to `date` in `configure.ac`, `Makefile.am`, etc.

The [[https://anonscm.debian.org/cgit/reproducible/misc.git/tree/clean-notes|clean-notes]] script in the `misc` repository will detect outdated notes and re-order packages by alphabetical order. It should be run before committing changes to the `notes` repository.

= Fixing issues =

Fixing reproducibility issues falls into two categories: either the problem is specific to a single package or the cause is the output of another package (then referenced as “toolchain” package).

== Fixing a single package ==

The usual steps are:

 1. Use `debcheckout` or `apt-get source` to retrieve the source code.
 2. Do the changes. With packages using the `3.0 (quilt)` format, `dpkg-source --commit` can be useful.
 3. Update `debian/changelog`. New version is usually original version with `.0~reproducible1`.
 4. Use `dpkg-buildpackage -S` to create source package.
 5. Use the [[ReproducibleBuilds/ExperimentalToolchain#Usage_example|prebuilder]] script to test reproducibility. If the package is not reproducible, examine DebianPts:diffoscope output `logs/PACKAGE.diffoscope.html` or compare build logs `logs/PACKAGE.build1` and `logs/PACKAGE.build2`, then repeat from step 2 unless the issue comes from another package. In that case, see about “toolchain” packages below.
 6. Use `debdiff` or `git format-patch` to create patches.
 7. [[ReproducibleBuilds/Contribute#How_to_report_bugs|Create a new bug report]], and don't forget to attach the patch!
 8. Add an entry or reference the bug in `packages.yml` in `notes.git`.

== Fixing a toolchain package ==

Fixing an issue in a package that affects the reproducibility of other packages requires some more steps, but the general process is the same:

 1. Use `debcheckout` or `apt-get source` to retrieve the source code.
 2. Do the changes. With packages using the `3.0 (quilt)` format, `dpkg-source --commit` can be useful.
 3. Update `debian/changelog`. New version is usually original version with `.0~reproducible1`.
 4. Use `pdebuild` or `gbp buildpackage` to build the package.
 5. Backup `base-reproducible.tgz`.
 6. Use `pbuilder --login --save-after-exec --basetgz base-reproducible.tgz` to install the newly built package.
 7. Test a package affected with `prebuilder`. If the issue is still not fixed, repeat from step 2.
 8. If the package is in Git, use SSH to login on [[https://alioth.debian.org/projects/reproducible/|alioth.debian.org]]. Go to `/git/reproducible`. Use `./setup-repository` to create a new repository. Push your changes to a (rebasable) `pu/reproducible_builds` branch.
 9. Subscribe to the `upload-source` notification for the package on the [[https://packages.qa.debian.org/|Package Tracking System]]. This is needed so you don't forget to update the custom package when a new version hits the archive.
 10. [[../ExperimentalToolchain#Adding_a_package_to_the_APT_archive|Upload]] the package to the reproducible APT repository.
 11. Document the changes on the [[ReproducibleBuilds/ExperimentalToolchain#Modified_packages|wiki]].
 12. Reference the bug in `issues.yml` in `notes.git` and on the wiki page about the issue if there's one.
 13. Everybody in the reproducible builds team can schedule source packages to be rebuilt by running the script `reschedule.sh` in `/srv/home/groups/reproducible/` on alioth. If you are not part of the team you can find somebody in the #debian-reproducible IRC channel.
 14. If the changes don't break anything, [[ReproducibleBuilds/Contribute#How_to_report_bugs|create a new bug report]]. Don't forget to attach patches and to use the `toolchain` usertag.

= Working on the continuous integration platform =

Several jobs have been created to regularly test packages (from sid main) on [[https://jenkins.debian.net|jenkins.debian.net]]. As a result there is the [[https://reproducible.debian.net|reproducible build overview of packages]].

The setup is explained in [[http://layer-acht.org/thinking/blog/20140925-reproducible-builds/|this blog post]] only, but this post is somewhat outdated by now and needs to be amended.

See the various `reproducible_*` scripts in the [[http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/|Jenkins Git repository]].

= Working on installation media or live systems =

Having installation and live systems which can be built reproducibly would also be great. There is an `analyze_image` bash script that creates sha512 hashes of all files included within an image, access rights, symlinks, partition table, bootloader and more. Doing this with two images that should match and comparing the reports the script creates can help to identify sources of non-determinism in images.

See also:

  * [[ReproducibleInstalls|Reproducible installs]]
  * [[http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20131209/000009.html|Announcing Whonix's First Implementation of Verifiable Builds]]
  * [[https://www.whonix.org/wiki/Verifiable_Builds|Whonix Verifiable Builds]]
  * [[https://github.com/adrelanos/Whonix/blob/master/help-steps/analyze_image|analyze_image bash script]]: Does not have iso support yet. The author (Patrick Schleizer) is interested to generalize the script for more generic, Debian use cases.
  * [[https://tails.boum.org/blueprint/reproducible_builds/|Tails reproducible builds blueprint]]
  * [[https://github.com/lamby/debootstrap/commits/pu/source-date-epoch|reproducible debootstrap]]
[[https://reproducible-builds.org/contribute/|This page has been moved to our more-general and Debian-unspecific website]]