Differences between revisions 43 and 44
Revision 43 as of 2015-07-22 10:48:20
Size: 11945
Editor: ?MariaMarin
Comment: Modified my last entry to be less confusing
Revision 44 as of 2015-07-22 11:11:05
Size: 11958
Editor: HolgerLevsen
Comment: mention Bug:792201 again
Deletions are marked like this. Additions are marked like this.
Line 86: Line 86:
A version of DebianPts:doxygen which contains a patch to honour SOURCE_DATE_EPOCH has been uploaded to the reproducible builds git repository. See [[https://bugzilla.gnome.org/show_bug.cgi?id=751984|Upstream Bug]] for patch and discussion. A version of DebianPts:doxygen which contains a patch to honour SOURCE_DATE_EPOCH (Bug:792201) has been uploaded to the reproducible builds git repository. See [[https://bugzilla.gnome.org/show_bug.cgi?id=751984|Upstream Bug]] for patch and discussion.

Current experiments are done in Debian unstable. We maintain a set of modified packages needed to make other packages reproducible. These packages allow to conduct experiments and test modifications locally or in our continuous integration platform. The goal is to get our changes accepted by the respective maintainers after proving them bug-free and fruitful.

APT repository

Our modified packages can be found in the following APT archive, which is signed by 49B6 5747 36D0 B637 CC37 01EA 5DB7 CA67 EA59 A31F:

deb http://reproducible.alioth.debian.org/debian/ ./
deb-src http://reproducible.alioth.debian.org/debian/ ./

Git repositories

Several Git repositories have been created on Alioth. Commit notifications are sent to a dedicated mailing list.

The QA infrastructure itself is managed in a different git repository: qa/jenkins.debian.net.git.

Straightforward patches — or packages not using Git as their version control system — have often been sent directly through the BTS.

Modified packages

The following packages have been modified to enable reproducible builds of other packages:

debhelper

The pu/reproducible_builds debhelper branch in the reproducible project contains several fixes and calls dh_strip_nondeterminism (see below) will be called before dh_compress in dh. It also sets the environment varialbe SOURCE_DATE_EPOCH with the last debian/changelog timestamp as proposed in standard timestamp and sets TZ=UTC to normalize timezones. See the changelog for details.

dpkg

The pu/reproducible_builds dpkg branch in the reproducible repository makes:

  1. file order deterministic in control and data part of the .deb,
  2. uses a single timestamp for .deb ar members
  3. preset the aforementioned timestamp to the latest changelog entry
  4. add -Wdate-time as part of CPPFLAGS in dpkg-buildflags

  5. add support for .buildinfo files

strip-nondeterminism

strip-nondeterminism is a post-processing tool that will normalize various file types. dh_strip_nondeterminism will be run by debhelper at the end of the build process.

cdbs

The pu/reproducible_builds cdbs branch in the reproducible project contains a fix for 764478 which makes cdbs call the newly introduced dh_strip_nondeterminism commands.

fontforge

fontforge needs a patch to propagate creation and modification times from source file. See 774148 and the pu/reproducible_builds branch.

libxslt

libxslt needs a patch to make generate-id() return identifiers in a deterministic way. See the pu/reproducible_builds branch.

libxslt needs a patch to honour SOURCE_DATE_EPOCH. See 791815.

python-support

python-support needs a patch to sort file lists in /usr/share/python-support/*.private. See 775786

gtk-doc

Patched gtk-doc to generate its links in a stable order.

wheel

wheel needs a patch to create reproducible wheel (.whl) files. See 776026.

docbook-to-man

docbook-to-man needs a patch to remove timestamps from the generated manpages. See 776143.

txt2man

txt2man needs a patch to honour SOURCE_DATE_EPOCH. See 790801.

epydoc

epydoc needs a patch to honour SOURCE_DATE_EPOCH. See 790899.

gettext

The tool xgettext from gettext needs a patch to honour SOURCE_DATE_EPOCH. See 792687.

doxygen

A version of doxygen which contains a patch to honour SOURCE_DATE_EPOCH (792201) has been uploaded to the reproducible builds git repository. See Upstream Bug for patch and discussion.

Usage example

If you have a pbuilder already setup, it's fairly easy to setup an environment with the custom toolchain:

## for pbuilder
sudo cp /var/cache/pbuilder/base.tgz /var/cache/pbuilder/base-reproducible.tgz
sudo pbuilder --login --save-after-exec --basetgz /var/cache/pbuilder/base-reproducible.tgz
## for cowbuilder
sudo cowbuilder --create --distribution sid --basepath /var/cache/pbuilder/base-reproducible.cow
sudo cowbuilder --login --save-after-exec --basepath /var/cache/pbuilder/base-reproducible.cow
## then, for both:
echo 'deb http://reproducible.alioth.debian.org/debian/ ./' > /etc/apt/sources.list.d/reproducible.list
apt-get install busybox
busybox wget -O- http://reproducible.alioth.debian.org/reproducible.gpg | apt-key add -
apt-get purge busybox
apt-key fingerprint | grep '49B6 5747 36D0 B637 CC37  01EA 5DB7 CA67 EA59 A31F' || { echo 'Something is wrong' && exit 1; }
apt-get update
apt-get upgrade
apt-get install locales-all # needed by rebuild.sh script
exit 0 # exit the pbuilder/cowbuilder login shell

Once that's done, two options:

  1. Use the prebuilder script from the misc.git repository.

  2. Manually, through the following process:

apt-get source --download-only acl
mkdir b1 b2
sudo DEB_BUILD_OPTIONS=nocheck pbuilder --build --debbuildopts '-b' --basetgz /var/cache/pbuilder/base-reproducible.tgz acl_*.dsc
dcmd cp /var/cache/pbuilder/result/acl_*.changes b1
sudo dcmd rm /var/cache/pbuilder/result/acl_*.changes
sudo DEB_BUILD_OPTIONS=nocheck pbuilder --build --debbuildopts '-b' --basetgz /var/cache/pbuilder/base-reproducible.tgz acl_*.dsc
dcmd cp /var/cache/pbuilder/result/acl_*.changes b2
sudo dcmd rm /var/cache/pbuilder/result/acl_*.changes

(for cowbuilder, change pbuilder to cowbuilder, --basetgz to --basepath and $path.tgz to $path.cow.

debbindiff is useful to check the result:

debbindiff --html $output_file b1/*.changes b2/*.changes

Adding a package to the APT archive

On the machine where you built the binary package(s):

  1. Copy the package files to /home/groups/reproducible/htdocs/debian/ on alioth, for example by running the following after having built the package (this requires that you have done a source-full build where the .changes file includes the orig.tar and debian.tar. If not, you have to repeat this step with the .dsc instead of the .changes):

dcmd scp ../packagename_version\~reproducible_amd64.changes username@alioth.debian.org:/home/groups/reproducible/htdocs/debian/

On alioth.debian.org:

  1. Import the private signing key to your keyring on alioth, if you haven't already: gpg --import /home/groups/reproducible/private/reproducible-private.gpg

  2. Run make from within /home/groups/reproducible/htdocs/debian/ to run apt-ftparchive and gpg to refresh the Release file, the Packages and Sources files

Guidelines for adding a package to the APT archive

(FIXME: to be discussed?)

  • uploads should target unstable (we do test testing and experimental on jenkins using this repo too, but thats a nice side-effect...)
    • the version number should be the version number from sid appended by .0~reproducibleX where X is an integer.

  • include source and binary packages
  • sign the upload
  • only fix issues which are relevant to reproducible builds and have bugs filed in the BTS
  • put your changes in .git in the pu/reproducible_builds branch as long it's a propesed change and rename the branch to merged/reproducible_builds once it has been included in an upload to sid.

Scheduling packages to rebuild on jenkins

Packages can be scheduled for immediate rebuild by anybody in the reproducible builds team by running the script reschedule.sh in /srv/home/groups/reproducible/ on alioth.debian.org. The script requires the mandatory -s parameter specifying the suite. Example:

/srv/home/groups/reproducible/reschedule.sh -s unstable mypackage1 mypackage2

It also allows some interesting tweaking of the scheduling (e.g. enable/disable notification at the start/end of the build, saving artifacts, etc) and allows selecting what packages to rebuild by doing a query itself. Run the script with the --help switch to see which options and filters are available.

Finding packages to schedule for rebuild

After a toolchain problem is fixed, the affected packages should be rebuilt. To find which packages are to be rebuilt, you need to calculate the intersection of all unreproducible packages and those which should now be fixed or have their debbindiff changed as a result of the toolchain upload to the experimental repository.

If the packages affected by the fixed issue are categorized under a single issue, you can just go on and do (or ask for) a rescheduling. The rescheduling script has enough power to get all unreproducible packages affected by a given issue.

Otherwise:

1. get a list of all unreproducible source packages:

curl --location https://jenkins.debian.net/userContent/reproducible.json > reproducible.json
jq --raw-output '.[] | select(.status == "unreproducible") | .package' < reproducible.json | sort -u > all-unreproducible

2. get a list of all affected source packages by finding those that build-depend on the fixed binary package. For example, python-sphinx (you need to have the deb-src lines on your /etc/apt/sources.list):

grep-dctrl -F Build-Depends python-sphinx --or -F Build-Depends-Indep python-sphinx --or -F Build-Depends-Arch python-sphinx /var/lib/apt/lists/*_debian_dists_sid_main_source_Sources -s Package -n | sort -u > all-affected

3. calculate the intersection:

comm -12 all-affected all-unreproducible > schedule-to-rebuild

Other way to get a list of packages to schedule are grepping for some patter the .buildinfos or debbindiff files. For that you need either to download all the files locally (the misc.git repository contains scripts to keep a synced copy of those files) or ask to somebody with shell access to reproducible.d.n to run the grep for you.

Once in a way or another you obtain a list of packages to reschedule, you can now post it on IRC to get a rebuild triggered as soon as somebody with the required privileges reads your message.

You should not immediately reschedule packages for rebuild after a toolchain upload, since the build chroot has to be updated (or regenerated) to notice the fixed package versions. The chroot gets updated every 4 hours. Also it's better to check whether the chroot has knowledge of the new package by scheduling a single small affected packages, and check its .buildinfo to see whether the fixed package has been picked up.