Size: 13494
Comment: Declare GNOME images reproducible (under a specific condition)
|
Size: 13474
Comment: use message-id macro
|
Deletions are marked like this. | Additions are marked like this. |
Line 79: | Line 79: |
* Note 6: The file `live/filesystem.size` (the output of `du -B 1 -s`) is not reproducible. Files with identical content are sometimes reported with a different size. However, this file becomes reproducible when building on `/dev/shm`. See the mailing list: https://lists.debian.org/debian-live/2021/03/msg00008.html | * Note 6: The file `live/filesystem.size` (the output of `du -B 1 -s`) is not reproducible. Files with identical content are sometimes reported with a different size. However, this file becomes reproducible when building on `/dev/shm`. See <<mid(67f81b59-52fc-af27-1778-2c8f6bf8becd@rclobus.nl)>> |
Reproducible Live images
This page tracks the progress on creating reproducible live images, using live-build from DebianLive
Contents
Configuration
The computer that is used for generating live images:
- Debian/sid
apt-cacher-ng running on http://localhost:3142
The most recent checkout of the git repository of live-build (https://salsa.debian.org/live-team/live-build)
An additional line in /etc/apt-cacher-ng/acnf.conf: VfilePatternEx: /project/trace/ftp-master\.debian\.org$
Snapshot
In order to get a more reproducible image, a snapshot is used. At the time of focusing on Bullseye, the timestamp 20210101T083123Z (1609489883) was chosen.
Considerations
Building an image must be fast, because a lot of rebuilds will be required -> use /dev/shm
Reduce network traffic, don't download the same files over-and-over -> use apt-cacher-ng
While looking for (non-)reproducibility, the command line option --parent-mirror-binary is used. Official live images should not use this command line option (and will therefore use deb.debian.org in /var/lib/apt/lists)
Preparing the build environment
All command lines must be executed as root.
export LIVE_BUILD=/home/roland/git/live-build export SOURCE_DATE_EPOCH=1609489883 cd /dev/shm mount /dev/shm -odev,exec,remount mkdir live cd live
Building images for Debian Buster
Timestamp: 20210210T031935Z
export LIVE_BUILD=/home/roland/git/live-build export SOURCE_DATE_EPOCH=1612927175 cd /dev/shm mount /dev/shm -odev,exec,remount mkdir live cd live
Mini, with live installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210210T031935Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/202102101031935Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution buster --debian-installer live --cache-packages false
Building images for Debian Bullseye
Image |
Installer |
Reproducible |
Notes |
mini |
none |
yes |
|
mini |
live/text |
yes |
1 |
mini |
live/gui |
yes |
1 |
mini |
daily |
yes |
2 |
standard |
none |
yes |
3, 4 |
standard |
live |
yes |
1, 3, 4 |
GNOME |
none |
yes, but 6 |
3, 4, 5 |
GNOME |
live |
yes, but 6 |
1, 3, 4, 5 |
- Note 1: The installer fails, because the installer uses an older kernel. Use the daily installer instead, until the installer is updated
Note 2: Reproducibility tested by setting SOURCE_DATE_EPOCH during the first lb build and running the second build a few minutes after finishing the first build
- Note 3: fontconfig uses random GUIDs, which can be circumvented. Hooks 1000 and 1001 implement a work around
Note 4: mdadm adds a timestamp to the configuration file, which can be circumvented. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982607. Until mdadm is updated, hook 1002 implements a work around
Note 5: During update-initramfs, the hook of plymouth adds fonts to the initrds. Hooks 1000 and 1003 implement a work around
Note 6: The file live/filesystem.size (the output of du -B 1 -s) is not reproducible. Files with identical content are sometimes reported with a different size. However, this file becomes reproducible when building on /dev/shm. See <67f81b59-52fc-af27-1778-2c8f6bf8becd@rclobus.nl>
Command lines
Mini, without installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer none --cache-packages false
Mini, with live/text-only installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false
Mini, with latest installer
unset SOURCE_DATE_EPOCH lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/deb.debian.org/debian --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false --parent-debian-installer-distribution daily # When doing comparisons, use the following line for the first run: # export SOURCE_DATE_EPOCH=`date +%s`; lb build
Mini, with live installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false
Standard, without installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer none --cache-packages false echo "live-task-standard" > config/package-lists/desktop.list.chroot
Standard, with live installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false echo "live-task-standard" > config/package-lists/desktop.list.chroot
GNOME desktop, without installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer none --cache-packages false echo "live-task-gnome" > config/package-lists/desktop.list.chroot
GNOME desktop, with live installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false echo "live-task-gnome" > config/package-lists/desktop.list.chroot
The hook scripts that should be applied after any of the command lines above, to fix non-reproducible packages
cat > config/hooks/normal/1000-reproducible-function-uuid_generate_random.hook.chroot << EOF #!/bin/sh set -e # util-linux creates random UUIDs when uuid_generate_random is called # Use LD_PRELOAD to replace uuid_generate_random with a less random version # Don't run if gcc is not installed if [ ! -e /usr/bin/cc ]; then exit 0 fi cat > unrandomize_uuid_generate_random.c << END_OF_SOURCE #include <stdlib.h> #include <stdio.h> #define SEQUENCE_FILENAME "/var/cache/unrandomize_uuid_generate_random.sequence_number" /* https://tools.ietf.org/html/rfc4122 */ typedef unsigned char uuid_t[16]; /* Our pseudo-random version */ void uuid_generate_random(uuid_t out) { /* Nil UUID */ for (int i=0;i<16;i++) { out[i] = 0x00; } out[6]=0x40; /* UUID version 4 means randomly generated */ out[8]=0x80; /* bit7=1,bit6=0 */ /* The file doesn't need to exist yet */ FILE *f = fopen(SEQUENCE_FILENAME, "rb"); if (f) { fread(out+12, 4, 1, f); fclose(f); } /* Use the next number. Endianness is not important */ (*(unsigned long*)(out+12))++; unsigned long long epoch; /* Use SOURCE_DATE_EPOCH when provided */ char *date = getenv("SOURCE_DATE_EPOCH"); if (date) { epoch = strtoll(date, NULL, 10); } else { epoch = 0ll; } out[0] = (epoch & 0xFF000000) >> 24; out[1] = (epoch & 0x00FF0000) >> 16; out[2] = (epoch & 0x0000FF00) >> 8; out[3] = (epoch & 0x000000FF); /* Write the sequence number */ f = fopen(SEQUENCE_FILENAME, "wb"); if (f) { fwrite(out+12, 4, 1, f); fclose(f); } } END_OF_SOURCE /usr/bin/cc -shared -fPIC unrandomize_uuid_generate_random.c -Wall --pedantic -o /usr/lib/unrandomize_uuid_generate_random.so rm -f unrandomize_uuid_generate_random.c EOF cat > config/hooks/normal/1001-reproducible-fontconfig.hook.chroot << EOF #!/bin/sh set -e # fontconfig creates non-reproducible files with UUIDs # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082 # # Because the UUIDs should not be deleted, the proposed work-around is: # * Use LD_PRELOAD to replace uuid_generate_random with a less random version # Don't run if fontconfig is not installed if [ ! -e /usr/bin/fc-cache ]; then exit 0 fi # Don't run if the LD_PRELOAD module is not compiled if [ ! -e /usr/lib/unrandomize_uuid_generate_random.so ]; then exit 0 fi LD_PRELOAD=/usr/lib/unrandomize_uuid_generate_random.so /usr/bin/fc-cache --force --really-force --system-only --verbose EOF cat > config/hooks/normal/1002-reproducible-mdadm.hook.chroot << EOF #!/bin/sh set -e # mkconf of mdadm creates a file with a timestamp # A bug report with patch is available at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982607 # This script duplicates that patch # Don't run if mdadm is not installed if [ ! -e /usr/share/mdadm/mkconf ]; then exit 0 fi # If mkconf already contains references to SOURCE_DATE_EPOCH, there is no need to patch the file if grep -q SOURCE_DATE_EPOCH /usr/share/mdadm/mkconf; then exit 0 fi sed -i -e '/# This configuration was auto-generated on/cif [ -z \$SOURCE_DATE_EPOCH ]; then\n echo "# This configuration was auto-generated on \$(date -R) by mkconf"\nelse\n echo "# This configuration was auto-generated on \$(date -R --utc -d@\$SOURCE_DATE_EPOCH) by mkconf"\nfi' /usr/share/mdadm/mkconf EOF cat > config/hooks/normal/1003-reproducible-plymouth.hook.chroot << EOF #!/bin/sh set -e # The hook of plymouth in update-initramfs calls fc-cache # Don't run if plymouth is not installed if [ ! -e /usr/share/initramfs-tools/hooks/plymouth ]; then exit 0 fi # Don't patch if the LD_PRELOAD module is not compiled if [ ! -e /usr/lib/unrandomize_uuid_generate_random.so ]; then exit 0 fi # If the hook already contains references to LD_PRELOAD, there is no need to patch the file if grep -q LD_PRELOAD /usr/share/initramfs-tools/hooks/plymouth; then exit 0 fi sed -i -e 's|fc-cache -s|LD_PRELOAD=/usr/lib/unrandomize_uuid_generate_random.so fc-cache|' /usr/share/initramfs-tools/hooks/plymouth EOF
Checking reproducibility
Ideally reprotest will be used to check reproducibility. For now, 2 images are built and then compared with diffoscope.
Using diffoscope
# 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) lb build mv live-image-amd64.hybrid.iso run01.iso lb clean --purge lb config lb build mv live-image-amd64.hybrid.iso run02.iso diffoscope run01.iso run02.iso --html-dir html_run01_02
Using reprotest
reprotest requires full access on /tmp, so it must be remounted (when applicable)
# 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) mount /tmp -odev,exec,suid,remount # This is probably better, but bails at the moment # reprotest --store-dir store --variations=+all,-user_group --diffoscope-arg="--html-dir=html_{0}" "lb clean --purge&&lb config&&lb build" "*.iso" reprotest --store-dir store --variations=-user_group --diffoscope-arg="--html-dir=html_{0}" "lb clean --purge&&lb config&&lb build" "*.iso"
Note: this command line is not correct yet, but shows the basic idea.
See also
Older work: https://rclobus.nl/blog/?p=190
Page maintainer: Roland Clobus