Reproducible Live images
This page tracks the progress on creating reproducible live images, using live-build from DebianLive
Status
Debian bookworm
Closed:
The package dictionaries-common produced unstable content: #1000674 (fixed in 1.28.14) and #1000685 (fixed in 1.28.13)
The package fontconfig created non-reproducible files with UUIDs: #864082 (fixed in 2.13.1-4.4)
The package libxmlb2 created differences at the first bytes of /var/cache/app-info/cache/C-local-metainfo.xb and /var/cache/app-info/cache/C-os-catalog.xb. #1006358 (fixed in 0.3.8-1)
Open:
- See sid
Debian sid (unstable)
- Seen in the Cinnamon image:
Core issue: The package nemo recommends texlive-binaries instead of untex, which introduces unnecessary non-reproducible files #1006472 (fixed in 5.4.0-1 experimental)
Various issues due to texlive-*
Ordering differences in /var/lib/texmf/ls-R
Ticket: #1003449 (texlive-base)
- Some related differences, needs FORCE_SOURCE_DATE=1 and has randomness for hashes in Lua
Ordering differences in /var/lib/texmf/web2c/luahbtex/luahbtex.fmt, /var/lib/texmf/web2c/luatex/dviluatex.fmt, /var/lib/texmf/web2c/luatex/luatex.fmt
Embedded timestamps in /var/lib/texmf/web2c/metafont/mf.log, /var/lib/texmf/web2c/pdftex/etex.log, /var/lib/texmf/web2c/pdftex/pdfetex.log, /var/lib/texmf/web2c/pdftex/pdftex.log, /var/lib/texmf/web2c/tex/tex.log
Small changes in /var/lib/texmf/web2c/metafont/mf.base, /var/lib/texmf/web2c/pdftex/etex.fmt, /var/lib/texmf/web2c/pdftex/pdfetex.fmt, /var/lib/texmf/web2c/pdftex/pdftex.fmt, /var/lib/texmf/web2c/tex/tex.fmt
Ticket: #1009196 (texlive-binaries)
Embedded timestamps in /var/lib/texmf/web2c/updmap.log
- Ticket: no ticket yet, the issue is solved very specific for live-build
Debian Bullseye
Image |
Installer |
Reproducible |
Notes |
mini |
none |
yes |
|
mini |
live/text |
yes |
1 |
mini |
live/gui |
yes |
1 |
mini |
daily |
yes |
2 |
standard |
none |
yes |
3, 4 |
standard |
live |
yes |
1, 3, 4 |
GNOME |
none |
yes |
3, 4, 5 |
GNOME |
live |
yes |
1, 3, 4, 5 |
KDE |
live |
yes |
1, 3, 4, 5 |
Cinnamon |
live |
yes |
3, 4, 5, 6 |
Note 1: The installer fails, because the installer uses an older kernel. Use the daily installer instead, until the installer is updated -> this is resolved
Note 2: Reproducibility tested by setting SOURCE_DATE_EPOCH during the first lb build and running the second build a few minutes after finishing the first build
Note 3: fontconfig uses random GUIDs, which can be circumvented. Hooks 1000 and 1001 implement a work around -> This will be fixed in bookworm with fontconfig 2.13.1-4.4
Note 4: mdadm adds a timestamp to the configuration file, which can be circumvented. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982607. Until mdadm is updated, hook 1002 implements a work around. This will be fixed in bookworm, 4.2~rc2-4
Note 5: During update-initramfs, the hook of plymouth adds fonts to the initrds. Hooks 1000 and 8000 implement a work around -> This will be fixed in bookworm with fontconfig 2.13.1-4.4
Note 6: libxml-sax-perl has a random order in its generated files, which can be circumvented. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993444. Hooks 1004 and 9000 implement a work around
Configuration
Modified configuration files
Required entries in sudoers (assuming you are thisuser):
Defaults env_keep += "SOURCE_DATE_EPOCH" Defaults env_keep += "LIVE_BUILD" thisuser ALL=(root) NOPASSWD: /usr/bin/lb build thisuser ALL=(root) NOPASSWD: /usr/bin/lb clean --purge
Jenkins node osuosl173-amd64.debian.net
- Running Debian/stable
Package live-build is installed
The most recent checkout of the git repository of live-build is used instead
- Running tests for 'bullseye', 'bookworm' and 'sid'
Personal computer
The computer that is used for generating live images:
- Debian/sid
apt-cacher-ng running on http://127.0.0.1:3142
auto-apt-proxy
The most recent checkout of the git repository of live-build (https://salsa.debian.org/live-team/live-build)
An additional line in /etc/apt-cacher-ng/acnf.conf: VfilePatternEx: /project/trace/ftp-master\.debian\.org$
Snapshot
In order to get a reproducible image, there are two main options:
If the live image is expected to be reproducible, the image can be built twice in rapid succession from deb.debian.org (within its regular 6 hour update cycle)
- If the live image is expected to be non-reproducible, a snapshot server needs to be used.
- By using a snapshot server, it is easier to re-generate images to track the difference and to polish on the work-around hook.
Note that for live images in production, deb.debian.org needs to be used.
- By using a snapshot server, it is easier to re-generate images to track the difference and to polish on the work-around hook.
The server snapshot.notset.fr is used since 2021-09-05, because the service at snapshot.debian.org occasionally rejects traffic when many requests are made.
Historical note: debian.notset.fr was used until 2021-09-05, snapshot.debian.org was used until 2021-08-25
Considerations
Building an image must be fast, because a lot of rebuilds will be required -> use /dev/shm
- NB: The GNOME image requires about 17GB, so you'll need sufficient memory. If that is not available, use a fast disc (e.g. SSD) instead.
The mount point for the image needs to support mknod and must allow the execution within the chroot, so it must be mounted with dev and exec.
Reduce network traffic, don't download the same files over-and-over -> use apt-cacher-ng
While looking for (non-)reproducibility, the command line option --parent-mirror-binary is used. Official live images should not use this command line option (and will therefore use deb.debian.org in /var/lib/apt/lists)
Preparing the build environment
Preparation of the build directory
As root:
# You need `dev` and `exec` active on your mount point # Slightly more than 16GB is required (on a machine with 32GB memory /dev/shm get half of it) mount /dev/shm -odev,exec,remount,size=24G
As a regular user:
mkdir /dev/shm/live cd /dev/shm/live
Building
All command lines can be executed as a regular user.
Use case: rebuild an image with a given timestamp (as e.g. taken from a Jenkins job) from the snapshot server
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid 20220421T153504Z
Use case: get the latest image from the snapshot server
# Replace the timestamp with the word 'snapshot' to fetch the latest snapshot /home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid snapshot
Use case: while preparing new hooks, using the local git checkout of live-build
# When LIVE_BUILD is defined, the script will not checkout live-build from the timestamp, but will use the version pointed at by LIVE_BUILD export LIVE_BUILD=/home/roland/git.nobackup/live-build /home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid 20220421T153504Z
Use case: build an image from the current Debian repository
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid
- Notes:
- You have at most 6 hours to build a second image to check for reproducibility. After that time the archive will be re-synced and contains a new timestamp
Untested yet: Verification of such images might require difference values for SNAPSHOT_TIMESTAMP and EPOCH_SOURCE_DATE, because the timestamps offered by the snapshot server do not match the content of InRelease
Preparing the build environment (old)
These steps below have migrated to a dedicated script in live-build. It is recommended to use that instead. This section is kept for documentation purposes.
All command lines must be executed as root.
## ## Section that does not need to be modified, if you want to use /dev/shm ## cd /dev/shm mount /dev/shm -odev,exec,remount mkdir live cd live ## ## Section that you would like to adjust for your own setup ## export LIVE_BUILD=/home/roland/git.nobackup/live-build export http_proxy=http://127.0.0.1:3142/ export USE_DISTRIBUTION=sid ## ## Timestamp selection ## # Bullseye: 20210101T083123Z (1609489883) #export SOURCE_DATE_EPOCH=1609489883 # Buster: 20210210T031935Z (1612927175) #export SOURCE_DATE_EPOCH=1612927175 # Bookworm and sid: Use the timestamp of the latest mirror snapshot (if the timestamp was not set already) if [ -z "${SNAPSHOT_TIMESTAMP}" ]; then wget http://snapshot.notset.fr/mr/timestamp/debian/latest # # Extract the timestamp from the JSON file # # Input: # { # "_api": "0.3", # "_comment": "notset", # "result": "20210828T083909Z" # } # Output: # 20210828T083909Z # export SNAPSHOT_TIMESTAMP=$(cat latest | awk '/"result":/ { split($0, a, "\""); print a[4] }') fi # Convert SNAPSHOT_TIMESTAMP to Unix time (and insert suitable formatting first) export SOURCE_DATE_EPOCH=$(date -d $(echo $SNAPSHOT_TIMESTAMP | awk '{ printf "%s-%s-%sT%s:%s:%sZ", substr($0,1,4), substr($0,5,2), substr($0,7,2), substr($0,10,2), substr($0,12,2), substr($0,14,2) }') +%s) ## ## Prepare the basic configuration ## export MIRROR=http://snapshot.notset.fr/archive/debian/${SNAPSHOT_TIMESTAMP} # The basic configuration line: lb config --apt-http-proxy ${http_proxy} --parent-mirror-bootstrap ${MIRROR} --parent-mirror-binary ${MIRROR} --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution ${USE_DISTRIBUTION} --debian-installer live --cache-packages false ## ## Install the hooks for reproducible builds ## if [ -z "${LIVE_BUILD}" ]; then cp /usr/share/doc/live-build/examples/hooks/reproducible/* config/hooks/normal else cp $LIVE_BUILD/examples/hooks/reproducible/* config/hooks/normal fi
Package selection
Standard
echo "live-task-standard" > config/package-lists/desktop.list.chroot
GNOME desktop
echo "live-task-gnome" > config/package-lists/desktop.list.chroot
KDE desktop
echo "live-task-kde" > config/package-lists/desktop.list.chroot
Some variants
Note: these command lines use bullseye, they were written when bullseye was still unstable.
No installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer none --cache-packages false
Live/text-only installer
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false
Latest installer
unset SOURCE_DATE_EPOCH lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/deb.debian.org/debian --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false --parent-debian-installer-distribution daily # When doing comparisons, use the following line for the first run: # export SOURCE_DATE_EPOCH=`date +%s`; lb build
Live installer (to be used for production images)
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false
Building images for Debian Buster
Mini, with live installer
export SOURCE_DATE_EPOCH=1612927175 lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210210T031935Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/202102101031935Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution buster --debian-installer live --cache-packages false
Checking reproducibility
Ideally reprotest will be used to check reproducibility. For now, 2 images are built and then compared with diffoscope.
Using diffoscope
# 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) lb build mv live-image-amd64.hybrid.iso run01.iso lb clean --purge lb config lb build mv live-image-amd64.hybrid.iso run02.iso diffoscope run01.iso run02.iso --html-dir html_run01_02
Using reprotest
reprotest requires full access on /tmp, so it must be remounted (when applicable)
# 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) mount /tmp -odev,exec,suid,remount reprotest --variations=-all,+environment,+build_path,-kernel,+aslr,+num_cpus,+time,-user_group,-fileordering,-domain_host,+home,+locales,+exec_path,+timezone,-umask "lb clean --purge&&lb config&&lb build" "*.iso" # Disabled tests: # kernel The variant with kernel 2.6 is too old for debootstrap # user_group Live build only works for root, the variation needs a list # fileordering Disorderfs needs to be mounted with dev,exec,suid # domain_host Needs additional rights during debootstrap # umask Some files get a different umask, further investigation is required
See also
Older work: https://rclobus.nl/blog/?p=190
Page maintainer: Roland Clobus