#language en #~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: [[sr/DefaultTemplate|srpski]]-~ ---- = Reproducible Live images = This page tracks the progress on creating reproducible live images, using `live-build` from [[DebianLive]] <> == Status == === Debian bookworm === Reproducible === Debian sid (unstable) === Reproducible == Verification == * Automated tests for reproducible build of the live images are running on [[https://jenkins.debian.net/view/live/|Jenkins]] * Automated tests for the functionality of the live images are running on [[https://openqa.debian.net/group_overview/14|openQA]] = Configuration = == Modified configuration files == Required entries in `sudoers` (assuming you are `thisuser`): {{{ Defaults env_keep += "SOURCE_DATE_EPOCH" Defaults env_keep += "LIVE_BUILD" thisuser ALL=(root) NOPASSWD: /usr/bin/lb build thisuser ALL=(root) NOPASSWD: /usr/bin/lb clean --purge }}} == Jenkins node osuosl3-amd64.debian.net == * Running Debian/stable * Package `live-build` is installed * The most recent checkout of the git repository of `live-build` is used instead * Running tests for 'bullseye', 'bookworm', 'trixie' and 'sid' == Personal computer == The computer that is used for generating live images: * Debian/sid * `apt-cacher-ng` running on `http://127.0.0.1:3142` * `auto-apt-proxy` * The most recent checkout of the git repository of `live-build` (https://salsa.debian.org/live-team/live-build) * An additional line in `/etc/apt-cacher-ng/acng.conf`: `VfilePatternEx: /project/trace/ftp-master\.debian\.org$` === Snapshot === In order to get a reproducible image, there are two main options: 1. If the live image is expected to be reproducible, the image can be built twice in rapid succession from `deb.debian.org` (within its regular 6 hour update cycle) 2. If the live image is expected to be non-reproducible, a snapshot server needs to be used. By using a snapshot server, it is easier to re-generate images to track the difference and to polish on the work-around hook. Note that for live images in production, `deb.debian.org` needs to be used. The server `snapshot.notset.fr` is used since 2021-09-05, because the service at `snapshot.debian.org` occasionally rejects traffic when many requests are made. === Considerations === * Building an image must be fast, because a lot of rebuilds will be required -> use `/dev/shm` NB: The GNOME image requires about 17GB during building, so you'll need sufficient memory. If that is not available, use a fast disc (e.g. SSD) instead. * The mount point for the image needs to support `mknod` and must allow the execution within the `chroot`, so it must be mounted with `dev` and `exec`. * Reduce network traffic, don't download the same files over-and-over -> use `apt-cacher-ng` * While looking for (non-)reproducibility, the command line option `--parent-mirror-binary` is used. Official live images should not use this command line option (and will therefore use deb.debian.org in `/var/lib/apt/lists`) = Building live images = == Preparation of the build directory == As root: {{{ # You need `dev` and `exec` active on your mount point # Slightly more than 16GB is required (on a machine with 32GB memory /dev/shm gets half of it) mount /dev/shm -odev,exec,remount,size=24G }}} As a regular user: {{{ mkdir /dev/shm/live cd /dev/shm/live }}} == Building == All command lines can be executed as a regular user === Use case: build an image from the current Debian repository (testing, unstable or experimental) === {{{ /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration gnome --debian-version sid }}} Notes: * You have at most 6 hours to build a second image to check for reproducibility. After that time the archive will be re-synced and contains a new timestamp * Untested yet: Verification of such images might require difference values for `SNAPSHOT_TIMESTAMP` and `EPOCH_SOURCE_DATE`, because the timestamps offered by the snapshot server do not match the content of `InRelease` === Use case: build an image from the current Debian repository (stable) === {{{ /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration gnome --debian-version bookworm --debian-version-number 12.1.0 --timestamp archive --installer-origin archive }}} Notes: * This will use the pre-built Debian installer * See [[https://salsa.debian.org/images-team/live-setup/-/blob/master/available/live-build-binary#L129=|available/live-build-binary in the live-setup at Salsa]] for the invocation of rebuild.sh for officially released Debian live images * Untested yet: Verification of such images might require difference values for `SNAPSHOT_TIMESTAMP` and `EPOCH_SOURCE_DATE` === Use case: rebuild an image with a given timestamp === {{{ /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration gnome --debian-version sid --timestamp 20220421T153504Z }}} Notes: * The timestamp could be extracted from a [[https://jenkins.debian.net/view/live/|Jenkins job]]) === Use case: get the latest image from the snapshot server === {{{ # The magic value 'snapshot' queries the snapshot server and fetches the latest/newest snapshot /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration gnome --debian-version sid --timestamp snapshot }}} === Use case: while preparing new hooks, using the local git checkout of `live-build` === {{{ # When LIVE_BUILD is defined, the script will not checkout live-build from the timestamp, but will use the version pointed at by LIVE_BUILD export LIVE_BUILD=/home/roland/git.nobackup/live-build /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration gnome --debian-version sid --timestamp 20220421T153504Z }}} = Checking reproducibility = Ideally `reprotest` will be used to check reproducibility. For now, 2 images are built and then compared with `diffoscope`. == Using diffoscope == {{{ # 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) lb build mv live-image-amd64.hybrid.iso run01.iso lb clean --purge lb config lb build mv live-image-amd64.hybrid.iso run02.iso diffoscope run01.iso run02.iso --html-dir html_run01_02 }}} == Using reprotest == `reprotest` requires full access on `/tmp`, so it must be remounted (when applicable) {{{ # 1) Prepare the build environment # 2) Run the command line to create the configuration (see above) mount /tmp -odev,exec,suid,remount reprotest --variations=-all,+environment,+build_path,-kernel,+aslr,+num_cpus,+time,-user_group,-fileordering,-domain_host,+home,+locales,+exec_path,+timezone,-umask "lb clean --purge&&lb config&&lb build" "*.iso" # Disabled tests: # kernel The variant with kernel 2.6 is too old for debootstrap # user_group Live build only works for root, the variation needs a list # fileordering Disorderfs needs to be mounted with dev,exec,suid # domain_host Needs additional rights during debootstrap # umask Some files get a different umask, further investigation is required }}} == Disorderfs == To ensure that file ordering is not a cause for non-reproducible images, the tool 'disorderfs' will offer the files from the file system in a different order: {{{ # As root disorderfs live_regular live_reversed_order --reverse-dirents=yes --multi-user=yes -o suid,dev }}} = See also = * [[ReproducibleInstalls]] * Older work: https://rclobus.nl/blog/?p=190 Page maintainer: Roland Clobus ---- ## If this page belongs to an existing Category, add it below. ## CategorySomething | CategoryAnother