Differences between revisions 40 and 41
Revision 40 as of 2022-07-24 20:19:38
Size: 15101
Editor: ?RolandClobus
Comment: Building requires slightly more than 16GB
Revision 41 as of 2022-07-25 19:53:33
Size: 16089
Editor: ?RolandClobus
Comment: How to build without using a snapshot server
Deletions are marked like this. Additions are marked like this.
Line 93: Line 93:
In order to get a reproducible image, a snapshot of the Debian repository is used. Note that for live images in production, `deb.debian.org` needs to be used. In order to get a reproducible image, there are two main options:

 1. If the live image is expected to be reproducible, the image can be built twice in rapid succession from `deb.debian.org` (within its regular 6 hour update cycle)
 2. If the live image is expected to be non-reproducible, a snapshot server needs to be used.
   By using a snapshot server, it is easier to re-generate images to track the difference and to polish on the work-around hook.
   Note that for live images in production, `deb.debian.org` needs to be used.
Line 128: Line 133:
Default use case: rebuild an image with a given timestamp (as e.g. taken from a [[https://jenkins.debian.net/view/live/|Jenkins job]]) Use case: rebuild an image with a given timestamp (as e.g. taken from a [[https://jenkins.debian.net/view/live/|Jenkins job]]) from the snapshot server
Line 134: Line 139:
Use case: get the latest image

{{{
# Omit the timestamp to fetch the latest snapshot
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid
Use case: get the latest image from the snapshot server

{{{
# Replace the timestamp with the word 'snapshot' to fetch the latest snapshot
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid snapshot
Line 148: Line 153:

Use case: build an image from the current Debian repository
{{{
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid
}}}
 Notes:
  * You have at most 6 hours to build a second image to check for reproducibility. After that time the archive will be re-synced and contains a new timestamp
  * Untested yet: Verification of such images might require difference values for `SNAPSHOT_TIMESTAMP` and `EPOCH_SOURCE_DATE`, because the timestamps offered by the snapshot server do not match the content of `InRelease`


Reproducible Live images

This page tracks the progress on creating reproducible live images, using live-build from DebianLive

Status

Debian bookworm

Closed:

  • The package dictionaries-common produced unstable content: #1000674 (fixed in 1.28.14) and #1000685 (fixed in 1.28.13)

  • The package fontconfig created non-reproducible files with UUIDs: #864082 (fixed in 2.13.1-4.4)

  • The package libxmlb2 created differences at the first bytes of /var/cache/app-info/cache/C-local-metainfo.xb and /var/cache/app-info/cache/C-os-catalog.xb. #1006358 (fixed in 0.3.8-1)

Open:

  • See sid

Debian sid (unstable)

  • Seen in the Cinnamon image:
    • Core issue: The package nemo recommends texlive-binaries instead of untex, which introduces unnecessary non-reproducible files #1006472 (fixed in 5.4.0-1 experimental)

    • Various issues due to texlive-*

      • Ordering differences in /var/lib/texmf/ls-R

      • Some related differences, needs FORCE_SOURCE_DATE=1 and has randomness for hashes in Lua
        • Ordering differences in /var/lib/texmf/web2c/luahbtex/luahbtex.fmt, /var/lib/texmf/web2c/luatex/dviluatex.fmt, /var/lib/texmf/web2c/luatex/luatex.fmt

        • Embedded timestamps in /var/lib/texmf/web2c/metafont/mf.log, /var/lib/texmf/web2c/pdftex/etex.log, /var/lib/texmf/web2c/pdftex/pdfetex.log, /var/lib/texmf/web2c/pdftex/pdftex.log, /var/lib/texmf/web2c/tex/tex.log

        • Small changes in /var/lib/texmf/web2c/metafont/mf.base, /var/lib/texmf/web2c/pdftex/etex.fmt, /var/lib/texmf/web2c/pdftex/pdfetex.fmt, /var/lib/texmf/web2c/pdftex/pdftex.fmt, /var/lib/texmf/web2c/tex/tex.fmt

      • Embedded timestamps in /var/lib/texmf/web2c/updmap.log

Debian Bullseye

Image

Installer

Reproducible

Notes

mini

none

yes

mini

live/text

yes

1

mini

live/gui

yes

1

mini

daily

yes

2

standard

none

yes

3, 4

standard

live

yes

1, 3, 4

GNOME

none

yes

3, 4, 5

GNOME

live

yes

1, 3, 4, 5

KDE

live

yes

1, 3, 4, 5

Cinnamon

live

yes

3, 4, 5, 6

  • Note 1: The installer fails, because the installer uses an older kernel. Use the daily installer instead, until the installer is updated -> this is resolved

  • Note 2: Reproducibility tested by setting SOURCE_DATE_EPOCH during the first lb build and running the second build a few minutes after finishing the first build

  • Note 3: fontconfig uses random GUIDs, which can be circumvented. Hooks 1000 and 1001 implement a work around -> This will be fixed in bookworm with fontconfig 2.13.1-4.4

  • Note 4: mdadm adds a timestamp to the configuration file, which can be circumvented. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982607. Until mdadm is updated, hook 1002 implements a work around. This will be fixed in bookworm, 4.2~rc2-4

  • Note 5: During update-initramfs, the hook of plymouth adds fonts to the initrds. Hooks 1000 and 8000 implement a work around -> This will be fixed in bookworm with fontconfig 2.13.1-4.4

  • Note 6: libxml-sax-perl has a random order in its generated files, which can be circumvented. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993444. Hooks 1004 and 9000 implement a work around

Configuration

Modified configuration files

Required entries in sudoers (assuming you are thisuser):

Defaults env_keep += "SOURCE_DATE_EPOCH"
Defaults env_keep += "LIVE_BUILD"
thisuser ALL=(root) NOPASSWD: /usr/bin/lb build
thisuser ALL=(root) NOPASSWD: /usr/bin/lb clean --purge

Jenkins node osuosl173-amd64.debian.net

  • Running Debian/stable
  • Package live-build is installed

  • The most recent checkout of the git repository of live-build is used instead

  • Running tests for 'bullseye', 'bookworm' and 'sid'

Personal computer

The computer that is used for generating live images:

  • Debian/sid
  • apt-cacher-ng running on http://127.0.0.1:3142

  • auto-apt-proxy

  • The most recent checkout of the git repository of live-build (https://salsa.debian.org/live-team/live-build)

  • An additional line in /etc/apt-cacher-ng/acnf.conf: VfilePatternEx: /project/trace/ftp-master\.debian\.org$

Snapshot

In order to get a reproducible image, there are two main options:

  1. If the live image is expected to be reproducible, the image can be built twice in rapid succession from deb.debian.org (within its regular 6 hour update cycle)

  2. If the live image is expected to be non-reproducible, a snapshot server needs to be used.
    • By using a snapshot server, it is easier to re-generate images to track the difference and to polish on the work-around hook.

      Note that for live images in production, deb.debian.org needs to be used.

The server snapshot.notset.fr is used since 2021-09-05, because the service at snapshot.debian.org occasionally rejects traffic when many requests are made.

Historical note: debian.notset.fr was used until 2021-09-05, snapshot.debian.org was used until 2021-08-25

Considerations

  • Building an image must be fast, because a lot of rebuilds will be required -> use /dev/shm

    • NB: The GNOME image requires about 17GB, so you'll need sufficient memory. If that is not available, use a fast disc (e.g. SSD) instead.
  • The mount point for the image needs to support mknod and must allow the execution within the chroot, so it must be mounted with dev and exec.

  • Reduce network traffic, don't download the same files over-and-over -> use apt-cacher-ng

  • While looking for (non-)reproducibility, the command line option --parent-mirror-binary is used. Official live images should not use this command line option (and will therefore use deb.debian.org in /var/lib/apt/lists)

Preparing the build environment

Preparation of the build directory

As root:

# You need `dev` and `exec` active on your mount point
# Slightly more than 16GB is required (on a machine with 32GB memory /dev/shm get half of it)
mount /dev/shm -odev,exec,remount,size=24G

As a regular user:

mkdir /dev/shm/live
cd /dev/shm/live

Building

All command lines can be executed as a regular user.

Use case: rebuild an image with a given timestamp (as e.g. taken from a Jenkins job) from the snapshot server

/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid 20220421T153504Z

Use case: get the latest image from the snapshot server

# Replace the timestamp with the word 'snapshot' to fetch the latest snapshot
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid snapshot

Use case: while preparing new hooks, using the local git checkout of live-build

# When LIVE_BUILD is defined, the script will not checkout live-build from the timestamp, but will use the version pointed at by LIVE_BUILD
export LIVE_BUILD=/home/roland/git.nobackup/live-build
/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid 20220421T153504Z

Use case: build an image from the current Debian repository

/home/roland/git.nobackup/live-build/test/rebuild.sh gnome sid
  • Notes:
    • You have at most 6 hours to build a second image to check for reproducibility. After that time the archive will be re-synced and contains a new timestamp
    • Untested yet: Verification of such images might require difference values for SNAPSHOT_TIMESTAMP and EPOCH_SOURCE_DATE, because the timestamps offered by the snapshot server do not match the content of InRelease

Preparing the build environment (old)

These steps below have migrated to a dedicated script in live-build. It is recommended to use that instead. This section is kept for documentation purposes.

All command lines must be executed as root.

##
## Section that does not need to be modified, if you want to use /dev/shm
##
cd /dev/shm
mount /dev/shm -odev,exec,remount
mkdir live
cd live

##
## Section that you would like to adjust for your own setup
##
export LIVE_BUILD=/home/roland/git.nobackup/live-build
export http_proxy=http://127.0.0.1:3142/
export USE_DISTRIBUTION=sid

##
## Timestamp selection
##
# Bullseye: 20210101T083123Z (1609489883)
#export SOURCE_DATE_EPOCH=1609489883
# Buster: 20210210T031935Z (1612927175)
#export SOURCE_DATE_EPOCH=1612927175
# Bookworm and sid: Use the timestamp of the latest mirror snapshot (if the timestamp was not set already)
if [ -z "${SNAPSHOT_TIMESTAMP}" ]; then
  wget http://snapshot.notset.fr/mr/timestamp/debian/latest
  #
  # Extract the timestamp from the JSON file
  #
  # Input:
  # {
  #   "_api": "0.3",
  #   "_comment": "notset",
  #   "result": "20210828T083909Z"
  # }
  # Output:
  # 20210828T083909Z
  #
  export SNAPSHOT_TIMESTAMP=$(cat latest | awk '/"result":/ { split($0, a, "\""); print a[4] }')
fi
# Convert SNAPSHOT_TIMESTAMP to Unix time (and insert suitable formatting first)
export SOURCE_DATE_EPOCH=$(date -d $(echo $SNAPSHOT_TIMESTAMP | awk '{ printf "%s-%s-%sT%s:%s:%sZ", substr($0,1,4), substr($0,5,2), substr($0,7,2), substr($0,10,2), substr($0,12,2), substr($0,14,2) }') +%s)

##
## Prepare the basic configuration
##
export MIRROR=http://snapshot.notset.fr/archive/debian/${SNAPSHOT_TIMESTAMP}
# The basic configuration line:
lb config --apt-http-proxy ${http_proxy} --parent-mirror-bootstrap ${MIRROR} --parent-mirror-binary ${MIRROR} --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution ${USE_DISTRIBUTION} --debian-installer live --cache-packages false

##
## Install the hooks for reproducible builds
##
if [ -z "${LIVE_BUILD}" ]; then
  cp /usr/share/doc/live-build/examples/hooks/reproducible/* config/hooks/normal
else
  cp $LIVE_BUILD/examples/hooks/reproducible/* config/hooks/normal
fi

Package selection

Standard

echo "live-task-standard" > config/package-lists/desktop.list.chroot

GNOME desktop

echo "live-task-gnome" > config/package-lists/desktop.list.chroot

KDE desktop

echo "live-task-kde" > config/package-lists/desktop.list.chroot

Some variants

Note: these command lines use bullseye, they were written when bullseye was still unstable.

No installer

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer none --cache-packages false

Live/text-only installer

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false

Latest installer

unset SOURCE_DATE_EPOCH
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/deb.debian.org/debian --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --distribution bullseye --debian-installer live --cache-packages false --debian-installer-gui false --parent-debian-installer-distribution daily
# When doing comparisons, use the following line for the first run:
#   export SOURCE_DATE_EPOCH=`date +%s`; lb build

Live installer (to be used for production images)

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210101T083123Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20210101T083123Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution bullseye --debian-installer live --cache-packages false

Building images for Debian Buster

Mini, with live installer

export SOURCE_DATE_EPOCH=1612927175
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20210210T031935Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/202102101031935Z --security false --updates false --apt-options "--yes -o Acquire::Check-Valid-Until=false" --distribution buster --debian-installer live --cache-packages false

Checking reproducibility

Ideally reprotest will be used to check reproducibility. For now, 2 images are built and then compared with diffoscope.

Using diffoscope

# 1) Prepare the build environment
# 2) Run the command line to create the configuration (see above)
lb build
mv live-image-amd64.hybrid.iso run01.iso
lb clean --purge
lb config
lb build
mv live-image-amd64.hybrid.iso run02.iso
diffoscope run01.iso run02.iso --html-dir html_run01_02

Using reprotest

reprotest requires full access on /tmp, so it must be remounted (when applicable)

# 1) Prepare the build environment
# 2) Run the command line to create the configuration (see above)
mount /tmp -odev,exec,suid,remount
reprotest  --variations=-all,+environment,+build_path,-kernel,+aslr,+num_cpus,+time,-user_group,-fileordering,-domain_host,+home,+locales,+exec_path,+timezone,-umask  "lb clean --purge&&lb config&&lb build" "*.iso"
# Disabled tests:
#  kernel        The variant with kernel 2.6 is too old for debootstrap
#  user_group    Live build only works for root, the variation needs a list
#  fileordering  Disorderfs needs to be mounted with dev,exec,suid
#  domain_host   Needs additional rights during debootstrap
#  umask         Some files get a different umask, further investigation is required

See also

Page maintainer: Roland Clobus