Bisecting a bug using the Debian wayback machine

Introduction

Debian doesn't yet have an ostree based snapshot system so we can't do system bisecting like Fedora can, but Debian does have a marvellous wayback machine that you can easily install old packages and chroots from.

The Debian wayback machine has a JSON based API that can simply be queried with curl and processed with jq to find the relevant information.

Which archival dates to test on?

If you have a package that you think probably caused the issue, you can reduce the amount of installations you need to create by restricting the installation dates to those where each of the package versions were first seen. Otherwise, you will have to test all dates between two dates.

Finding dates for specific packages

Find out which versions of the package (perl for example) are available:

$ curl -s http://snapshot.debian.org/mr/package/perl/ |
  jq -j '.result[].version + "\u0000"' > versions
$ head -zn1 versions ; echo
5.30.0-6

Filter the versions to those between two releases:

$ rmadison -u debian -a amd64 -s stretch,buster perl 
perl       | 5.24.1-3+deb9u5 | oldstable  | amd64
perl       | 5.28.1-6        | stable     | amd64
$ < versions \
  xargs -0 -n1 \
   sh -c '
    dpkg --compare-versions "$1" ge 5.24.1-3 &&
    dpkg --compare-versions "$1" le 5.28.1-6 &&
    printf "%s\0" "$1"' sh \
    > filtered-versions
$ head -zn1 filtered-versions ; echo
5.28.1-6
$ tail -zn1 filtered-versions ; echo
5.24.1-3

Find out each of the .deb files for your architecture and find out which dates contain those files:

$ < filtered-versions \
  xargs -0 -I{} curl -s http://snapshot.debian.org/mr/binary/perl/{}/binfiles?fileinfo=1 |
  jq -j '.fileinfo[.result[] | select(.architecture=="amd64").hash][].first_seen + "\u0000"' |
  sort -zu \
  > filtered-dates
$ head -zn1 filtered-dates ; echo
20170521T033922Z
$ tail -zn1 filtered-dates ; echo
20190331T222337Z

Now you have a list of archival dates when new package versions appeared in Debian.

All archival dates between two dates

At the time of writing, the snapshot API does not support listing archival dates, but you can get them from the web interface, which lists archival dates for each month.

How to bisect the list of dates

Now that you have a list of dates to test, you can install Debian from the snapshots of the archive on those dates but installing from every single one of those dates would waste time and bandwidth on dates when there was no behaviour change. Instead you will want to use bisection to reduce the amount of steps to find out when the change occurred and only install each for step.

Since the most sophisticated bisect tools available are mostly within version control systems like git (although a command-line bisect tool exists), it is best to just use git for a non-git bisect situation.

First prepare a git repository with a file containing the snapshot date with one commit per change of the date:

$ git init bisect-dates
$ < filtered-dates \
  xargs -0 -n1 \
   sh -c '
    printf "%s" "$1" > bisect-dates/date &&
    git -C bisect-dates add date &&
    git -C bisect-dates commit -m "$1"
    ' sh

Doing the bisection

Now that you have a git repository with one commit for each date where the package you are interested in has changed, you can use git-bisect in the usual way to find out which date caused the regression.

For each bisection point you will want to create a chroot where you can test the package you are interested in, containing the package itself and any test dependencies you will need:

date="$(cat date)" &&
sudo debootstrap --include=perl,libfoo-perl,libbar-perl sid "chroot-$date" "https://snapshot.debian.org/archive/debian/$date/"

You will want to do git bisect skip when debootstrap fails. If you have automated your test with a script for git bisect run, then you can skip dates by exiting the script with return code 125, indicating the test could not be run and the date should be skipped, something like this:

date="$(cat date)" &&
sudo debootstrap --include=perl,libfoo-perl,libbar-perl sid "chroot-$date" "https://snapshot.debian.org/archive/debian/$date/" ||
exit 125

Downgrading instead of reinstalling

As a performance optimisation during testing, you can downgrade/upgrade all packages to specific snapshot dates and then test after each downgrade/upgrade.

Refining the bisection

Once you have found a pair of dates between which the issue started appearing, you can try to find out if the dates in between had any effect by repeating the bisection with the dates in between them.

Isolating the package

Once you have found the exact pair of dates between which the issue started appearing, you can try to find out which of the package changes between those two dates caused the issue. Install the first date, add the second date to the apt sources and then upgrade one package at a time and note which one caused the change in behaviour.

Bisecting upstream

At this point you can move on to bisecting the upstream project that caused the issue. Some projects have tutorials for bisecting them, for example the Linux kernel.


CategoryDebugging

ToDo: