This page describes the various processes and tools used by Lucas Nussbaum, David Suárez and others to do archive-wide testing.
Different kinds of tests are done:
archive rebuilds, to find packages that FTBFS. For those tests, a rather standard sbuild+schroot setup is used. main script
installation, removal and upgrade testing. A Ruby rewrite of piuparts (less polished, but generating more useful logs) is used.
- custom rebuilds, to test breakage on new toolchain versions, new compiler flags, etc
General overview of the process
- Amazon Web Services is used to process all tests and generate logs.
- The logs are scanned for failures.
Initiating rebuilds using Amazon Web Services
Scripts used are in https://salsa.debian.org/lucas/cloud-scripts. See https://salsa.debian.org/lucas/cloud-scripts/-/blob/master/README for instructions. The *master* host is reachable at aws-logs.debian.net. As a result, logs are generated in /tmp/logs.
You can look at https://salsa.debian.org/lucas/cloud-scripts/-/blob/master/README.oldway for the old README. Another way to launch the rebuild.
Scanning logs, generating list of failures
svn checkout svn+ssh://svn.debian.org/svn/collab-qa cd collab-qa/archive-rebuilds less fetch-and-process-results-aws export DATE=2013/05/12 # fixme # export ARCH=amd64 # export CHROOT=unstable ./fetch-and-process-results-aws -d /logs/dir
This scans the logs, and publish failed ones on aws-logs.debian.net.
Analyzing logs and filing bugs using collab-qa-tools
(See also the log of an /IRCTutorial)
collab-qa-tools is the set of tools used to scan logs, submit bugs, and track submitted bugs.
# In order to commit bugs lists, you need to be a member of the collab-qa alioth group # setup sudo apt install collab-qa-tools # get rebuild logs locally cd collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64 cat README then e.g rsync -avzP ec2:/see/readme/2011/08/22/ /tmp/cqa.amd64.2011-08-22/ # or wget -r -np -nd http://aws-logs.debian.net/ftbfs-logs/2011/08/22/ # the "todo file" is the file listing the current state of bugs filings # it first need to be created (see instructions in README) export TODOFILE=~lucas/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64/failed.2011-08-22.txt # look at $TODOFILE. note the TODO lines (bugs that need to be filed) # scan logs cqa-scanlogs # scan logs, only TODO bugs cqa-scanlogs -t $TODOFILE # get cache of "interesting" bugs (takes a while) ; creates .bugs.$package files cqa-fetchbugs -t $TODOFILE # setup mutt so that you can send mail locally # start filing bugs. Those matching GCC_ERROR first export DATE=2011/08/22 # date of the rebuild, used to generate the people.d.o link in the mail cqa-annotate -t $TODOFILE -r GCC_ERROR # For each matching failure, the script prompt for an action, which can be: s => skip that failure, 1,2,3,4... => indicate that the bug is one of those already filed, r => report a bug # cqa-annotate also fills $TODOFILE with NNN instead of TODO # It currently uses mutt, so you need it installed, as well as a way to send mail locally
Add bug numbers to $TODOFILE
~/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64$ cqa-importbugnumbers firstname.lastname@example.org qa-ftbfs-20110822 < failed.2011-08-22.txt > failed.2011-08-22.txt2 # check failed.2011-08-22.txt2 mv failed.2011-08-22.txt2 failed.2011-08-22.txt # svn commit
# get instest logs (partial) rsync -avzP people.debian.org:~lucas/public_html/logs/2011/07/26/ /tmp/cqa-inst.amd64.2011-07-26/ # or wget -r -np http://people.debian.org/~lucas/logs/2011/07/26/
The rest of the process is identical. Except the scripts are much less polished.
- Package instest (find a cool name?), maybe inside collab-qa-tools
- Improve instest to test more things (like what piuparts does)
- Use a public cloud infrastructure to make the execution of tests less Lucas-dependent
- Interesting archive rebuild ideas:
build-daemon from hell (rebuild inside chroot with many -dev packages) (references)