This page describes the various processes and tools used by Lucas Nussbaum (and others) to do archive-wide testing.
Two different kinds of tests are done:
archive rebuilds, to find packages that FTBFS. For those tests, a rather standard sbuild+schroot setup is used. main script
installation, removal and upgrade testing. A Ruby rewrite of piuparts (less polished, but generating more useful logs) is used.
General overview of the process
- Amazon Web Services is used to process all tests and generate logs.
- The logs are scanned for failures.
Someone gets all the logs locally, then process (+ file bugs) them using collab-qa-tools
Initiating rebuilds using Amazon Web Services
Scanning logs, generating list of failures
svn checkout svn+ssh://svn.debian.org/svn/collab-qa cd collab-qa/archive-rebuilds less fetch-and-process-results-aws export DATE=2013/05/12 # fixme export ARCH=amd64 export CHROOT=unstable ./fetch-and-process-results-aws
This scans the logs, and tries to push them to people.debian.org. Someone needs to hack the script so that the location can be configured.
Analyzing logs and filing bugs using collab-qa-tools
(See also the log of an /IRCTutorial)
collab-qa-tools is the set of tools used to scan logs, submit bugs, and track submitted bugs.
# In order to commit bugs lists, you need to be a member of the collab-qa alioth group # setup svn checkout svn+ssh://svn.debian.org/svn/collab-qa cd collab-qa/collab-qa-tools ; dpkg-buildpackage -us -uc ; sudo debi # get rebuild logs locally cd collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64 cat README then e.g rsync -avzP ravel.debian.org:~lucas/public_html/logs/2011/08/22/ /tmp/cqa.amd64.2011-08-22/ # or wget -r -np -nd http://people.debian.org/~lucas/logs/2011/08/22/ # the "todo file" is the file listing the current state of bugs filings # it first need to be created (see instructions in README) export TODOFILE=~lucas/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64/failed.2011-08-22.txt # look at $TODOFILE. note the TODO lines (bugs that need to be filed) # scan logs cqa-scanlogs # scan logs, only TODO bugs cqa-scanlogs -t $TODOFILE # get cache of "interesting" bugs (takes a while) ; creates .bugs.$package files cqa-fetchbugs -t $TODOFILE # setup mutt so that you can send mail locally # start filing bugs. Those matching GCC_ERROR first export DATE=2011/08/22 # date of the rebuild, used to generate the people.d.o link in the mail cqa-annotate -t $TODOFILE -r GCC_ERROR # For each matching failure, the script prompt for an action, which can be: s => skip that failure, 1,2,3,4... => indicate that the bug is one of those already filed, r => report a bug # cqa-annotate also fills $TODOFILE with NNN instead of TODO # It currently uses mutt, so you need it installed, as well as a way to send mail locally
Add bug numbers to $TODOFILE
~/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64$ cqa-importbugnumbers firstname.lastname@example.org qa-ftbfs-20110822 < failed.2011-08-22.txt > failed.2011-08-22.txt2 # check failed.2011-08-22.txt2 mv failed.2011-08-22.txt2 failed.2011-08-22.txt # svn commit
# get instest logs (partial) rsync -avzP ravel.debian.org:~lucas/public_html/logs/2011/07/26/ /tmp/cqa-inst.amd64.2011-07-26/ # or wget -r -np http://people.debian.org/~lucas/logs/2011/07/26/
The rest of the process is identical. Except the scripts are much less polished.
- Package collab-qa-tools
- Package instest (find a cool name?), maybe inside collab-qa-tools
- Improve instest to test more things (like what piuparts does)
- Use a public cloud infrastructure to make the execution of tests less Lucas-dependent
- Interesting archive rebuild ideas:
- * clang
- * build-daemon from hell (rebuild inside chroot with many -dev packages)