Differences between revisions 4 and 5
Revision 4 as of 2011-07-27 14:57:47
Size: 3562
Comment:
Revision 5 as of 2011-07-27 14:59:31
Size: 3631
Comment:
Deletions are marked like this. Additions are marked like this.
Line 53: Line 53:
# rebuid logs
Line 55: Line 54:

The rest of the process is identical. Except the scripts are much less polished.

This page describes the various processes and tools used by Lucas Nussbaum (and others) to do archive-wide testing.

Two different kinds of tests are done:

  1. archive rebuilds, to find packages that FTBFS. For those tests, a rather standard sbuild+schroot setup is used. main script

  2. installation, removal and upgrade testing. A Ruby rewrite of piuparts (less polished, but generating more useful logs) is used.

General overview of the process

  1. Lucas uses Grid'5000 to run the tests on each package, and generates logs. The scripts to do that are quite Grid'5000-specific, and quite dirty too. They can be found in the collab-qa SVN repo, debcluster dir.

  2. Logs are parsed on a Grid'5000 machine to generate a summary list. Then logs are pushed to people.d.o, and the summary list is committed to collab-qa (archive rebuilds ; installation testing)

  3. Joe (not Lucas, preferably) gets all the logs locally, then process (+ file bugs) them using collab-qa-tools

Using collab-qa-tools

collab-qa-tools is the set of tools used to scan logs, submit bugs, and track submitted bugs.

Filing bugs

# setup
svn checkout svn+ssh://svn.debian.org/svn/collab-qa
cd collab-qa/collab-qa-tools ; sudo ruby setup.rb (installs stuff in /usr/local/)
# get rebuild logs locally
rsync -avzP ravel.debian.org:~lucas/public_html/logs/2011/07/27/ /tmp/cqa.amd64.2011-07-27/
# or
wget -r -np  http://people.debian.org/~lucas/logs/2011/07/27/
# the "todo file" is the file listing the current state of bugs filings
export TODOFILE=~lucas/dev/collab-qa/archive-rebuilds/2011-07-27-lsid64-amd64/failed.2011-07-27.txt
# look at $TODOFILE. note the TODO lines (bugs that need to be filed)
# scan logs
cqa-scanlogs
# scan logs, only TODO bugs
cqa-scanlogs -t $TODOFILE
# get cache of "interesting" bugs (takes a while)  ; creates .bugs.$package files
cqa-fetchbugs -t $TODOFILE
# start filing bugs. Those matching GCC_ERROR first
export DATE=2011/07/27 # date of the rebuild, used to generate the people.d.o link in the mail
cqa-annotate -t $TODOFILE -r GCC_ERROR
# this fills $TODOFILE with NNN instead of TODO

Add bug numbers to $TODOFILE

~/dev/collab-qa/archive-rebuilds/2011-07-27-lsid64-amd64$ cqa-importbugnumbers debian-qa@lists.debian.org qa-ftbfs-20110727 < failed.2011-07-27.txt > failed.2011-07-27.txt2
# check failed.2011-07-27.txt2
mv failed.2011-07-27.txt2 failed.2011-07-27.txt
# svn commit

installation testing

# get instest logs (partial)
rsync -avzP ravel.debian.org:~lucas/public_html/logs/2011/07/26/ /tmp/cqa-inst.amd64.2011-07-26/
# or
wget -r -np  http://people.debian.org/~lucas/logs/2011/07/26/

The rest of the process is identical. Except the scripts are much less polished.

TODO

  • Package collab-qa-tools
  • Package instest (find a cool name?), maybe inside collab-qa-tools
  • Improve instest to test more things (like what piuparts does)
  • Use a public cloud infrastructure to make the execution of tests less Lucas-dependent
  • Interesting archive rebuild ideas:
  • * clang
  • * build-daemon from hell (rebuild inside chroot with many -dev packages)