Differences between revisions 21 and 23 (spanning 2 versions)
⇤ ← Revision 21 as of 2023-01-15 08:18:03 →
Size: 4825
Editor: William Desportes
Comment: Fix links
← Revision 23 as of 2023-01-19 10:01:17 → ⇥
Size: 4824
Editor: PaulWise
Comment: minor cleanup
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
Line 12: Line 13:
 1. Someone gets all the logs locally, then process (+ file bugs) them using [[https://tracker.debian.org/pkg/collab-qa-tools|collab-qa-tools]], [[http://anonscm.debian.org/viewvc/collab-qa/collab-qa-tools/|VCS]]  1. Someone gets all the logs locally, then process (+ file bugs) them using [[https://tracker.debian.org/pkg/collab-qa-tools|collab-qa-tools]] ([[https://salsa.debian.org/lucas/collab-qa-tools|VCS]])
Line 15: Line 16:
Line 21: Line 23:
Line 33: Line 36:
Line 38: Line 42:
Line 41: Line 46:
sudo aptitude install collab-qa-tools sudo apt install collab-qa-tools
Line 68: Line 73:
Line 76: Line 82:
Line 86: Line 93:

This page describes the various processes and tools used by Lucas Nussbaum, David Suárez and others to do archive-wide testing.

Different kinds of tests are done:

  1. archive rebuilds, to find packages that FTBFS. For those tests, a rather standard sbuild+schroot setup is used. main script

  2. installation, removal and upgrade testing. A Ruby rewrite of piuparts (less polished, but generating more useful logs) is used.

  3. custom rebuilds, to test breakage on new toolchain versions, new compiler flags, etc

General overview of the process

  1. Amazon Web Services is used to process all tests and generate logs.
  2. The logs are scanned for failures.

  3. Someone gets all the logs locally, then process (+ file bugs) them using collab-qa-tools (VCS)

Initiating rebuilds using Amazon Web Services

Scripts used are in https://salsa.debian.org/lucas/cloud-scripts. See https://salsa.debian.org/lucas/cloud-scripts/-/blob/master/README for instructions. The *master* host is reachable at aws-logs.debian.net. As a result, logs are generated in /tmp/logs.

You can look at https://salsa.debian.org/lucas/cloud-scripts/-/blob/master/README.oldway for the old README. Another way to launch the rebuild.

Scanning logs, generating list of failures

svn checkout svn+ssh://svn.debian.org/svn/collab-qa
cd collab-qa/archive-rebuilds
less fetch-and-process-results-aws
export DATE=2013/05/12 # fixme
# export ARCH=amd64
# export CHROOT=unstable
./fetch-and-process-results-aws -d /logs/dir

This scans the logs, and publish failed ones on aws-logs.debian.net.

Analyzing logs and filing bugs using collab-qa-tools

(See also the log of an ?/IRCTutorial)

collab-qa-tools is the set of tools used to scan logs, submit bugs, and track submitted bugs.

Filing bugs

# In order to commit bugs lists, you need to be a member of the collab-qa alioth group
# setup
sudo apt install collab-qa-tools
# get rebuild logs locally
cd collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64
cat README
then e.g rsync -avzP ec2:/see/readme/2011/08/22/ /tmp/cqa.amd64.2011-08-22/
# or
wget -r -np -nd http://aws-logs.debian.net/ftbfs-logs/2011/08/22/
# the "todo file" is the file listing the current state of bugs filings
# it first need to be created (see instructions in README)
export TODOFILE=~lucas/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64/failed.2011-08-22.txt
# look at $TODOFILE. note the TODO lines (bugs that need to be filed)
# scan logs
cqa-scanlogs
# scan logs, only TODO bugs
cqa-scanlogs -t $TODOFILE
# get cache of "interesting" bugs (takes a while)  ; creates .bugs.$package files
cqa-fetchbugs -t $TODOFILE
# setup mutt so that you can send mail locally
# start filing bugs. Those matching GCC_ERROR first
export DATE=2011/08/22 # date of the rebuild, used to generate the people.d.o link in the mail
cqa-annotate -t $TODOFILE -r GCC_ERROR
# For each matching failure, the script prompt for an action, which can be: s => skip that failure, 1,2,3,4... => indicate that the bug is one of those already filed, r => report a bug
# cqa-annotate also fills $TODOFILE with NNN instead of TODO
# It currently uses mutt, so you need it installed, as well as a way to send mail locally

Add bug numbers to $TODOFILE

~/dev/collab-qa/archive-rebuilds/2011-08-22-lsid64-amd64$ cqa-importbugnumbers debian-qa@lists.debian.org qa-ftbfs-20110822 < failed.2011-08-22.txt > failed.2011-08-22.txt2
# check failed.2011-08-22.txt2
mv failed.2011-08-22.txt2 failed.2011-08-22.txt
# svn commit

installation testing

# get instest logs (partial)
rsync -avzP people.debian.org:~lucas/public_html/logs/2011/07/26/ /tmp/cqa-inst.amd64.2011-07-26/
# or
wget -r -np  http://people.debian.org/~lucas/logs/2011/07/26/

The rest of the process is identical. Except the scripts are much less polished.

TODO

  • Package instest (find a cool name?), maybe inside collab-qa-tools
  • Improve instest to test more things (like what piuparts does)
  • Use a public cloud infrastructure to make the execution of tests less Lucas-dependent
  • Interesting archive rebuild ideas:

    • build-daemon from hell (rebuild inside chroot with many -dev packages) (references)