Testing NMUs before uploading

In #700849#15 Andreas wrote:

Here is the script I used recently to do install and upgrade tests for packages I NMUed (targetting wheezy via sid, or t-p-u, or s-p-u) But I always copied it, adjusted LOCAL and logprefix and set FROM/VIA/TO as appropriate ... (setting only FROM or TO switches from distupgrade to install+purge and install-upgrade-purge test)

I don't know how to do this with a script that only takes a package as argument. You don't really want to test single packages. You probably want to test all packages from a .changes file. And each of these you want to test individually. And they usually have versioned dependencies on each other. Passing individial .debs or .changes to piuparts is not really helpful for this case (by default passing a .changes would install all .debs it lists with dpkg and ignoring dependency ordering). Therefore I implemented --testdebs-repo and --distupgrade-to-testdebs.

Oh, and it uses several options not yet in the public GIT, just drop them.

MIRROR=<SET THIS>
TARBALLDIR=../../basetgz

LOCAL=/tmp/000
#URL=http://
SUBDIR=.
logprefix=xx000

#FROM=lenny
FROM=squeeze

#VIA=squeeze
#VIA=squeeze-backports
#VIA=wheezy

#TO=squeeze
TO=wheezy
#TO=sid

#NO_UPGRADE_TEST=--no-upgrade-test
#WARN_ON_LEFTOVERS_AFTER_PURGE=--warn-on-leftovers-after-purge

### END OF CONFIGURABLE PART

DISTRO=${FROM:-$TO}

if [ -n "$LOCAL" ]; then
        URL=file://$LOCAL
        GET_PACKAGES="cat $LOCAL/Packages"
else
        GET_PACKAGES="wget $URL/$SUBDIR/Packages -O -"
fi

DEBS=$($GET_PACKAGES | perl -ne '$p = $1 if /^Package: (.*)/; print "$p=$1\n" if /^Version: (.*)/;')

for p in $DEBS
do
        v=${p#*=}
        p=${p%=*}
        l=${logprefix}${FROM:+-$FROM}${VIA:+-$VIA}${TO:+-$TO}_$p
        test ! -f $l.log || continue
        rm -f $l.log $l.faillog
        sudo nice \
                env PYTHONPATH=/org/piuparts.debian.org/lib/python2.6/dist-packages:/org/piuparts.debian.org/lib/python2.7/dist-packages \
                ~/bin/piuparts \
                --skip-logrotatefiles-test \
                --scriptsdir /etc/piuparts/scripts \
                --warn-on-others \
                --allow-database \
                ${MIRROR:+--mirror $MIRROR} \
                --tmpdir /tmp/piupartss \
                -b ${TARBALLDIR:-.}/$DISTRO.tar.gz \
                ${FROM:+-d $FROM} \
                ${VIA:+-d $VIA} \
                ${TO:+-d $TO} \
                --log-file $l.log \
                ${LOCAL:+--bindmount $LOCAL} \
                --testdebs-repo "deb $URL $SUBDIR/" \
                --do-not-verify-signatures \
                --distupgrade-to-testdebs \
                --dpkg-noforce-unsafe-io \
                $NO_UPGRADE_TEST \
                $WARN_ON_LEFTOVERS_AFTER_PURGE \
                --apt $p=$v
        test $? = 0 || mv $l.log $l.faillog
done

Howto setup a piuparts test-instance for a custom archive

*This is still incomplete.*

Setup a private repo using reprepro

To be able to run piuparts on packages outside the archive, they need to be in a repository, which is available via http.

Install reprepro and create a repo in /var/www/debian:

Codename: sid
Architectures: amd64 i386 source
Components: main contrib non-free
UDebComponents: main

Codename: wheezy
Architectures: amd64 i386 source
Components: main contrib non-free
UDebComponents: main

Codename: squeeze
Architectures: amd64 i386 source
Components: main contrib non-free
UDebComponents: main

Codename: lenny
Architectures: amd64 i386 source
Components: main contrib non-free
UDebComponents: main

reprepro -b /var/www/debian export lenny
reprepro -b /var/www/debian export squeeze
reprepro -b /var/www/debian export wheezy
reprepro -b /var/www/debian export sid

reprepro -b /var/www/debian -C main includedeb wheezy *deb

reprepro -b /var/www/debian -C main remove ${PKG}

Create pbuilder tarballs

This step can be skipped, but if you create tarballs with pbuilder first, creating the initial chroot is faster, because there is no need for a debootstrap each time.

For each distro (eg wheezy) and each arch (eg amd64), create a configfile.

/etc/pbuilder/pbuilderrc.i386-wheezy:

ARCHITECTURE=i386
DISTRIBUTION=wheezy
BASETGZ=/var/cache/pbuilder/base-$DISTRIBUTION-$ARCHITECTURE.tgz
BUILDRESULT=/var/cache/pbuilder/result-$DISTRIBUTION-$ARCHITECTURE

MIRRORSITE=http://ftp.de.debian.org/debian

PKGNAME_LOGFILE_EXTENTION=.buidlog

Then just run

pbuilder --create --configfile /etc/pbuilder/pbuilderrc.i386-wheezy

run install test

TODO

Run upgrade test squeeze2wheezy

In this example, /ram is a ramdisk which is large enough to do the piuparts run.

run_piuparts_squeeze2wheezy:

#! /bin/bash

PKG=$1
LOCALREPO=http://localhost/debian

piuparts \
 --tmpdir=/ram \
 --warn-on-others \
 --no-eatmydata \
 --warn-on-leftovers-after-purge \
 --skip-logrotatefiles-test \
 --scriptsdir /etc/piuparts/scripts \
 -d squeeze \
 -d wheezy \
 -m http://ftp.de.debian.org/debian/ \
 -m ${LOCALREPO} \
 --do-not-verify-signatures \
 -b /var/cache/pbuilder/base-squeeze-i386.tgz \
 --apt \
 $PKG | tee ${PKG}.`date +%s`.log

Run upgrade test lenny2squeeze2wheezy

As lenny isn't available on mirrors anymore, it needs to be downloaded from archive.debian.org. Currently, piuparts doesn't really have a nice way to deal with this (see bug #699028), but this patch works for now:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=699028#5

run run_piuparts_lenny2squeeze2wheezy:

PKG=$1
LOCALREPO=http://localhost/debian
PIUPARTSDIR=/path/to/piuparts/dir

${PIUPARTSDIR}/piuparts.lenny \
 --tmpdir=/ram \
 --warn-on-others \
 --no-eatmydata \
 --warn-on-leftovers-after-purge \
 --skip-logrotatefiles-test \
 --scriptsdir /etc/piuparts/scripts \
 --dpkg-noforce-unsafe-io \
 --no-eatmydata \
 -d lenny \
 -d squeeze \
 -d wheezy \
 -m http://ftp.de.debian.org/debian/ \
 -m ${LOCALREPO} \
 --do-not-verify-signatures \
 -b /var/cache/pbuilder/base-lenny-i386.tgz \
 --apt \
 $PKG | tee ${PKG}.`date +%s`.log

Run custom scripts

TODO: describe this:


CategoryPermalink