System upgrade testing

• Mentor: LarsWirzenius or possibly Michael Vogt

• Summary: Automatic testing of upgrades of entire running systems, using QEMU

• Required skills: python, understanding of the packaging system, python-apt would be good

Description:

Develop a tool that can be pointed at apt package repositories and which will test upgrades from one release to another (where either release may be tentative). That is, from sarge to etch, or etch to unstable, or etch to a custom repository with packages that have been modified for a big library transition. The tests should be run under QEMU (or some other emulator/virtual machine) so that packages can safely start daemons and so on. The actual backend should be plugable so that e.g. a chroot/XEN/vserver target can be used as well to get quick results.

The system should be able to bootstrap the initial guest system based on various policies. Those include "minimal system", "all packages of priority standard and above", "all optional packages as well", or "this particular set of packages" and "try to install as many packages as do not conflict (taking priority into account to make sure we have the important ones installed) and then upgrade from this set".

Removals on the upgrade need to be checked in some way to see what caused them. Any errors during the upgrade should be collected and output at the end of the upgrade test.

In addition to error detection it should detect spurious conffile changes. It needs to deal with packages that do direct prompting on the terminal. Python-apt supports getting error and conffile messages from dpkg.

The goal is to have a tool that Debian can run automatically as often as required to be able to catch upgrade problems early. Ubuntu has something on these lines written in python/python-apt (see http://people.ubuntu.com/~mvo/automatic-upgrade-testing/ and http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/), but based on chroots, not an emulator.

A further idea: For ease of use, having a way of creating the virtual disk/environment from an existing environment would be cool. I'm thinking primarily of sysadmin users here: they could run the make-virtualization-snapshot-of-running-system command on their existing server, then use the snapshot to test upgrades. The snapshot should include not just the same versions of the packages, but also the same configurations.