There are two ways that the tests are run.
unstable is tested via debci-batch from the debci package
- testing is tested by britney, the migration software maintained by the Release Team.
This page explains the latter setup.
For several years already, maintainers can add autopkgtest test cases to their packages, which are then run on the continuous integration platform on ci.d.n. debci, the framework that runs that site has an API to test individual packages from unstable in testing. britney (the software that determines which packages are allowed to migrate) uses this API to trigger tests for packages that are candidate for migration from unstable to testing and use the results to influence the required age in unstable before the package can migrate to testing. The idea is that a package that is candidate for migration is updated in testing to its candidate version and that the autopkgtest case(s) of the package and those of all reverse dependencies are run. Regression in the results with respect to the current situation in testing will add to the required time a package needs to be in unstable (via the age policy). This extra time can then be used to investigate the situation and file proper bugs to the appropriate package(s), if there is a bug somewhere. On the other hand, if a package has (a) successful autopkgtest(s) and no regression otherwise, it will be rewarded with a reduced required age. Information on what happens is added to the excuses.
It is the intention that in the (far) future regressions will become blocking for migration, but until then the added age will probably be raised over time as a semi-block.
There is one important note to make here on how to go about regressions in test cases from reverse dependencies. We recommend communication between the maintainers of the involved packages as one party has insight in what changed and the other party insight in what is being tested. More information is available in the britney documentation.
Note about migration-reference/0: this is a trigger that has special meaning for britney. britney keeps track of historical PASS results. If a regression migrates to testing britney doesn't know. However, when it encounters the result of a migratio-reference/0 trigger, it stores that as it current reference. These triggers are run with only packages from testing.
Delta to Ubuntu
People may have experience with the way Ubuntu runs the autopkgtests in their framework. There a couple of differences.
- In Debian, we don't block migration (yet), we merely delay migration in case of regression.
- In Debian's setup, only packages that are explicitly enabled are allowed from unstable, while in Ubuntu's setup, there is a fall back where all packages from the proposed pocket are allowed (autopkgtest --no-apt-fallback)
- Debian's britney2 tries to calculate which packages from unstable are needed to make a package (including the test dependencies) install-able. I.e. when a package coming from unstable has a versioned dependency only in unstable, it will be added to the triggers. Same for versioned Breaks.
- For Debian, the reference is the current status in testing (we don't care for the history). To know the current status we trigger migration-reference/0 runs with ONLY testing enabled. Ubuntu regresses if a test passed anywhere in the history on a specific architecture.
- In the Debian setup we retry regressions once a day.
- Debian only runs autopkg tests on amd64, while Ubuntu runs them on all architectures built by Ubuntu.
- Debian doesn't run autopkg tests for binNMUs. E.g. for a library transition, Debian only runs the tests triggered by the library package, it doesn't run the autopkg tests for all the packages built with the new library version. Things known not to be catched in Debian:
- A library rebuild causes an ABI change in another library. E.g. when boost is rebuilt with a new version of icu, it changes ABI (that seems to be not the case in recent boost versions).
- binNMUs picking up unrelated changes, and failing. E.g. dh-python now generates dependencies on python2 instead of python, but the autopkg tests still call python.