Debian packages can specify autopkgtest tests. These tests are run on infrastructure maintained by the Debian CI team (https://ci.debian.net).
Note: This project is different from Salsa CI
There are multiple ways that the tests are triggered.
everybody that has a salsa account can schedule jobs
elbrus schedules tests in a monthly cadence on all suites and most architectures
The Release Team has jobs scheduled automatically by the britney code; the migration software. These jobs run on all suites, with different purposes. The main one, for unstable-to-testing migration is described below, but additionally there's more.
experimental (results in unstable), the results are summarized at the pseudo-excuses
stable-proposed-updates (results in stable) or oldstable-proposed-updates (results in oldstable), the results are used by the upgrade viewer: stable / oldstable
- The security team also uses the setup to test (embargoed) security uploads
This page explains the Release Team britney setup.
For several years already, maintainers can add autopkgtest test cases to their packages, which are then run on the continuous integration platform on ci.d.n. debci, the framework that runs that site has an API to test individual packages from unstable in testing. britney (the software that determines which packages are allowed to migrate) uses this API to trigger tests for packages that are candidate for migration from unstable to testing and use the results to influence the migration to testing. The idea is that a package that is candidate for migration is updated in testing to its candidate version and that the autopkgtest case(s) of the package and those of all reverse dependencies are run. Regression in the results with respect to the current situation in testing will block the package from migrating. On the other hand, if a package has (a) successful autopkgtest(s) and no regression otherwise, it will be rewarded with a reduced required age. Information on what happens is added to the excuses.
There is one important note to make here on how to go about regressions in test cases from reverse dependencies. We recommend communication between the maintainers of the involved packages as one party has insight in what changed and the other party insight in what is being tested. More information is available in the britney documentation.
In case of issues, please contact the release team and ci team: debian-release@lists.debian.org debian-ci@lists.debian.org
Note about migration-reference/0: this is a trigger that has special meaning for britney. britney keeps track of historical PASS results. If a regression migrates to testing britney doesn't know, it could falsely block packages. However, when it encounters the result of a migration-reference/0 trigger, it stores that as it current reference. These triggers are run with only packages from testing.
Delta to Ubuntu
People may have experience with the way Ubuntu runs the autopkgtests in their framework. There a couple of differences.
- Debian's britney2 tries to calculate which packages from unstable are needed to make a package (including the test dependencies) install-able. I.e. when a package coming from unstable has a versioned dependency only in unstable, it will be added to the triggers. Same for versioned Breaks.
- For Debian, the reference is the current status in testing (we don't care for the history). To know the current status we trigger migration-reference/0 runs with ONLY testing enabled. Ubuntu regresses if a test passed anywhere in the history on a specific architecture.
- In the Debian setup we retry regressions once a day, migration-reference/0 runs expire after a week.
- Debian doesn't run autopkgtests for binNMUs. E.g. for a library transition, Debian only runs the tests triggered by the library package, it doesn't run the autopkgtests for all the packages built with the new library version. Things known not to be caught in Debian:
- A library rebuild causes an ABI change in another library. E.g. when boost is rebuilt with a new version of icu, it changes ABI (that seems to be not the case in recent boost versions).
- binNMUs picking up unrelated changes, and failing. E.g. dh-python now generates dependencies on python2 instead of python, but the autopkgtests still call python.