This page collects best practices for Debian autopkgtests.
Specifications
https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
Recommendations
- Use upstream test-suite if they have as-installed test
- Try to use standard test solutions available to the language
- Examples: TAP, nosetest (Python),
Avoid build-needed as much as possible.
For large builds it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ sub-directory). It is often possible to run make -C tests instead, or copy the test code to $AUTOPKGTEST_TMP and build it there with some custom commands.
- Makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages.
- Make output verbose
- Other people may want to investigate regressions without knowing the package
- Avoid hard coding a reference that changes with new code but doesn't actually break the package. E.g. white space is nearly always irrelevant.
- Keep autopkg tests and tests during the build in sync, especially
- Output what you are doing (make prints the command itself, a shell script doesn't)
- If you blacklist a test during the build, also blacklist it in the autopkg tests
If you need network access during your test, make sure that you are robust against temporal network issues. I.e. retry access at least once, but probably a couple of times after waiting some time (autopkgtest current waits 10 seconds itself on places where it itself needs network). Please also re-read the Network access section of the spec.
Suggestions
Use set -x in scripts when the are written specially for Debian.
Considerations
- Avoid very long test suites. Hardly any test suite is worth hours of testing.
- Time outs can often be worked around by breaking tests up into smaller tests
- autopkgtest is meant to test *installed* packages
Don't
Have one test with only Test-command: /bin/true (or similar). We have piuparts for that.
- Don't disable testing during the build, and then rebuild the package to be able to test it in the autopkg tests. This basically disables ANY blocking on test failures on amd64, and disables ANY testing on all the other architectures. This is how KDE is packaged ...
- Use the same test system for build testing and autopkg testing. See yapf, nose for the build, unittest for debci.