This page collects best practices for Debian autopkgtests.
Specifications
https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
Important
autopkgtest is meant to test installed packages, not some build artifact or the source of the package.
Recommendations
- Use upstream test-suite if they have as-installed test
- Try to use standard test solutions available to the language
Examples: TAP, nosetest (Python), prove (Perl and not only Perl), sharness (Shell) and shunit2 (Shell)
Avoid build-needed as much as possible.
For large builds it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ sub-directory). It is often possible to run make -C tests instead, or copy the test code to $AUTOPKGTEST_TMP and build it there with some custom commands.
- Makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages.
- Make output verbose
- Other people may want to investigate regressions without knowing the package
- Avoid hard coding a reference that changes with new code but doesn't actually break the package. E.g. white space is nearly always irrelevant, the exact text in a warning or error message is hardly ever relevant (consider typo fixes, or style fixes).
- Keep autopkgtest and tests during the build in sync, especially
Output what you are doing (make prints the command itself, a shell script doesn't)
- If you blacklist a test during the build, also blacklist it in the autopkgtest
If you need network access during your test, make sure that you are robust against temporal network issues. I.e. retry access at least once, but probably a couple of times after waiting some time (autopkgtest current waits 10 seconds itself on places where it itself needs network). Please also re-read the Network access section of the spec.
Disable deprecation warnings if your test doesn't have the allow-stderr restriction enabled. autopkgtest is not the framework to be warned about deprecation, that's a task for the upstream CI-framework.
Suggestions
Use set -x in scripts when they are written specially for Debian.
- Use autodep8 if it support the language of your package, but extend it where possible with more/real tests.
- For Python related packages test for all supported versions (not always feasible for applications, but do it if you can)
Include python3-all in the test depends to ensure all supported versions are installed
Use py3versions with the -s flag to loop over the supported versions
Considerations
- Avoid very long test suites. Hardly any test suite is worth hours of testing.
Time outs can often be worked around by breaking tests up into smaller tests. I.e. each test stanza in debian/tests/control has its own time out.
Test can change the system they run on; there are restrictions to tell how severe the system is changed (i.e. breaks-testbed, isolation-container, isolation-machine)
Mark simple tests as superficial.
Don't
Don't have only one test with only Test-command: /bin/true (or similar). We have piuparts for that.
- Don't disable testing during the build, and then rebuild the package to be able to test it in the autopkgtest. This basically disables testing on all the architectures not supported by ci.d.n.
Don't use different testing tools for build testing and autopkgtest testing. See yapf, nose during build, unittest for autopkgtest.
- Don't install packages during the test unless:
- it's part of the functionality of the package
- or if configuration of some package is needed before installing other packages.
- Don't run code checkers on your source. If the code checker improves, it can't migrate if it finds issues in your package. These checkers are meant for development CI.