This page collects best practices for using autopkgtest in Debian. To learn how to use autopkgtest, see autopkgtest. For a discussion of how autopkgtest fits in to Debian's continuous integration framework, see ContinuousIntegration/autopkgtest. For information about Debian's continuous integration framework, see ContinuousIntegration.
Specifications
DEP-8 (draft)
Recommendations
autopkgtest is meant to test installed packages, not some build artifact or the source of the package.
- use the upstream test-suite if they have an "as-installed" test
prefer standard testing frameworks (e.g. ones that output TAP)
Python: pytest
Perl: prove (also works with other languages)
shell: sharness, shunit2
avoid build-needed wherever possible
for large builds, it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ sub-directory)
if possible, run make -C tests instead, or copy the test code to $AUTOPKGTEST_TMP and build it there with custom commands
- makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages
- make output verbose
- other people should be able to investigate regressions without knowing the package
- avoid hard-coding values that can change without breaking the package
- e.g. whitespace is nearly always irrelevant, the exact text in a warning or error message could change due to a typo fix, style change etc.
- ensure autopkgtests stay in sync with build tests
output what you are doing (make prints the command itself, a shell script doesn't)
- if you blacklist a build test, also blacklist the associated autopkgtest
- if you need network access during your test, make your test robust against temporary network issues
- e.g. retry access at least once, but probably a couple of times with a pause in between
autopkgtest itself currently waits 10 seconds whenever it needs to use the network
please also re-read the "Network access" section of the spec
- e.g. retry access at least once, but probably a couple of times with a pause in between
disable deprecation warnings if your test doesn't have the allow-stderr restriction
autopkgtest is the wrong place to warn about deprecations - that's a task for the upstream CI framework
Suggestions
use set -x in scripts when they are written specially for Debian
use autodep8 if it supports your language, but add more/real tests if possible
- for Python-related packages, test for all supported versions (not always feasible for applications, but do it if you can)
include python3-all in the test depends to ensure all supported versions are installed
use py3versions with the -s flag to loop over the supported versions
Considerations
- avoid very long test suites - hardly anything is worth hours of testing
- timeouts can often be worked around by breaking tests up into smaller tests
each stanza in debian/tests/control has its own timeout
- tests can change the system they run on
there are restrictions to tell how severely the system is changed (i.e. breaks-testbed, isolation-container, isolation-machine)
mark simple tests as superficial
Don't
don't create a test suite with just Test-command: /bin/true (or similar)
we have piuparts for that
- don't disable testing during the build, then rebuild the package to test it in the autopkgtest
this basically disables testing on all the architectures not supported by ci.d.n.
- don't use different testing tools for build testing and autopkgtest testing
e.g. yapf, pytest during build, then unittest for autopkgtest
- don't install packages in the test command itself
- unless you're testing your program's ability to install packages
- or you're testing what happens when one package is configured before another is installed
- don't run code-checkers on your source
- improvements to the code checker might cause tests to unexpectedly start failing
autopkgtest is the wrong place to check code style - that's a task for the upstream CI framework