This page collects best practices for Debian [[ContinuousIntegration/autopkgtest|autopkgtest]]s.

= Specifications =
https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

= Important =
autopkgtest is meant to test installed packages, not some build artifact or the source of the package.

= Recommendations =
 * Use upstream test-suite if they have as-installed test
 * Try to use standard test solutions available to the language
  * Examples: `TAP`, `nosetest` (Python), `prove` (Perl and not only Perl), `sharness` (Shell) and `shunit2` (Shell)
 * Avoid `build-needed` as much as possible.
  * For large builds it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ sub-directory). It is often possible to run `make -C tests` instead, or copy the test code to `$AUTOPKGTEST_TMP` and build it there with some custom commands.
  * Makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages.
 * Make output verbose
  * Other people may want to investigate regressions without knowing the package
 * Avoid hard coding a reference that changes with new code but doesn't actually break the package. E.g. white space is nearly always irrelevant, the exact text in a warning or error message is hardly ever relevant (consider typo fixes, or style fixes).
 * Keep autopkgtest and tests during the build in sync, especially
   * Output what you are doing (`make` prints the command itself, a shell script doesn't)
   * If you blacklist a test during the build, also blacklist it in the autopkgtest
 * If you need network access during your test, make sure that you are robust against temporal network issues. I.e. retry access at least once, but probably a couple of times after waiting some time (autopkgtest current waits 10 seconds itself on places where it itself needs network). Please also re-read the `Network access` section of the spec.
 * Disable deprecation warnings if your test doesn't have the `allow-stderr` restriction enabled. autopkgtest is not the framework to be warned about deprecation, that's a task for the upstream CI-framework.

= Suggestions =
 * Use `set -x` in scripts when they are written specially for Debian.
 * Use autodep8 if it support the language of your package, but extend it where possible with more/real tests. 
 * For Python related packages test for all supported versions (not always feasible for applications, but do it if you can)
   * Include `python3-all` in the test depends to ensure all supported versions are installed
   * Use `py3versions` with the `-s` flag to loop over the supported versions

= Considerations =
 * Avoid very long test suites. Hardly any test suite is worth hours of testing.
 * Time outs can often be worked around by breaking tests up into smaller tests. I.e. each test stanza in `debian/tests/control` has its own time out.
 * Test can change the system they run on; there are restrictions to tell how severe the system is changed (i.e. `breaks-testbed`, `isolation-container`, `isolation-machine`)
 * Mark simple tests as `superficial`.

= Don't =
 * Don't have only one test with only `Test-command: /bin/true` (or similar). We have [[https://piuparts.debian.org/|piuparts]] for that.
 * Don't disable testing during the build, and then rebuild the package to be able to test it in the autopkgtest. This basically disables testing on all the architectures not supported by ci.d.n.
 * Don't use different testing tools for build testing and autopkgtest testing. See `yapf`, `nose` during build, `unittest` for autopkgtest.
 * Don't install packages during the test unless:
  * it's part of the functionality of the package
  * or if configuration of some package is needed before installing other packages.
 * Don't run code checkers on your source. If the code checker improves, it can't migrate if it finds issues in your package. These checkers are meant for development CI.