This page documents tips and strategies to debug regressions in the ci.d.n framework for testing.
src_trigger is the package that tries to migrate from unstable to testing and triggers a regression.
src_broken is the package containing the autopkgtest that starts failing when using the unstable binary packages of src_trigger in testing.
- Check the error message in the autopkgtest log file.
- Sometimes it is obvious what the problem is and which package needs fixing.
- Check if the tests of src_broken also started failing in unstable recently.
- New version of src_broken: If it started failing with a new version of src_broken, this is a strong hint for a regression in the package's autopkgtest.
Solution: update src_broken
- Same version of src_broken: If is started failing consistently while it succeeded before with the same version, this is strong hint that some dependency caused the regression.
Solution: fix src_trigger to not cause the regression
Solution: fix src_broken to adapt to the new situation + possibly add Breaks: src_broken (<< fixed-version) to src_trigger
Solution: if the regression already propagated to testing, retrigger the migration-reference/0 test
- New version of src_broken: If it started failing with a new version of src_broken, this is a strong hint for a regression in the package's autopkgtest.
- Check if the version of src_broken is the same in unstable and testing.
- If the autopkgtest in unstable also fails, this most likely means that the reference needs updating.
Solution: if the regression already propagated to testing, retrigger the migration-reference/0 test. However, there is a cron job running, that tries to make sure that the reference is not older than a week, so this happens automatically.
- If the autopkgtest in unstable also fails, this most likely means that the reference needs updating.
- Check the changelog file for src_trigger.
- Sometimes the changelog already mentions a change that explains the regression
Solution: fix src_broken to adapt to the new situation + possibly add Breaks: src_broken (<< fixed-version) to src_trigger
- Sometimes the changelog already mentions a change that explains the regression
- Check if src_broken is part of a transition.
- Transitions lead to uninstallability issues in unstable.
Solution: wait some time and retrigger the test. However, there is a cron job running, that retries regressions daily.
- Transitions lead to uninstallability issues in unstable.
- Check if the installation of the (test) dependencies had installability issues.
If that is the case, while unstable is fine, this typically means some (versioned) (test) dependency missing or one of the packages coming from unstable is missing a versioned Breaks somewhere. Also (versioned) dependencies pulled in via autodep8 control files are missed. (See also https://lists.debian.org/debian-ci/2018/09/msg00040.html)
Solution: if the package uses autodep8, contact elbrus@d.o
Solution: fix versioned dependencies; in the package (debian/control and/or in the test (debian/tests/control)
Check if a package has needs-recommends as test Restrictions and check that the regression is not caused due to missing packages
autopkgtest asks APT to install the recommends, but APT doesn't fail and does that silently if it can't install recommends (bug #896698)
Solution: stop using needs-recommends but make sure that the required test dependencies are in the list of test dependencies. (bug #896698 may still be fixed on day by changing the way needs-recommends is implemented)
- Check the history of the autopkgtest in both unstable and testing.
- If the failure already migrated to testing, there is no regression
Solution: one can always do this: retrigger the migration-reference/0 test. However, there is a cron job running, that tries to make sure that the reference is not older than a week.
- If the failure already migrated to testing, there is no regression
Links:
Retrigger past tests: https://ci.debian.net/api/v1/retry/0
Fix information for recent regressions