This page documents tips and strategies to debug regressions in the [[https://ci.debian.net|ci.d.n ]] framework for testing.

src_trigger is the package that tries to migrate from unstable to testing and triggers a regression.

src_broken is the package containing the autopkgtest that starts failing when using the unstable binary packages of src_trigger in testing.

 * Check the error message in the autopkgtest log file.
  * Sometimes it is obvious what the problem is and which package needs fixing.
 * Check if the tests of src_broken also started failing in unstable recently.
  * New version of src_broken: If it started failing with a new version of src_broken, this is a strong hint for a regression in the package's autopkgtest.
   * '''Solution''': update src_broken
  * Same version of src_broken: If is started failing consistently while it succeeded before with the same version, this is strong hint that some dependency caused the regression.
   * '''Solution''': fix src_trigger to not cause the regression
   * '''Solution''': fix src_broken to adapt to the new situation + possibly add Breaks: src_broken (<< fixed-version) to src_trigger
   * '''Solution''': if the regression already propagated to testing, retrigger the migration-reference/0 test
 * Check if the version of src_broken is the same in unstable and testing.
  * If the autopkgtest in unstable also fails, this most likely means that the reference needs updating.
   * '''Solution''': if the regression already propagated to testing, retrigger the migration-reference/0 test.  However, there is a cron job running, that tries to make sure that the reference is not older than a week, so this happens automatically.
 * Check the changelog file for src_trigger.
  * Sometimes the changelog already mentions a change that explains the regression
   * '''Solution''': fix src_broken to adapt to the new situation + possibly add Breaks: src_broken (<< fixed-version) to src_trigger
 * Check if src_broken is part of a transition.
  * Transitions lead to uninstallability issues in unstable. 
   * '''Solution''': wait some time and retrigger the test. However, there is a cron job running, that retries regressions daily.
 * Check if the installation of the (test) dependencies had installability issues.
  * If that is the case, while unstable is fine, this typically means some (versioned) (test) dependency missing or one of the packages coming from unstable is missing a versioned Breaks somewhere. Also (versioned) dependencies pulled in via autodep8 control files are missed. (See also https://lists.debian.org/debian-ci/2018/09/msg00040.html)
   * '''Solution''': if the package uses autodep8, contact elbrus@d.o
   * '''Solution''': fix ''versioned'' dependencies; in the package (debian/control and/or in the test (debian/tests/control)
 * Check if a package has '''needs-recommends''' as test Restrictions and check that the regression is not caused due to missing packages
  * autopkgtest asks APT to install the recommends, but APT doesn't fail and does that silently if it can't install recommends ([[https://bugs.debian.org/896698|bug #896698]])
   * '''Solution''': stop using '''needs-recommends''' but make sure that the required test dependencies are in the list of test dependencies. (bug #896698 may still be fixed on day by changing the way needs-recommends is implemented)
 * Check the history of the autopkgtest in both unstable and testing.
  * If the failure already migrated to testing, there is no regression
   * '''Solution''': one can always do this: retrigger the migration-reference/0 test. However, there is a cron job running, that tries to make sure that the reference is not older than a week.

Links: 
 * Retrigger past tests: [[https://ci.debian.net/api/v1/retry/0]]

= Fix information for recent regressions =

 * [[ContinuousIntegration/TriagingTips/openssl-1.1.1|2018-08-27 openssl 1.1.1 runtime issues]]
 * [[ContinuousIntegration/TriagingTips/openmpi3-oversubscription|2018-06-27 openmpi3 oversubscription failures]]
 * [[ContinuousIntegration/TriagingTips/numpy-1.14-doctests|2018-05-09 python-numpy doctest failures]]

= Historical tips for regressions =