Differences between revisions 1 and 298 (spanning 297 versions)
Revision 1 as of 2014-04-25 11:06:30
Size: 856
Editor: ?MuehlenhoffUnivention
Comment:
Revision 298 as of 2022-07-24 08:48:16
Size: 41999
Editor: ?ThorstenAlteholz
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Debian LTS development =

== Example apt source for squeeze-lts ==

deb http://ftp.de.debian.org/debian squeeze-lts main
deb-src http://ftp.de.debian.org/debian squeeze-lts main

== Uploads ==

To upload a package simply add "squeeze-lts" to the target distribution. If you use dput-ng, you need to apply the patch from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745806.
After that "dput CHANGES file" is sufficient.

== Versioning of updates ==

The versioning follows the conventions already used in security.debian.org. Historically codenames have been used as version numbers, but this was changed some time ago as version numbers are more deterministic.

If a package already e.g. had a +squeeze1 update, use +squeeze2 for the next update.

If a package hasn't seen an update, use +deb6u1 for the next update.
#language en

~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: English - [[es/LTS/Development|Español]] - [[fr/LTS/Development|Français]] - [[ru/LTS/Development|Русский]]-~

<<BR>> <<BR>> <<BR>> <<BR>>

|| ||<:99%> '''''This page has been moved to [[https://lts-team.pages.debian.net/wiki/LTS-Development.html|the Development page within the LTS team pages]]'''''|| ||

<<BR>> <<BR>> <<BR>> <<BR>>

= 7 steps for publishing a DLA =
''Debian Long Term Support Security Advisory (DLA)''

How-to prepare security updates for LTS
 * [[LTS/Development#Claim_the_issue_in_the_security_tracker_.28in_dla-needed.txt.29|Claim the issue in the security tracker (in dla-needed.txt)]]
 * [[LTS/Development#Build_the_update|Build the update]]
 * [[LTS/Development#Test_the_update|Test the update]]
 * [[LTS/Development#Upload_the_update|Upload the update]]
 * [[LTS/Development#Claim_a_DLA_ID_in_DLA.2Flist|Claim a DLA ID in DLA/list]]
 * [[LTS/Development#Announce_the_update|Announce the update]]
 * [[LTS/Development#Prepare_an_update_for_the_website|Prepare an update for the website]]

= Contributing to Debian Long Term Support =

The Debian LTS team is always looking for more volunteers to do a better job. With more resources, we could for example:
 * support packages that are currently unsupported
 * take care of all Debian packages and not only the most popular ones
 * fix lower priority issues

If you want to get involved in the LTS team and help keep Debian packages secure for 5 years, have a look at this page. We assume that you are already familiar with the repository layout described in [[LTS/Using]] and that you are subscribed to the LTS mailing list: http://lists.debian.org/debian-lts/

You can help in many ways:

<<TableOfContents(3)>>

== Test updated packages and report regressions ==

As a simple user, you can test packages that have been updated (or that are in the process of being updated). If you find a regression (i.e. something that used to work and that no longer works), then please report it to debian-lts@lists.debian.org and put the person who prepared the update in copy (in case they are not subscribed to the list).

Many LTS contributors are looking for testers of their updated packages. They send call for tests on the mailing list, so please [[http://lists.debian.org/debian-lts/|subscribe to it]] and test the packages they provide when you can, and report back whether they worked for you.

== Debian Security Tracker ==

The Debian LTS team makes extensive use of the [[https://security-tracker.debian.org/tracker/|Debian Security Tracker]]. This is a database of all known security issues in Debian.

Developers check out the Git source (see [[#checkout-security-tracker-repo|instructions]]), commit changes, and the website automatically updates to reflect the changes.

LTS Developers are strongly encouraged to keep the data in the security tracker up-to-date. This includes updating the following files:

 * data/CVE/list: Should be updated with broader information that affects more than just LTS (i.e., stable, testing). For example, consider adding to this file links to bug reports related to the security issue, links to upstream commits or new releases which address the vulnerability, links to public mailing list posts, upstream reports, upstream exploits, information regarding if the exploits do or do not work, etc. Any information that could be useful for the security team as well as the LTS team.

 * data/dla-needed.txt: Should be updated to reflect LTS specific information, such as who is working on a package, possible issues that are specific to LTS, test packages available, requests for help, problems encountered, etc.

 * data/DLA/list: In most cases there is no need to manually change this file, as it is maintained automatically by bin/gen-DLA which is described in following sections.

Make sure to read
[[https://security-team.debian.org/security_tracker.html|its documentation]] on the Debian Security Team website.

== Prepare security updates for LTS ==

=== Claim the issue in the security tracker (in dla-needed.txt) ===

In order to prevent duplication of effort, make sure the issue is listed in the [[https://salsa.debian.org/security-tracker-team/security-tracker/blob/master/data/dla-needed.txt|data/dla-needed.txt]] file from the security tracker and add your name to it. Normally, issues are triaged by the frontdesk in `dla-needed.txt` at first. If not, you should add the issue to `dla-needed.txt` yourself. Look at how that is done [[#Guidelines_for_CVE_triaging|below]] before adding an issue or adding your name to an issue.

{{{#!wiki note
We give package maintainers 24 to 72 hours delay before taking on updates ourselves, depending on the severity of the security issue.
}}}

As mentioned in the previous section, make sure to also review and update the relevant information in data/CVE/list when claiming a package for LTS work.

{{{#!wiki caution
You shall only claim it for the time you do active work. You are expected to actively handle the problem within 2-3 days and then you are expected to announce your intention to upload so others can test it for a few (3-4) more days. If you can not solve the problem within the 2-3 days, then please mention why it takes more time, how far you have gone, the preliminary results and/or free it for someone else to work on it. We do not want people to work on a package for a few hours, wait a few days, work again on it, wait more and so on.
}}}

<<Anchor(checkout-security-tracker-repo)>>
Checkout the git repository of the security tracker: (Using a shallow clone because otherwise the operation will take half an hour. The clone will still need a few minutes to complete.)
{{{
$ git clone --depth 1 git@salsa.debian.org:security-tracker-team/security-tracker
$ cd security-tracker
$ sudo apt install jq python3-apt
$ bin/setup-repo
}}}

{{{setup-repo}}} installs a pre-commit hook to check data file syntax.

Then edit the data/dla-needed.txt file:
{{{
$ editor data/dla-needed.txt
$ git commit -m "Claim foo in dla-needed.txt" data/dla-needed.txt
$ git push
}}}

<<Anchor(security-tracker)>>
{{{#!wiki note
'''Write access to the security-tracker repository'''

Debian Developers automatically have commit access to the security-tracker repository. Otherwise you need to be a member of the security-tracker project on salsa.debian.org, please request membership through the [[https://salsa.debian.org/security-tracker-team/|security tracker team gitlab project]], [[https://lists.debian.org/debian-security-tracker/|security-tracker mailinglist]] or [[LTS/Contact#debian-lts|debian-lts mailinglist]].
}}}

=== Build the update ===

Backport the fix to the version in LTS (in case there's already been an earlier update). Make sure that the fix was approved by the upstream project (e.g. committed to their development repository), sometimes early patches proposals are erroneous or incomplete. If there is no fix, it is fine to work on a new one: be a good citizen and contribute the patch back upstream and make sure the maintainer and regular security teams will be aware of them, by adding it to a bug report against the package and linking that bug report in the security tracker. (If the issue being addressed requires !AddressSanitizer in order to reproduce the vulnerability and confirm the fix, then [[LTS/Development/Asan|this page]] may be helpful).

If the fix comes with new binary files (e.g. test cases), they won't be accepted by quilt, but you can reference them in {{{debian/source/include-binaries}}} and they'll be included in the {{{.debian.tar.xz}}} file directly ([[https://lists.debian.org/debian-lts/2021/02/msg00008.html|discussion]]).

In the changelog, you need to set the target distribution in debian/changelog to "'''stretch-security'''". The versioning follows the conventions already used in security.debian.org. Historically codenames have been used as version numbers, but this was changed some time ago as version numbers are more deterministic.

 * If you use dch (pre-buster) to make the changelog entry, note that it may include the text ''Non-maintainer upload by the Security Team.'' It is important to either remove the line, remove ''by the Security Team'', or replace it with something referring to the LTS Team, to avoid giving the appearance of support by the Security Team. Cf DebianBug:762715 ("Non-maintainer upload by the LTS Security Team.").
 * If a package already e.g. had a +deb9u1 update, use +deb9u2 for the next update.
 * If a package hasn't seen an update, use +deb9u1 for the next update.
 * For the rare cases where new upstream versions are introduced, make sure to use a lower version than in stretch. See [[https://lists.debian.org/debian-lts/2020/06/msg00008.html|discussion]]. Just don't use +stretchX as this will cause problems!
 * For the rare case where backports of the entire package from newer distributions is required (as opposed to just the fix), it is expected that the changelog will not be in incrementing version order.

BTS numbers can be referenced but are currently [[https://lists.debian.org/debian-lts/2020/02/msg00048.html|not processed]].

Note that for non-native packages, uploads to security-master need to include the source tarball (even if it's already present in the distribution you're uploading to) by using 'dpkg-buildpackage -sa'.

Make sure you use a clean chroot for building the binary packages. Using [[sbuild]]/[[pbuilder]] is recommended. For instance:
{{{
# Init
sudo pbuilder --create --basetgz /var/cache/pbuilder/base-stretch.tgz --distribution stretch --othermirror 'deb http://security.debian.org/ stretch/updates main contrib'
sudo pbuilder --update --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild source and binary packages from Stretch (in extracted source)
export DEB_BUILD_OPTIONS="parallel=$(nproc)"
pdebuild --use-pdebuild-internal --buildresult .. --debbuildopts '-S' -- --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild binary packages from Stretch, reproducing buildd's separate build-indep/build-arch
# - first security upload:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep --debbuildopts '-sa' package+deb9u1.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log --buildresult . --binary-arch --debbuildopts '-sa' package+deb9u1.dsc
# - later uploads:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep package+deb9u2.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log --buildresult . --binary-arch package+deb9u2.dsc
}}}

{{{#!wiki tip
To actually build the binary package, you may want to use a porter box, especially if you need to fix a regression that concerns a specific architecture you may not have access to, see PorterBoxHowToUse for instructions on how to use porterboxes.
}}}

See also:
 * [[https://www.debian.org/doc/manuals/developers-reference/pkgs.html#preparing-packages-to-address-security-issues|Debian Developer's Reference - Preparing packages to address security issues]]
 * [[https://release.debian.org/wanna-build.txt|Requesting wanna-build actions]] if your package failed to build and you need a gb/giveback/give-back (and the build failure can't actually be fixed cleanly)

=== Test the update ===

See [[LTS/TestSuites|TestSuites]] for package-specific test instructions.

If there's a built-in test suite, make sure it runs fine at build-time in the LTS environment. It is acceptable to enable or even backport test suites from later releases if they are available.

There can also be [[https://dep-team.pages.debian.net/deps/dep8/|DEP-8]] tests in {{{debian/tests}}}, to run with [[https://packages.debian.org/autopkgtest|autopkgtest]]. See for instance [[LTS/TestSuites/rails|TestSuites/rails]] for a direct run or [[LTS/TestSuites/sane-backends|TestSuites/sane-backends]] for a full LXC environment.

If there are no test suites available, you will need to run and test the package directly. Prefer a full environment (virtualized or sometimes even physical).

Do check the reverse-dependencies ({{{find-rdeps}}}, [[https://packages.debian.org/apt-rdepends|apt-rdepends]]{{{ -r}}}, [[https://manpages.debian.org/stretch/dose-extra/dose-ceve.1.en.html#EXAMPLES|dose-ceve]] from [[https://packages.debian.org/dose-extra|dose-extra]]) and install some of them, to check if your package still runs correctly with them.

It is worth testing with both a 32- and 64-bit architecture, see e.g. [[https://lists.debian.org/debian-lts/2019/11/msg00005.html|CVE-2019-14866]].

Check your package update with generic tools as well:
{{{
# check for common packaging issues in last build
# from extracted source after build, stretch host (only check new errors)
lintian -i
# inspect source changes
debdiff package+deb9u3.dsc package+deb9u4.dsc | diffstat
debdiff package+deb9u3.dsc package+deb9u4.dsc | less
# inspect binary changes
debdiff --from deb9u3/*.deb --to deb9u4/*.deb
# test package upgrade
sudo piuparts -d stretch \
  --extra-repo='deb http://security.debian.org/ stretch/updates main contrib' \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  --single-changes-list package+deb9u4_{all,amd64}.changes \
  | grep -P '(INFO|ERROR):'
# also consider --install-remove-install
}}}

Depending on the urgency of the security fix, and your confidence on your own work, you might want to seek reviews from interested parties (such as upstream authors, Debian maintainers, security team, LTS team, etc.). If you ask for reviews, you should supply the following information:

 1. A generated debdiff against the previous version in LTS.
 1. A link to a test package that can be downloaded and tested.

It is common for LTS workers to upload their packages to a personal repository in [[https://www.debian.org/doc/manuals/developers-reference/ch04.en.html#servers-people|people.debian.org]], but any public repository or even simple webserver will do. A few guidelines for those temporary uploads:

 * sign the packages (using `debsign`, for example) so that users can verify package integrity
 * make sure to have the target distribution set to UNRELEASED so that nobody else can upload them before they are ready.

=== Upload the update ===

{{{#!wiki note
Uploads to Debian LTS '''do not wait''' for a DLA or any other manual intervention before being installed into the Debian Archive. In this regard, they are different from stable security uploads.
}}}

If you're satisfied with your tests, upload to security-master, using DebianPackage:dput or DebianPackage:dput-ng:

{{{
dput security-master hello_1.0-1+deb9u1_source.changes
}}}

Once uploaded the package will be auto-built for all the available architectures (if it's an arch:any package). The architectures currently supported are amd64, i386, armel and armhf. Check the build logs at:
https://buildd.debian.org/status/package.php?suite=stretch-security&p=SOURCE_PACKAGE

''Built-Using refers to non-existing source package'': if your package is rejected with this reason, open a bug against package {{{ftp.debian.org}}} requesting to add the missing package, see e.g. DebianBug:974877 or DebianBug:974954.

=== Claim a DLA ID in DLA/list ===

Once your upload has been built successfully, you can go ahead with the DLA.

Run bin/gen-DLA in the top directory of the Git repository. It automatically generates an entry in data/DLA/list and asks you to commit the changes to ensure that no IDs are used twice. The following command would add an entry for src:hello fixing CVE-2014-0666 and creates an advisory template in the top directory of your security-tracker checkout.

{{{
$ DEBFULLNAME=xxx bin/gen-DLA --save hello CVE-2014-0666
# OR
$ DEBFULLNAME=xxx bin/gen-DLA --save .../hello_1.2-3+deb9u4_source.changes
}}}

That list of CVE will make sure that the security tracker is aware of the fix and mark the issues as resolved for that specific package.

{{{gen-DLA}}} will automatically remove prior suite triage for the fixed vulnerabilities (e.g. {{{[stretch] - package <postponed>}}}). However, the {{{{DLA-XXXX-1}}}} annotation will be added by the bi-daily cron, and the fixed version will be inferred automatically (don't add it).

=== Special case: Upload non-free packages ===

Currently (20220119) source-only uploads of non-free packages to security-master are not possible (according to Aurélien Jarno this is a bug). The buildd will not build the package for different architectures. The LTS team still maintains packages like [[https://packages.debian.org/amd64-microcode|amd64-microcode]] and [[https://packages.debian.org/firmware-linux-nonfree|firmware-linux-nonfree]].

New upstream releases should include the full "source-code" and the amd64 binary packages (build with -sa and upload the amd64 changes file).

After that you need to upload only the i386 binary packages as well. This can be achieved by passing the -b flag to dpkg-buildpackage. You can also use the changestool from [[https://packages.debian.org/reprepro|reprepro]] to modify your changes file and to exclude unwanted files from it.

=== Announce the update ===

Once the upload is completed, an email will be sent to you and to the [[https://lists.debian.org/debian-lts-changes|debian-lts-changes]] mailing list, confirming the update has been released into the Debian archive. The `BTS` bot will also announce it in the [[ircs://irc.oftc.net/debian-devel-changes|#debian-devel-changes]] channel.

Send mail to the [[https://lists.debian.org/debian-lts-announce/|debian-lts-announce]] mailing list. The mail needs to be signed by a GPG key in the debian or debian-maintainers keyring. Inline signatures always work (PGP/MIME works with mutt but fails with Enigmail. Thunderbird fails starting from version 76).

The advisory template has been created by bin/gen-DLA (see before) and generally looks like this:

{{{
Subject: [SECURITY] [DLA $DLA-1] $SOURCEPACKAGENAME security update

Package : $SOURCEPACKAGENAME
Version : $LTS_VERSION
CVE ID : CVE-2014-0001 CVE-2014-0002
Debian Bug : 12345

DLA text goes here
[...]
}}}

{{{#!wiki note
In addition to describing the vulnerabilities, remember to add a short package description, so the audience can better understand the context and whether they are affected.
}}}

Please also don't append your personal signatures to DLAs.

If you use mutt, a simple way to send it is to use {{{
mutt -e "set signature=/dev/null" -H DLA-123
}}}

Simple "mail" command works: {{{
cat DLA-XXXX.txt | gpg --clearsign | mail -s "[SECURITY] [DLA XXXX-X] $SOURCEPACKAGENAME security update" -a "From: Max Mustermann <max.mustermann@debian.org>" -r max.mustermann@debian.org debian-lts-announce@lists.debian.org
}}}

Only when you have confirmed that the package was processed after upload (once you get the accept email) should you send the DLA to the mailing list.

When all this is done, you should remove the package from other locations it was uploaded to in the test section, if any.

See also:
 * [[https://www.debian.org/doc/manuals/developers-reference/pkgs.html#security-advisories|Debian Developer's Reference - Security Advisories]]
 * [[DebianSecurity/AdvisoryCreation/SecFull#Sending_out_the_announcement_to_debian-security-announce]]

=== Prepare an update for the website ===

After sending the email, you must also prepare an update for the main `debian.org` web site with the new announcement. The benefits of posting to the web site are;
 * permalinks: https://www.debian.org/lts/security/20XX/dla-XXX
 * i18n: mark for translation (if the translators wish to)
 * DLAs are considered to be equally as important as DSAs.

For this you need a copy of the website source. This is a one-time setup:

/!\ We recommend you request membership to the salsa webwml repo. Alternatively you can do merge requests. This workflow is work-in-progress.

{{{
git clone https://salsa.debian.org/webmaster-team/webwml
# or: salsa fork webmaster-team/webwml
# /usr/bin/salsa is part of devscripts in sid and stretch-bpo
cd webwml/
}}}

If you have made a salsa fork you can do the following (once) to prepare it so that you can branch off from the webmaster-team/webwml instead of your fork. This helps if you make multiple (with time in-between) website updates.

{{{
git remote add upstream https://salsa.debian.org/webmaster-team/webwml
}}}

Then, assuming the advisory number is DLA-1234-1 and the email template is in `~/DLA-1234-1`, you should do the following:

{{{
git checkout -b DLA-1234-1
}}}
or if you have prepared it to have a separate upstream:
{{{
git fetch upstream
git checkout -b DLA-1234-1 upstream/master
}}}
Do the update
{{{
cd english/lts/security
./parse-dla.pl ~/DLA-1234-1
$EDITOR 2022/dla-1234* # make sure everything looks good
                       # Other option is, instead of looking at the html code, doing
                       # $ make -C 2022 dla-123.en.html
                       # and open the resulting html file with a web browser.
git add 2022/dla-1234.{data,wml}
git commit -m 'DLA-1234-1'
}}}
If you are a member of the debian-www project on salsa:
{{{
git push -u origin
}}}
If you are not a member you need to provide a merge request.
First push it to your fork and then make the merge request.
{{{
git push --set-upstream origin DLA-1234-1
# Do this either one the command-line with 'salsa mr'
salsa mr
# or you need to be logged into the web ui and then go to
# https://salsa.debian.org/webmaster-team/webwml/merge_requests and press the green button labeled "new merge request"
}}}

QA: LTS Coordinator runs [[https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/security/find-missing-advisories|find-missing-advisories]] [[https://lists.debian.org/debian-lts/2020/08/msg00032.html|weekly]].

== Prepare regression updates for LTS ==

If an upload introduces a regression, it needs to be fixed as soon as possible.

=== Steps for a regression update ===

 1. Fix the package.
 1. Do a test build and test installation. Verify the regression issue and also the original issue are really resolved now.
 1. Make sure you use a clean chroot for building. Using [[sbuild]]/[[pbuilder]] is recommended.
 1. Upload the fixed package to stretch-security.
 1. Use gen-DLA script to add a DLA entry to data/DLA/list and to provide you with an announcement email template. Two possible options:
   a. The regression fixes the previous upload of the package:{{{
$ bin/gen-DLA --save $SOURCEPACKAGENAME regression
}}}
   a. The regression fixes an earlier than the previous upload of the package:{{{
$ bin/gen-DLA --save $PREVIOUS_DLA_ID $SOURCEPACKAGENAME regression
# /!\ $PREVIOUS_DLA_ID needs to be incremented (-1 -> -2)...
}}}
 1. If this is a security regression (e.g. incomplete or missing fix), reference the CVE ID in the {{{data/DLA/list}}} entry, otherwise (functional regression) do not reference it.
 1. Wait for the uploaded package to arrive in the stretch-security repository.
 1. In the base folder of the security-tracker Git repository, you can find an auto-generated LTS announcement template file with name DLA-$DLA-{2,3,4...}. Use this file as the template for your regression fix announcement. Don't commit this file to Git(!). Copy+paste the announcement template to your email application, complete/edit it manually and finally send the regression fix DLA email to debian-lts-announce@lists.debian.org.
 1. Prepare an [[#Prepare_an_update_for_the_website|update for the website]]
 1. Note: if a DLA requires a fix in another package, use the same DLA major number (see [[https://lists.debian.org/debian-lts/2019/12/msg00072.html|mail]])

== Prepare other (non-security-related) updates for LTS ==

Same procedure as a security update, but it is recommended to edit the title of the announcement e-mail accordingly, for instance:

 * https://lists.debian.org/debian-lts-announce/2020/12/msg00010.html "switch to ACMEv2 API"
 * https://lists.debian.org/debian-lts-announce/2020/12/msg00040.html "new upstream release"

== Triage new security issues ==

The security team and the LTS team cooperate to process the flow of new security issues that are reported. Since each security issue is assigned a CVE number, we often call this "CVE triage".

If you want to help with this, you must first get familiar with the security tracker. It's the infrastructure we use to track how security issues are affecting Debian packages. Please read carefully this page: http://security-team.debian.org/security_tracker.html

You will need [[#security-tracker|write access to the security-tracker git repository]].

=== CVE triaging in the LTS release ===

Now let's cover additional, LTS-specific workflow. We start from the list of open issues in the LTS release.

Sources of CVEs to triage:
 * You can start with the list on the web at this url: https://security-tracker.debian.org/tracker/status/release/oldstable (tick the top-level checkboxes accordingly; "issues to be checked" means "undetermined")
 * Or you can run {{{bin/lts-cve-triage.py}}} (script in the security-tracker repository) to get a pre-filtered list (packages in dla-needed.txt are excluded, and the issues are grouped by "status")
 * Follow [[https://lists.debian.org/debian-security-announce/|debian-security-announce@l.d.o]] for new DSAs (may include batches of unmonitored postponed vulnerabilities)
 * Check [[https://lists.debian.org/debian-announce/|debian-announce@l.d.o]] for new point releases (often fix batches of unmonitored postponed vulnerabilities)

For each CVE/package listed on these sources, there are only 2 desirable outcomes:
 * the package is added to dla-needed.txt so that someone else will have to prepare an updated package and close the security issue(s)
 * the CVE does not trigger an immediate package update:
   * we tag it "postponed" when it's a minor issue that is not urgent to fix, but may be fixed later. Since LTS does not have point releases, this means when there is more serious issue, or when lots of minor issues piled up.
   * we tag it "ignored" when there is a valid/strong reason *not* to fix it (e.g. it is only in not compiled and not shipped sources; it's non-critical and too hard to backport; it's non-critical and requires an API change affecting reverse dependencies)
   * we tag it "not-affected" when the version in the LTS release is not affected (usually because the vulnerable code is not present in the version); it is good practice, and a good test, to reference the related commit (cf. "NOTE: Introduced" in {{{data/CVE/list}}}), vulnerable code often gets renamed or split between upstream releases
   * we tag it "end-of-life" when the package is not supported in the LTS release (the list of unsupported packages for Stretch is currently available at [[https://salsa.debian.org/debian/debian-security-support/blob/master/security-support-ended.deb9|debian-security-support:security-support-ended.deb9]]. Update that file and add {{{debian-security-support}}} to {{{dla-needed.txt}}} if needed -- cf. its [[https://salsa.debian.org/debian/debian-security-support/-/blob/master/debian/README.source|README.source]].)
   * we tag it as fixed in a specific version (assuming that the version in the LTS release is higher or equal, it will be automatically assumed to be fixed too)
   * the severity was set to "unimportant" (that should be seldom used and is generally done by the security team anyway)
   * "no-dsa" is a more vague tag that includes "postponed" and "ignored", see [[https://security-team.debian.org/security_tracker.html#issues-not-warranting-a-security-advisory|security tracker doc]]. Debian Security team tends to mark non-Internet-facing vulnerabilities as "minor", because they are not directly exploitable, however they allow greater penetration when the attacker put one foot in the victim's infrastructure.

There are some differences between LTS triaging and Debian Security triaging:
 * LTS does not have any point releases
 * Fewer maintainers want to help with the LTS fixes

This mainly has an implication on when CVEs are marked as no-dsa. If the Debian Security team marks the package as no-dsa, the package maintainer may still decide to fix it. This explains why some packages have minor issues fixed while other packages do not, fixed vulnerabilities needn't be critical.

Moreover, a security update may impact the end users' services (e.g. requires restarting a daemon, or simply a sysadmin to supervise each upgrade in a large network), so grouping "postponed" issues is usually a good idea (neither too often, to avoid these impacts; neither too infrequent, to avoid dragging a large backlog of changes with a more important security fix).

Finally, part of the triage work includes filing undocumented bugs for security issues in the Debian [[BTS]], using the [[https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/bin/report-vuln|bin/report-vuln]] script. Bugs should be filed unless the issue is already fixed in unstable, avoiding duplicates of course.

=== Guidelines for CVE triaging ===

Front desk is responsible for the '''initial''' triaging of CVE. Nobody should circumvent this procedure without a very good reason.

These two reasonable exceptions are:

 1. There is a critical CVE vulnerability which should be fixed ASAP. The issue is often embargoed and affects a major package with priority important and higher. Those are rare, so this should be an exception really.

 2. A team member is also the maintainer of a package and they are already aware of the vulnerability and could confirm it. Please let people handle their own packages if they are the package maintainer or a regular contributor to a team maintained package.


The first step of CVE triaging is to determine whether a certain version of the package is affected at all.

Now the tricky part. How do you decide whether an issue should be added to dla-needed.txt or if it should be tagged in some other way? There's no perfect answer, you should use your best judgment after having *carefully* analyzed the issue, the risks involved, the trouble caused by admins to apply the update, etc. If in doubt check with the other LTS maintainers, the package maintainer (the [[https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/bin/report-vuln|bin/report-vuln]] script is a good way to get contact with the maintainer) or the security team.

That said the security team is taking similar decisions all the time and we usually follow their decisions. If the security team has added a vulnerable package to "data/dsa-needed.txt", you can usually safely add it to "data/dla-needed.txt" (if the bug is present in the LTS version of the package as well). If the security team has tagged the issue "postponed", "ignored" or "no-dsa" with "Minor issue" description, then you can usually use the same status for LTS. An exception is that when the reason picked by the security team is "Maintainer will update via stable-proposed-updates", in this case we must prepare an update for LTS since there is no lts-proposed-updates (yet). In addition, you can always fix no-dsa issues if you want to.

We can and we want to fix CVE in Debian. If a package has too many "postponed" or "ignored" CVE, it is usually a sign that now is a good point to fix all of them together.

If in doubt rather add a package to dla-needed.txt, with a note about the doubt, and let another team member double-check the vulnerability.

You should not add packages to dla-needed.txt, if we have consistently ignored certain type of CVEs for a certain package in the past or the CVE is really minor.

If you add a package to dla-needed.txt you can add an optional NOTE to explain your thought process.


'''Examples:'''

 * 20201124: I believe the issue is severe but patches don't exist yet. (name)
 * 20201124: I believe the issue is minor and can be postponed (name)
 * 20201124: I believe the issue is minor but there are several other ignored and postponed issues already. Let's fix all of them now? (name)


To highlight the fact that a team member is also the maintainer of a package.

 * 20201124: Tim is the maintainer. (name)

If you triage an issue that was not researched by the security team yet and it's unclear whether the issue warrants a DLA/DSA, it's good to document this in a note attached to the entry that you might add to dla-needed.txt.

If it's not clear whether the issue affects sid you should triage that too and open a corresponding bug in the BTS to inform the maintainer (except for embargoed issues). Maintainers often know best how severe an issue is and if an immediate update is actually needed.

Note that some packages have special instructions. Checkout {{{packages/*.txt}}} in the security-tracker repository. This might influence your decision and possibly prompt you to behave differently (for example by not contacting the maintainer since some predetermined agreement is already in place).

=== Contact the maintainer ===

Whenever you take a decision for a new issue (that is real and affecting the LTS release), you should inform the corresponding package maintainers to offer them a chance to help us, if they are listed in '''data/packages/lts-do-call-me'''.
Add the package to dla-needed.txt and add a note when you have contacted the maintainer. Then wait for a response and act accordingly by updating the notes or removing the package again if the maintainer feels no security update is required.

Do not contact the maintainer, if they are listed in '''data/packages/lts-do-not-call'''.

In all other cases you can decide whether you want to contact the maintainer or not. In the past the majority of maintainers were glad if security updates were handled by the LTS team.

We use the {{{bin/contact-maintainers}}} script in the security-tracker repository to this end. Only do this if the bug is already recorded in the BTS. If it's not there yet you can use *bin/report-vuln* to open it. Discussions about severity and urgency are best done in the bugreport since all involved parties (LTS team, security team and package maintainers) can get involved.

When we add a package to dla-needed.txt, you can use it this way for example:
{{{
$ bin/contact-maintainers --lts sudo CVE-2014-9680 CVE-2014-0106
}}}

When we tag issues as "no-dsa", and don't plan to take care of the updates by ourselves, we usually do not contact maintainers. However, if it is necessary, there is another template available:
{{{
$ bin/contact-maintainers --lts --no-dsa sudo CVE-2014-9680 CVE-2014-0106
}}}

Note that the list of CVE is optional and that this script assumes that you use mutt to send mails. If not, you should consider using the {{{--mailer}}} option to use something else (see {{{bin/contact-maintainer --help}}} for details). You must customize the mail sent (at least to drop the list of uploaders and possibly to adjust the recipients list).

In order to avoid undue pressure on the maintainer, there are some exceptions to consider:
 * If the package was recently added to dsa-needed.txt you probably want to re-formulate the email to make it clear that this is long term security and not the usual security team email.
 * Make sure a bug report regarding the problem is also in place (if the security issue is public) before this email is sent.
 * Check who was involved in making the security team aware of the issue. If it was the maintainer you need to re-formulate the email as well.

== Frontdesk duties ==

As described in [[https://lists.debian.org/debian-lts/2015/06/msg00032.html|this email]], there is an assigned person for the "LTS frontdesk". The allocations are managed in the [[https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/org/lts-frontdesk.2022.txt|lts-frontdesk.2022.txt]] file.


Frontdesk work currently involves [[LTS/Development#Triage_new_security_issues|CVE triage]] and making sure queries on the mailing list or on IRC get an answer. In particular, make sure to followup on package upload sponsorship requests and questions from new contributors to make sure things flow smoothly. Following up on stale and stalled issues is also a good idea. Contributors should feel free to contact frontdesk for any questions regarding LTS.

Frontdesk is also responsible for making sure packages claimed in `data/dla-needed.txt` are taken care of. When a package has been claimed for too long without activity, notify the claimant and, if there is no response within 24 hours, remove the lock. Note: currently (2022) LTS coordinator does it. Claims history can be inspected with:

{{{
bin/review-update-needed --lts
}}}

Finally, there should always be someone available in any given week on the frontdesk duty. When starting frontdesk, make sure the next 4 weeks are claimed by someone in the frontdesk roster. If not, send an email to the mailing list to ask for volunteers.

=== Checklist ===

Here's a quick summary checklist of the above tasks:

 * [[LTS/Development#Triage_new_security_issues|CVE triage]], which includes:
   * add package to `dla-needed.txt` or mark as `fixed/postponed/ignored/not-affected/end-of-life/unimportant/...`. As a first note to the package should stay the used programming language.
   * reporting bugs (unfixed in unstable) in the BTS
   * contact maintainers
 * followup on sponsorship requests
 * respond to unanswered questions on the mailing list
 * review claimed packages without activity (unless LTS Coordinator did their weekly semi-automatic unclaim already)
 * ensure next 4 weeks frontdesk duties are claimed

== Keeping track of LTS related bugs ==
We're tracking LTS related bugs in the Debian bug tracker using two usertags:

 * [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ease-lts;users=debian-lts@lists.debian.org|ease-lts]]: Bugs that ease long term support like enabling autopkgtests or internal test suites
 * [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=infrastructure;users=debian-lts@lists.debian.org|infrastructure]]: LTS related issues in Debian infrastructure.

To add a certain usertag to a bug you can use the ''bts'' command:

{{{
bts user debian-lts@lists.debian.org , usertags <bugnumber> ease-lts
}}}

== Switching to the next LTS release ==

 * Send a reminder to debian-lts-announce@lists.debian.org two months before the official support for the last LTS ends.
 * Contact the [[Teams/Publicity]] and draft an [[Teams/Publicity/Announcements|announcement]] that the old LTS release cycle has come to an end and a new one has started. [[https://salsa.debian.org/publicity-team/announcements/blob/eba424150df49898f5033e04602a828583a57488/en/2016/201602XX.wml| Example1 ]], [[https://salsa.debian.org/publicity-team/announcements/commit/d816ef401c55297904868a4b8d0b7f18d5bc9154| Example2 ]]
 * Ensure that the infrastructure is ready for the new LTS release.
 * Contact the FTP team and ask them to wait four weeks with archiving the old LTS release to give users a "grace period" to update their systems.
 * Contact the Stable Release Managers for coordinating the last point release (and so that they can clean up the proposed updates queues).
 * Update all relevant wiki pages especially [[LTS/Using]] and this [[LTS/Development]] itself.
 * check/update/merge security-support-ended.debX in src:debian-security-support
 * Reference [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=release.debian.org@packages.debian.org;tag=pu|pending proposed-updates]] in data/dla-needed.txt to prevent conflicts

== Switching to the next Stable release ==

 * Update {{{debian-archive-keyring}}} e.g. https://lists.debian.org/debian-lts/2021/08/msg00037.html

== Housekeeping Tasks ==
There are several tasks where we base our work on the work of the security team and where we can support them - especially when there are no open issues in dla-needed.txt. This will lead to more packages to fix:

 * Easy: Add bug references for issues unfixed in unstable. Check [[https://security-tracker.debian.org/tracker/status/unreported | unreported issues in unstable]], recheck them and file a bug if not already there. This makes sure we don't regress in unstable with bugs fixed in stable, oldstable. You can use the [[https://salsa.debian.org/security-tracker-team/security-tracker/blob/master/bin/report-vuln| bin/report-vuln]] script to generate a template report to be further edited with reportbug or a MUA. In case you don't use reportbug to further submit the bugreport, please X-Debbugs-CC team@security.debian.org and secure-testing-team@lists.alioth.debian.org. This is done automatically if using reportbug.
 * Hard: Check [[https://security-tracker.debian.org/tracker/status/todo | TODO items]] from the tracker. Be sure to only mark items as NOT-FOR-US when you're sure the software is not contained in Debian in any form.

= Tips & tools =

 * [[https://github.com/agx/emacs-tools/blob/master/debian.el|Emacs shortcuts to deal with the security-tracker]]
 * [[LTS/TestSuites]]: tips to test difficult packages

----
CategoryLts

Translation(s): English - Español - Français - Русский





This page has been moved to the Development page within the LTS team pages





7 steps for publishing a DLA

Debian Long Term Support Security Advisory (DLA)

How-to prepare security updates for LTS

Contributing to Debian Long Term Support

The Debian LTS team is always looking for more volunteers to do a better job. With more resources, we could for example:

  • support packages that are currently unsupported
  • take care of all Debian packages and not only the most popular ones
  • fix lower priority issues

If you want to get involved in the LTS team and help keep Debian packages secure for 5 years, have a look at this page. We assume that you are already familiar with the repository layout described in LTS/Using and that you are subscribed to the LTS mailing list: http://lists.debian.org/debian-lts/

You can help in many ways:

Test updated packages and report regressions

As a simple user, you can test packages that have been updated (or that are in the process of being updated). If you find a regression (i.e. something that used to work and that no longer works), then please report it to debian-lts@lists.debian.org and put the person who prepared the update in copy (in case they are not subscribed to the list).

Many LTS contributors are looking for testers of their updated packages. They send call for tests on the mailing list, so please subscribe to it and test the packages they provide when you can, and report back whether they worked for you.

Debian Security Tracker

The Debian LTS team makes extensive use of the Debian Security Tracker. This is a database of all known security issues in Debian.

Developers check out the Git source (see instructions), commit changes, and the website automatically updates to reflect the changes.

LTS Developers are strongly encouraged to keep the data in the security tracker up-to-date. This includes updating the following files:

  • data/CVE/list: Should be updated with broader information that affects more than just LTS (i.e., stable, testing). For example, consider adding to this file links to bug reports related to the security issue, links to upstream commits or new releases which address the vulnerability, links to public mailing list posts, upstream reports, upstream exploits, information regarding if the exploits do or do not work, etc. Any information that could be useful for the security team as well as the LTS team.
  • data/dla-needed.txt: Should be updated to reflect LTS specific information, such as who is working on a package, possible issues that are specific to LTS, test packages available, requests for help, problems encountered, etc.
  • data/DLA/list: In most cases there is no need to manually change this file, as it is maintained automatically by bin/gen-DLA which is described in following sections.

Make sure to read its documentation on the Debian Security Team website.

Prepare security updates for LTS

Claim the issue in the security tracker (in dla-needed.txt)

In order to prevent duplication of effort, make sure the issue is listed in the data/dla-needed.txt file from the security tracker and add your name to it. Normally, issues are triaged by the frontdesk in dla-needed.txt at first. If not, you should add the issue to dla-needed.txt yourself. Look at how that is done below before adding an issue or adding your name to an issue.

We give package maintainers 24 to 72 hours delay before taking on updates ourselves, depending on the severity of the security issue.

As mentioned in the previous section, make sure to also review and update the relevant information in data/CVE/list when claiming a package for LTS work.

You shall only claim it for the time you do active work. You are expected to actively handle the problem within 2-3 days and then you are expected to announce your intention to upload so others can test it for a few (3-4) more days. If you can not solve the problem within the 2-3 days, then please mention why it takes more time, how far you have gone, the preliminary results and/or free it for someone else to work on it. We do not want people to work on a package for a few hours, wait a few days, work again on it, wait more and so on.

Checkout the git repository of the security tracker: (Using a shallow clone because otherwise the operation will take half an hour. The clone will still need a few minutes to complete.)

$ git clone --depth 1 git@salsa.debian.org:security-tracker-team/security-tracker
$ cd security-tracker
$ sudo apt install jq python3-apt
$ bin/setup-repo

setup-repo installs a pre-commit hook to check data file syntax.

Then edit the data/dla-needed.txt file:

$ editor data/dla-needed.txt
$ git commit -m "Claim foo in dla-needed.txt" data/dla-needed.txt
$ git push

Write access to the security-tracker repository

Debian Developers automatically have commit access to the security-tracker repository. Otherwise you need to be a member of the security-tracker project on salsa.debian.org, please request membership through the security tracker team gitlab project, security-tracker mailinglist or debian-lts mailinglist.

Build the update

Backport the fix to the version in LTS (in case there's already been an earlier update). Make sure that the fix was approved by the upstream project (e.g. committed to their development repository), sometimes early patches proposals are erroneous or incomplete. If there is no fix, it is fine to work on a new one: be a good citizen and contribute the patch back upstream and make sure the maintainer and regular security teams will be aware of them, by adding it to a bug report against the package and linking that bug report in the security tracker. (If the issue being addressed requires AddressSanitizer in order to reproduce the vulnerability and confirm the fix, then this page may be helpful).

If the fix comes with new binary files (e.g. test cases), they won't be accepted by quilt, but you can reference them in debian/source/include-binaries and they'll be included in the .debian.tar.xz file directly (discussion).

In the changelog, you need to set the target distribution in debian/changelog to "stretch-security". The versioning follows the conventions already used in security.debian.org. Historically codenames have been used as version numbers, but this was changed some time ago as version numbers are more deterministic.

  • If you use dch (pre-buster) to make the changelog entry, note that it may include the text Non-maintainer upload by the Security Team. It is important to either remove the line, remove by the Security Team, or replace it with something referring to the LTS Team, to avoid giving the appearance of support by the Security Team. Cf 762715 ("Non-maintainer upload by the LTS Security Team.").

  • If a package already e.g. had a +deb9u1 update, use +deb9u2 for the next update.
  • If a package hasn't seen an update, use +deb9u1 for the next update.
  • For the rare cases where new upstream versions are introduced, make sure to use a lower version than in stretch. See discussion. Just don't use +stretchX as this will cause problems!

  • For the rare case where backports of the entire package from newer distributions is required (as opposed to just the fix), it is expected that the changelog will not be in incrementing version order.

BTS numbers can be referenced but are currently not processed.

Note that for non-native packages, uploads to security-master need to include the source tarball (even if it's already present in the distribution you're uploading to) by using 'dpkg-buildpackage -sa'.

Make sure you use a clean chroot for building the binary packages. Using sbuild/pbuilder is recommended. For instance:

# Init
sudo pbuilder --create --basetgz /var/cache/pbuilder/base-stretch.tgz --distribution stretch --othermirror 'deb http://security.debian.org/ stretch/updates main contrib'
sudo pbuilder --update --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild source and binary packages from Stretch (in extracted source)
export DEB_BUILD_OPTIONS="parallel=$(nproc)"
pdebuild --use-pdebuild-internal --buildresult .. --debbuildopts '-S' -- --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild binary packages from Stretch, reproducing buildd's separate build-indep/build-arch
# - first security upload:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep --debbuildopts '-sa' package+deb9u1.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log  --buildresult . --binary-arch  --debbuildopts '-sa' package+deb9u1.dsc
# - later uploads:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep package+deb9u2.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log  --buildresult . --binary-arch  package+deb9u2.dsc

To actually build the binary package, you may want to use a porter box, especially if you need to fix a regression that concerns a specific architecture you may not have access to, see PorterBoxHowToUse for instructions on how to use porterboxes.

See also:

Test the update

See TestSuites for package-specific test instructions.

If there's a built-in test suite, make sure it runs fine at build-time in the LTS environment. It is acceptable to enable or even backport test suites from later releases if they are available.

There can also be DEP-8 tests in debian/tests, to run with autopkgtest. See for instance TestSuites/rails for a direct run or TestSuites/sane-backends for a full LXC environment.

If there are no test suites available, you will need to run and test the package directly. Prefer a full environment (virtualized or sometimes even physical).

Do check the reverse-dependencies (find-rdeps, apt-rdepends -r, dose-ceve from dose-extra) and install some of them, to check if your package still runs correctly with them.

It is worth testing with both a 32- and 64-bit architecture, see e.g. CVE-2019-14866.

Check your package update with generic tools as well:

# check for common packaging issues in last build
# from extracted source after build, stretch host (only check new errors)
lintian -i
# inspect source changes
debdiff package+deb9u3.dsc package+deb9u4.dsc | diffstat
debdiff package+deb9u3.dsc package+deb9u4.dsc | less
# inspect binary changes
debdiff --from deb9u3/*.deb --to deb9u4/*.deb
# test package upgrade
sudo piuparts -d stretch \
  --extra-repo='deb http://security.debian.org/ stretch/updates main contrib' \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  --single-changes-list package+deb9u4_{all,amd64}.changes \
  | grep -P '(INFO|ERROR):'
# also consider --install-remove-install

Depending on the urgency of the security fix, and your confidence on your own work, you might want to seek reviews from interested parties (such as upstream authors, Debian maintainers, security team, LTS team, etc.). If you ask for reviews, you should supply the following information:

  1. A generated debdiff against the previous version in LTS.
  2. A link to a test package that can be downloaded and tested.

It is common for LTS workers to upload their packages to a personal repository in people.debian.org, but any public repository or even simple webserver will do. A few guidelines for those temporary uploads:

  • sign the packages (using debsign, for example) so that users can verify package integrity

  • make sure to have the target distribution set to UNRELEASED so that nobody else can upload them before they are ready.

Upload the update

Uploads to Debian LTS do not wait for a DLA or any other manual intervention before being installed into the Debian Archive. In this regard, they are different from stable security uploads.

If you're satisfied with your tests, upload to security-master, using dput or dput-ng:

dput security-master hello_1.0-1+deb9u1_source.changes

Once uploaded the package will be auto-built for all the available architectures (if it's an arch:any package). The architectures currently supported are amd64, i386, armel and armhf. Check the build logs at: https://buildd.debian.org/status/package.php?suite=stretch-security&p=SOURCE_PACKAGE

Built-Using refers to non-existing source package: if your package is rejected with this reason, open a bug against package ftp.debian.org requesting to add the missing package, see e.g. 974877 or 974954.

Claim a DLA ID in DLA/list

Once your upload has been built successfully, you can go ahead with the DLA.

Run bin/gen-DLA in the top directory of the Git repository. It automatically generates an entry in data/DLA/list and asks you to commit the changes to ensure that no IDs are used twice. The following command would add an entry for src:hello fixing CVE-2014-0666 and creates an advisory template in the top directory of your security-tracker checkout.

$ DEBFULLNAME=xxx bin/gen-DLA --save hello CVE-2014-0666
# OR
$ DEBFULLNAME=xxx bin/gen-DLA --save .../hello_1.2-3+deb9u4_source.changes

That list of CVE will make sure that the security tracker is aware of the fix and mark the issues as resolved for that specific package.

gen-DLA will automatically remove prior suite triage for the fixed vulnerabilities (e.g. [stretch] - package <postponed>). However, the {DLA-XXXX-1} annotation will be added by the bi-daily cron, and the fixed version will be inferred automatically (don't add it).

Special case: Upload non-free packages

Currently (20220119) source-only uploads of non-free packages to security-master are not possible (according to Aurélien Jarno this is a bug). The buildd will not build the package for different architectures. The LTS team still maintains packages like amd64-microcode and firmware-linux-nonfree.

New upstream releases should include the full "source-code" and the amd64 binary packages (build with -sa and upload the amd64 changes file).

After that you need to upload only the i386 binary packages as well. This can be achieved by passing the -b flag to dpkg-buildpackage. You can also use the changestool from reprepro to modify your changes file and to exclude unwanted files from it.

Announce the update

Once the upload is completed, an email will be sent to you and to the debian-lts-changes mailing list, confirming the update has been released into the Debian archive. The BTS bot will also announce it in the #debian-devel-changes channel.

Send mail to the debian-lts-announce mailing list. The mail needs to be signed by a GPG key in the debian or debian-maintainers keyring. Inline signatures always work (PGP/MIME works with mutt but fails with Enigmail. Thunderbird fails starting from version 76).

The advisory template has been created by bin/gen-DLA (see before) and generally looks like this:

Subject: [SECURITY] [DLA $DLA-1] $SOURCEPACKAGENAME security update

Package        : $SOURCEPACKAGENAME
Version        : $LTS_VERSION
CVE ID         : CVE-2014-0001 CVE-2014-0002
Debian Bug     : 12345

DLA text goes here
[...]

In addition to describing the vulnerabilities, remember to add a short package description, so the audience can better understand the context and whether they are affected.

Please also don't append your personal signatures to DLAs.

If you use mutt, a simple way to send it is to use

mutt -e "set signature=/dev/null" -H DLA-123

Simple "mail" command works:

cat DLA-XXXX.txt | gpg --clearsign | mail -s "[SECURITY] [DLA XXXX-X] $SOURCEPACKAGENAME security update" -a "From: Max Mustermann <max.mustermann@debian.org>" -r max.mustermann@debian.org debian-lts-announce@lists.debian.org

Only when you have confirmed that the package was processed after upload (once you get the accept email) should you send the DLA to the mailing list.

When all this is done, you should remove the package from other locations it was uploaded to in the test section, if any.

See also:

Prepare an update for the website

After sending the email, you must also prepare an update for the main debian.org web site with the new announcement. The benefits of posting to the web site are;

For this you need a copy of the website source. This is a one-time setup:

/!\ We recommend you request membership to the salsa webwml repo. Alternatively you can do merge requests. This workflow is work-in-progress.

git clone https://salsa.debian.org/webmaster-team/webwml
# or: salsa fork webmaster-team/webwml
# /usr/bin/salsa is part of devscripts in sid and stretch-bpo
cd webwml/

If you have made a salsa fork you can do the following (once) to prepare it so that you can branch off from the webmaster-team/webwml instead of your fork. This helps if you make multiple (with time in-between) website updates.

git remote add upstream https://salsa.debian.org/webmaster-team/webwml

Then, assuming the advisory number is DLA-1234-1 and the email template is in ~/DLA-1234-1, you should do the following:

git checkout -b DLA-1234-1

or if you have prepared it to have a separate upstream:

git fetch upstream
git checkout -b DLA-1234-1 upstream/master

Do the update

cd english/lts/security
./parse-dla.pl ~/DLA-1234-1
$EDITOR 2022/dla-1234* # make sure everything looks good
                       # Other option is, instead of looking at the html code, doing
                       # $ make -C 2022 dla-123.en.html
                       # and open the resulting html file with a web browser.
git add 2022/dla-1234.{data,wml}
git commit -m 'DLA-1234-1'

If you are a member of the debian-www project on salsa:

git push -u origin

If you are not a member you need to provide a merge request. First push it to your fork and then make the merge request.

git push --set-upstream origin DLA-1234-1
# Do this either one the command-line with 'salsa mr'
salsa mr
# or you need to be logged into the web ui and then go to
# https://salsa.debian.org/webmaster-team/webwml/merge_requests and press the green button labeled "new merge request"

QA: LTS Coordinator runs find-missing-advisories weekly.

Prepare regression updates for LTS

If an upload introduces a regression, it needs to be fixed as soon as possible.

Steps for a regression update

  1. Fix the package.
  2. Do a test build and test installation. Verify the regression issue and also the original issue are really resolved now.
  3. Make sure you use a clean chroot for building. Using sbuild/pbuilder is recommended.

  4. Upload the fixed package to stretch-security.
  5. Use gen-DLA script to add a DLA entry to data/DLA/list and to provide you with an announcement email template. Two possible options:
    1. The regression fixes the previous upload of the package:

      $ bin/gen-DLA  --save $SOURCEPACKAGENAME regression
    2. The regression fixes an earlier than the previous upload of the package:

      $ bin/gen-DLA  --save $PREVIOUS_DLA_ID $SOURCEPACKAGENAME regression
      # /!\ $PREVIOUS_DLA_ID needs to be incremented (-1 -> -2)...
  6. If this is a security regression (e.g. incomplete or missing fix), reference the CVE ID in the data/DLA/list entry, otherwise (functional regression) do not reference it.

  7. Wait for the uploaded package to arrive in the stretch-security repository.
  8. In the base folder of the security-tracker Git repository, you can find an auto-generated LTS announcement template file with name DLA-$DLA-{2,3,4...}. Use this file as the template for your regression fix announcement. Don't commit this file to Git(!). Copy+paste the announcement template to your email application, complete/edit it manually and finally send the regression fix DLA email to debian-lts-announce@lists.debian.org.

  9. Prepare an update for the website

  10. Note: if a DLA requires a fix in another package, use the same DLA major number (see mail)

Prepare other (non-security-related) updates for LTS

Same procedure as a security update, but it is recommended to edit the title of the announcement e-mail accordingly, for instance:

Triage new security issues

The security team and the LTS team cooperate to process the flow of new security issues that are reported. Since each security issue is assigned a CVE number, we often call this "CVE triage".

If you want to help with this, you must first get familiar with the security tracker. It's the infrastructure we use to track how security issues are affecting Debian packages. Please read carefully this page: http://security-team.debian.org/security_tracker.html

You will need write access to the security-tracker git repository.

CVE triaging in the LTS release

Now let's cover additional, LTS-specific workflow. We start from the list of open issues in the LTS release.

Sources of CVEs to triage:

  • You can start with the list on the web at this url: https://security-tracker.debian.org/tracker/status/release/oldstable (tick the top-level checkboxes accordingly; "issues to be checked" means "undetermined")

  • Or you can run bin/lts-cve-triage.py (script in the security-tracker repository) to get a pre-filtered list (packages in dla-needed.txt are excluded, and the issues are grouped by "status")

  • Follow debian-security-announce@l.d.o for new DSAs (may include batches of unmonitored postponed vulnerabilities)

  • Check debian-announce@l.d.o for new point releases (often fix batches of unmonitored postponed vulnerabilities)

For each CVE/package listed on these sources, there are only 2 desirable outcomes:

  • the package is added to dla-needed.txt so that someone else will have to prepare an updated package and close the security issue(s)
  • the CVE does not trigger an immediate package update:
    • we tag it "postponed" when it's a minor issue that is not urgent to fix, but may be fixed later. Since LTS does not have point releases, this means when there is more serious issue, or when lots of minor issues piled up.
    • we tag it "ignored" when there is a valid/strong reason *not* to fix it (e.g. it is only in not compiled and not shipped sources; it's non-critical and too hard to backport; it's non-critical and requires an API change affecting reverse dependencies)
    • we tag it "not-affected" when the version in the LTS release is not affected (usually because the vulnerable code is not present in the version); it is good practice, and a good test, to reference the related commit (cf. "NOTE: Introduced" in data/CVE/list), vulnerable code often gets renamed or split between upstream releases

    • we tag it "end-of-life" when the package is not supported in the LTS release (the list of unsupported packages for Stretch is currently available at debian-security-support:security-support-ended.deb9. Update that file and add debian-security-support to dla-needed.txt if needed -- cf. its README.source.)

    • we tag it as fixed in a specific version (assuming that the version in the LTS release is higher or equal, it will be automatically assumed to be fixed too)
    • the severity was set to "unimportant" (that should be seldom used and is generally done by the security team anyway)
    • "no-dsa" is a more vague tag that includes "postponed" and "ignored", see security tracker doc. Debian Security team tends to mark non-Internet-facing vulnerabilities as "minor", because they are not directly exploitable, however they allow greater penetration when the attacker put one foot in the victim's infrastructure.

There are some differences between LTS triaging and Debian Security triaging:

  • LTS does not have any point releases
  • Fewer maintainers want to help with the LTS fixes

This mainly has an implication on when CVEs are marked as no-dsa. If the Debian Security team marks the package as no-dsa, the package maintainer may still decide to fix it. This explains why some packages have minor issues fixed while other packages do not, fixed vulnerabilities needn't be critical.

Moreover, a security update may impact the end users' services (e.g. requires restarting a daemon, or simply a sysadmin to supervise each upgrade in a large network), so grouping "postponed" issues is usually a good idea (neither too often, to avoid these impacts; neither too infrequent, to avoid dragging a large backlog of changes with a more important security fix).

Finally, part of the triage work includes filing undocumented bugs for security issues in the Debian BTS, using the bin/report-vuln script. Bugs should be filed unless the issue is already fixed in unstable, avoiding duplicates of course.

Guidelines for CVE triaging

Front desk is responsible for the initial triaging of CVE. Nobody should circumvent this procedure without a very good reason.

These two reasonable exceptions are:

  1. There is a critical CVE vulnerability which should be fixed ASAP. The issue is often embargoed and affects a major package with priority important and higher. Those are rare, so this should be an exception really.
  2. A team member is also the maintainer of a package and they are already aware of the vulnerability and could confirm it. Please let people handle their own packages if they are the package maintainer or a regular contributor to a team maintained package.

The first step of CVE triaging is to determine whether a certain version of the package is affected at all.

Now the tricky part. How do you decide whether an issue should be added to dla-needed.txt or if it should be tagged in some other way? There's no perfect answer, you should use your best judgment after having *carefully* analyzed the issue, the risks involved, the trouble caused by admins to apply the update, etc. If in doubt check with the other LTS maintainers, the package maintainer (the bin/report-vuln script is a good way to get contact with the maintainer) or the security team.

That said the security team is taking similar decisions all the time and we usually follow their decisions. If the security team has added a vulnerable package to "data/dsa-needed.txt", you can usually safely add it to "data/dla-needed.txt" (if the bug is present in the LTS version of the package as well). If the security team has tagged the issue "postponed", "ignored" or "no-dsa" with "Minor issue" description, then you can usually use the same status for LTS. An exception is that when the reason picked by the security team is "Maintainer will update via stable-proposed-updates", in this case we must prepare an update for LTS since there is no lts-proposed-updates (yet). In addition, you can always fix no-dsa issues if you want to.

We can and we want to fix CVE in Debian. If a package has too many "postponed" or "ignored" CVE, it is usually a sign that now is a good point to fix all of them together.

If in doubt rather add a package to dla-needed.txt, with a note about the doubt, and let another team member double-check the vulnerability.

You should not add packages to dla-needed.txt, if we have consistently ignored certain type of CVEs for a certain package in the past or the CVE is really minor.

If you add a package to dla-needed.txt you can add an optional NOTE to explain your thought process.

Examples:

  • 20201124: I believe the issue is severe but patches don't exist yet. (name)
  • 20201124: I believe the issue is minor and can be postponed (name)
  • 20201124: I believe the issue is minor but there are several other ignored and postponed issues already. Let's fix all of them now? (name)

To highlight the fact that a team member is also the maintainer of a package.

  • 20201124: Tim is the maintainer. (name)

If you triage an issue that was not researched by the security team yet and it's unclear whether the issue warrants a DLA/DSA, it's good to document this in a note attached to the entry that you might add to dla-needed.txt.

If it's not clear whether the issue affects sid you should triage that too and open a corresponding bug in the BTS to inform the maintainer (except for embargoed issues). Maintainers often know best how severe an issue is and if an immediate update is actually needed.

Note that some packages have special instructions. Checkout packages/*.txt in the security-tracker repository. This might influence your decision and possibly prompt you to behave differently (for example by not contacting the maintainer since some predetermined agreement is already in place).

Contact the maintainer

Whenever you take a decision for a new issue (that is real and affecting the LTS release), you should inform the corresponding package maintainers to offer them a chance to help us, if they are listed in data/packages/lts-do-call-me. Add the package to dla-needed.txt and add a note when you have contacted the maintainer. Then wait for a response and act accordingly by updating the notes or removing the package again if the maintainer feels no security update is required.

Do not contact the maintainer, if they are listed in data/packages/lts-do-not-call.

In all other cases you can decide whether you want to contact the maintainer or not. In the past the majority of maintainers were glad if security updates were handled by the LTS team.

We use the bin/contact-maintainers script in the security-tracker repository to this end. Only do this if the bug is already recorded in the BTS. If it's not there yet you can use *bin/report-vuln* to open it. Discussions about severity and urgency are best done in the bugreport since all involved parties (LTS team, security team and package maintainers) can get involved.

When we add a package to dla-needed.txt, you can use it this way for example:

$ bin/contact-maintainers --lts sudo CVE-2014-9680 CVE-2014-0106

When we tag issues as "no-dsa", and don't plan to take care of the updates by ourselves, we usually do not contact maintainers. However, if it is necessary, there is another template available:

$ bin/contact-maintainers --lts --no-dsa sudo CVE-2014-9680 CVE-2014-0106

Note that the list of CVE is optional and that this script assumes that you use mutt to send mails. If not, you should consider using the --mailer option to use something else (see bin/contact-maintainer --help for details). You must customize the mail sent (at least to drop the list of uploaders and possibly to adjust the recipients list).

In order to avoid undue pressure on the maintainer, there are some exceptions to consider:

  • If the package was recently added to dsa-needed.txt you probably want to re-formulate the email to make it clear that this is long term security and not the usual security team email.
  • Make sure a bug report regarding the problem is also in place (if the security issue is public) before this email is sent.
  • Check who was involved in making the security team aware of the issue. If it was the maintainer you need to re-formulate the email as well.

Frontdesk duties

As described in this email, there is an assigned person for the "LTS frontdesk". The allocations are managed in the lts-frontdesk.2022.txt file.

Frontdesk work currently involves CVE triage and making sure queries on the mailing list or on IRC get an answer. In particular, make sure to followup on package upload sponsorship requests and questions from new contributors to make sure things flow smoothly. Following up on stale and stalled issues is also a good idea. Contributors should feel free to contact frontdesk for any questions regarding LTS.

Frontdesk is also responsible for making sure packages claimed in data/dla-needed.txt are taken care of. When a package has been claimed for too long without activity, notify the claimant and, if there is no response within 24 hours, remove the lock. Note: currently (2022) LTS coordinator does it. Claims history can be inspected with:

bin/review-update-needed --lts

Finally, there should always be someone available in any given week on the frontdesk duty. When starting frontdesk, make sure the next 4 weeks are claimed by someone in the frontdesk roster. If not, send an email to the mailing list to ask for volunteers.

Checklist

Here's a quick summary checklist of the above tasks:

  • CVE triage, which includes:

    • add package to dla-needed.txt or mark as fixed/postponed/ignored/not-affected/end-of-life/unimportant/.... As a first note to the package should stay the used programming language.

    • reporting bugs (unfixed in unstable) in the BTS
    • contact maintainers
  • followup on sponsorship requests
  • respond to unanswered questions on the mailing list
  • review claimed packages without activity (unless LTS Coordinator did their weekly semi-automatic unclaim already)
  • ensure next 4 weeks frontdesk duties are claimed

We're tracking LTS related bugs in the Debian bug tracker using two usertags:

  • ease-lts: Bugs that ease long term support like enabling autopkgtests or internal test suites

  • infrastructure: LTS related issues in Debian infrastructure.

To add a certain usertag to a bug you can use the bts command:

bts user debian-lts@lists.debian.org , usertags <bugnumber> ease-lts

Switching to the next LTS release

  • Send a reminder to debian-lts-announce@lists.debian.org two months before the official support for the last LTS ends.

  • Contact the Teams/Publicity and draft an announcement that the old LTS release cycle has come to an end and a new one has started. Example1, Example2

  • Ensure that the infrastructure is ready for the new LTS release.
  • Contact the FTP team and ask them to wait four weeks with archiving the old LTS release to give users a "grace period" to update their systems.
  • Contact the Stable Release Managers for coordinating the last point release (and so that they can clean up the proposed updates queues).
  • Update all relevant wiki pages especially LTS/Using and this LTS/Development itself.

  • check/update/merge security-support-ended.debX in src:debian-security-support
  • Reference pending proposed-updates in data/dla-needed.txt to prevent conflicts

Switching to the next Stable release

Housekeeping Tasks

There are several tasks where we base our work on the work of the security team and where we can support them - especially when there are no open issues in dla-needed.txt. This will lead to more packages to fix:

  • Easy: Add bug references for issues unfixed in unstable. Check unreported issues in unstable, recheck them and file a bug if not already there. This makes sure we don't regress in unstable with bugs fixed in stable, oldstable. You can use the bin/report-vuln script to generate a template report to be further edited with reportbug or a MUA. In case you don't use reportbug to further submit the bugreport, please X-Debbugs-CC team@security.debian.org and secure-testing-team@lists.alioth.debian.org. This is done automatically if using reportbug.

  • Hard: Check TODO items from the tracker. Be sure to only mark items as NOT-FOR-US when you're sure the software is not contained in Debian in any form.

Tips & tools


CategoryLts