Translation(s): English - Español - Français - Русский





This page has been moved to the Development page within the LTS team pages

























































7 steps for publishing a DLA

Debian Long Term Support Security Advisory (DLA)

How-to prepare security updates for LTS

Contributing to Debian Long Term Support

The Debian LTS team is always looking for more volunteers to do a better job. With more resources, we could for example:

If you want to get involved in the LTS team and help keep Debian packages secure for 5 years, have a look at this page. We assume that you are already familiar with the repository layout described in LTS/Using and that you are subscribed to the LTS mailing list: http://lists.debian.org/debian-lts/

You can help in many ways:

Test updated packages and report regressions

As a simple user, you can test packages that have been updated (or that are in the process of being updated). If you find a regression (i.e. something that used to work and that no longer works), then please report it to debian-lts@lists.debian.org and put the person who prepared the update in copy (in case they are not subscribed to the list).

Many LTS contributors are looking for testers of their updated packages. They send call for tests on the mailing list, so please subscribe to it and test the packages they provide when you can, and report back whether they worked for you.

Debian Security Tracker

The Debian LTS team makes extensive use of the Debian Security Tracker. This is a database of all known security issues in Debian.

Developers check out the Git source (see instructions), commit changes, and the website automatically updates to reflect the changes.

LTS Developers are strongly encouraged to keep the data in the security tracker up-to-date. This includes updating the following files:

Make sure to read its documentation on the Debian Security Team website.

Prepare security updates for LTS

Claim the issue in the security tracker (in dla-needed.txt)

In order to prevent duplication of effort, make sure the issue is listed in the data/dla-needed.txt file from the security tracker and add your name to it. Normally, issues are triaged by the frontdesk in dla-needed.txt at first. If not, you should add the issue to dla-needed.txt yourself. Look at how that is done below before adding an issue or adding your name to an issue.

We give package maintainers 24 to 72 hours delay before taking on updates ourselves, depending on the severity of the security issue.

As mentioned in the previous section, make sure to also review and update the relevant information in data/CVE/list when claiming a package for LTS work.

You shall only claim it for the time you do active work. You are expected to actively handle the problem within 2-3 days and then you are expected to announce your intention to upload so others can test it for a few (3-4) more days. If you can not solve the problem within the 2-3 days, then please mention why it takes more time, how far you have gone, the preliminary results and/or free it for someone else to work on it. We do not want people to work on a package for a few hours, wait a few days, work again on it, wait more and so on.

Checkout the git repository of the security tracker: (Using a shallow clone because otherwise the operation will take half an hour. The clone will still need a few minutes to complete.)

$ git clone --depth 1 git@salsa.debian.org:security-tracker-team/security-tracker
$ cd security-tracker
$ sudo apt install jq python3-apt
$ bin/setup-repo

setup-repo installs a pre-commit hook to check data file syntax.

Then edit the data/dla-needed.txt file:

$ editor data/dla-needed.txt
$ git commit -m "Claim foo in dla-needed.txt" data/dla-needed.txt
$ git push

Write access to the security-tracker repository

Debian Developers automatically have commit access to the security-tracker repository. Otherwise you need to be a member of the security-tracker project on salsa.debian.org, please request membership through the security tracker team gitlab project, security-tracker mailinglist or debian-lts mailinglist.

Build the update

Backport the fix to the version in LTS (in case there's already been an earlier update). Make sure that the fix was approved by the upstream project (e.g. committed to their development repository), sometimes early patches proposals are erroneous or incomplete. If there is no fix, it is fine to work on a new one: be a good citizen and contribute the patch back upstream and make sure the maintainer and regular security teams will be aware of them, by adding it to a bug report against the package and linking that bug report in the security tracker. (If the issue being addressed requires AddressSanitizer in order to reproduce the vulnerability and confirm the fix, then this page may be helpful).

If the fix comes with new binary files (e.g. test cases), they won't be accepted by quilt, but you can reference them in debian/source/include-binaries and they'll be included in the .debian.tar.xz file directly (discussion).

In the changelog, you need to set the target distribution in debian/changelog to "stretch-security". The versioning follows the conventions already used in security.debian.org. Historically codenames have been used as version numbers, but this was changed some time ago as version numbers are more deterministic.

BTS numbers can be referenced but are currently not processed.

Note that for non-native packages, uploads to security-master need to include the source tarball (even if it's already present in the distribution you're uploading to) by using 'dpkg-buildpackage -sa'.

Make sure you use a clean chroot for building the binary packages. Using sbuild/pbuilder is recommended. For instance:

# Init
sudo pbuilder --create --basetgz /var/cache/pbuilder/base-stretch.tgz --distribution stretch --othermirror 'deb http://security.debian.org/ stretch/updates main contrib'
sudo pbuilder --update --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild source and binary packages from Stretch (in extracted source)
export DEB_BUILD_OPTIONS="parallel=$(nproc)"
pdebuild --use-pdebuild-internal --buildresult .. --debbuildopts '-S' -- --basetgz /var/cache/pbuilder/base-stretch.tgz
# Rebuild binary packages from Stretch, reproducing buildd's separate build-indep/build-arch
# - first security upload:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep --debbuildopts '-sa' package+deb9u1.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log  --buildresult . --binary-arch  --debbuildopts '-sa' package+deb9u1.dsc
# - later uploads:
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-indep.log --buildresult . --binary-indep package+deb9u2.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-stretch.tgz --source-only-changes --logfile build-arch.log  --buildresult . --binary-arch  package+deb9u2.dsc

To actually build the binary package, you may want to use a porter box, especially if you need to fix a regression that concerns a specific architecture you may not have access to, see PorterBoxHowToUse for instructions on how to use porterboxes.

See also:

Test the update

See TestSuites for package-specific test instructions.

If there's a built-in test suite, make sure it runs fine at build-time in the LTS environment. It is acceptable to enable or even backport test suites from later releases if they are available.

There can also be DEP-8 tests in debian/tests, to run with autopkgtest. See for instance TestSuites/rails for a direct run or TestSuites/sane-backends for a full LXC environment.

If there are no test suites available, you will need to run and test the package directly. Prefer a full environment (virtualized or sometimes even physical).

Do check the reverse-dependencies (find-rdeps, apt-rdepends -r, dose-ceve from dose-extra) and install some of them, to check if your package still runs correctly with them.

It is worth testing with both a 32- and 64-bit architecture, see e.g. CVE-2019-14866.

Check your package update with generic tools as well:

# check for common packaging issues in last build
# from extracted source after build, stretch host (only check new errors)
lintian -i
# inspect source changes
debdiff package+deb9u3.dsc package+deb9u4.dsc | diffstat
debdiff package+deb9u3.dsc package+deb9u4.dsc | less
# inspect binary changes
debdiff --from deb9u3/*.deb --to deb9u4/*.deb
# test package upgrade
sudo piuparts -d stretch \
  --extra-repo='deb http://security.debian.org/ stretch/updates main contrib' \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  --single-changes-list package+deb9u4_{all,amd64}.changes \
  | grep -P '(INFO|ERROR):'
# also consider --install-remove-install

Depending on the urgency of the security fix, and your confidence on your own work, you might want to seek reviews from interested parties (such as upstream authors, Debian maintainers, security team, LTS team, etc.). If you ask for reviews, you should supply the following information:

  1. A generated debdiff against the previous version in LTS.
  2. A link to a test package that can be downloaded and tested.

It is common for LTS workers to upload their packages to a personal repository in people.debian.org, but any public repository or even simple webserver will do. A few guidelines for those temporary uploads:

Upload the update

Uploads to Debian LTS do not wait for a DLA or any other manual intervention before being installed into the Debian Archive. In this regard, they are different from stable security uploads.

If you're satisfied with your tests, upload to security-master, using dput or dput-ng:

dput security-master hello_1.0-1+deb9u1_source.changes

Once uploaded the package will be auto-built for all the available architectures (if it's an arch:any package). The architectures currently supported are amd64, i386, armel and armhf. Check the build logs at: https://buildd.debian.org/status/package.php?suite=stretch-security&p=SOURCE_PACKAGE

Built-Using refers to non-existing source package: if your package is rejected with this reason, open a bug against package ftp.debian.org requesting to add the missing package, see e.g. 974877 or 974954.

Claim a DLA ID in DLA/list

Once your upload has been built successfully, you can go ahead with the DLA.

Run bin/gen-DLA in the top directory of the Git repository. It automatically generates an entry in data/DLA/list and asks you to commit the changes to ensure that no IDs are used twice. The following command would add an entry for src:hello fixing CVE-2014-0666 and creates an advisory template in the top directory of your security-tracker checkout.

$ DEBFULLNAME=xxx bin/gen-DLA --save hello CVE-2014-0666
# OR
$ DEBFULLNAME=xxx bin/gen-DLA --save .../hello_1.2-3+deb9u4_source.changes

That list of CVE will make sure that the security tracker is aware of the fix and mark the issues as resolved for that specific package.

gen-DLA will automatically remove prior suite triage for the fixed vulnerabilities (e.g. [stretch] - package <postponed>). However, the {DLA-XXXX-1} annotation will be added by the bi-daily cron, and the fixed version will be inferred automatically (don't add it).

Special case: Upload non-free packages

Currently (20220119) source-only uploads of non-free packages to security-master are not possible (according to Aurélien Jarno this is a bug). The buildd will not build the package for different architectures. The LTS team still maintains packages like amd64-microcode and firmware-linux-nonfree.

New upstream releases should include the full "source-code" and the amd64 binary packages (build with -sa and upload the amd64 changes file).

After that you need to upload only the i386 binary packages as well. This can be achieved by passing the -b flag to dpkg-buildpackage. You can also use the changestool from reprepro to modify your changes file and to exclude unwanted files from it.

Announce the update

Once the upload is completed, an email will be sent to you and to the debian-lts-changes mailing list, confirming the update has been released into the Debian archive. The BTS bot will also announce it in the #debian-devel-changes channel.

Send mail to the debian-lts-announce mailing list. The mail needs to be signed by an OpenPGP key in the debian or debian-maintainers keyring. Inline signatures always work (PGP/MIME works with mutt but fails with Enigmail. Thunderbird fails starting from version 76).

The advisory template has been created by bin/gen-DLA (see before) and generally looks like this:

Subject: [SECURITY] [DLA $DLA-1] $SOURCEPACKAGENAME security update

Package        : $SOURCEPACKAGENAME
Version        : $LTS_VERSION
CVE ID         : CVE-2014-0001 CVE-2014-0002
Debian Bug     : 12345

DLA text goes here
[...]

In addition to describing the vulnerabilities, remember to add a short package description, so the audience can better understand the context and whether they are affected.

Please also don't append your personal signatures to DLAs.

If you use mutt, a simple way to send it is to use

mutt -e "set signature=/dev/null" -H DLA-123

Simple "mail" command works:

cat DLA-XXXX.txt | gpg --clearsign | mail -s "[SECURITY] [DLA XXXX-X] $SOURCEPACKAGENAME security update" -a "From: Max Mustermann <max.mustermann@debian.org>" -r max.mustermann@debian.org debian-lts-announce@lists.debian.org

Only when you have confirmed that the package was processed after upload (once you get the accept email) should you send the DLA to the mailing list.

When all this is done, you should remove the package from other locations it was uploaded to in the test section, if any.

See also:

Prepare an update for the website

After sending the email, you must also prepare an update for the main debian.org web site with the new announcement. The benefits of posting to the web site are;

For this you need a copy of the website source. This is a one-time setup:

/!\ We recommend you request membership to the salsa webwml repo. Alternatively you can do merge requests. This workflow is work-in-progress.

git clone https://salsa.debian.org/webmaster-team/webwml
# or: salsa fork webmaster-team/webwml
# /usr/bin/salsa is part of devscripts in sid and stretch-bpo
cd webwml/

If you have made a salsa fork you can do the following (once) to prepare it so that you can branch off from the webmaster-team/webwml instead of your fork. This helps if you make multiple (with time in-between) website updates.

git remote add upstream https://salsa.debian.org/webmaster-team/webwml

Then, assuming the advisory number is DLA-1234-1 and the email template is in ~/DLA-1234-1, you should do the following:

git checkout -b DLA-1234-1

or if you have prepared it to have a separate upstream:

git fetch upstream
git checkout -b DLA-1234-1 upstream/master

Do the update

cd english/lts/security
./parse-dla.pl ~/DLA-1234-1
$EDITOR 2024/dla-1234* # make sure everything looks good
                       # Other option is, instead of looking at the html code, doing
                       # $ make -C 2024 dla-123.en.html
                       # and open the resulting html file with a web browser.
git add 2024/dla-1234.{data,wml}
git commit -m 'DLA-1234-1'

If you are a member of the debian-www project on salsa:

git push -u origin

If you are not a member you need to provide a merge request. First push it to your fork and then make the merge request.

git push --set-upstream origin DLA-1234-1
# Do this either one the command-line with 'salsa mr'
salsa mr
# or you need to be logged into the web ui and then go to
# https://salsa.debian.org/webmaster-team/webwml/merge_requests and press the green button labeled "new merge request"

QA: LTS Coordinator runs find-missing-advisories weekly.

Prepare regression updates for LTS

If an upload introduces a regression, it needs to be fixed as soon as possible.

Steps for a regression update

  1. Fix the package.
  2. Do a test build and test installation. Verify the regression issue and also the original issue are really resolved now.
  3. Make sure you use a clean chroot for building. Using sbuild/pbuilder is recommended.

  4. Upload the fixed package to stretch-security.
  5. Use gen-DLA script to add a DLA entry to data/DLA/list and to provide you with an announcement email template. Two possible options:
    1. The regression fixes the previous upload of the package:

      $ bin/gen-DLA  --save $SOURCEPACKAGENAME regression
    2. The regression fixes an earlier than the previous upload of the package:

      $ bin/gen-DLA  --save $PREVIOUS_DLA_ID $SOURCEPACKAGENAME regression
      # /!\ $PREVIOUS_DLA_ID needs to be incremented (-1 -> -2)...
  6. If this is a security regression (e.g. incomplete or missing fix), reference the CVE ID in the data/DLA/list entry, otherwise (functional regression) do not reference it.

  7. Wait for the uploaded package to arrive in the stretch-security repository.
  8. In the base folder of the security-tracker Git repository, you can find an auto-generated LTS announcement template file with name DLA-$DLA-{2,3,4...}. Use this file as the template for your regression fix announcement. Don't commit this file to Git(!). Copy+paste the announcement template to your email application, complete/edit it manually and finally send the regression fix DLA email to debian-lts-announce@lists.debian.org.

  9. Prepare an update for the website

  10. Note: if a DLA requires a fix in another package, use the same DLA major number (see mail)

Prepare other (non-security-related) updates for LTS

Same procedure as a security update, but it is recommended to edit the title of the announcement e-mail accordingly, for instance:

Triage new security issues

The security team and the LTS team cooperate to process the flow of new security issues that are reported. Since each security issue is assigned a CVE number, we often call this "CVE triage".

If you want to help with this, you must first get familiar with the security tracker. It's the infrastructure we use to track how security issues are affecting Debian packages. Please read carefully this page: http://security-team.debian.org/security_tracker.html

You will need write access to the security-tracker git repository.

CVE triaging in the LTS release

Now let's cover additional, LTS-specific workflow. We start from the list of open issues in the LTS release.

Sources of CVEs to triage:

For each CVE/package listed on these sources, there are only 2 desirable outcomes:

There are some differences between LTS triaging and Debian Security triaging:

This mainly has an implication on when CVEs are marked as no-dsa. If the Debian Security team marks the package as no-dsa, the package maintainer may still decide to fix it. This explains why some packages have minor issues fixed while other packages do not, fixed vulnerabilities needn't be critical.

Moreover, a security update may impact the end users' services (e.g. requires restarting a daemon, or simply a sysadmin to supervise each upgrade in a large network), so grouping "postponed" issues is usually a good idea (neither too often, to avoid these impacts; neither too infrequent, to avoid dragging a large backlog of changes with a more important security fix).

Finally, part of the triage work includes filing undocumented bugs for security issues in the Debian BTS, using the bin/report-vuln script. Bugs should be filed unless the issue is already fixed in unstable, avoiding duplicates of course.

Guidelines for CVE triaging

Front desk is responsible for the initial triaging of CVE. Nobody should circumvent this procedure without a very good reason.

These two reasonable exceptions are:

  1. There is a critical CVE vulnerability which should be fixed ASAP. The issue is often embargoed and affects a major package with priority important and higher. Those are rare, so this should be an exception really.
  2. A team member is also the maintainer of a package and they are already aware of the vulnerability and could confirm it. Please let people handle their own packages if they are the package maintainer or a regular contributor to a team maintained package.

The first step of CVE triaging is to determine whether a certain version of the package is affected at all.

Now the tricky part. How do you decide whether an issue should be added to dla-needed.txt or if it should be tagged in some other way? There's no perfect answer, you should use your best judgment after having *carefully* analyzed the issue, the risks involved, the trouble caused by admins to apply the update, etc. If in doubt check with the other LTS maintainers, the package maintainer (the bin/report-vuln script is a good way to get contact with the maintainer) or the security team.

That said the security team is taking similar decisions all the time and we usually follow their decisions. If the security team has added a vulnerable package to "data/dsa-needed.txt", you can usually safely add it to "data/dla-needed.txt" (if the bug is present in the LTS version of the package as well). If the security team has tagged the issue "postponed", "ignored" or "no-dsa" with "Minor issue" description, then you can usually use the same status for LTS. An exception is that when the reason picked by the security team is "Maintainer will update via stable-proposed-updates", in this case we must prepare an update for LTS since there is no lts-proposed-updates (yet). In addition, you can always fix no-dsa issues if you want to.

If a CVE has been fixed in Debian Stable it should, in general, be fixed in LTS as well, or marked as ignored. It does not make sense to have such CVEs marked as postponed or no-dsa since either the Debian Security team or the maintainer have decided that it was worth fixing.

We can and we want to fix CVE in Debian. If a package has too many "postponed" or "ignored" CVE, it is usually a sign that now is a good point to fix all of them together.

If in doubt rather add a package to dla-needed.txt, with a note about the doubt, and let another team member double-check the vulnerability.

You should not add packages to dla-needed.txt, if we have consistently ignored certain type of CVEs for a certain package in the past or the CVE is really minor.

If you add a package to dla-needed.txt you can add an optional NOTE to explain your thought process.

Examples:

To highlight the fact that a team member is also the maintainer of a package.

If you triage an issue that was not researched by the security team yet and it's unclear whether the issue warrants a DLA/DSA, it's good to document this in a note attached to the entry that you might add to dla-needed.txt.

If it's not clear whether the issue affects sid you should triage that too and open a corresponding bug in the BTS to inform the maintainer (except for embargoed issues). Maintainers often know best how severe an issue is and if an immediate update is actually needed.

Note that some packages have special instructions. Checkout packages/*.txt in the security-tracker repository. This might influence your decision and possibly prompt you to behave differently (for example by not contacting the maintainer since some predetermined agreement is already in place).

Contact the maintainer

Whenever you take a decision for a new issue (that is real and affecting the LTS release), you should inform the corresponding package maintainers to offer them a chance to help us, if they are listed in data/packages/lts-do-call-me. Add the package to dla-needed.txt and add a note when you have contacted the maintainer. Then wait for a response and act accordingly by updating the notes or removing the package again if the maintainer feels no security update is required.

Do not contact the maintainer, if they are listed in data/packages/lts-do-not-call.

In all other cases you can decide whether you want to contact the maintainer or not. In the past the majority of maintainers were glad if security updates were handled by the LTS team.

We use the bin/contact-maintainers script in the security-tracker repository to this end. Only do this if the bug is already recorded in the BTS. If it's not there yet you can use *bin/report-vuln* to open it. Discussions about severity and urgency are best done in the bugreport since all involved parties (LTS team, security team and package maintainers) can get involved.

When we add a package to dla-needed.txt, you can use it this way for example:

$ bin/contact-maintainers --lts sudo CVE-2014-9680 CVE-2014-0106

When we tag issues as "no-dsa", and don't plan to take care of the updates by ourselves, we usually do not contact maintainers. However, if it is necessary, there is another template available:

$ bin/contact-maintainers --lts --no-dsa sudo CVE-2014-9680 CVE-2014-0106

Note that the list of CVE is optional and that this script assumes that you use mutt to send mails. If not, you should consider using the --mailer option to use something else (see bin/contact-maintainer --help for details). You must customize the mail sent (at least to drop the list of uploaders and possibly to adjust the recipients list).

In order to avoid undue pressure on the maintainer, there are some exceptions to consider:

Frontdesk duties

As described in this email, there is an assigned person for the "LTS frontdesk". The allocations are managed in the lts-frontdesk.2024.txt file.

Frontdesk work currently involves CVE triage and making sure queries on the mailing list or on IRC get an answer. In particular, make sure to followup on package upload sponsorship requests and questions from new contributors to make sure things flow smoothly. Following up on stale and stalled issues is also a good idea. Contributors should feel free to contact frontdesk for any questions regarding LTS.

Frontdesk is also responsible for making sure packages claimed in data/dla-needed.txt are taken care of. When a package has been claimed for too long without activity, notify the claimant and, if there is no response within 24 hours, remove the lock. Note: currently (2022) LTS coordinator does it. Claims history can be inspected with:

bin/review-update-needed --lts

Finally, there should always be someone available in any given week on the frontdesk duty. When starting frontdesk, make sure the next 4 weeks are claimed by someone in the frontdesk roster. If not, send an email to the mailing list to ask for volunteers.

Checklist

Here's a quick summary checklist of the above tasks:

We're tracking LTS related bugs in the Debian bug tracker using two usertags:

To add a certain usertag to a bug you can use the bts command:

bts user debian-lts@lists.debian.org , usertags <bugnumber> ease-lts

Switching to the next LTS release

Switching to the next Stable release

Housekeeping Tasks

There are several tasks where we base our work on the work of the security team and where we can support them - especially when there are no open issues in dla-needed.txt. This will lead to more packages to fix:

Tips & tools


CategoryLts