Translation(s): English - Español - Français - Русский

7 steps for publishing a DLA

How-to prepare security updates for LTS

Contributing to Debian Long Term Support

The Debian LTS team is always looking for more volunteers to do a better job. With more resources, we could for example:

If you want to get involved in the LTS team and help keep Debian packages secure for 5 years, have a look at this page. We are assuming that you are familiar with the repository layout described in LTS/Using and that you are subscribed to the mailing list: http://lists.debian.org/debian-lts/

You can help in many ways:

Test updated packages and report regressions

As a simple user, you can test packages that have been updated (or that are in the process of being updated). If you find a regression (i.e. something that used to work and that no longer works), then please report it to debian-lts@lists.debian.org and put the person who prepared the update in copy (in case they are not subscribed to the list).

Many LTS contributors are looking for testers of their updated packages. They send call for tests on the mailing list, so please subscribe to it and test the packages they provide when you can, and report back whether they worked for you.

Debian Security Tracker

The Debian LTS team makes extensive use of the Debian Security Tracker. This is a database of all known security issues in Debian.

Developers check out the Git source (see instructions), commit changes, and the website automatically updates to reflect the changes.

LTS Developers are strongly encouraged to keep the data in the security tracker up-to-date. This includes updating the following files:

Make sure to read its documentation on the Debian Security Team website.

Prepare security updates for LTS

Claim the issue in the security tracker (in dla-needed.txt)

In order to prevent duplication of effort, make sure the issue is listed in the data/dla-needed.txt file from the security tracker and add your name to it. Normally, issues are triaged by the frontdesk in dla-needed.txt at first. If not, you should add the issue to dla-needed.txt yourself. Look at how that is done below before adding an issue or adding your name to an issue.

We give package maintainers 24 to 72 hours delay before taking on updates ourselves, depending on the severity of the security issue.

As mentioned in the previous section, make sure to also review and update the relevant information in data/CVE/list when claiming a package for LTS work.

You shall only claim it for the time you do active work. You are expected to actively handle the problem within 2-3 days and then you are expected to announce your intention to upload so others can test it for a few (3-4) more days. If you can not solve the problem within the 2-3 days, then please mention why it takes more time, how far you have gone, the preliminary results and/or free it for someone else to work on it. We do not want people to work on a package for a few hours, wait a few days, work again on it, wait more and so on.

Checkout the git repository of the security tracker: (Using a shallow clone because otherwise the operation will take half an hour. The clone will still need a few minutes to complete.)

$ git clone --depth 1 https://salsa.debian.org/security-tracker-team/security-tracker
$ cd security-tracker
$ sudo apt install python-apt
$ bin/setup-repo

*setup/repo* installs a pre-commit hook to check data file syntax.

Then edit the data/dla-needed.txt file:

$ editor data/dla-needed.txt
$ git commit -m "Claim foo in dla-needed.txt" data/dla-needed.txt
$ git push

Write access to the security-tracker repository

Debian Developers automatically have commit access to the security-tracker repository. Otherwise you need to be a member of the security-tracker project on salsa.debian.org, please request membership through the security tracker team gitlab project, security-tracker mailinglist or debian-lts mailinglist.

Build the update

Backport the fix to the version in LTS (in case there's already been an earlier update). Make sure that the fix was approved by the upstream project (e.g. committed to their development repository), sometimes early patches proposals are erroneous or incomplete. If there is no fix, it is fine to work on a new one: be a good citizen and contribute the patch back upstream and make sure the maintainer and regular security teams will be aware of them, by adding it to a bug report against the package and linking that bug report in the security tracker. (If the issue being addressed requires AddressSanitizer in order to reproduce the vulnerability and confirm the fix, then this page may be helpful).

In the changelog, you need to set the target distribution in debian/changelog to "jessie-security". The versioning follows the conventions already used in security.debian.org. Historically codenames have been used as version numbers, but this was changed some time ago as version numbers are more deterministic.

BTS numbers can be referenced but are currently not processed.

Note that for non-native packages, uploads to security-master need to include the source tarball (even if it's already present in the distribution you're uploading to) by using 'dpkg-buildpackage -sa'.

Make sure you used a clean chroot for building the binary packages because the binary packages that you upload end up in the repository (they are not discarded and rebuilt). Using sbuild/pbuilder is recommended. For instance:

# Init (note: jessie->jessie buggy https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806377)
sudo pbuilder --create --basetgz /var/cache/pbuilder/base-jessie.tgz --distribution jessie --othermirror 'deb http://security.debian.org/ jessie/updates main contrib'
sudo pbuilder --update --basetgz /var/cache/pbuilder/base-jessie.tgz
# Rebuild source and binary packages from Jessie (in extracted source)
export DEB_BUILD_OPTIONS="parallel=$(nproc)"
pdebuild --use-pdebuild-internal --buildresult .. --debbuildopts '-S' -- --basetgz /var/cache/pbuilder/base-jessie.tgz
# Rebuild binary packages from Jessie
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-jessie.tgz --buildresult . --debbuildopts '-sa' package+deb8u1.dsc
sudo --preserve-env=DEB_BUILD_OPTIONS pbuilder --build --basetgz /var/cache/pbuilder/base-jessie.tgz --buildresult . package+deb8u2.dsc

To actually build the binary package, you may want to use a porter box, especially if you need to fix a regression that concerns a specific architecture you may not have access to, see PorterBoxHowToUse for instructions on how to use porterboxes.

See also: Debian Developer's Reference - Preparing packages to address security issues

Test the update

Now build the package and run tests. You may do this by making sure a built-in test suite runs at build-time in the LTS environment. It is acceptable to enable or even backport test suites from later releases if they are available.

If there are no test suites available, you will need to run the package directly. The complexity of this will vary with the size of the package: smaller packages can just be ran in a chroot, others will need a full virtualization environment, install procedures and everything. It may be difficult for you to test the whole package, especially if you are not familiar with the package first hand.

It is worth testing with both a 32- and 64-bit architecture, see e.g. CVE-2019-14866.

Check your package update with generic tools as well:

# check for common packaging issues in last build
# from extracted source after build, jessie host (only check new errors)
lintian -i
# inspect source changes
debdiff package+deb8u3.dsc package+deb8u4.dsc
# inspect binary changes
debdiff --from deb8u3/*.deb --to deb8u4/*.deb
# test package upgrade
sudo piuparts -d jessie \
  --extra-repo='deb http://security.debian.org/ jessie/updates main contrib' \
  -l piuparts-package.log \
  -I :etc/buggy-dep \
  package+deb8u4_amd64.changes \
  | grep -P '(INFO|ERROR):'
# also consider --install-remove-install

Depending on the urgency of the security fix, and your confidence on your own work, you might want to seek reviews from interested parties (such as upstream authors, Debian maintainers, security team, LTS team, etc.). If you ask for reviews, you should supply the following information:

  1. A generated debdiff against the previous version in LTS.
  2. A link to a test package that can be downloaded and tested.

It is common for LTS workers to upload their packages to a personal repository in people.debian.org, but any public repository or even simple webserver will do. A few guidelines for those temporary uploads:

Upload the update

Uploads to Debian LTS do not wait for a DLA or any other manual intervention before being installed into the Debian Archive. In this regard, they are different from stable security uploads.

If you're satisfied with your tests, upload to security-master, using dput or dput-ng:

dput security-master hello_1.0-1+deb8u1_amd64.changes

Once uploaded the package will be auto-built for the architectures not covered by your upload (if it's an arch:any package). The architectures currently supported are amd64, i386, armel and armhf. Check the build logs at: https://buildd.debian.org/status/package.php?suite=jessie-security&p=source_package

Claim a DLA ID in DLA/list

Run bin/gen-DLA in the top directory of the Git repository. It automatically generates an entry in data/DLA/list and asks you to commit the changes to ensure that no IDs are used twice. The following command would add an entry for src:hello fixing CVE-2014-0666 and creates an advisory template in the top directory of your security-tracker checkout.

$ bin/gen-DLA --save hello CVE-2014-0666
$ bin/gen-DLA --save .../hello_1.2-3+deb8u4_amd64.changes

That list of CVE will make sure that the security tracker is aware of the fix and mark the issues as resolved for that specific package.

In data/CVE/list, you need to manually remove prior suite triage for the fixed vulnerabilities (e.g. [jessie] - package <postponed>). However, the {DLA-XXXX-1} annotation will be added by the bi-daily cron, and the fixed version will be inferred automatically (don't add it).

Announce the update

Once the upload is completed, an email will be sent to you and to the debian-lts-changes mailing list, confirming the update has been released into the Debian archive. The BTS bot will also announce it in the #debian-devel-changes channel.

Send mail to the debian-lts-announce mailing list. The mail needs to be signed by a GPG key in the debian or debian-maintainers keyring. Inline signatures always work (PGP/MIME works with mutt but fails with Enigmail).

The advisory template has been created by bin/gen-DLA (see before) and generally looks like this:

Subject: [SECURITY] [DLA $DLA-1] $SOURCEPACKAGENAME security update

Package        : $SOURCEPACKAGENAME
Version        : $LTS_VERSION
CVE ID         : CVE-2014-0001 CVE-2014-0002
Debian Bug     : 12345

DLA text goes here
[...]

In addition to describing the vulnerabilities, remember to add a short package description, so the audience can better understands the context and whether they are affected.

Please also don't append your personal signatures to DLAs.

If you use mutt, a simple way to send it is to use mutt -e "set signature=/dev/null" -H DLA-123.

Only when you have confirmed that the package was processed after upload (once you get the accept email) should you send the DLA to the mailing list.

When all this is done, you should remove the package from other locations it was uploaded to in the test section, if any.

See also: Debian Developer's Reference - Security Advisories

Prepare an update for the website

After sending the email, you must also prepare an update for the main debian.org website with the new announcement. Benefits:

For this you need a copy of the website source. This is a one-time setup:

/!\ We recommend you request membership to the salsa webmaster-team group. Alternatively you can do merge requests. This workflow is work-in-progress.

git clone https://salsa.debian.org/webmaster-team/webwml
# or: salsa fork webmaster-team/webwml
# /usr/bin/salsa is part of devscripts in sid and stretch-bpo
cd webwml/

If you have made a salsa fork you can do the following (once) to prepare it so that you can branch off from the webmaster-team/webwml instead of your fork. This helps if you make multiple (with time in-between) website updates.

git remote add upstream https://salsa.debian.org/webmaster-team/webwml

Then, assuming the advisory number is DLA-1234-1 and the email template is in ~/DLA-1234-1, you should do the following:

git checkout -b DLA-1234-1

or if you have prepared it to have a separate upstream:

git fetch upstream
git checkout -b DLA-1234-1 upstream/master

Do the update

cd english/lts/security
./parse-dla.pl ~/DLA-1234-1
$EDITOR 2020/dla-1234* # make sure everything looks good
                       # Other option is, instead of looking at the html code, doing
                       # $ make -C 2020 dla-123.en.html
                       # and open the resulting html file with a web browser.
git add 2020/dla-1234.{data,wml}
git commit -m 'DLA-1234-1'

If you are a member of the debian-www project on salsa:

git push -u origin

If you are not a member you need to provide a merge request. First push it to your fork and then make the merge request.

git push --set-upstream origin DLA-1234-1
# Do this either one the command-line with 'salsa mr'
salsa mr
# or you need to be logged into the web ui and then go to
# https://salsa.debian.org/webmaster-team/webwml/merge_requests and press the green button labeled "new merge request"

Prepare regression updates for LTS

If an upload introduces a regression, it needs to be fixed as soon as possible.

Steps for a regression update

  1. Fix the package.
  2. Do a test build and test installation. Verify the regression issue and also the original issue are really resolved now.
  3. Make sure you used a clean chroot for building the binary packages because the binary packages you upload end up in the repository (they are not discarded and rebuilt). Using sbuild/pbuilder is recommended.

  4. Upload the fixed package to jessie-security.
  5. Use gen-DLA script to add a DLA entry to data/DLA/list and to provide you with an announcement email template. Two possible options:
    1. The regression fixes the previous upload of the package:

      $ bin/gen-DLA  --save $SOURCEPACKAGENAME regression
    2. The regression fixes an earlier than the previous upload of the package:

      $ bin/gen-DLA  --save $PREVIOUS_DLA_ID $SOURCEPACKAGENAME regression

/!\ $PREVIOUS_DLA_ID needs to be incremented...

  1. Commit the updated data/DLA/list to the security-tracker Git repository.

  2. Wait for the uploaded package to arrive in the jessie-security repository.
  3. In the base folder of the security-tracker Git repository, you can find an auto-generated LTS announcement template file with name DLA-$DLA-{2,3,4...}. Use this file as the template for your regression fix announcement. Don't commit this file to Git(!). Copy+paste the announcement template to your email application, complete/edit it manually and finally send the regression fix DLA email to debian-lts-announce@lists.debian.org.

  4. Note: if a DLA requires a fix in another package, use the same DLA major number (see mail)

Triage new security issues

The security team and the LTS team cooperate to process the flow of new security issues that are reported. Since each security issue is assigned a CVE number, we often call this "CVE triage".

If you want to help with this, you must first get familiar with the security tracker. It's the infrastructure we use to track how security issues are affecting Debian packages. Please read carefully this page: http://security-team.debian.org/security_tracker.html

You will need write access to the security-tracker git repository.

CVE triaging in the LTS release

Now let's cover additional, LTS-specific workflow. We start from the list of open issues in the LTS release.

You can start with the list on the web at this url: https://security-tracker.debian.org/tracker/status/release/oldstable (tick "include issues to be checked") Or you can run bin/lts-cve-triage.py (script in the security-tracker repository) to get a pre-filtered list (packages in dla-needed.txt are excluded, and the issues are grouped by "status").

For each CVE/package listed on this page, there are only 2 desirable outcomes:

There are some differences between LTS triaging and Debian Security triaging:

This mainly has an implication on when CVEs are marked as no-dsa. If the Debian Security team marks the package as no-dsa, the package maintainer may still decide to fix it. This explains why some packages have minor issues fixed while other packages do not, fixed vulnerabilities needn't be critical.

Moreover, a security update may impact the end users' services (e.g. requires restarting a daemon, or simply a sysadmin to supervise each upgrade in a large network), so grouping "postponed" issues is usually a good idea (neither too often, to avoid these impacts; neither too infrequent, to avoid dragging a large backlog of changes with a more important security fix).

Finally, part of the triage work includes filing undocumented bugs for security issues in the Debian BTS, using the bin/report-vuln script. Bugs should be filed unless the issue is already fixed in unstable, avoiding duplicates of course.

Guidelines for CVE triaging

Now the tricky part. How do you decide whether an issue should be added to dla-needed.txt or if it should be tagged in some other way? There's no perfect answer, you should use your best judgment after having *carefully* analyzed the issue, the risks involved, the trouble caused by admins to apply the update, etc. If in doubt check with the other LTS maintainers, the package maintainer or the security team.

That said the security team is taking similar decisions all the time and we usually follow their decisions. If the security team has added a vulnerable package to "data/dsa-needed.txt", you can usually safely add it to "data/dla-needed.txt" (if the bug is present in the LTS version of the package as well). If the security team has tagged the issue "postponed" with "Minor issue" description, you can safely do the same. An exception is that when the reason picked by the security team is "Maintainer will update via stable-proposed-updates", in this case we must prepare an update for LTS since there is no lts-proposed-updates (yet). In addition, you can always fix no-dsa issues if you want to.

If you triage an issue that was not researched by the security team yet and it's unclear whether the issue warrants a DLA/DSA, it's good to document this in a note attached to the entry that you might add to dla-needed.txt. It's usually pointless to fix bugs in LTS that will never be fixed in stable or even unstable. Granted, exceptional cases exist, such as when the issue has a greater impact on LTS release.

If it's not clear whether the issue affects sid you should triage that too and open a corresponding bug in the BTS to inform the maintainer (except for embargoed issues). Maintainers often know best how severe an issue is and if an immediate update is actually needed.

Note that some packages have special instructions. Checkout packages/*.txt in the security-tracker repository. This might influence your decision and possibly prompt you to behave differently (for example by not contacting the maintainer since some predetermined agreement is already in place).

Contact the maintainer

Whenever you take a decision for a new issue (that is real and affecting the LTS release), you should inform the corresponding package maintainers to offer them a chance to help us, if they are listed in data/packages/lts-do-call-me. Add the package to dla-needed.txt and add a note when you contacted the maintainer. Then wait for a response and act accordingly by updating the notes or removing the package again if the maintainer feels no security update is required.

Do not contact the maintainer, if he or she is listed in data/packages/lts-do-not-call.

In all other cases you can decide whether you want to contact the maintainer or not. In the past the majority of maintainers were glad if security updates were handled by the LTS team.

We use the bin/contact-maintainers script in the security-tracker repository to this end. Only do this if the bug is already recorded in the BTS. If it's not there yet you can use *bin/report-vuln* to open it. Discussions about severity and urgency are best done in the bugreport since all involved parties (LTS team, security team and package maintainers) can get involved.

When we add a package to dla-needed.txt, you can use it this way for example:

$ bin/contact-maintainers --lts sudo CVE-2014-9680 CVE-2014-0106

When we tag issues as "no-dsa", and don't plan to take care of the updates by ourselves, we usually do not contact maintainers. If necessary though, there is another template available:

$ bin/contact-maintainers --lts --no-dsa sudo CVE-2014-9680 CVE-2014-0106

Note that the list of CVE is optional and that this script assumes that you use mutt to send mails. If not, you should consider using the --mailer option to use something else (see bin/contact-maintainer --help for details). You must customize the mail sent (at least to drop the list of uploaders and possibly to adjust the recipients list).

In order to avoid maintainer to feel pushed there are some exceptions to consider:

Frontdesk duties

As described in this email, there is an assigned person for the "LTS frontdesk". The allocations are managed in the lts-frontdesk.2020.txt file.

Frontdesk work currently involves CVE triage and making sure queries on the mailing list get an answer. In particular, make sure to followup on package upload sponsorship requests and questions from new contributors to make sure things flow smoothly. Following up on stale and stalled issues is also a good idea. Contributors should feel free to contact frontdesk for any questions regarding LTS.

Frontdesk is also responsible for making sure packages claimed in data/dla-needed.txt are taken care of. When a package has been claimed for too long without activity, notify the claimant and, if there is no response within 24 hours, remove the lock. Note: currently (2019-04) Holger does this as part of LTS coordination. Claim's history can be inspected with:

bin/review-update-needed --lts

Finally, there should always be someone available in any given week on the frontdesk duty. When starting frontdesk, make sure the next 4 weeks are claimed by someone in the frontdesk roster. If not, send an email to the mailing list to ask for volunteers.

Checklist

Here's a quick summary checklist of the above tasks:

We're tracking LTS related bugs in the Debian bug tracker using two usertags:

To add a certain usertag to a bug you can use the bts command:

bts user debian-lts@lists.debian.org , usertags <bugnumber> ease-lts

Switching to the next LTS release

Housekeeping Tasks

There are several tasks where we base our work on the work of the security team and where we can support them - especially when there are no open issues in dla-needed.txt. This will lead to more packages to fix:

Tips & tools


CategoryLts