Differences between revisions 1 and 53 (spanning 52 versions)
Revision 1 as of 2006-01-06 22:48:28
Size: 12423
Editor: JoeyHess
Comment:
Revision 53 as of 2009-05-07 05:37:28
Size: 22077
Editor: FranklinPiat
Comment: CategoryPermalink
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#language en
Line 2: Line 3:

Recently Debian's unstable and testing branches have begun to use
strong crypro to validate downloaded packages. This is commonly called
"secure apt" and was implemented in version 0.6. Since the documentation is fairly slim on how this all works from an administrator's point of view, this document will try to explain in detail how it secure apt works and how to use it.

[[TableOfContents([2])]]
In recent releases, Debian has been using strong crypto to validate downloaded packages. This is commonly called "secure apt" (or "apt-secure") and was implemented in Apt version 0.6 in 2003, which Debian migrated to in 2005. Since the documentation ([[http://www.enyo.de/fw/software/apt-secure/|here]] and [[http://www.syntaxpolice.org/apt-secure/|here]]) is fairly slim on how this all works from an administrator's point of view, this document will try to explain in detail how secure apt works and how to use it.

<<TableOfContents(2)>>
Line 10: Line 8:

Here are a few basic concepts that you'll need to understand for the rest
of this document.

A checksum is a method of taking a file and boiling it down to a reasonably
short number that uniquely identifies the content of the file. This is a
lot harder t
o well than it might seem, and the most commonly used type of
checksum, the md5sum, is in the process of being broken.

Public key cryptography is based on pairs of keys, each consisting of one
public key and one
private key. The public key is given out to the world;
the private key must be kept a secret. Anyone possessing one key can
encrypt a message so that it can only be read by someone possessing the
other
key. It's also possible to use a key to only sign a file, not
encrypt it. If a private key is used to sign a file, then anyone who has
the public key can check that the file was signed by that key. No one who
doesn't have the private key can forge such a signature.

These keys are quite long numbers (1024 to 2048 digets or more long), and to
make it
easier to work with them they have a key id, which is a shorter, 8
or 16 diget number that can be used to refer to them.

gpg is the tool used in secure apt to sign files and check their
signatures.

apt-key is a program that is used to manage a keyring of gpg keys for
secure apt. The keyring is kept in the file /etc/apt/trusted.gpg (not to be
confused with the related but not very interesting /etc/apt/trustdb.gpg).
apt-key can be used to show the keys in the keyring, and to add or remove a
key.
Here are a few basic concepts that you'll need to understand for the rest of this document.

A checksum is a method of taking a file and boiling it down to a reasonably short number that uniquely identifies the content of the file. This is a lot harder to do well than it might seem, and the most commonly used type of checksum, [[http://en.wikipedia.org/wiki/MD5|MD5]], is now a broken hash function, and should be replaced for all security-minded usages.

Public key cryptography is based on pairs of keys, a public key and a private key. The public key is given out to the world; the private key must be kept a secret. Anyone possessing the public key can encrypt a message so that it can only be read by someone possessing the private key. It's also possible to use a private key to sign a file, not encrypt it. If a private key is used to sign a file, then anyone who has the public key can check that the file was signed by that key. Anyone who doesn't have the private key can't forge such a signature.

These keys are quite long numbers (at least 1024 bits, i.e. 256 or more hex digits), and to make them easier to work with they have a key id, which is a shorter, 8 or 16 digit number that can be used to refer to them.

gpg is the tool used in secure apt to sign files and check their signatures.

apt-key is a program that is used to manage a keyring of gpg keys for secure apt. The keyring is kept in the file /etc/apt/trusted.gpg (not to be confused with the related but not very interesting /etc/apt/trustdb.gpg). apt-key can be used to show the keys in the keyring, and to add or remove a key.
Line 42: Line 21:

A Debian archive contains a Release file, which is updated each time any
of the packages in the archive change. Amoung other things, the Release
file contains some md5sums of other files in the archive. An excerpt of an
example Release
file:
A Debian archive contains a Release file, which is updated each time any of the packages in the archive change. Among other things, the Release file contains some checksums of other files in the archive. An excerpt of an example Release file:
Line 56: Line 31:
(The Release files also include sha1 checksums, which will be useful
once md5sums become fully broken, however apt doesn't use them yet.)

Now if we look inside a Packages file, we'll find more md5sums, once for
each package listed in it. For example:
Now if we look inside a Packages file, we'll find more checksums, one for each package listed in it. For example:
Line 70: Line 41:

These two checksums allow apt to verify that it's downloaded a correct copy
of the Packages file, with a md5sum that matches the one in the Release
file. And when it downloads an individual package, it can also check
its md5sum against the content of the Packages file.
If apt fails at either
of these steps, it will abort.

None of this is new in secure apt, but it does provide the foundation.
Notice that so far there is one file that apt doesn't have a way to
check: The Release file. Secure apt is all about making apt verify the
Release file before it does anything else with it, and plugging this hole,
so that there is a chain of verification from the package that you are
going to install all the way back to the provider of the package.
These two checksums allow apt to verify that it's downloaded a correct copy of the Packages file, with an checksum that matches the one in the Release file. And when it downloads an individual package, it can also check its checksum against the content of the Packages file. If apt fails at either of these steps, it will abort.

None of this is new in secure apt, but it does provide the foundation. Notice that so far there is one file that apt doesn't have a way to check: The Release file. Secure apt is all about making apt verify the Release file before it does anything else with it, and plugging this hole, so that there is a chain of verification from the package that you are going to install all the way back to the provider of the package.
Line 85: Line 46:

To plug the hole, secure apt adds a gpg signature for the Release file.
This is put in a file named Release.gpg that's shipped alongside the
Release file. It looks something like this, although only gpg actually
looks at its contents normally:
To plug the hole, secure apt adds a gpg signature for the Release file. This is put in a file named Release.gpg that's shipped alongside the Release file. It looks something like this, although only gpg actually looks at its contents normally:
Line 94: Line 51:
Line 100: Line 56:
Line 104: Line 59:

Secure apt always downloads Release.gpg files when it's downloading Release
files, and if it cannot download the Release.gpg, or if the signature is bad,
it will complain, and will make note that the Packages files that
the Release file points to, and all the packages listed therin, are from
an untrusted source. Here's how it looks during an apt-get update:
Secure apt always downloads Release.gpg files when it's downloading Release files, and if it cannot download the Release.gpg, or if the signature is bad, it will complain, and will make note that the Packages files that the Release file points to, and all the packages listed therein, are from an untrusted source. Here's how it looks during an apt-get update:
Line 113: Line 63:
 couldn't be verified because the public key is not available: NO_PUBKEY 0109083
12D230C5F
}}}

Note that the second half of the long number is the key id of the key
that apt doesn't know about, in this case that's 2D230C5F.

If you ignore that warning and try to install a package later, apt
will warn again:
 couldn't be verified because the public key is not available: NO_PUBKEY 010908312D230C5F
}}}
Note that the second half of the long number is the key id of the key that apt doesn't know about, in this case that's 2D230C5F.

If you ignore that warning and try to install a package later, apt will warn again:
Line 128: Line 74:

If you say Y here you have no way to know if the file you're getting is
package you're supposed to install, or if it's something else entirely
that a black hat has arranged for you, containing a nasty suprise.

Note that you can diable these checks by running apt with
--allow-unauthenticated.

It's also worth noting that newer versions of the Debian installer use the
same signed Release file mechanism during their debootstap of the Debian
base system, before apt is available, and that the installer even uses
this system to verify peices of itself that it downloads from the net.
Also, Debian does not currently sign the Release files on its CDs; apt can
be configured to always trust packages from CDs so this is not a large
problem.
If you say Y here you have no way to know if the file you're getting is the package you're supposed to install, or if it's something else entirely that a black hat has arranged for you, containing a nasty suprise.

Note that you can disable these checks by running apt with --allow-unauthenticated.

It's also worth noting that newer versions of the Debian installer use the same signed Release file mechanism during their debootstrap of the Debian base system, before apt is available, and that the installer even uses this system to verify pieces of itself that it downloads from the net. Also, Debian does not currently sign the Release files on its CDs; apt can be configured to always trust packages from CDs so this is not a large problem.
Line 145: Line 81:

So the security of the whole system depends on there being a Release.gpg
file, which signs a Release file, and of apt checking that signature using
gpg. To check the signature, it has to know the public key of the person
who signed the file. These keys are kept in apt's own keyring
(/etc/apt/trusted.gpg), and managing the keys is where secure apt comes in.

By default, Debian systems come preconfigured with the Debian archive key
in the keyring.
So the security of the whole system depends on there being a Release.gpg file, which signs a Release file, and of apt checking that signature using gpg. To check the signature, it has to know the public key of the person who signed the file. These keys are kept in apt's own keyring (/etc/apt/trusted.gpg), and managing the keys is where secure apt comes in.

By default, Debian systems come preconfigured with the Debian archive key in the keyring.
Line 163: Line 93:

Here 4F368D5D is the key id, and notice that this key was only valid
for a one year period. Debian rotates these keys as a last line of defense
against some sort of security breach breaking a key.

That will make apt trust the official Debian archive, but if you add some
other apt repository to /etc/apt/sources.list, you'll also have to give apt
it's key if you want apt to trust it. Once you have the key and have
verified it, it's a simple matter of "apt-key add file" to add i
t. Getting
the key and verifying it are the trickier part.
Here 4F368D5D is the key id, and notice that this key was only valid for a one year period. Debian occasionally rotates these keys as a last line of defense against some sort of security breach breaking a key.

That will make apt trust the official Debian archive, but if you add some other apt repository to /etc/apt/sources.list, you'll also have to give apt its key if you want apt to trust it. Once you have the key and have verified it, it's a simple matter of "apt-key add file" to add it. Getting the key and verifying it are the trickier part.
Line 175: Line 98:

There is not yet a standard location where you can find the key for a given
apt repository. There's a rough standard of putting the key up on the web
page for the repository or as a file in the repository itself, but no
real standard, so you might have to hunt for it.

The Debian archive signing key is available at
http://ftp-master.debian.org/ziyi_key_2006.asc (replace 2006 with current
year). (Nobody seems to be telling what "ziyi" means.)

gpg itself has a standard way to distribute keys, using a keyserver that
gpg can download a key from and add it to its keyring. For example:
The debian-archive-keyring package is used to distribute keys to apt. Upgrades to this package can add (or remove) gpg keys for the main Debian archive.

For other archives, there is not yet a standard location where you can find the key for a given apt repository.
There's a rough standard of putting the key up on the web page for the repository or as a file in the repository itself, but no real standard, so you might have to hunt for it.

The Debian archive "signing" key is available at http://ftp-master.debian.org/ziyi_key_2006.asc (replace 2006 with current year). (the tool that is used for signing on the Debian servers is called "ziyi", after [[http://en.wikipedia.org/wiki/Zhang_Ziyi|Ziyi Zhang]].) This key is not the secret signing key as its name might suggest, but the public testing key.

gpg itself has a standard way to distribute keys, using a keyserver that gpg can download a key from and add it to its keyring. For example:
Line 196: Line 114:
Line 204: Line 121:

(What does the "gpg: no ultimately trusted keys found" warning mean?)
(What does the "gpg: no ultimately trusted keys found" warning mean? --> The Warning: "no ultimately trusted keys found" means that gpg was not configured to ultimately trust a specific key. Trust settings are part of OpenPGPs Web-of-Trust which does not apply here. So there is no problem with this warning. In usual setups the users own key is ultimately trusted.)
Line 208: Line 124:

By adding a key to apt's keyring, you're telling apt to trust everything
signed by the key, and this lets you know for sure that apt won't install
anything not signed by the person who possesses the private key. But if
you're sufficiently paranoid, you can see that this just pushes things up
a level, now instead of having to worry if a package, or a Release file
is valid, you can worry about whether you've actually gotten the right key.
Is the http://ftp-master.debian.org/ziyi_key_2006.asc mentiond above really
Debian's archive signing key, or is this document a clever trap?

It's good to be paranoid in security, but verifying things from here is
harder. gpg has the concept of a chain of trust, which can start at someone
you're sure of, who signs someone's key, who signs some other key, etc
until you get to the archive key. If you're sufficiently paranoid you'll
want to check that your archive key is signed by a key that you can trust,
with a trust chain that goes back to someone you know personally. If you
want to do this, visit a Debian conference.

(Note: Not all apt repoisitory keys are signed at all by another key. Maybe
the person setting up the repository doesn't have another key, or maybe they
don't feel comfortable signing such a role key with their main key.)

If you can't afford this level of paranoia, do whatever feels appropriate
to you when adding a new apt source and a new key. Maybe you'll want to
mail the person providing the key and verify it, or maybe you're willing to
take your chances with downloading it and assuming you got the real thing.
The important thing is that by reducing the problem to what archive
keys to trust, secure apt lets you be as careful and secure as it suites
you to be.

== Debian archive key yearly rotation ==

As mentioned above, the Debian archive signing key is changed each year,
in January. Since secure apt is young, we don't have a great deal of
experience with changing the key and there are still rough spots.

In January 2006, a new key for 2006 was made and the Release file began to
be signed by it, but to try to avoid breaking systems that had the old 2005
key, the Release file was signed by that as well. The intent was that apt
would accept one signature or the other depending on the key it had, but
apt turned out to be buggy and refused to trust the file unless it had both
keys and was able to check both signatures. This was fixed in apt version
0.6.43.1. There was also confusion about how the key was distributed to
users who already had systems using secure apt; initially it was uploaded to
the web site with no announcement and no real way to verify it and users
were forced to download it by hand.

Based on this experience, here's how things could work in 2007:

 * Early in January a new key for 2007 will be created. Perhaps with an
   annoucement and a well-defined chain of trust this time.
 * The Release file will be signed by this key, while also being signed
   still by the 2006 key. apt and other tools will accept either signature.
 * A new package, debain-server-keyring, will have been installed on
   everyone's system before hand. It will be updated to include the 2007
   key. When users upgrade to the new version, it will use apt-key to
   update their keyring, removing the 2006 key and adding the 2007 key.
 * The 2006 key expires on January 31st, 2007.
 * Still uncertian is what will happen to anyone who doesn't upgrade at all in
   January, and how this upgrade will be handled for people running stable,
   once secure apt is available there.

But it's really too early to tell.
By adding a key to apt's keyring, you're telling apt to trust everything signed by the key, and this lets you know for sure that apt won't install anything not signed by the person who possesses the private key. But if you're sufficiently paranoid, you can see that this just pushes things up a level, now instead of having to worry if a package, or a Release file is valid, you can worry about whether you've actually gotten the right key. Is the http://ftp-master.debian.org/ziyi_key_2006.asc mentioned above really Debian's archive testing key, or is this document a clever trap?

It's good to be paranoid in security, but verifying things from here is harder. gpg has the concept of a chain of trust, which can start at someone you're sure of, who signs someone's key, who signs some other key, etc., until you get to the archive key. If you're sufficiently paranoid you'll want to check that your archive key is signed by a key that you can trust, with a trust chain that goes back to someone you know personally. If you want to do this, visit a Debian conference or perhaps a local LUG for a key signing.

(Note: Not all apt repository keys are signed at all by another key. Maybe the person setting up the repository doesn't have another key, or maybe they don't feel comfortable signing such a role key with their main key.)

If you can't afford this level of paranoia, do whatever feels appropriate to you when adding a new apt source and a new key. Maybe you'll want to mail the person providing the key and verify it, or maybe you're willing to take your chances with downloading it and assuming you got the real thing. The important thing is that by reducing the problem to what archive keys to trust, secure apt lets you be as careful and secure as it suits you to be.

Here's a [[http://blog.madduck.net/debian/2006.01.08-apt-updates.html|blog post]] with a procedure to verify the key's integrity.

== Debian archive key expiry ==
Since secure apt was introduced, the keys used to sign the main Debian archive have changed a couple of times. Since secure apt is young, we don't have a great deal of experience with changing the key and there are still rough spots.

In January 2006, a new key for 2006 was made and the Release file began to be signed by it, but to try to avoid breaking systems that had the old 2005 key, the Release file was signed by that as well. The intent was that apt would accept one signature or the other depending on the key it had, but apt turned out to be buggy and refused to trust the file unless it had both keys and was able to check both signatures. This was fixed in apt version 0.6.43.1. There was also confusion about how the key was distributed to users who already had systems using secure apt; initially it was uploaded to the web site with no announcement and no real way to verify it and users were forced to download it by hand. This was fixed by the introduction of the debian-archive-keyring package, which manages apt keyring updates.

In late 2006, a new key was created that will be used to sign the archive for the lifetime of the Debian 4.0 release (until 2009-07-01). The archive began to be signed by this new key in addition to the yearly signing key for 2006. That was a bit confusing, because the key began to be used before it was announced and before debian-archive-keyring was updated to include it! Apt's warning message in this situation is slightly opaque to end users. There's obviously still room for improvement in how we roll out new keys. This new key does answer the question of how users of the 4.0 (etch) release will be able to validate their software for the lifetime of that release. This new key is also being used to sign other versions of debian (like unstable).

On February 7th 2007, the 2006 key expired. Currently the only known breakage of this is that it broke rc1 of the etch installer, since the installer images only know about the 2006 key. Daily builds of the installer have the 2007 key and continue to work.

Most recently, a new Etch stable release key has been added. This key is an offline key that will be used to sign releases of Etch (including point releases).

== Other problems ==
 * One not so obvious gotcha is that if your clock is very far off, secure apt will not work. If it's set to a date in the past, such as 1999, apt will fail with an unhelpful message such as this:
 {{{
  W: GPG error: http://archive.progeny.com sid Release: Unknown error executing gpg
 }}}
 Although apt-key list will make the problem plain:
 {{{
gpg: key 2D230C5F was created 192324901 seconds in the future (time warp or clock problem)
gpg: key 2D230C5F was created 192324901 seconds in the future (time warp or clock problem)
pub 1024D/2D230C5F 2006-01-03
uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
 }}}
 If it's set to a date too far in the future, apt will treat the keys as expired.
 * Another problem you may encounter if using testing or unstable is that if you have not run apt-get update lately and apt-get install a package, apt might complain that it cannot be authenticated (why does it do this?). apt-get update will fix this.
 * If apt gives a warning like this:
 {{{
W: There are no public key available for the following key IDs:
A70DAF536070D3A1
 }}}
  . This means that the archive has begun to be signed by a new key, which your system does not know about. In this example, the new key is a dedicated key that will be used to sign the release of Debian 4.0. Since the archive was still signed by another key that apt knows about, this is just a warning, and once the system is fed the new key (by upgrading the debian-archive-keyring package), the warning will go away.
 * If you have the debsig-verify package installed, you might run into errors like this one:
 {{{
dpkg: error
processing /var/cache/apt/archives/anjuta-common_1.2.4a-2_all.deb (--unpack):
 Verification on package /var/cache/apt/archives/anjuta-common_1.2.4a-2_all.deb failed!
Authenticating /var/cache/apt/archives/anjuta_1.2.4a-2_i386.deb ...
debsig: Origin Signature check failed. This deb might not be signed.
 }}}
 This actually has nothing to do with secure apt. debsig-verify checks for signatures embedded inside individual Debian packages. Since such signatures are not widely used (we use secure apt instead), it doesn't work very well to install this, and removing the debsig-verify package will fix the problem.
 * If apt-get update outputs this
 {{{
W: GPG error: http://non-us.debian.org stable/non-US Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F1D53D8C4F368D5D
W: You may want to run apt-get update to correct these problems
 }}}
 remove non-us from /etc/apt/sources.
Line 273: Line 181:

TODO
From {{{man apt-secure}}}

If you want to provide archive signatures in an archive under your maintenance you have to:

 * Create a toplevel Release file. if it does not exist already. You can do this by running apt-ftparchive release (provided inftp apt-utils).
 * Sign it. You can do this by running gpg -abs -o Release.gpg Release.
 * Publish the key fingerprint, that way your users will know what key they need to import in order to authenticate the files in the archive.
Whenever the contents of the archive changes (new packages are added or removed) the archive maintainer has to follow the first two steps previously outlined.

== History ==
Conectiva implemented something similar in their fork of APT. Debian Developers Colin Walters and Isaac Jones implemented APT Secure for Debian in 2003. Around Christmas 2003, Matt Zimmerman(?) integrated this patch into APT 0.6. In February 2005, Debian started [[http://www.enyo.de/fw/software/apt-secure/|migrating]] to apt-secure.
Line 277: Line 194:
Line 280: Line 196:
Debian isn't Ubuntu and won't have sudo installed by default. It might be worth changing the usage of sudo, and making the examples use the root account explicitly -- SteveKemp

Given the failure modes I've seen from gpg --recv-keys, suggesting that a user run it as root doesn't seem wise to me. But I've never actually audited it either.. Despite sudo not being installed by default (in sarge), I think that most of the audience of this page are familiar with it, or can skip over it. -- JoeyHess

Does it make any sense to pre-install debian-server-keyring on every debian system? At least the user should be asked if s/he trusts this key. Until now (2006-01-07) the Debian Archive Automatic Signing Key is not in the strong set of keys. Thus is it not possible to test via pathfinders if there is a trust path from my key to the Debian Archive Automatic Signing Key. (See http://pgp.cs.uu.nl/ or http://www.lysator.liu.se/~jc/wotsap/search.html)

Yes, the debian-archive-keyring package will be our key upgrade path for all Debian systems, so it should be installed on all of them, and I assume will be in standard. -- JoeyHess

Can we somehow integrate [[http://lists.debian.org/debian-devel/2004/06/msg01499.html|this idea]] into this page? I think it's important we also move in this direction. -- [[madduck]]

I tried making a local repository using apt-move, serving machines running testing and stable. I could not use the same repository for both stable and testing due to incopatibilities between apt v5 and v6. You might highlight this when you get around to writing the bit about crating a repository. I could get v6 working for testing but not for sarge. --MartinHodges

What does it mean for md5sum to be broken? Since it's a checksum, I thought the only way it can be broken is that it fail to compute the proper checksum. I have a feeling some other meaning is intended. -- RossBoylan

**it is broken as people were able to actually create a fake certificate that could sign anything and was trusted, they did this by finding a collision, they created a certificate that had the same md5 sum as the certificate they were issued, and where thereby able to give themselves right other than they were granted.--Scientes

***apt has supported sha256 checksums since version 0.7.7, so these will be used in lenny and future releases. --JoeyHess

The idea is that generating a checksum from a file is easy, but recreating a file from a checksum (or making another file generate the same checksum) is very hard (ideally, it would be impossible). "md5sum is being broken" means that that "very hard" path becomes quite possible. In other words, it is becoming (or maybe even has become) feasible to create a "rogue" Debian package that still generates the same checksum as the original true package. --JohnZaitseff

I'd say "theoretically possible in certian cases which may or may not include the Debian Packages files", not "quite possible" --JoeyHess

Does Secure APT cover the possibility that the archive machine itself is broken into? What is there to stop someone "inserting" a rogue version of a Debian package and simply regenerating the Packages and Release/Release.gpg files? I strongly suspect this has been considered, but this document does not mention it. Perhaps a link to appropriate documentation, if such exists? --JohnZaitseff

If someone has root access to your mchine they allready have everything, they do not need to break apt. Detecting if your machine has been broken into is another topic and something extremely difficult, although there are certainly some things debian could do better.-Scientes

Yes, if this happens we can revoke the archive key, and introduce a new key for the new install of ftp-master and rollback of the archive to its last known good state that we'd have to do after such an incident. --JoeyHess

It might be worth linking to http://www.debian.org/doc/manuals/securing-debian-howto/ch7.en.html#s-check-non-debian-releases as it explains briefly how to create the Release and Release.gpg files for repositories. --JohnLamb

I had a problem where apt gave the dreaded "WARNING: The following packages cannot be authenticated" message despite having debian-archive-keyring installed and having run "apt-get update". It turned out that my sources list contained {{{deb http://security.debian.org/ stable updates/main}}} instead of {{{deb http://security.debian.org/ stable/updates main}}} Apt found the packages file and could find all the debs, but it was not locating the Releases and Releases.gpg files. --AlexKing
Line 281: Line 229:

This document is copyright 2006 JoeyHess and others under the terms of the
GNU GPL.
This document is copyright 2006 JoeyHess and others under the terms of the GNU GPL.

----
## This page is referenced from http://www.debian.org/releases/etch/i386/release-notes/ch-whats-new.html and /usr/share/doc/debian-archive-keyring/README.gz
CategoryPermalink

All about secure apt

In recent releases, Debian has been using strong crypto to validate downloaded packages. This is commonly called "secure apt" (or "apt-secure") and was implemented in Apt version 0.6 in 2003, which Debian migrated to in 2005. Since the documentation (here and here) is fairly slim on how this all works from an administrator's point of view, this document will try to explain in detail how secure apt works and how to use it.

Basic concepts

Here are a few basic concepts that you'll need to understand for the rest of this document.

A checksum is a method of taking a file and boiling it down to a reasonably short number that uniquely identifies the content of the file. This is a lot harder to do well than it might seem, and the most commonly used type of checksum, MD5, is now a broken hash function, and should be replaced for all security-minded usages.

Public key cryptography is based on pairs of keys, a public key and a private key. The public key is given out to the world; the private key must be kept a secret. Anyone possessing the public key can encrypt a message so that it can only be read by someone possessing the private key. It's also possible to use a private key to sign a file, not encrypt it. If a private key is used to sign a file, then anyone who has the public key can check that the file was signed by that key. Anyone who doesn't have the private key can't forge such a signature.

These keys are quite long numbers (at least 1024 bits, i.e. 256 or more hex digits), and to make them easier to work with they have a key id, which is a shorter, 8 or 16 digit number that can be used to refer to them.

gpg is the tool used in secure apt to sign files and check their signatures.

apt-key is a program that is used to manage a keyring of gpg keys for secure apt. The keyring is kept in the file /etc/apt/trusted.gpg (not to be confused with the related but not very interesting /etc/apt/trustdb.gpg). apt-key can be used to show the keys in the keyring, and to add or remove a key.

Secure apt groundwork: checksums

A Debian archive contains a Release file, which is updated each time any of the packages in the archive change. Among other things, the Release file contains some checksums of other files in the archive. An excerpt of an example Release file:

MD5Sum:
 6b05b392f792ba5a436d590c129de21f            3453 Packages
 1356479a23edda7a69f24eb8d6f4a14b            1131 Packages.gz
 2a5167881adc9ad1a8864f281b1eb959            1715 Sources
 88de3533bf6e054d1799f8e49b6aed8b             658 Sources.gz

Now if we look inside a Packages file, we'll find more checksums, one for each package listed in it. For example:

Package: uqm
Priority: optional
...
Filename: unstable/uqm_0.4.0-1_i386.deb
Size: 580558
MD5sum: 864ec6157c1eea88acfef44d0f34d219

These two checksums allow apt to verify that it's downloaded a correct copy of the Packages file, with an checksum that matches the one in the Release file. And when it downloads an individual package, it can also check its checksum against the content of the Packages file. If apt fails at either of these steps, it will abort.

None of this is new in secure apt, but it does provide the foundation. Notice that so far there is one file that apt doesn't have a way to check: The Release file. Secure apt is all about making apt verify the Release file before it does anything else with it, and plugging this hole, so that there is a chain of verification from the package that you are going to install all the way back to the provider of the package.

Signed Release files

To plug the hole, secure apt adds a gpg signature for the Release file. This is put in a file named Release.gpg that's shipped alongside the Release file. It looks something like this, although only gpg actually looks at its contents normally:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
iD8DBQBCqKO1nukh8wJbxY8RAsfHAJ9hu8oGNRAl2MSmP5+z2RZb6FJ8kACfWvEx
UBGPVc7jbHHsg78EhMBlV/U=
=x6og
-----END PGP SIGNATURE-----

(Technically speaking, this is an ascii-armored detached gpg signature.)

How apt uses Release.gpg

Secure apt always downloads Release.gpg files when it's downloading Release files, and if it cannot download the Release.gpg, or if the signature is bad, it will complain, and will make note that the Packages files that the Release file points to, and all the packages listed therein, are from an untrusted source. Here's how it looks during an apt-get update:

W: GPG error: http://ftp.us.debian.org testing Release: The following signatures
 couldn't be verified because the public key is not available: NO_PUBKEY 010908312D230C5F

Note that the second half of the long number is the key id of the key that apt doesn't know about, in this case that's 2D230C5F.

If you ignore that warning and try to install a package later, apt will warn again:

WARNING: The following packages cannot be authenticated!
  libglib-perl libgtk2-perl
Install these packages without verification [y/N]?

If you say Y here you have no way to know if the file you're getting is the package you're supposed to install, or if it's something else entirely that a black hat has arranged for you, containing a nasty suprise.

Note that you can disable these checks by running apt with --allow-unauthenticated.

It's also worth noting that newer versions of the Debian installer use the same signed Release file mechanism during their debootstrap of the Debian base system, before apt is available, and that the installer even uses this system to verify pieces of itself that it downloads from the net. Also, Debian does not currently sign the Release files on its CDs; apt can be configured to always trust packages from CDs so this is not a large problem.

How to tell apt what to trust

So the security of the whole system depends on there being a Release.gpg file, which signs a Release file, and of apt checking that signature using gpg. To check the signature, it has to know the public key of the person who signed the file. These keys are kept in apt's own keyring (/etc/apt/trusted.gpg), and managing the keys is where secure apt comes in.

By default, Debian systems come preconfigured with the Debian archive key in the keyring.

joey@dragon:~>sudo apt-key list
/etc/apt/trusted.gpg
--------------------
pub   1024D/4F368D5D 2005-01-31 [expires: 2006-01-31]
uid                  Debian Archive Automatic Signing Key (2005) <ftpmaster@debi
an.org>

Here 4F368D5D is the key id, and notice that this key was only valid for a one year period. Debian occasionally rotates these keys as a last line of defense against some sort of security breach breaking a key.

That will make apt trust the official Debian archive, but if you add some other apt repository to /etc/apt/sources.list, you'll also have to give apt its key if you want apt to trust it. Once you have the key and have verified it, it's a simple matter of "apt-key add file" to add it. Getting the key and verifying it are the trickier part.

How to find a key

The debian-archive-keyring package is used to distribute keys to apt. Upgrades to this package can add (or remove) gpg keys for the main Debian archive.

For other archives, there is not yet a standard location where you can find the key for a given apt repository. There's a rough standard of putting the key up on the web page for the repository or as a file in the repository itself, but no real standard, so you might have to hunt for it.

The Debian archive "signing" key is available at http://ftp-master.debian.org/ziyi_key_2006.asc (replace 2006 with current year). (the tool that is used for signing on the Debian servers is called "ziyi", after Ziyi Zhang.) This key is not the secret signing key as its name might suggest, but the public testing key.

gpg itself has a standard way to distribute keys, using a keyserver that gpg can download a key from and add it to its keyring. For example:

joey@dragon:~>gpg --keyserver pgpkeys.mit.edu --recv-key 2D230C5F
gpg: requesting key 2D230C5F from hkp server pgpkeys.mit.edu
gpg: key 2D230C5F: public key "Debian Archive Automatic Signing Key (2006) <ftpm
aster@debian.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1

You can then export that key from your own keyring and feed it to apt-key:

joey@dragon:~>gpg -a --export 2D230C5F | sudo apt-key add -
gpg: no ultimately trusted keys found
OK

(What does the "gpg: no ultimately trusted keys found" warning mean? --> The Warning: "no ultimately trusted keys found" means that gpg was not configured to ultimately trust a specific key. Trust settings are part of OpenPGPs Web-of-Trust which does not apply here. So there is no problem with this warning. In usual setups the users own key is ultimately trusted.)

How to safely add a key

By adding a key to apt's keyring, you're telling apt to trust everything signed by the key, and this lets you know for sure that apt won't install anything not signed by the person who possesses the private key. But if you're sufficiently paranoid, you can see that this just pushes things up a level, now instead of having to worry if a package, or a Release file is valid, you can worry about whether you've actually gotten the right key. Is the http://ftp-master.debian.org/ziyi_key_2006.asc mentioned above really Debian's archive testing key, or is this document a clever trap?

It's good to be paranoid in security, but verifying things from here is harder. gpg has the concept of a chain of trust, which can start at someone you're sure of, who signs someone's key, who signs some other key, etc., until you get to the archive key. If you're sufficiently paranoid you'll want to check that your archive key is signed by a key that you can trust, with a trust chain that goes back to someone you know personally. If you want to do this, visit a Debian conference or perhaps a local LUG for a key signing.

(Note: Not all apt repository keys are signed at all by another key. Maybe the person setting up the repository doesn't have another key, or maybe they don't feel comfortable signing such a role key with their main key.)

If you can't afford this level of paranoia, do whatever feels appropriate to you when adding a new apt source and a new key. Maybe you'll want to mail the person providing the key and verify it, or maybe you're willing to take your chances with downloading it and assuming you got the real thing. The important thing is that by reducing the problem to what archive keys to trust, secure apt lets you be as careful and secure as it suits you to be.

Here's a blog post with a procedure to verify the key's integrity.

Debian archive key expiry

Since secure apt was introduced, the keys used to sign the main Debian archive have changed a couple of times. Since secure apt is young, we don't have a great deal of experience with changing the key and there are still rough spots.

In January 2006, a new key for 2006 was made and the Release file began to be signed by it, but to try to avoid breaking systems that had the old 2005 key, the Release file was signed by that as well. The intent was that apt would accept one signature or the other depending on the key it had, but apt turned out to be buggy and refused to trust the file unless it had both keys and was able to check both signatures. This was fixed in apt version 0.6.43.1. There was also confusion about how the key was distributed to users who already had systems using secure apt; initially it was uploaded to the web site with no announcement and no real way to verify it and users were forced to download it by hand. This was fixed by the introduction of the debian-archive-keyring package, which manages apt keyring updates.

In late 2006, a new key was created that will be used to sign the archive for the lifetime of the Debian 4.0 release (until 2009-07-01). The archive began to be signed by this new key in addition to the yearly signing key for 2006. That was a bit confusing, because the key began to be used before it was announced and before debian-archive-keyring was updated to include it! Apt's warning message in this situation is slightly opaque to end users. There's obviously still room for improvement in how we roll out new keys. This new key does answer the question of how users of the 4.0 (etch) release will be able to validate their software for the lifetime of that release. This new key is also being used to sign other versions of debian (like unstable).

On February 7th 2007, the 2006 key expired. Currently the only known breakage of this is that it broke rc1 of the etch installer, since the installer images only know about the 2006 key. Daily builds of the installer have the 2007 key and continue to work.

Most recently, a new Etch stable release key has been added. This key is an offline key that will be used to sign releases of Etch (including point releases).

Other problems

  • One not so obvious gotcha is that if your clock is very far off, secure apt will not work. If it's set to a date in the past, such as 1999, apt will fail with an unhelpful message such as this:
      W: GPG error: http://archive.progeny.com sid Release: Unknown error executing gpg
    Although apt-key list will make the problem plain:
    gpg: key 2D230C5F was created 192324901 seconds in the future (time warp or clock problem)
    gpg: key 2D230C5F was created 192324901 seconds in the future (time warp or clock problem)
    pub   1024D/2D230C5F 2006-01-03
    uid                  Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
    If it's set to a date too far in the future, apt will treat the keys as expired.
  • Another problem you may encounter if using testing or unstable is that if you have not run apt-get update lately and apt-get install a package, apt might complain that it cannot be authenticated (why does it do this?). apt-get update will fix this.
  • If apt gives a warning like this:
    W: There are no public key available for the following key IDs:
    A70DAF536070D3A1
    • This means that the archive has begun to be signed by a new key, which your system does not know about. In this example, the new key is a dedicated key that will be used to sign the release of Debian 4.0. Since the archive was still signed by another key that apt knows about, this is just a warning, and once the system is fed the new key (by upgrading the debian-archive-keyring package), the warning will go away.
  • If you have the debsig-verify package installed, you might run into errors like this one:
    dpkg: error
    processing /var/cache/apt/archives/anjuta-common_1.2.4a-2_all.deb (--unpack):
     Verification on package /var/cache/apt/archives/anjuta-common_1.2.4a-2_all.deb failed!
    Authenticating /var/cache/apt/archives/anjuta_1.2.4a-2_i386.deb ...
    debsig: Origin Signature check failed. This deb might not be signed.
    This actually has nothing to do with secure apt. debsig-verify checks for signatures embedded inside individual Debian packages. Since such signatures are not widely used (we use secure apt instead), it doesn't work very well to install this, and removing the debsig-verify package will fix the problem.
  • If apt-get update outputs this
    W: GPG error: http://non-us.debian.org stable/non-US Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F1D53D8C4F368D5D
    W: You may want to run apt-get update to correct these problems
    remove non-us from /etc/apt/sources.

Setting up a secure apt repository

From man apt-secure

If you want to provide archive signatures in an archive under your maintenance you have to:

  • Create a toplevel Release file. if it does not exist already. You can do this by running apt-ftparchive release (provided inftp apt-utils).
  • Sign it. You can do this by running gpg -abs -o Release.gpg Release.
  • Publish the key fingerprint, that way your users will know what key they need to import in order to authenticate the files in the archive.

Whenever the contents of the archive changes (new packages are added or removed) the archive maintainer has to follow the first two steps previously outlined.

History

Conectiva implemented something similar in their fork of APT. Debian Developers Colin Walters and Isaac Jones implemented APT Secure for Debian in 2003. Around Christmas 2003, Matt Zimmerman(?) integrated this patch into APT 0.6. In February 2005, Debian started migrating to apt-secure.

Comments and questions

(Add any here.)

Debian isn't Ubuntu and won't have sudo installed by default. It might be worth changing the usage of sudo, and making the examples use the root account explicitly -- SteveKemp

Given the failure modes I've seen from gpg --recv-keys, suggesting that a user run it as root doesn't seem wise to me. But I've never actually audited it either.. Despite sudo not being installed by default (in sarge), I think that most of the audience of this page are familiar with it, or can skip over it. -- JoeyHess

Does it make any sense to pre-install debian-server-keyring on every debian system? At least the user should be asked if s/he trusts this key. Until now (2006-01-07) the Debian Archive Automatic Signing Key is not in the strong set of keys. Thus is it not possible to test via pathfinders if there is a trust path from my key to the Debian Archive Automatic Signing Key. (See http://pgp.cs.uu.nl/ or http://www.lysator.liu.se/~jc/wotsap/search.html)

Yes, the debian-archive-keyring package will be our key upgrade path for all Debian systems, so it should be installed on all of them, and I assume will be in standard. -- JoeyHess

Can we somehow integrate this idea into this page? I think it's important we also move in this direction. -- madduck

I tried making a local repository using apt-move, serving machines running testing and stable. I could not use the same repository for both stable and testing due to incopatibilities between apt v5 and v6. You might highlight this when you get around to writing the bit about crating a repository. I could get v6 working for testing but not for sarge. --?MartinHodges

What does it mean for md5sum to be broken? Since it's a checksum, I thought the only way it can be broken is that it fail to compute the proper checksum. I have a feeling some other meaning is intended. -- ?RossBoylan

**it is broken as people were able to actually create a fake certificate that could sign anything and was trusted, they did this by finding a collision, they created a certificate that had the same md5 sum as the certificate they were issued, and where thereby able to give themselves right other than they were granted.--Scientes

***apt has supported sha256 checksums since version 0.7.7, so these will be used in lenny and future releases. --JoeyHess

The idea is that generating a checksum from a file is easy, but recreating a file from a checksum (or making another file generate the same checksum) is very hard (ideally, it would be impossible). "md5sum is being broken" means that that "very hard" path becomes quite possible. In other words, it is becoming (or maybe even has become) feasible to create a "rogue" Debian package that still generates the same checksum as the original true package. --JohnZaitseff

I'd say "theoretically possible in certian cases which may or may not include the Debian Packages files", not "quite possible" --JoeyHess

Does Secure APT cover the possibility that the archive machine itself is broken into? What is there to stop someone "inserting" a rogue version of a Debian package and simply regenerating the Packages and Release/Release.gpg files? I strongly suspect this has been considered, but this document does not mention it. Perhaps a link to appropriate documentation, if such exists? --JohnZaitseff

If someone has root access to your mchine they allready have everything, they do not need to break apt. Detecting if your machine has been broken into is another topic and something extremely difficult, although there are certainly some things debian could do better.-Scientes

Yes, if this happens we can revoke the archive key, and introduce a new key for the new install of ftp-master and rollback of the archive to its last known good state that we'd have to do after such an incident. --JoeyHess

It might be worth linking to http://www.debian.org/doc/manuals/securing-debian-howto/ch7.en.html#s-check-non-debian-releases as it explains briefly how to create the Release and Release.gpg files for repositories. --?JohnLamb

I had a problem where apt gave the dreaded "WARNING: The following packages cannot be authenticated" message despite having debian-archive-keyring installed and having run "apt-get update". It turned out that my sources list contained deb http://security.debian.org/ stable updates/main instead of deb http://security.debian.org/ stable/updates main Apt found the packages file and could find all the debs, but it was not locating the Releases and Releases.gpg files. --AlexKing

This document is copyright 2006 JoeyHess and others under the terms of the GNU GPL.


CategoryPermalink