Differences between revisions 60 and 61
Revision 60 as of 2008-05-14 00:47:45
Size: 16290
Editor: ?ArcadyGenkin
Comment:
Revision 61 as of 2008-05-14 00:49:03
Size: 16514
Editor: ?KenBloom
Comment:
Deletions are marked like this. Additions are marked like this.
Line 50: Line 50:
 Are you telling me that the only source of entropy is the process' PID, and some uninitialized memory that may even be filled in a predictable manner by the execution trace leading up to use of the buffer? -- KenBloom
Line 183: Line 185:

/!\ As this page is changing rapidly, please be careful not to introduce conflicts; respecting edit locks would be a good idea...


In [http://www.debian.org/security/2008/dsa-1571 Debian Security Advisory 1571] (New openssl packages fix predictable random number generator), the Debian Security Team disclosed a vulnerability in the openssl package that makes many cryptographic keys that are used for authentication (e.g. through SSH) or signing (e.g. web server certificates) potentially vulnerable.

End User Summary

The scope of the problem includes:

  • weak keys for both clients and servers (see section "Identifying Weak Keys below")
  • all key types that were generated using openssl (this includes RSA and DSA keys)
  • compromise of other keys or passwords that were transmitted over an encrypted link that was set up using weak keys. Note that this last point means that passwords transmitted over ssh to a server with a weak dsa server key could be compromised too; see the [http://lists.debian.org/debian-devel-announce/2008/05/msg00003.html Debian project's reaction to this].

The following cryptographic tools are unaffected:

  • LUKS (doesn't use openssl)
  • GNUTLS
  • GnuPG

Identifying Weak Keys

Characteristics of potentially vulnerable keys:

  • generated since 2006-09-17
  • generated with etch, lenny or sid (sarge is not vulnerable)
  • generated using openssl, ssh-keygen, or 'openvpn --keygen' (GnuPG and GNUTLS are not affected)

Additionally, some keys may be compromised in the following situations:

  • key generated with broken openssl = bad
  • key generated with good openssl and used to ssh from a machine with bad ssl = bad
  • key generated with good openssl and used to ssh from a machine with good ssl = good

Applications/protocols known to use these keys:

  • openssh (both server and user keys)
  • OpenVPN
  • DNSSEC
  • key material for X.509
  • encfs
  • Tor

To fix this, first aptitude update && aptitude upgrade to install the new version of the openssl and libssl0.9.8 packages (the vulnerability is fixed in version 0.9.8c-4etch3 for etch and version 0.9.8g-9 for lenny/sid).

If you choose not to use the above aptitude command, note that all of the following packages must be upgraded (they all come from the same source package):

  • openssl
  • libssl0.9.8
  • libssl-dev

Then, regenerate and distribute any potentially vulnerable keys. Instructions for how to regenerate the keys for these applications are below. You can also test to see if keys are vulnerable using the dowkd.pl utility as described below.

How weak?

Is there any information on how weak these keys are? Are we talking brute-forceable in minutes, or days, or millennia? -- AsheeshLaroia

It looks likely that the only remaining source of entropy in the generated keys comes from the PID of the process. This is 16 bits, typically much less effective entropy. So there may, in fact, be just a few thousand possibilities for a specific key size. Looks like 'dowkd.pl' lists about 262,000 entries. -- ?MarshRay

  • Are you telling me that the only source of entropy is the process' PID, and some uninitialized memory that may even be filled in a predictable manner by the execution trace leading up to use of the buffer? -- ?KenBloom

    A perl script is available that tests keys for weakness. This perl script is <10 MiB in size and contains a pre-generated list of keys. This suggests that the key search space is very, very small. -- ["Mentor"]

OpenSSH (Server)

rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server

Note that your users will see a "IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!" warning when they next log on to your ssh server because the key has changed. They will need to edit $HOME/.ssh/known_hosts to remove the offending line before continuing; checking that the key fingerprint is correct, of course (the fingerprint of your new key can be found with ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key). You can remove the key from known_hosts by running "ssh-keygen -R hostname"

Also, note that your existing ssh connections shouldn't be interrupted.

OpenSSH (Client)

You will need to have a list of the openssh keys that you currently have and where they have been copied to. For each key that is vulnerable:

cd ~/.ssh
ssh-keygen -t rsa -f filename
ssh-copy-id -i filename hostname 

Replacing rsa by dsa if you prefer dsa keys and replacing filename and hostname with appropriate values.

Generic PEM Generation

This is just a reminder for those who generate PEM encoded certificates. Your site probably has other policies in place about how to manage keys which you should follow. Additionally, you may need to get the certificates signed again by a 3rd party Certificate Authority rather than by using a self-signed certificate as shown below:

cd /etc/ssl/private
openssl genrsa 1024 > mysite.pem
cd /etc/ssl/certs
openssl req -new -key ../private/mysite.pem -x509 -days 9999 -out mysite.pem

OpenVPN

Backup /etc/openvpn/openvpn.secretkey.* (I placed them in a new subdirectory called vulnerable-DSA-1571-1)

recreate the keys using: openvpn --genkey --secret openvpn.secretkey.system1-system2 

copy the shared secret keys to the other hosts

restart the vpn with /etc/init.d/openvpn force-reload on each host.

BIND9

dovecot

Generate a new PEM as shown in "Generic PEM Generation" above (making sure that it is placed in the correct place according to your dovecot configuration, then restart dovecot:  invoke-rc.d dovecot restart .

postfix

Generate a new PEM as shown in "Generic PEM Generation" above (making sure that it is placed in the correct place according to your postfix configuration, then restart postfix:  invoke-rc.d postfix restart .

cyrus imapd

courier imap/pop3

Follow the "Generic PEM Generation" instructions and add a openssl gendh >> mysite.pem.

Or

rm /etc/courier/imapd.pem
dpkg-reconfigure courier-imap-ssl

and dpkg generate back an imapd.pem file.

Apache2 SSL

According to folks in #debian-security, if you have generated an SSL key (normally the step just prior to generating the CSR, and then sending it off to your SSL certificate provider), then the certificate should be considered vulnerable.

dropbear

rm /etc/dropbear/*_host_key
dpkg-reconfigure dropbear

Cfengine

Tor Onion Router / Hidden Service Keys

Please see the vulnerability announcement at [http://archives.seul.org/or/talk/May-2008/msg00049.html].

If you are an onion router (middleman or exit), please delete your private keys in /var/lib/tor/keys/secret_*.

If you are hidden service provider, please delete your hidden service's private key. This will change your hidden service's host name and may require reconfiguring your servers.

If you are running 0.2.0.x, an upgrade is highly recommended -- 3 of the 6 v3 authority servers have compromised keys; however 0.1.2.x (using v2 authorities) is OK.

Testing keys using dowkd.pl

The security team has prepared a utility for you to test your keys to see if they have a fingerprint that matches a list of known-weak finger prints. Download [http://security.debian.org/project/extra/dowkd/dowkd.pl.gz dowkd.pl.gz ] ([http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc gpg signature]):

cd /tmp
wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc
gpg --keyserver subkeys.pgp.net --recv-keys 02D524BE
gpg --verify dowkd.pl.gz.asc
gunzip dowkd.pl.gz

/!\ Note that dowkd.pl produces many false positives and that the authors suggest that it may also produce false negatives. On that basis, you might decide that it is simply easier to regenerate your keys as described above.

/!\ Can we get a version of this that also checks 4096 bit keys, or a version that can crank out new databases?

Use dowkd.pl in the following ways:

SSH Server Host Key

Key is OK:

perl dowkd.pl host localhost
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9

Key is weak:

perl dowkd.pl host localhost
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
localhost: weak key
localhost: weak key

SSH User Key

To check one user (someuser):

perl dowkd.pl user someuser
/home/someuser/.ssh/authorized_keys:1: warning: unparsable line
/home/someuser/.ssh/authorized_keys:3: warning: unparsable line
/home/someuser/.ssh/authorized_keys:4: weak key
/home/someuser/.ssh/authorized_keys:5: weak key

Notes:

  • The complaints about lines 1 and 3 are because these lines are comment lines (starting with #) in the ~/.ssh/authorized_keys files (which are allowed, see man 8 sshd for the syntax).

  • Line 2 has a non-weak dsa key that should be replaced anyway according to the information provided above.
  • Lines 4 and 5 are rsa keys that are reported as being weak (note that both rsa and dsa keys are affected by this vulnerability)

To check all users:

perl dowkd.pl user
  • The script does not check the ~/.ssh/authorized_keys2 files, but it should. To fix this, add the following two lines to the from_user() subroutine:

$file = "$dir/.ssh/authorized_keys2";
from_ssh_key_file $file if -r $file;

Other keys?

The tool is unable to parse PEM files at present.

While the tool is unable to parse PEM files, there is a way to convert PEM files to SSH-style public keys, which the tool can parse. I've not verified this method completely, it comes from information found at: http://webjob.sourceforge.net/Files/Recipes/openssl-convert-ssl-key-to-ssh-keypair.txt

cp /etc/apache/ssl.key/your.keyfile.key ~/.ssh/id_ssl
ssh-keygen -y -f ~/.ssh/id_ssl  > ~/.ssh/id_ssl.pub
./dowkd.pl file ~/.ssh/id_ssl.pub

I have as yet not determined how accurate this method is, but it seems to work reliably. Someone more familiar with the workings of ssh-keygen can confirm the reliability of this method. Scripting this test should be easy enough.

* If the code for generating the key in your PEM file used OpenSSL's random number generator even slightly differently than ssh-keygen from openssh, it's likely to have keys that are equally-weak but not contained in that blacklist. I wouldn't recommend drawing any conclusions from this method. -- ?MarshRay

  • someone who actually understands this tool and its output, please insert details here!

I don't know what kind of information is being requested, but it seems fairly easy. On first invocation of the tool, it generates a DB file from the "blacklist" contained in the __DATA__ section of the script. If the DB exists, it is opened for reading. Once the DB exists, it uses ssh-keygen to retrieve the fingerprint of the public key presented to it by the executor of the script, and compares the fingerprint that it retrieves to the blacklist. If there is a match, then the script prints 'weak key'.

This is perhaps irrelevant now -- an earlier version of this tool didn't say "weak key" but just printed the key name. It also completely failed to parse correctly formed authorized_key files. Perhaps it has now been sufficiently refined as to now be understandable. This page has been a work in progress throughout the day and getting the bones of the content here so that it could then be added to by others was the priority.

A patch for testing PEM files

I created a patch to parse PEM files. fw told me to look at the Modulus of the RSA key, so I am doing that. Even though I tested more than 20 certificates generated on an affected machine, I did not find a single weak key. This could mean that

  • the list is incomplete
  • I am looking in the wrong place
  • the certificates are not, in fact, affected by this. Note that I did not generate them myself as I do not have any vulnerable hosts left

As I manually verified the results against, I do not think there is an error in my patch. It is worth noting that none of the certs showed any hits when I converted them into SSH format. Please test this and report back to me!

--- 1/dowkd.pl  2008-05-13 16:12:42.000000000 +0200
+++ dowkd.pl    2008-05-14 02:10:07.360602871 +0200
@@ -4,9 +4,16 @@
 #
 # Written by Florian Weimer <fw@deneb.enyo.de>, with blacklist data
 # from Kees Cook, Peter Palfrader and James Strandboge.
+# Modified by Richard Hartmann <richih-debian@net.in.tum.de> to support PEM
+#
+# As we need version numbers so people do not get too confused, here goes:
+#
+# Version 0.2
+#
 #
 # Patches and comments are welcome.
 
+
 use strict;
 use warnings;
 
@@ -20,6 +27,7 @@
   host: examine the specified hosts for weak SSH keys
   user: examine user SSH keys for weakness; examine all users if no
         users are given
+  pem : examine a PEM formatted x509 certificate
   help: show this help screen
 
 OPTIONS is one pf:
@@ -120,8 +128,19 @@
        check_hash $name, $hash;
     } else {
        warn "$name: warning: no suitable blacklist\n";
-    }
-                     }
+       }
+}
+
+sub check_pem_file ($) {
+       my $name = shift;
+       my $data = safe_backtick qw/openssl x509 -noout -text -in/, $name;
+       defined $data or return ();
+       my @rsa_details = $data =~ /Modulus \((\d+) bit\):\n\s{20}(.*)\n\s{20}([0-9a-f]{2})/m;
+       if (@rsa_details == 3) {
+               $rsa_details[1] =~ s/://g;
+               return @rsa_details;
+       }
+}
 
 sub from_ssh_key_file ($) {
     my $name = shift;
@@ -189,6 +208,44 @@
     }
 }
 
+sub from_pem_file ($) {
+       my $name = shift;
+       my $pem;
+       unless (open $pem, '<', $name) {
+               warn "$name:0: open failed: $!\n";
+               return 1;
+       }
+       my $tmp = new File::Temp;
+       my $type;
+       my $done;
+       while (my $line = <$pem>) {
+               if ($line =~ /-----BEGIN CERTIFICATE-----/) {
+                       $type = 'cert';
+                       print $tmp $line;
+               } elsif ($line =~ /-----END CERTIFICATE-----/) {
+                       print $tmp $line;
+                       $done = 1;
+               } else {
+                       print $tmp $line;
+               }
+               if ($done) {
+                       my $lineno = $.;
+                       $done = 0;
+                       my ($bits, $rsa_head, $rsa_tail) = check_pem_file($tmp);
+                       if ($bits == 1024 || $bits == 2048) {
+                               check_hash("$name:$lineno", check_pem_file($tmp));
+                       } else {
+                               warn "$name:$.: warning: no suitable blacklist\n";
+                       }
+                       return 1;
+               }
+       }
+       if ($done || !$type) {
+               warn "$name:$.: warning: illegal PEM file format\n";
+               return 1;
+       }
+}
+
 sub from_ssh_host (@) {
     my @names = @_;
     my @lines;
@@ -244,13 +301,15 @@
            from_ssh_auth_file $name;
        }
     } elsif ($cmd eq 'host') {
-       from_ssh_host @ARGV;
+               from_ssh_host @ARGV;
     } elsif ($cmd eq 'user') {
        if (@ARGV) {
            from_user $_ for @ARGV;
        } else {
            from_user_all;
        }
+    } elsif ($cmd eq 'pem') {
+               from_pem_file $_ for @ARGV;
     } elsif ($cmd eq 'help') {
        help;
     } else {

Technical Summary

(If you want to add more technical details that an end-user doesn't need to know or isn't likely to understand, please add them here rather than making the above summary impossible for the average user to understand.)

Causes

It is important to understand that this problem was caused by trying to remove valgrind warnings related to the use of uninitialised memory within the openssl libraries. This was done to try to make it easier to debug C applications that use the openssl libraries which is a good thing to do.

A discussion of why this change was made can be found at [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 #363516] and also on the [http://marc.info/?t=114651088900003&r=1&w=2 openssl-dev list] although it appears that a certain [http://www.links.org/?p=327 revisionist attitude] is being held with regards to that conversation.

Acknowledgements

This document was put together by the folks on [:] to help with user support. It would have been nice to have had the chance to prepare this (and the key rollover page) in the 5 days between the commit being made to pkg-openssl and the DSA being released. Particular credit to themill, dondelelcaro, stew, Maulklin, iobound for reading, writing and suggesting bits for this page.