/!\ As this page is changing rapidly, please be careful not to introduce conflicts; respecting edit locks would be a good idea and please do not use the GUI editor, as it introduces huge numbers of whitespace changes which make it harder to keep track of what is changing on this page (Changing one line with the GUI editor can make a 17KB diff as it arbitrarily reformats the text). Also don't check the "remove trailing whitespace" checkbox as it also introduces many changes.


In [http://www.debian.org/security/2008/dsa-1571 Debian Security Advisory 1571] (New openssl packages fix predictable random number generator), the Debian Security Team disclosed a vulnerability in the openssl package that makes many cryptographic keys that are used for authentication (e.g. through SSH) or signing (e.g. web server certificates) potentially vulnerable.

End User Summary

The scope of the problem includes:

The following cryptographic tools are unaffected:

Identifying Weak Keys

Characteristics of potentially vulnerable keys:

Additionally, simply the use of DSA keys may have compromised them. A strong key (i.e., generated with a 'good' OpenSSL) but used locally on a machine with a 'bad' OpenSSL must be considered to be compromised. This is due to an 'attack' on DSA that allows the secret key to be found if the nonce used in the signature is reused or known.

If you have used the vulnerable openssl version to generate an SSL key and then created a CSR from that key and sent it to a SSL certificate provider, then the certificate should be considered vulnerable.

A full list of 'weak' keys has been generated by the metasploit project: http://metasploit.com/users/hdm/tools/debian-openssl/

Applications/protocols known to use these keys:

To fix this, first aptitude update && aptitude upgrade to install the new version of the openssl and libssl0.9.8 packages (the vulnerability is fixed in version 0.9.8c-4etch3 for etch and version 0.9.8g-9 for lenny/sid). You probably want to also pick up the new openssh packages that include the blacklist of known weak keys, but you will need to aptitude dist-upgrade for that in order to install the new openssh-blacklist package.

If you choose not to use the above aptitude command, note that all of the following packages must be upgraded (they all come from the same source package):

Then, regenerate and distribute any potentially vulnerable keys. Instructions for how to regenerate the keys for these applications are below. You can also test to see if keys are vulnerable using the dowkd.pl utility as described below.

How weak?

Is there any information on how weak these keys are? Are we talking brute-forceable in minutes, or days, or millennia? -- AsheeshLaroia

It looks likely that the only remaining source of entropy in the generated keys comes from the PID of the process. This is 16 bits, typically much less effective entropy. So there may, in fact, be just a few thousand possibilities for a specific key size. Looks like 'dowkd.pl' lists about 262,000 entries. -- ?MarshRay

Application Details

BIND9

Cfengine

For each cfengine host, remove the old keys and generate new keys:

rm /var/lib/cfengine2/ppkeys/localhost.priv
rm /var/lib/cfengine2/ppkeys/localhost.pub
/usr/sbin/cfkey
Then restart cfservd:
/etc/init.d/cfengine2 restart

Once the keys are regenerated, exchange keys between hosts as necessary to reestablish 2-way trusts.

courier imap/pop3

Follow the "Generic PEM Generation" instructions and add a openssl gendh >> mysite.pem.

Or

rm /etc/courier/imapd.pem
dpkg-reconfigure courier-imap-ssl

and let dpkg generate back an imapd.pem file.

csync2

As described in /usr/share/doc/csync2/README.Debian

/etc/csync2_ssl*
openssl genrsa -out /etc/csync2_ssl_key.pem 1024
openssl req -new -key /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.csr
openssl x509 -req -days 600 -in /etc/csync2_ssl_cert.csr \
        -signkey /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.pem

Then the keys must be distributed out to each host prior to running csync2 again.

cyrus imapd

To find out which certificates/keys are in use, see the directives whose names contain "key_file" or "cert_file" in /etc/imapd.conf.

Generate new private keys and certificates as shown in "Generic PEM Generation" above, then restart the service:

invoke-rc.d cyrus21 restart

if you are using version 2.1 (if you are using another version, please examine /etc/init.d for the correct service name)

dovecot

Generate a new PEM as shown in "Generic PEM Generation" above (making sure that it is placed in the correct place according to your dovecot configuration, then restart dovecot:  invoke-rc.d dovecot restart .

dropbear

If you have /etc/ssh/*host* keys, either remove them, or follow the openssh instructions above, first, before updating dropbear's keys. Dropbear's postinst imports any existing openssh keys if they exist.

rm /etc/dropbear/*_host_key
dpkg-reconfigure dropbear

Note that keys that have been generated by Dropbear itself are not vulnerable (it uses libtomcrypt rather than OpenSSL).

exim4

If TLS is in use, generate a new PEM using /usr/share/doc/exim4-base/examples/exim-gencert --force to get a self-signed certificate. Per default it has a three years expiration duration.

Generic PEM Generation

This is just a reminder for those who generate PEM encoded certificates. Your site probably has other policies in place about how to manage keys which you should follow. Additionally, you may need to get the certificates signed again by a 3rd party Certificate Authority rather than by using a self-signed certificate as shown below:

cd /etc/ssl/private
openssl genrsa 1024 > mysite.pem
cd /etc/ssl/certs
openssl req -new -key ../private/mysite.pem -x509 -days 9999 -out mysite.pem

OpenSSH (Server)

See also [http://www.debian.org/security/key-rollover/#openssh Official Key-Rollover page].

Updated packages for openssh that have a blacklist of known weak keys are now available See [http://www.debian.org/security/2008/dsa-1576 DSA 1576] for more information. Installing these packages on hosts with weak keys will cause the ssh server to regenerate its keys. Weak user keys being used for a connection will also be rejected where possible.

/!\ Note that you will have to use aptitude dist-upgrade (or apt-get dist-upgrade) to install these packages rather than just upgrade because this update will cause the new package openssh-blacklist to be installed.

You can also update your openssh-server keys manually.

rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server

/!\ Note that in either case, your users will see a "IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!" warning when they next log on to your ssh server because the key has changed. They will need to edit $HOME/.ssh/known_hosts to remove the offending line before continuing; checking that the key fingerprint is correct, of course (the fingerprint of your new key can be found with ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key). You can remove the key from known_hosts by running "ssh-keygen -R hostname"

Also, note that your existing ssh connections shouldn't be interrupted.

OpenSSH (Client)

See also [http://www.debian.org/security/key-rollover/#openssh Official Key-Rollover page].

You will need to have a list of the openssh keys that you currently have and where they have been copied to. For each key that is vulnerable:

cd ~/.ssh
ssh-keygen -t rsa -f filename
ssh-copy-id -i filename hostname 

Replacing rsa by dsa if you prefer dsa keys and replacing filename and hostname with appropriate values.

Also remember to remove compromized keys from you .ssh/authorized_keys file!

OpenSWAN / StrongSWAN

See [http://www.debian.org/security/key-rollover/#openswan Official Key-Rollover page].

OpenVPN

Backup /etc/openvpn/openvpn.secretkey.* (I placed them in a new subdirectory called vulnerable-DSA-1571-1)

recreate the keys using: openvpn --genkey --secret openvpn.secretkey.system1-system2 

copy the shared secret keys to the other hosts

restart the vpn with /etc/init.d/openvpn force-reload on each host.

postfix

Generate a new PEM as shown in "Generic PEM Generation" above (making sure that it is placed in the correct place according to your postfix configuration, then restart postfix:  invoke-rc.d postfix restart .

puppet

  1. You need to wipe and regenerate your CA info:
    /etc/init.d/puppetmaster stop
    rm $vardir/ssl/*
    /etc/init.d/puppetmaster start 
    However, if you are running mongrel, instead of starting puppetmaster from the initscript, you will need to do:
    puppetmasterd --daemonize
    sleep 30
    killall -9 puppetmasterd 
    because for some reason when running with mongrel, puppetmaster will not regenerate its CA.
  2. Wipe all the client certs
    /etc/init.d/puppet stop
    rm $vardir/ssl/* 
  3. On the puppetmaster do:
    puppetca --clean --all
  4. Have each client request a new cert:
    puppetd --onetime --debug --ignorecache --no-daemonize
  5. Once all the requests have rolled in, you can sign them all at once:
    puppetca --sign --all
  6. Start up your puppet clients:
    /etc/init.d/puppet start

You could also enable autosign temporarily, if you are comfortable with that.

The following capistrano recipe will accomplish these steps for you, the top of the recipe includes instructions on how to use it: http://pastie.org/197239

ssl-cert

For the snakeoil certificate /etc/ssl/certs/ssl-cert-snakeoil.pem, just call:

make-ssl-cert generate-default-snakeoil --force-overwrite

tinc

Remove all suspect public and private key files:

  1. Remove rsa_key.priv.
  2. Edit all files in the hosts/ directory and remove the public key blocks.

Generate a new public/private key pair:

tincd -n <netname> -K

Exchange your host config file with the new public key with other members of your VPN. Do not forget to restart your tinc daemons.

Tor Onion Router / Hidden Service Keys

Please see the vulnerability announcement at http://archives.seul.org/or/talk/May-2008/msg00049.html.

Upgrading to 0.1.2.19-3 or 0.2.0.26-rc-1 is recommended. If you run a relay these versions will force new server keys (/var/lib/tor/keys/secret_*.) being generated.

If you are hidden service provider, please delete your hidden service's private key. This will change your hidden service's host name and may require reconfiguring your servers.

If you are running 0.2.0.x, an upgrade is highly recommended -- 3 of the 6 v3 authority servers have compromised keys; however 0.1.2.x (using v2 authorities) is OK.

xrdp

rm /etc/xrdp/rsakeys.ini
/etc/init.d/xrdp restart

SSL Certificate Reissuance

If you paid good money to have a vulnerable key signed by a Certificate Authority (CA), chances are your CA can re-issue a certificate for free, provided all information in the CSR is identical to the original CSR. Create a new key with a non-vulnerable OpenSSL installation, re-create the CSR with the same information as your original (vulnerable) key's CSR, and submit it to your CA according to their reissuance policy

Testing keys using ssh-vulnkey

In the latest openssh-client packages for etch, ssh-vulnkey is included along with a blacklist of known weak keys. This tool appears to be similar to dowkd.pl (below) although it is restricted to checking ssh keys (not that dowkd.pl can do much more than that anyway).

Run ssh-vulnkey as a user to check your host and your own keys.

Run ssh-vulnkey -a as root to check all users keys (as well as the host key).

Sample output:

ssh-vulnkey -a
Not blacklisted: 2048 fa:2e:1d:a6:84:64:a1:80:c4:31:68:5a:b0:1a:cb:fe /etc/ssh/ssh_host_rsa_key.pub
Not blacklisted: 1024 f4:34:04:85:58:a0:6b:0a:a1:b9:2d:3b:e6:19:5a:76 /etc/ssh/ssh_host_dsa_key.pub
COMPROMISED: 2048 5c:10:8a:c0:55:8c:1f:d9:4b:05:f0:35:0a:0d:2f:5c /home/someuser/.ssh/authorized_keys
Not blacklisted: 2048 a7:b4:3e:41:18:cb:f7:68:5e:4f:ae:30:14:d2:17:fd /home/someuser/.ssh/authorized_keys

/!\ Note: about output is one line per key not linewrapped as shown here in the wiki

Testing keys using dowkd.pl

The security team has prepared a utility for you to test your keys to see if they have a fingerprint that matches a list of known-weak finger prints. Download [http://security.debian.org/project/extra/dowkd/dowkd.pl.gz dowkd.pl.gz ] ([http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc gpg signature]):

cd /tmp
wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc
gpg --keyserver subkeys.pgp.net --recv-keys 02D524BE
gpg --verify dowkd.pl.gz.asc
gunzip dowkd.pl.gz

/!\ Note that dowkd.pl produces many false positives and that the authors suggest that it may also produce false negatives. On that basis, you might decide that it is simply easier to regenerate your keys as described above.

Use dowkd.pl in the following ways:

SSH Server Host Key

Key is OK:

perl dowkd.pl host localhost
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9

Key is weak:

perl dowkd.pl host localhost
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
# localhost SSH-2.0-OpenSSH_4.3p2 Debian-9
localhost: weak key
localhost: weak key

A remote check based on the keys generated by HD Moore (http://metasploit.com/users/hdm/tools/debian-openssl/ ) is available at http://itsecurity.net. debian_ssh_scan_v3 now includes fingerprints of all weak DSA 1024, RSA 2048 and RSA 4096 bit keys.

root@box:~/debian_ssh_scan_v2# ./debian_ssh_scan_v2.py 10.128.15.110
65536 fingerprints loaded.
10.128.15.110:22 sshd fingerprint 9cf71acb1b0dff0dceef4f755f721e9d VULNERABLE (RSA 2048 bit key, pid 5252)

SSH User Key

To check one user (someuser):

perl dowkd.pl user someuser
/home/someuser/.ssh/authorized_keys:1: warning: unparsable line
/home/someuser/.ssh/authorized_keys:3: warning: unparsable line
/home/someuser/.ssh/authorized_keys:4: weak key
/home/someuser/.ssh/authorized_keys:5: weak key

Notes:

To check all users:

perl dowkd.pl user

$file = "$dir/.ssh/authorized_keys2";
from_ssh_key_file $file if -r $file;

PEM keys

The dowkd.pl tool is unable to parse PEM files at present.

The Ubuntu team has created a package which will verify if PEM files are vulnerable. I used it against PEM keys created on a known vulnerable system and the utility highlighted them as vulnerable. The package is called "openssl-blacklist". The package is available [http://security.ubuntu.com/ubuntu/pool/main/o/openssl-blacklist/openssl-blacklist_0.1-0ubuntu0.8.04.1_all.deb here]. It can be installed on etch using dpkg -i --force-all.

openssl-blacklist repackaged for Debian: attachment:openssl-blacklist_0.1-0%7Edebian-1_all.deb . Built for etch/stable, but should install cleanly on any later debian versions. I'm not uploading this to Debian proper at present, because I haven't checked whether anyone else is already doing that. For the (signed) sources, see http://xillion.org/openssl-blacklist/ . -- PaulCannon

While the dowkd.pl tool is unable to parse PEM files, there is a way to convert PEM files to SSH-style public keys, which the tool can parse. I've not verified this method completely, it comes from information found at: http://webjob.sourceforge.net/Files/Recipes/openssl-convert-ssl-key-to-ssh-keypair.txt

cp /etc/apache/ssl.key/your.keyfile.key ~/.ssh/id_ssl
ssh-keygen -y -f ~/.ssh/id_ssl  > ~/.ssh/id_ssl.pub
./dowkd.pl file ~/.ssh/id_ssl.pub

I have as yet not determined how accurate this method is, but it seems to work reliably. Someone more familiar with the workings of ssh-keygen can confirm the reliability of this method. Scripting this test should be easy enough.

someone who actually understands this tool and its output, please insert details here!

I created a patch to parse PEM files. fw told me to look at the Modulus of the RSA key, so I am doing that. Even though I tested more than 20 certificates generated on an affected machine, I did not find a single weak key. This could mean that

As I manually verified the results against, I do not think there is an error in my patch. It is worth noting that none of the certs showed any hits when I converted them into SSH format. Please test this and report back to me!

To test this for yourself, do this

for i in $(seq 1 100)
do
    openssl genrsa 1024 > server$i.key 2>/dev/null
    echo -e "\n\n\n\n\n\n\n\n\n\n" | \
            openssl req -new -key server$i.key -days 365 -out server$i.req 2>/dev/null
    openssl x509 -req -days 365 -in server$i.req -signkey server$i.key -out server$i.crt 2>/dev/null
    chmod 600 server$i*
    ssh-keygen -y -f server$i.key > server$i.pub 2>/dev/null
    perl dowkd.pl file server$i.pub
    perl dowkd.pl pem server$i.crt
done

and please post your results.

Here is a small patch of mine checking also authorized_keys2 and known_hosts: attachment:dowkd.pl.ke.patch.gz (also available [http://www.ethgen.de/~klaus/dowkd.pl.ke.patch.gz here]) -- Klaus Ethgen

I have hacked a little shell script using the tables from openssl-blacklist from ubuntu which allows the user to check the ssl key of a webhost from remote by just calling "chksslkey hostname". You can find it here: attachment:chksslkey.tar.bz2 -- Michael Holzt

Technical Summary

(If you want to add more technical details that an end-user doesn't need to know or isn't likely to understand, please add them here rather than making the above summary impossible for the average user to understand.)

Causes

It is important to understand that this problem was caused by trying to remove valgrind warnings related to the use of uninitialised memory within the openssl libraries. This was done to try to make it easier to debug C applications that use the openssl libraries which is a good thing to do.

A discussion of why this change was made can be found at [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 #363516] and also on the [http://marc.info/?t=114651088900003&r=1&w=2 openssl-dev list]. Judging from the [http://www.links.org/?p=327#comment-176642 discussion there], the main culprit seems to be a misunderstanding about which is the right list to ask this question on, followed by misleading answers from the list.

A bit more detail

In an effort to clear up confusion about this bug, here's a bit more technical description. This was caused by an overzealous, well-intentioned elimination of code that was believed to have no impact on security. (Please do note from the above links that this was discussed with the !OpenSSL team and that no objections were raised at the time.)

So here's the problem: the Debian maintainer wanted very much to get rid of valgrind errors while using OpenSSL; certainly a noble cause, right? As you can see here, there are two identical lines,  MD_Update(&m, buf, j);  in SSL's md_rand.c file that were commented out way back in 2006: http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c

The second of these is in  ssleay_rand_bytes  where  buf  is used as an output buffer. It had already been marked as a bad idea when -DPURIFY was in effect, because Purify (and valgrind, naturally) dislike this use of an output buffer as input. This use of  MD_Update  is dubious but shouldn't hurt as long as the mixing function of the PRNG is "sufficiently good". The removal of this call to MD_Update should not meaningfully alter the entropy available in the pool -- that is, OpenSSL does not depend upon uninitialized memory for its correct operation.

The first call, however, is in  ssleay_rand_add  where  buf  is used as an INPUT buffer, to add entropy to the pool. Failing to call  MD_Update  there means that the pool will never actually get the entropy intended for it.

Acknowledgements

This document was put together by the folks on [:] to help with user support. It would have been nice to have had the chance to prepare this (and the key rollover page) in the 5 days between the commit being made to pkg-openssl and the DSA being released. Particular credit to themill, dondelelcaro, stew, Maulklin, iobound for reading, writing and suggesting bits for this page. No thanks to the people who keep using the wiki's GUI editor and keep generating 15 to 20 KB diffs from when it randomly reformats the entire page when you only change one line.

See Also

http://www.debian.org/security/key-rollover/