Differences between revisions 56 and 57
Revision 56 as of 2018-08-05 09:06:39
Size: 24965
Editor: BenHutchings
Comment: Add references to 2017 and 2018 events
Revision 57 as of 2018-08-29 19:12:34
Size: 24965
Editor: BenHutchings
Comment: Fix JSON syntax errors
Deletions are marked like this. Additions are marked like this.
Line 147: Line 147:
    "trusted_certs": ["079646974bce09b1f04da67bd722d1fb0947ae4c4010bccdbba52d5b23cbf1a2"],
    "trusted_certs": ["079646974bce09b1f04da67bd722d1fb0947ae4c4010bccdbba52d5b23cbf1a2"]

Requirements for signing

  • Blockers
    • Persistent log of all signatures (package, version, timestamp, hash of binary, signature)
    • All signatures must be traceable to an upload, or otherwise explained (e.g. failed upload)
    • dak upload processing must not wait very long for signing - signing probably needs to be asynchronous
    • Must be able to sign embargoed packages (that is, signing before publishing)
  • Nice-to-have
    • Signed packages are reproducible (e.g. through detached signatures in "source")
    • Fully automatic signing after unsigned binaries accepted by dak
    • Don't make excessively weird binary packages

Agreed design

  1. Package uploaded to whatever queue
  2. process-upload matches it to a configured list of packages that need signing, and triggers signing service
  3. Signing service unpacks binary and generates detached signatures
    • The list of things to be signed is part of the source package template (see below)
  4. Signing service updates and uploads a source package with the detached signatures
    • The source package is generated from a template in the binary package. This is slightly weird, but it can't get out of sync and makes reproducibility easy. It allows binNMUs to work without adding too much complexity to the signing service.

Template organization

  • The original source package will build a "binary" template package per architecture (linux-image-amd64-signed-template; arch:amd64)
  • The template package will contain the list of binary packages and the files for each of those that needs to be signed
  • The signed binary packages needs to use Built-Using to declare what source packages were used to build the signed packages, e.g. for linux:
    • debian/control:
      • source section:
        • Build-Depends: linux-image-1.2.3-amd64-unsigned (= 1.2.3-1)
      • binary sections:
        • Built-Using: linux (= 1.2.3-1)

The signed source package will be a native package, so any "-" in the binary version must be replaced by another character (such as "+" or ".") in the template changelog. For binNMUs, the "+bX" at the end of the binary version must be mapped to ".bX" in the template changelog. The versions of signed binary package may be overridden to undo these changes.

dak (config) changes

  • Add keyring with signing service's source upload key
  • Permissons for this key - can only upload specific source packages (which may be NEW)
  • Add post-accept trigger mechanism
  • Configure post-accept trigger for binary template packages that need signing
    • Update mapping of packages to be signed - (archive, suite, source or binary (template package), architecture) -> version

      • Possibly via a file in the buildd archive?
  • Expose mapping of packages to be signed, to be polled by signing service
  • In the common case (not embargoed), dak will published the unsigned version of the packages without waiting for the signed version
  • Package can be rejected from the embargo queue and re-use the same version number, which was already signed - the new package then won't be signed again!
    • So maintainers for these packages must not re-use version numbers
    • we should annotate the audit log that those binaries never made it to public anyway

The mappings of packages to be signed are being published at https://incoming.debian.org/debian-buildd/project/external-signatures/requests.json and https://security-master.debian.org/debian-security-buildd/project/external-signatures/requests.json (restricted access).

signing box requirements

  • API (input):
    • Mapping of packages to be signed, generated by dak

#  (archive, suite, source or binary (template package), architecture) 
  "packages": [
      "suite": "unstable", 
      "version": "0.1-1", 
      "architecture": "all", 
      "package": "linux-image-m68k-signed-template"
      "suite": "unstable", 
      "version": "0.1-1", 
      "architecture": "all", 
      "package": "package"
  • Storage:
    • Mapping of latest packages that have been signed
      • Ideally on Postgres server which is backed up
    • audit log: append only to some database table
      • package name, version, archive, suite
      • timestamp
      • hash of binary, signature
      • sequence number
    • audit log questions:
      • should it be local or have sockup mechanism? Or a service in a third host?
      • should it be only writable by the signing box? Or dak would write something there?
      • would we have the reason why a package got rejected?
      • possible status or -signed packages:
        • published
        • rejected
        • waiting
  • Output:
    • Generate new source package for each updated template package, and upload to the archive
  • method of operation
    • Poll dak about outstanding unsigned packages (e.g. every ~1min)
    • Download binary template package, unpack into a tree
    • Extract list of binary packages that needs signing, download those and extract into separate trees
    • Sign binary files and collect the detached signatures into the unpacked template package
    • Build the signed source package, sign this and upload it to the correct archive
      • Worth documenting:
      • It will not build the -signed binary packages

      • The new source will be uploaded to dak in the normal process and then binaries built as normal
      • We don't want to execute debian/rules in the signing service
    • Record that the package has been signed (in own database).
  • State table (by template)
    • Template package name, version
    • archive, suite, architecture
    • timestamp
    • Possible states:
      • signed
      • failed ("we tried to sign this three times and it failed each time" → ignore for signing purposes and add alerting)
      • incomplete
      • submitted
    • Ignore packages that are submitted or failed more then 3 times
    • Filter packages with same (suite, architecture, template name)
  • ISSUE:
    • package from security-master are copied to ftp-master, we need to make sure we don't sign them twice and upload twice

Source template inside a binary package


  • files.json: Contains description of files to be signed, in JSON format. This includes a complete list of the files, not a pattern as initially discussed.

    • Rough schema:

{$pkg_name: {
    "files": [
         {"sig_type": "efi" | "linux-module", "file": "file-name-relative-to-package-root"}, ...
  • source-template: Unpacked source package, complete except for signatures

    • Signatures are added in source-template/debian/signatures/original-binary-package-name/complete-path-name.sig

    • Signing service should check that debian/source/format is exactly "3.0 (native)" and debian/source/options and debian/source/local-options don't exist
    • Signing service should check that files.json doesn't use absolute paths or parent ("..") components

Describing the trust chain

BenHutchings proposes the following addition, to avoid accidentally creating a trust chain from production to test signing certificates.

For every binary package mentioned in files.json, there must be an additional "trusted_certs" key. The value for this key is an array identifying all certificates that are built-in to the signed code as roots of trust for verifying additional privileged code, excluding any certificates for which the private key was generated and discarded during the build process. The certificates are identified by their SHA-256 fingerprint, i.e. the SHA-256 digest of the certificate in DER format, formatted as a hexadecimal string.

For example, for a binary package "foo" that trusts only Debian's current UEFI CA, the following would be added:

{"foo": {
    "trusted_certs": ["079646974bce09b1f04da67bd722d1fb0947ae4c4010bccdbba52d5b23cbf1a2"]

The value for "trusted_certs" may be an empty array (in case there are no such certificates) but the key must still be present.

The signing service must verify that this key is present and that all certificate fingerprints in the array are in the permitted (configurable) set of trusted certificates.

Open questions

  • Can the signature on shim be detached from the binary and combined with it during the build? This would allow reproducible builds.
    • - Yes, sbattach in sbsigntool supports this
  • How to verify that Microsoft hasn't tampered with the shim binary before signing?
    • - Applying 'sbattach --remove' to signed shim should yield the unsigned version we sent
  • How many people and who should hold the private keys for the certificate(s)? DSA? Secure Boot team? How should we backup the private keys?
  • Do we want to use key sharding or a HSM?

  • What is the verification process for the EV cert, which identity is being verified?
  • Sign shim by just MS key or one package for the Debian key, one for the MS key?
  • Do we want a canary like the canary for Fedora/RHEL/shim upstream.

  • We need to figure out what revocation looks like, the story is not great yet
  • What/how to do root key rotation? So far "don't do that", but we should only need to rotate this if the key is believed to be insecure, either through compromise or because crypto.

Secure Boot General Information

The idea is to use the latest shim signed by Microsoft that will enable us to bootstrap into a later boot loader that can be signed by Debian. grub itself will need to be signed by a FTP master, so we will need to sort out key signing. Same for the kernel.

Roughly matches existing deployment in Ubuntu, although hopefully with some further improvements on top.


The shim is an EFI executable that is in a format that is acceptable for signing by microsoft. UEFI has a database of keys that can be used for signing, the shim is to be used by a 3rd party.

It supports its own user-modifiable key database (?MokManager) so an end user can install their own key (requires the user to be physically present). Grub will call into the shim to verify the kernel, so you get a fully signed root of trust. Could theoretically verify initramfs and root filesystem. You can also disable sig validation so shim will launch anything you give it (again requiring physical presence)... from then on it will boot any copy of grub/kernel, without disabling secure boot entirely... allows for kernel and grub development without having to jump through a lot of hoops.

The aim is the least worst that still respects user's freedoms. Local key management was implemented by SUSE, the rest by redhat.

Potential improvements: fall-back bootloader so if a system loses all boot entries, the shim will re-enroll and register them and boot normally.


Grub has many patches from redhat/mjg causing it to operate in a secure boot way. When grub core is signed it refuses to load any modules that are unsigned. Secure boot core images are larger as a result. That code is in debian, some of it is configured off by default, but that is trivial to change. There is code to build custom uploads. For more information, please see: http://thread.gmane.org/gmane.linux.debian.devel.boot/143954

Involved and related software/packages: shim grub efilinux linux openssl mokutil pesign sbsigntool secureboot-db efitools vboot-kernel-utils

MSFT key requirements

MSFT has a short list of requirements, it boils down to these critical points:

  • EV cert: this requires identify verification
  • code must not be subject to GPLv3, "or any license that purports to give someone the right to demand authorization keys to be able to install modified forms of the code on a device. Code that is subject to such a license that has already been signed might have that signature revoked. For example, GRUB 2 is licensed under GPLv3 and won’t be signed."
  • because of the above, we use a shim (which hands off executation to another bootloader)...this is a model where the vendor cert is embedded in a new place in the executable. Even if you receive a binary from debian, you can still verify the executables are identical. This will allow MSFT to verify the binary is the same, with only the certificate changed.
  • code that hasn't been SecureBoot "enlightened" won't be signed

  • code signing keys must be backed up, stored, and recovered only by personnel in trusted roles, using at least dual-factor authorization in a physically secured environment.
  • The private key must be protected with a hardware cryptography module. This includes but is not limited to HSMs, smart cards, smart card–like USB tokens, and TPMs. The operating environment must achieve a level of security at least equal to FIPS 140-2 Level 2.
  • submitter must design and implement a strong revocation mechanism for everything the shim loads, directly and subsequently.
  • some shims are known to present weaknesses into the SecureBoot system. For a faster signing turnaround, we recommend that you use source code of 0.8 or higher from shim - GitHub branch.

TODO: Ubuntu does a key sharding process, required some kind of approval from Microsoft, find out what this procedure is

Pre-submission testing

Task list

See the tracking bug 820036.

Packages needing signing

  • source: linux
    • binaries: linux-image-{amd64,i386,arm64}-signed-template
    • Implemented in version 4.16-1~exp1
    • deb [trusted=yes] https://people.debian.org/~benh/packages/secure-boot/ unstable main

  • source: grub2
  • source: shim
    • binaries: shim-{amd64,i386,arm64}-signed-template [nice to have]
    • mokmanager and fall-back App are currently signed by an ephemeral key embedded inside shim for only this process. This breaks reproducible builds. Instead we should sign mm$EFI_ACH.efi and fb$EFI_ARCH.efi by our signiningbox, too, or whitelist their hashes in shim.
    • https://salsa.debian.org/pmhahn/shim/tree/master

    • https://salsa.debian.org/pmhahn/shim-signed

    • deb [trusted=yes] https://people.debian.org/~pmhahn/packages/secure-boot/ unstable main

  • source: fwupdate
    • binaries: fwupdate-amd64-signed-template fwupdate-i386-signed-template fwupdate-arm64-signed-template fwupdate-armhf-signed-template
    • deb [trusted=yes] https://www.einval.com/debian/efi/ unstable main

Suites enabled for signing

  • ftp-master: experimental, unstable (initially), testing-proposed-updates, stable-proposed-updates, oldstable-proposed-updates, oldoldstable-proposed-updates, stable-backports, oldstable-backports (later, as needed)
  • security-master: testing, stable, oldstable (later, as needed)


Earlier proposed signing architectures

First option: by-hand script in dak

signing architecture with dak

The main idea is to have a signing-box with access to the signing usb keys.

Basically, the steps can be described as:

  • The maintainer of the package (grub/linux) uploads it to Dak
  • A by-hand script inside dak is called, which will send the binaries to be signed by the signing-box
  • The signing-box (which runs as user 'codesign' in the diagram above) uses the ?YubiKey to sign the binaries inside the tarball

  • The signing-box sends a tarball back to the by-hand script in dak with all the detached signatures
  • The by-hand script sends the detached signatures to a machine accessible by the DDs
  • Another (or the same) DD retrieves the detached signatures and makes a signed version of the package
  • The maintainer uploads the signed package to dak

Ideally dak would upload the -signed version of the package automatically, but this can be done later. Right now, we currently have the ?YubiKeys plugged into the fasolo machine, so the signing-box would also run in fasolo for now. For security and better key management, the signing-box would run as another user that we called codesign in the diagram above.

One of the suggestions is to use a machine called coccia to host the detached signatures for the DDs to retrieve them.

what we have

signing-box code:

dak patch:

what we need

  • To patch dak's code, adding the by-hand script
  • Install the signing-box scripts on fasolo

Second option: use buildd + debhelper instead of dak

The idea is that instead of changing dak, add the signing logic to a helper script and use that in build process on the buildd.

Signing architecture with buildd

Basically, the steps can be described as:

  1. The maintainer uploads the source package to dak (without the binary packages)
  2. dak/wanna-build sends the package to the buildd as usual; the buildd works as normal and start the build process of the package in an isolated environment (chroot)
  3. During the package build, it uses the dh_signcli tool (which doesn't exist yet) to request signatures on various files.
  4. dh_signcli communicates with dh_signd (which also doesn't exist yet) outside of the chroot environment to request signatures. This communication can be done via some method (maybe d-bus, maybe unix socket?), so that dh_signcli can work even if the chroot environment doesn't have network enabled. dh_signd checks if the package is allowed to request signatures.
  5. dh_signd sends a tarball of binaries to the signing-box through ssh
  6. The signing-box uses the ?YubiKey to sign the binaries

  7. The signing-box sends a tarball back to dh_signd with the detached signatures
  8. dh_signd send the detached signatures back to the dh_signcli
  9. The package build continues, using the output of dh_signcli to assemble packages that include signatures.
  10. buildd uploads the build output into the archive as normal.

Handling errors

  • What to do if dh_signd is not running or if signing-box fail or can't be reached? dh_signcli will fail and we fail the whole build?


1. When the package enters in the NEW queue, the binary package sent by the maintainer is not discarded, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798000 Possible solutions:

  • Upload source+all to the NEW queue, but it won't work with other packages as grub which doesn't provide a build for "all", we could add a grub-doc (sounds like a hack)
  • Modify the NEW queue policy and accept source only uploads
  • Modify the NEW queue policy for only some specific packages/maintainers
  • Upload the binary packages without the -signed version of it, but make buildd to rebuild it again and discard the binaries uploaded by the maintainer

Or actually fix the bug.

2. Builds are going to stall waiting on the signing service to do stuff. That could take some time for Linux, for example, with many many signs needed for individual modules.

  • What does "some time" mean? This would be an even greater issue when using byhand as it would block ftp-master from processing (as opposed to blocking a single buildd).
  • Julien Cristau found it took ~20 minutes per kernel flavour to sign modules. So maximum of 1 hour. This should be OK.

3. Signing during build process isn't reproducible.

Perhaps we could record signatures in the buildinfo file, and have a dummy signing service that replays them from there.

4. Signing service may be exploitable by compromised buildd.

what we have

signing-box code:

what we need

  • Install/configure the signing-box scripts on fasolo
  • Develop the dh_sign scripts
  • Install/configure x86_64 buildd machine with the scripts
  • Solve the NEW queue issue described above
  • Adapt grub/kernel to this new process

Wrap-up of the discussions so far

  1. Cons about the byhand approach:
    • 1.1. byhand files for security-master break the security-master -> ftp-master sync.

    • 1.2. The autobyhand scripts don't work for uploads that go to NEW
    • 1.3. byhand files are not that nice; I would rather not add more of them
    • 1.4. byhand files aren't publish in the public archive, so harder to see what was actually signed. Nothing enforces the binaries in the *.deb and the *.tar.gz are related after all. Or that a *.deb is present.
    • 1.5. Not clear how to publish signed binaries for the manual step in preparing uploads for the security archive.
  2. About the Buildd approach:
    • 2.1. The exact same -signed.deb package is not reproducible, but the signature can be easily striped away and we can compare if the generated package is the same. While the byhand approach we would have two source packages, and the source for the -signed package would be shipped with the detached signatures.
    • 2.2. Signing things automatically is dangerous as an attacker can attack the infrastructure and get things signed (even if we have a whitelist to limit our signing process to some packages as grub and the kernel), so we will need to revoke signatures or the key. We could develop a mechanism to easily revoke things.
    • 2.3. Ftp masters don't want verify each package by to say it is safe to sign
    • 2.4. As buildd will access an external signing service, it makes it hard to isolate the builds and restrict the potentially evil things only to this build