SameKernel (Deterministic Kernel) is software created in Mempo project for Debian that allows to build Kernel in Verifiable (reproducible, deterministic) way. The resulting .deb are identical each time, so you can be sure no backdoor/trojan was addedd - because your friends confirm they also are getting same checksum from this sources.

You can very easily download the .deb of our verifiable-build kernel and #install it, -or- (for advanced) #build and then help by testifying checksum of your resulting .deb.

This page describes two connected topics:

  1. Building deterministic kernels for Debian (.deb), on Debian
  2. building kernel by default in hardened variant (grsecurity) ready for fast and easy installation on Debian

Current status

icon/ver8.png beta-testing, but usable, as of 2014-04-04:

News: https://github.com/mempo/deterministic-kernel/commits/master :) and #mempo

DOWNLOAD release

New versions will appear on our github, and on grsecurity.net web page, if we are not updating given package you can rather easily update yourself following instructions on github README.md how to increase grsecurity version, or use the provided there devel-up* script.

Download (or #download_deepnet), #verify and #install this ready hardened kernel: version mempo-v0.1.38-05-release of kernel 3.2.57 with grsecurity and built in deterministic way:

See also #download_old for older versions.

Deepnet

Mempo team maintaining this site also provides secure darknet download of most files. Using Freenet to make it hard to find downloaders, uploaders of these files, and making it close to impossible to censor/delete the files (DDoS, server break-in etc).

After installing Freenet software you could access this alternative download linked from: freesite USK@jgyPvA.../ icon/Freenet.png (?)

Remember to also check signatures on darknet downloads. While in Freenet mirrors there is no server to be hacked, still the uploader key could be compromised, so better doublecheck with PGP. In if any doubt just #build yourself after checking at least new (our scripts) sources.

Tor onion and i2p bittorrent are also experimented with as additional distribution platform.

Goal

Users should be able to rebuild identical kernel that matches the distributed .deb files, to know that they do not contain a virus added on top of the source code.

Optional Grsecurity

In our SameKernel script, grsecurity is enabled by default;

To disable it, just remove all lines containing the word grsecurity from script's sources.list and then run the #build again. (In future we would like to also provide here the .deb for non-grsecurity kernels if this is requested).

/!\ To fully use Grsecurity kernel you must follow steps in #install!

(Note to editors: please leave this section/anchor even if this topic would be moved)

Install kernel from SameKernel

Progress

Steps of this project:

See general instructions for all programs: ReproducibleBuilds .

We created a script that does this deterministic build in a bit more automated way with added:

This page should be usable for everyone in Debian, and script we're writing will be later easy to run in pure-Debian mode too.

Crossbuild

Depending on your computer and target computer there are options.

Remark: remember that amd64 means both Intel and AMD standard 64 bit CPUs - see amd64.

Build

Build from sources easily - get the software and publish (PGP-signed) cheksums confirmation so that other users can trust the binary *.deb we publish - please consider doing: #create_verification.

If you are building something else then on amd64 for amd64 then see #xbuild.

/!\ Extra tools that are not-standard-wheezy and are installed globally:

/!\ Extra tools that do not need any non-standard global changes:

Prepare dependencies for Build

As root :

apt-get install faketime time git build-essential libncurses5-dev libncursesw5-dev kernel-package md5deep gcc-4.7-plugin-dev g++ make time automake pkg-config flex

Actual and authorized depencies are in README.md

Create user for kernel build

/!\ At this moment, to get the same *.deb kernel files on different machines, build must be ran on the same user!

Create unix user kernelbuild.

Get and build dpkg

We are using Lunar's branch of dpkg: pu/reproducible_builds. We need GNU gettext >= 0.18.2, therefore on wheezy please add (skip this step if you're building .deb files on Jessie!!!):

deb http://''YOURMIRROR''.debian.org/debian wheezy-backports main

to /etc/apt/sources.list and run:

# aptitude update
# aptitude install -t wheezy-backports gettext autopoint

Please use Mempo's script: https://github.com/mempo/mempo-deb/tree/master/pack/dpkg that will fetch, build and install reproducible-dpkg locally (no need to upgrade your system wide dpkg) (it will delete our old directory in home without asking)

cd ~
rm -rf mempo-deb/
git clone https://github.com/mempo/mempo-deb
cd mempo-deb/pack/dpkg

Verify version of git repository with the command (check if this is ?Mempo/#pgp signed correctly)

LANG=C git tag -v `git describe --tags`

Build and install reproducible-dpkg.

./build-and-install-locally.sh

Run kernel compilation

In home directory run (it will delete our old directory in home without asking)

cd ~
rm -rf deterministic-kernel/
git clone https://github.com/mempo/deterministic-kernel.git
cd deterministic-kernel/

Verify version of git repository with the command (check if this is ?Mempo/#pgp signed correctly):

LANG=C git tag -v `git describe --tags`

Build:

bash run.sh

press ENTER to confirm (e.g. the download) and then kernel should build :)

/!\ (Or better execute this by hand and check sha1sum of git version)

How this works

Following fixes are applied:

Verify

First your friend, trusted person, or even you yourself should #build the kernel from sources and publish resulting checksums of deb files.

Then everyone can download the .deb files, verify if the checksum is as published and only then install them (also verify auxiliary files like the fsattr script here).

Then you can be sure that the .deb file at least is matching the source code, without hidden surprises that would be not visible from reviewing the sources.

Verification process

We (Mempo) are looking for various people, known to community as freedom or FOSS supporters, to help this project in multi-verification of the code.

This are multi-signatures - the idea to have multiple parties providing independent signatures for given .deb file(s), so that user could choose to wait for few separate confirmations from various people before installing a package in order to avoid binary malicious or backdoored version even when the 1 person doing compilation&signature gets compromised (it would then require to compromise many or even all verification developers).

It will probably work better with signing individual .deb files, then when signing entire repository. Therefore it was suggested (<pabs>) to possibly check out dpkg-sig (e.g see this article).

One idea is to provide files like:

linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb.verified.by-bob.gpg
linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb.verified.by-alice.gpg
linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb.verified.by-rfree.gpg

This files would be gpg signed text files with expected checksums and some other information, including checksum of the source code that was built.

Doing this is possible today, thought there is no tool yet to download/verify this data automatically.

Help in verification

The work is quite trivial - please do the following to help us and all other users:

software=mempo/deterministic-kernel
sources-url=https://github.com/mempo/deterministic-kernel.git
sources-sha1=..........
sources-tag=..........
sources-sign-by-fpr=..........
my-name-or-pseudonym=..........
my-code-review=.......... (see below)
my-grsecurity-sigchecked=.......... (see below)
my-checksum=.......... ..........
my-checksum=.......... ..........
my-computer-system=..........
my-computer-kernel=..........
my-computer-arch=..........
my-computer-type=..........
my-computer-trust=.......... (see below)
my-tools-sha1-mempo-deb-dpkg=..........
[details]
<human>
--Following "cannary" lines scan be deleted, but it's preferred to have it. Also DELETE THIS comment line please.--
<cannary>I hereby freely say that I am not, and I never was, and I do not suspect to be in foreseeable future, under ANY form of NDA or gag order 
(nor ANY influence) that could in any way affect this verification testimony. (I am in law jurisdiction: <j>.........</j>).</cannary>

I, ...... (known from/related to project:  ......) can confirm, 
that I downloaded the source code with git sha1 sum as in line sources-sha1= above, and I built it 
on my system as described above, and that these sources did in fact produced the files with checksums as listed above in my-checksum= lines, 
and that the said build was done under following conditions:
* value of line my-computer-trust= means security of computer used by me for this: 0-50 is regular, 
  50-100 is hardened, 100-200 is +always airgap, 200-300 is +openhardware computer, best self-build
* value of my-code-review= means how I reviewed the primary source code 
  (the sources that have checksum sources-sha1=) as follows:
  0=no check; 2=glanced over the code; 4=reviewed by comparing to previously
  reviewed versions; 8=reviewed from scratch. 10=did deep review.
  (for all this values, the review applies to files as downloaded from git, with EXCLUSION of the grsecurity*patch file).
  WARNING: all extra code (e.g. downloaded by running this script) should be 
  checksummed on download by primary source code, otherwise the compilation
  outcome is not guaranteed to be meaningful.
* my-tools-sha1... list sha1sums (e.g. git name) of other sources that I build in order to build this code (e.g. the mempo-deb dpkg).
* as in my-grsecurity-sigchecked I did[/not] verified grsecurity*path files as correctly signed by the Spender's pubkey from http://grsecurity.net
</human>

The information to fill in about sources is result of command  LANG=C git tag -v git describe --tags  and also of lsb_release -a

In the line my-tools-sha1-mempo-deb-dpkg= you should paste the git version of the dpkg that you installed as described in #build (use git log in the mempo-deb repository, where you build our version of dpkg).

Please fill it exactly as in following example, because we plan to auto-process this files in future (and paste the <human>...</human> and read before signing)

software=mempo/deterministic-kernel
sources-url=https://github.com/mempo/deterministic-kernel.git
sources-sha1=bdfae2b8144c4cf03722fccdf14e8b571320d7a5
sources-tag=v0.1.30-03-rc
sources-sign-by-fpr=21A5 9D31 7421 F02E C3C3  81F3 4623 E8F7 4595 3F23
my-name-or-pseudonym=John Doe
my-code-review=4 (see below)
my-checksum=linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb 19a34375ff5ed5f8ef4577b8679fd9ac278b6fd13876e0d2f5990d14d89d74d9
my-checksum=linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb 9ab993f142a1fb37d762c0969d73f18e4f6a813b0332d4ba2c18e745fa855f6b
my-computer-system=Debian 7.4
my-computer-kernel=3.2.55-grsec-mempo.good.0.1.28
my-computer-arch=amd64
my-computer-type=intel
my-computer-trust=100
my-tools-dpkg-from-sources=3b754a0b57fb235d32648abef8ec176f996d3920
[details]
<human>
<cannary>I can say that during writing, signing, publishing this testimony I am NOT under ANY form of gag order (nor ANY pressure nor ANY influence) that could in any way 
affect this code verification testimony. (I am in law jurisdiction: <j>Poland, dependency of EU</j>).</cannary>

I, John Doe (e.g. related to John Doe Inc.) can confirm, 
[... snip ...]
</human>

This text should be saved in file named like: thecreated.deb.verified.by-bob and then gpg signed to create thecreated.deb.verified.by-bob.gpg

Trust chain

For Mempo-Kernel: obtain the *.deb from mempo repository, then check checksum of it with trustworthy people who did build *.deb from source and check if it produces same binary as in repository.

So the full chain trust is:

But how do you verify is security@mempo.org is not compromised nor malicious?

In time, we will upgrade the build script to have the final *.deb file identical, then procedure is even faster (just run run.sh and publish sha512 of your *.deb).

Mempo project might release the *.deb in own repository for easy installation for people that want this.

Ultimately, Debian.org might one day officially include Mempo-kernel in Debian repository (though, even then checksum based verification will be more secure, in case of ftp master key being compromised).

FAQ

FAQ posts solutions to common questions and problems:

Bugs

bug1 [low] [fixed] gzip problem while building from source

#bug1 Error unknown option '-' to gzip:: error

Thanks to benbrown we managed to locate problem. Output of strace:

execve("/bin/gzip", ["gzip", "-n", "-c-1"], [/* 15 vars */]) = 0

This bug is caused by passing -c%d instead -c9 to gzip compress in dpkg, since we fixed it only for xz compression. Problem appears on machines without installed zlib1g-dev when compiling dpkg.

Solutions:

bug2 [very-low] [fixed] Headers checksum not matching since fixdep has other gcc name

#bug2 (fixed before 2014-02-29) not matching checksum of the deb with headers, just the gcc name differs

Solution: you must have identical version of tools that have their version embed in binaries during compilation - especially gcc and libc6. Now script will partially try to detect/warn you. Simply run system update before building. If you try to build older kernel it might be harder, you would need correct versions of libs - or instead compare by hand as explained in next paragraphs.

Impact: just makes it harder to produce-verification (but despite that you can just trust that other people your trust said source==kernel)

Info: the headers deb is sometimes different, between systems; It turn out that some of binary programs there like fixdep have embbed other exact name/version of gcc compiler.

News: seems to be result of differences in libc library package, will add check for it in run.sh

Workaround: update fully your system before building (and in retrospection - use identical version that developers used). Workaround: unpack your deb and the orginal deb and inspect diff, only the binaries should differ, and with command you can check the problem:

<vyrly@oftc> Can check GCC version easly using command:  readelf -p .comment fixdep
<tefnoot@oftc> String dump of section '.comment':
<tefnoot@oftc>   [     0]  GCC: (Debian 4.7.2-5) 4.7.2
<tefnoot@oftc>   [    1c]  GCC: (Debian 4.4.7-2) 4.4.7

also you can hexdump the both binaries and diff them, it should be like:

<rfree> -000022f0  20 28 44 65 62 69 61 6e  20 34 2e 34 2e 37 2d 33  | (Debian 4.4.7-3|
<rfree> +000022f0  20 28 44 65 62 69 61 6e  20 34 2e 34 2e 37 2d 32  | (Debian 4.4.7-2|

but also this line, it's probably same info about version but in binary (int?)

-00000240  14 00 00 00 03 00 00 00  47 4e 55 00 9c 99 41 b2  |........GNU...A.|
-00000250  36 53 75 44 d1 54 57 84  cd 01 f0 2b 84 17 a2 27  |6SuD.TW....+...'|
+00000240  14 00 00 00 03 00 00 00  47 4e 55 00 ad 80 0e 61  |........GNU....a|
+00000250  2c ee 29 52 92 10 8f f0  09 90 e3 e3 42 2d 4b 36  |,.)R........B-K6|

Though above is not 100% guaranteed the difference is not a trojan. Perhaps further assembler analysis of that section can prove it (readelf)

Research: this happens even if dpkg says gcc, gcc-4.7 are same. Some other packages are involved.

More details:

bug3 [very-low] [fixed] not setting concurency

#bug3 was reported by ?SeekingFor@irc2p; was fixed in 47d5cb24945c52122a51f8b4837728498b604cf1 + regression fixed 7fcba6d3cdc048b758ff3c3156721b7a41a96115

<SeekingFo> some small bug @ build.sh script
<SeekingFo> its always using CONCURRENCY_LEVEL == 8
<SeekingFo> reason, its one time CONCURRENCY_LEVEL for export with fixed assignment of 8, and CONCURENCY_LEVEL for the actual cpu core check
<SeekingFo> so the ones at nproc check just misses its second "R"
<SeekingFo> deterministic-kernel/kernel-build/linux-mempo/build.sh that is
[...?...]
<SeekingFo> K1773R, right. the script does that already but uses a wrong variable to store its result

bug4 [very-low] [opened] grsec: eth0 not auto starting

#bug4 was reported by trolly@irc2p

It was reported that with grsecurity kernel (compared to stock debian's?) the eth0 card is not auto starting and needs ifdown/ifup.

This might be juts a case of compatibility issues, and fix could be to manually load the module (modprobe ..., or config files?)

TODO: check in dmesg and /var/log/messages for message related to grsec or to blocked module autoloading.

It could be also that this version of kernel (not grsec) supports this card worse, could be compared with identical SameKernel but without grsec. That is unlikely option.

bug5 [very-low] [wip] missing depends libc6

#bug5 was reported by benbrown; now fixed in 190ae7853be8ede91ab25dadf747f2aa4f2665c9

During one of his builds, (wheezy in chroot in jessie) he got other control file content:

 Installed-Size: 41656
-Depends: libc6 (>= 2.7)
 Recommends: libc6-dev | libc-dev, gcc | c-compiler, make (>= 3.80-10), binutils (>= 2.12), util-linux (>= 2.10o)

#bug6 was reported by rfree When we switched in 190ae7853be8ede91ab25dadf747f2aa4f2665c9 from copying to linking of the dpkg status (ln -s /var ~/.local/var....) then script could fail to delete the old copy (no -r recursive) and would silently make a wrong link in subdir causing dpkg data to not be updated. Only would happened if you would use older versions of this script there first.

When building this old version, after building using even older versions, you should first cleanly delete entire ~/.local/var/ related to dpkg status/info.

This bug probably never affected anyone (this conditions above, plus there would need to be change in dpkg versions of relevant packages libc etc between these 2 versions) but still full disclosure. Is being fixed in version on 2014.03.31.

bug7 [HIGH] [resolved] config without UDEREF or KERNEXEC

#bug7 was reported [hidden], first found/announced by comex, it is not really a bug, and does NOT affect grsecurity in general.

Mempo-related only, Grsecurity-configuration, Security related.

In Mempo's default settings for our kernel we chosen to enable XEN support (as in Debian) at cost of disabling UDEREF / KERNEXEC - since CONFIG_XEN=y is the default for Debian kernels.

In next mempo versions, even in the desktop config-good (the default configuration) will disable XEN and enable UDEREF / KERNEXEC because they are important protections.

We will probably release config-mediumxen as separate kernel variant for people who need it.

(Initially we thought that was overlooked somehow by Mempo team, but then it was clarified that XEN setting was copied, as other options, from Debian default kernel config. We will remove more security sensitive options in future as we test their compatibility impact).

Test

Tests - please test this script and report any problems to Mempo and on IRC #mempo also add here to wiki in /Test :

Downloads Old

Archive of older downloads.

Download, #verify and #install ready hardened kernel: version mempo-v0.1.36-06-release of kernel 3.2.55 with grsecurity and built in deterministic way:

/!\ be aware of #bug7 that affects all versions below.

is in progress. reported checksums below.

<tefnut@freenode> c701c085a4245b76092354dc974eadc43e3f7f6d057c6b8c3c5d7323842ef797 kernel-build/linux-mempo/linux-headers-3.2.55-grsec-mempo.good.0.1.35_02_amd64.deb

<tefnut@freenode> d836976f4a1e413d506cd5c31e9cdcf6ecb7a76373ec25677409a31390e8ae3a kernel-build/linux-mempo/linux-image-3.2.55-grsec-mempo.good.0.1.35_02_amd64.deb

This was never released. The checksums can differ because the used concurrency level during compilation was later saved into .deb.

Download, #verify and #install ready hardened kernel: version mempo-v0.1.31-01-rc of kernel 3.2.55 with grsecurity and built in deterministic way:

Download, #verify and #install ready hardened kernel: version mempo-v0.1.28-rc2 of kernel 3.2.55 with grsecurity and built in deterministic way:

Download, #verify and #install ready hardened kernel: version mempo-v0.1.25-rc2 of kernel 3.2.54 with grsecurity and built in deterministic way:

CategoryKernel