Differences between revisions 164 and 165
Revision 164 as of 2014-02-27 14:50:36
Size: 18510
Editor: ?Mempo
Comment:
Revision 165 as of 2014-02-27 14:51:56
Size: 18514
Editor: ?Mempo
Comment:
Deletions are marked like this. Additions are marked like this.
Line 35: Line 35:
  * linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:{{{19a34375ff5ed5f8ef4577b8679fd9ac278b6fd13876e0d2f5990d14d89d74d9}}} [[http://p.suchdig.com/dhn-linux-image-3_2_55-grsec-mempo_good_0_1_28_02_amd64_deb.deb|mirror1]] or [[http://127.0.0.1:8888/CHK@T8g41atyoP9tvvC9qASZzcE5a8l7ZvktEfD0WWWabuI,5EdHpCgv8O1XvgPg0lMMAA35feZgOPb8ZGzNuwzG6sE,AAMC--8/linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb| mirror2]] {{attachment:icon/Freenet.png}} [[Freenet#download|(?)]]
  * linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:{{{9ab993f142a1fb37d762c0969d73f18e4f6a813b0332d4ba2c18e745fa855f6b}}} [[http://p.suchdig.com/dhd-linux-headers-3_2_55-grsec-mempo_good_0_1_28_02_amd64_deb.deb|mirror1]] or [[http://127.0.0.1:8888/CHK@G8QJ5hu3RW9agjnO4IPpDcN~oIJrmaLgSMCLZk8iGHs,afuymV10Gsz5R0Bf3MJEsMnplo5573ns9RP~nJdEG-E,AAMC--8/linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb|mirror2]] {{attachment:icon/Freenet.png}} [[Freenet#download|(?)]]
  * linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:{{{19a34375ff5ed5f8ef4577b8679fd9ac278b6fd13876e0d2f5990d14d89d74d9}}} [[http://p.suchdig.com/p/dhn-linux-image-3_2_55-grsec-mempo_good_0_1_28_02_amd64_deb.deb|mirror1]] or [[http://127.0.0.1:8888/CHK@T8g41atyoP9tvvC9qASZzcE5a8l7ZvktEfD0WWWabuI,5EdHpCgv8O1XvgPg0lMMAA35feZgOPb8ZGzNuwzG6sE,AAMC--8/linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb| mirror2]] {{attachment:icon/Freenet.png}} [[Freenet#download|(?)]]
  * linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:{{{9ab993f142a1fb37d762c0969d73f18e4f6a813b0332d4ba2c18e745fa855f6b}}} [[http://p.suchdig.com/p/dhd-linux-headers-3_2_55-grsec-mempo_good_0_1_28_02_amd64_deb.deb|mirror1]] or [[http://127.0.0.1:8888/CHK@G8QJ5hu3RW9agjnO4IPpDcN~oIJrmaLgSMCLZk8iGHs,afuymV10Gsz5R0Bf3MJEsMnplo5573ns9RP~nJdEG-E,AAMC--8/linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb|mirror2]] {{attachment:icon/Freenet.png}} [[Freenet#download|(?)]]

SameKernel - Deterministic Kernel - allows to build Kernel in Verifiable (reproducible, deterministic) way - in order to be sure that .deb indeed matches the source code of kernel without heaving added backdoors/viruses during or after compilation.

Current status

icon/ver8.png beta-testing as of 2014-02-10:

  • This is a beta-version. ?Mempo team will help you on irc://irc.oftc.net/#mempo (or chat-webpage(sometimes allows Tor) or see: ?anonymous) please wait up to 48 hours for our reply!

  • Linux kernel only, testing on amd64 (and i386?)
  • Includes by default grsecurity, but that can be easily removed

News: https://github.com/mempo/deterministic-kernel/commits/master :) and #mempo

Goal

Users should be able to rebuild identical kernel that matches the distributed .deb files, to know that they do not contain a virus added on top of the source code.

Optional Grsecurity

In our SameKernel script, grsecurity is enabled by default;

To disable it, just remove grsecurity line in script's sources.list) and then run the build.

/!\ To fully use Grsecurity kernel you must follow steps in #install!

(Note to editors: please leave this section/anchor even if this topic would be moved)

Releases and Downloads

Version v0.1.28-rc2

  • wheezy, with grsecurity/mempo, variant "?good"

    • linux-image-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:19a34375ff5ed5f8ef4577b8679fd9ac278b6fd13876e0d2f5990d14d89d74d9 mirror1 or mirror2 icon/Freenet.png (?)

    • linux-headers-3.2.55-grsec-mempo.good.0.1.28_02_amd64.deb = sha256:9ab993f142a1fb37d762c0969d73f18e4f6a813b0332d4ba2c18e745fa855f6b mirror1 or mirror2 icon/Freenet.png (?)

Version v0.1.25-rc2

  • wheezy, with grsecurity/mempo, variant "?good"

    • linux-image-3.2.54-grsec-mempo.good.0.1.25_02_amd64.deb = sha256:589a30c6902679d0e9e359ae218a03d1876bcc6eb1049b60c823a45191e9cad9 mirror1 or mirror2 icon/Freenet.png (?)

    • linux-headers-3.2.54-grsec-mempo.good.0.1.25_02_amd64.deb = sha256:2c52b8d6c4a9f0de9371ce16e256cbac781c15191b67cf997e17426b96043d82 mirror1 or mirror2 icon/Freenet.png (?)

  • wheezy, without grsecurity
    • not tested yet (TODO)

Install kernel from SameKernel

  • Get the .deb files either from download or build.

  • Read about known bugs

  • Install them with dpkg -i thefiles.deb

  • /!\ For grsecurity version you must run the setfattr script for grsecurity (and other instructions there - enable user_xattr) or otherwise JIT-using programs like java, python, firefox, icedove etc., will be prohibited from running! (tip: if you do not have internet or browser is not working already, remember the script to fix firefox is located also in your local copy of SameKernel/deterministic-kernel, search for that filename e.g. in directory apps/.)

  • if you are interested in security for more kernel-related and other tools see: ?Mempo#install

Progress

Steps of this project:

  • (./) Step 1 configure kernel build to create identical intermittent files: .o

  • (./) Step 2 configure kernel build to create identical final files: .ko

  • (./) Step 3 identical .gz files (docs?)

  • (./) Step 4 have identical all files and vmlinux including vmlinux .notes

  • (./) Step 5 have identical .deb files on the same machine

  • (./) Step 6 have identical .deb files on different machines :-) (this works, provided username kernelbuild and directory /home/kernelbuild/deterministic-kernel/)

  • (./) Step 6b fix varying xz header/tail between some machines ( /Test#v0.1.23-rc1_buildA )

  • Step 7a remove requirement of having same directory
  • Step 7b remove requirement of having same username
  • Step 8 test if hostname, and timezone are ignored
  • Step 9 any other differences between alpha-testers doing the verification

See general instructions for all programs: ReproducibleBuilds .

We created a script that does this deterministic build in a bit more automated way with added:

  • verification of downloaded sources (check against hardcoded in script list of expected checksums of sources; also check PGP signature with hardcoded public key of kernel developers)
  • apply security patches for the ?Mempo subproject

  • grsecurity patch - http://grsecurity.net/

  • misc patches if needed (e.g. quick fixes regarding security)

This page should be usable for everyone in Debian, and script we're writing will be later easy to run in pure-Debian mode too.

Build

Build from sources easily - get the software and publish (PGP-signed) cheksums confirmation so that other users can trust the binary *.deb we publish.

/!\ Extra tools that are not-standard-wheezy and are installed globally:

  • GNU gettext >= 0.18.2 (install them from official debian-wheezy backports)

/!\ Extra tools that do not need any non-standard global changes:

  • User local installation of dpkg pu/reproducible_builds - instructions below

  • As root from normal repository install faketime (in wheezy) and other common programs

Prepare dependencies for Build

As root :

apt-get install faketime time git build-essential libncurses5-dev libncursesw5-dev kernel-package md5deep gcc-4.7-plugin-dev g++ make time automake pkg-config flex

Create user for kernel build

/!\ At this moment, to get the same *.deb kernel files on different machines, build must be ran on the same user!

Create unix user kernelbuild.

Get and build dpkg

We are using Lunar's branch of dpkg: pu/reproducible_builds. We need GNU gettext >= 0.18.2, therefore on wheezy please add:

deb http://''YOURMIRROR''.debian.org/debian wheezy-backports main

to /etc/apt/sources.list and run:

# aptitude update
# aptitude install -t wheezy-backports gettext autopoint

Please use ?Mempo's script: https://github.com/mempo/mempo-deb/tree/master/pack/dpkg that will fetch, build and install reproducible-dpkg locally (no need to upgrade your system wide dpkg) (it will delete our old directory in home without asking)

cd ~
rm -rf mempo-deb/
git clone https://github.com/mempo/mempo-deb
cd mempo-deb/pack/dpkg

Verify version of git repository with the command:

git tag -v `git describe --tags`

Build and install reproducible-dpkg.

./build-and-install-locally.sh

Run kernel compilation

In home directory run (it will delete our old directory in home without asking)

cd ~
rm -rf deterministic-kernel/
git clone https://github.com/mempo/deterministic-kernel.git
cd deterministic-kernel/

Verify version of git repository with the command:

git tag -v `git describe --tags`

Build:

bash run.sh

press ENTER to confirm (e.g. the download) and then kernel should build :)

/!\ (Or better execute this by hand and check sha1sum of git version)

How this works

Following fixes are applied:

  • (TODO update this doc)
  • (./) Set anv options like USER, HOST

  • (./) Faketime

  • (./) Replace TIME with given time in kernel sources (maybe not needed since faketime, however it's cleaner solution to the problem)

  • (./) Overlay fix-md5sums-sort.patch

  • (./) Overlay fix-remove-debuglink.patch (probably not needed because of pu/reproducible_builds debhelper)

  • (./) Overlay fix-kpkg-gz.patch (probably not needed because of pu/reproducible_builds dpkg)

  • (./) Overlay fix-deterministic-buildinfo.patch

  • (./) build-id

    • (./) Change to build-id=none for vmlinux to fix it quickly

    • (./) Check exact version of libc6, gcc, and other tools that embed their name in elf binaries in headers package

    • Turn back on buildid for vmlinux, if the exact versions are forced then it should be the same each time? (TODO)
  • (./) Tar --mtime and sorting files are managed by reproducible dpkg

  • (./) Deterministic ar is set by reproducible dpkg

  • (./) Force always same XZ options: CRC and compression in ?our dpkg from Mempo

Trust chain

For ?Mempo-Kernel: obtain the *.deb from mempo repository, then check checksum of it with trustworthy people who did build *.deb from source and check if it produces same binary as in repository.

  • You: check checksum of *.deb and look for trusted 3rd party signed message that such *.deb is fine.
  • Volunteers: will spend the 2-3 hours needed to build and verify our *.deb and then confirm you can trust the *.deb with given checksum.

So the full chain trust is:

  • You should install the *.deb of Kernel
  • If this is the official Debian Kernel, after Debian will use the script/technique described here, then this *.deb is signed by apt-get GPG key so you trust it implicitly - all done and you are able to repeat the build process to verify if the apt-get GPG key was not compromised.
  • If this is for the ?Mempo-Kernel, then until this flavour is hopefully included in Official Debian as another kernel option, it is not signed by GPG key of Debian official apt-get.

  • In this case, verify if the *.deb file you downloaded is signed by security@mempo.org

But how do you verify is security@mempo.org is not compromised nor malicious?

  • For this verification, you can build the kernel yourself as described here
  • Compare the content of provided and built *.deb with each other
  • If matching, you can publish that "Kernel *.deb file with checksum sha512:........ was unpacked and verified to match results of compilation from source by me" so others can do it easier.

In time, we will upgrade the build script to have the final *.deb file identical, then procedure is even faster (just run run.sh and publish sha512 of your *.deb).

?Mempo project might release the *.deb in own repository for easy installation for people that want this.

Ultimately, Debian.org might one day officially include Mempo-kernel in Debian repository (though, even then checksum based verification will be more secure, in case of ftp master key being compromised).

FAQ

FAQ posts solutions to common questions and problems:

  • Bugs - read the known #bugs below first.

  • Using kernel: Firefox and other programs do not run

    you run then #grsecurity version of kernel? You forgot to run the setfattr script, see #grsecurity and grsecurity/setfattr. Currently this script needs to be run AGAIN after reinstalling, upgrading.

    Using kernel: Still some things do not run

    special programs, some hardware drivers, opengl, opencl could be blocked on this security settings. Other settings of grsecurity in ?future could be more permissive. Ask us on mempo as well as at #grsecurity.

    Using kernel: Programs run slower

    Grsecurity at high security levels is auditing many system calls and internal operations to make sure even new unknown exploits would be usually blocked. This takes time. We are preparing ?grsecurity/#performance review. Roughly, task like compiling a program could be 20-50% slower. We will prepare more light versions of grsecurity settings in ?future

    Build: Wrong version of something

    if libc differs a bit then it should work but the binary files can differ similar to what happened in #bug2 (that is not our bug - just you need to use correct version of libs). If dpkg, gettext etc - then you must install them as described here.

    Build: Not enough free space

    It will probably run out of disk space. If you would try to fix that and create space on another partition then move (or symlink) entire home or the top directory, in the end the working directory must be as expected (at least as of 2014-02-14) script will warn you if the work dir is incorrect.

    Build: Wrong checksum on file (on linux kernel)

    probably file was corrupted in the cached download on your hard drive, in ~/Downloads/linux... or in kernel-sources/ where you build SameKernel. Delete this partially download file, and script will re-download. -or- in rare cases it could mean network download error, or actual attack on you (DNS spoof/network takeover - and sending malicious file instead), in such case back up the file and report people you trust / security researchers

    Build: Wrong checksum on file (other file, included in SameKernel)
    files corrupted on disk, or mistake in our script/sources listing. Contact us
    Build: Wrong dpkg version
    do as the error message says.
    Can not run as root
    do as the error message says.
    Build: Wrong username
    do as the error message says.
    Build: Wrong directory
    do as the error message says.

Bugs

bug1 [low] [open] gzip problem while building from source

#bug1 Error unknown option '-' to gzip:: error caused by unknown problem, seen by us 3 time so far, please tell us on IRC if you see it too.

Please help us debug it if you can, e.g. strace -s 8192 -fff -o strace run.sh and look how is gzip executed, does it get invalid arguments.

It was said it could be related to redirections and redirecting to null could help (but perhaps not).

bug2 [very-low] [fixed?] Headers checksum not matching since fixdep has other gcc name

#bug2 not matching checksum of the deb with headers, just the gcc name differs

Solution: you must have identical version of tools that have their version embed in binaries during compilation - especially gcc and libc6. Now script will partially try to detect/warn you. Simply run system update before building. If you try to build older kernel it might be harder, you would need correct versions of libs - or instead compare by hand as explained in next paragraphs.

Impact: just makes it harder to produce-verification (but despite that you can just trust that other people your trust said source==kernel)

Info: the headers deb is sometimes different, between systems; It turn out that some of binary programs there like fixdep have embbed other exact name/version of gcc compiler.

News: seems to be result of differences in libc library package, will add check for it in run.sh

Workaround: update fully your system before building (and in retrospection - use identical version that developers used). Workaround: unpack your deb and the orginal deb and inspect diff, only the binaries should differ, and with command you can check the problem:

<vyrly@oftc> Can check GCC version easly using command:  readelf -p .comment fixdep
<tefnoot@oftc> String dump of section '.comment':
<tefnoot@oftc>   [     0]  GCC: (Debian 4.7.2-5) 4.7.2
<tefnoot@oftc>   [    1c]  GCC: (Debian 4.4.7-2) 4.4.7

also you can hexdump the both binaries and diff them, it should be like:

<rfree> -000022f0  20 28 44 65 62 69 61 6e  20 34 2e 34 2e 37 2d 33  | (Debian 4.4.7-3|
<rfree> +000022f0  20 28 44 65 62 69 61 6e  20 34 2e 34 2e 37 2d 32  | (Debian 4.4.7-2|

but also this line, it's probably same info about version but in binary (int?)

-00000240  14 00 00 00 03 00 00 00  47 4e 55 00 9c 99 41 b2  |........GNU...A.|
-00000250  36 53 75 44 d1 54 57 84  cd 01 f0 2b 84 17 a2 27  |6SuD.TW....+...'|
+00000240  14 00 00 00 03 00 00 00  47 4e 55 00 ad 80 0e 61  |........GNU....a|
+00000250  2c ee 29 52 92 10 8f f0  09 90 e3 e3 42 2d 4b 36  |,.)R........B-K6|

Though above is not 100% guaranteed the difference is not a trojan. Perhaps further assembler analysis of that section can prove it (readelf)

Research: this happens even if dpkg says gcc, gcc-4.7 are same. Some other packages are involved.

More details:

Test

Tests - please test this script and report any problems to ?Mempo and on IRC #mempo also add here to wiki in /Test :

CategoryKernel