Differences between revisions 113 and 114
Revision 113 as of 2014-02-07 15:18:42
Size: 10080
Editor: ?Mempo
Comment: scripts
Revision 114 as of 2014-02-07 15:20:05
Size: 10072
Editor: ?Mempo
Comment: format
Deletions are marked like this. Additions are marked like this.
Line 88: Line 88:
Please use [[Mempo]]'s script: [[https://github.com/mempo/mempo-deb/tree/master/pack/dpkg]] that will fetch, build and install reproducible-dpkg '''locally''' (no need to upgrade your system wide dpkg) ''''(it will '''delete''' our old directory in home without asking)'''' Please use [[Mempo]]'s script: [[https://github.com/mempo/mempo-deb/tree/master/pack/dpkg]] that will fetch, build and install reproducible-dpkg '''locally''' (no need to upgrade your system wide dpkg) ''(it will '''delete''' our old directory in home without asking)''
Line 99: Line 99:
In home directory run ''''(it will '''delete''' our old directory in home without asking)'''' In home directory run ''(it will '''delete''' our old directory in home without asking)''

This allows to build Kernel in verificable way to know if .deb indeed matches the source code of kernel without added backdoors/viruses.

To use it just execute the 2 simple commands described in section "Install" below.

To get all needed files, on Debian use command: git clone https://github.com/mempo/deterministic-kernel.git

This is work in progress, ?Mempo team will help you in using/testing this software, contact us on #mempo and #debian-kernel on irc.oftc.net (see http://www.oftc.net/) or use this chat-webpage (for anon. use I2P irc #mempo) or see ?Mempo-Contact

As of now (2014-02-03) this work is:

  • about Linux kernel only, testing on amd64 and i386
  • work in alpha-testing
  • includes grsecurity, but that can be easily removed (just remove grsecurity line in script's sources.list)

News: https://github.com/mempo/deterministic-kernel/commits/master :) and #mempo

(./) (./) (./) Good news, everyone! it worked, as of 2013-02-03 we are producing binary identical .deb files (last fix: XZ CRC fix) :) (testing pending! ignore the wrong-patch warning in RC3) (./) (./) (./)

Progress

(!) Issues currenty worked on:

  • Need to remove path from .ko files - for now we need to use same path(/user home)

Steps of this project:

  • (./) Step1 configure kernel build to create identical intermittent files: .o

  • (./) Step2 configure kernel build to create identical final files: .ko

  • (./) Step3 identical .gz files (docs?)

  • (./) Step4 have identical all files and vmlinux including vmlinux .notes

  • (./) Step5 have identical .deb files on the same machine

  • (./) Step6 have identical .deb files on different machines :-) (this works, provided username kernelbuild and directory /home/kernelbuild/deterministic-kernel/)

  • Step6b fix varying xz header/tail between some machines ( /Test#v0.1.23-rc1_buildA )

  • Step7a remove requirement of having same directory
  • Step7b remove requirement of having same username
  • Step8 test if hostname, and timezone are ignored
  • Step9 any other differences between alpha-testers doing the verification

See general instructions for all programs: ReproducibleBuilds .

We are trying to create a script that does this deterministic build in a bit more automated way with added:

  • verification of downloaded sources (check against hardcoded in script list of expected checksums of sources; also check PGP signature with hardcoded public key of kernel developers)
  • apply security patches for the ?Mempo subproject

  • grsecurity patch - http://grsecurity.net/

  • misc patches if needed (e.g. quick fixes regarding security)

This page should be usable for everyone in Debian, and script we're writing will be later easy to run in pure-Debian mode too.

Install

You can help yourself and this project easily!

/!\ Extra tools that are not-standard-wheezy and are installed globally:

  • GNU gettext >= 0.18.2 (install them from official debian-wheezy backports)

/!\ Extra tools that do not need any non-standard global changes:

  • User local installation of dpkg pu/reproducible_builds - instructions below

  • As root from normal repository install faketime (in wheezy) and other common programs

Install dependencies

As root :

apt-get install faketime time git build-essential libncurses5-dev libncursesw5-dev kernel-package md5deep gcc-4.7-plugin-dev g++ make time automake pkg-config flex

Create user for kernel build

/!\ At this moment, to get the same *.deb kernel files on different machines, build must be run on the same user!

Create unix user kernelbuild.

Get and build dpkg

We are using Lunar's branch of dpkg: pu/reproducible_builds. We need GNU gettext >= 0.18.2, therefore on wheezy please add:

http://YOURMIRROR.debian.org/debian wheezy-backports main

to /etc/apt/sources.list and run:

# aptitude update
# aptitude install -t wheezy-backports gettext autopoint

Please use ?Mempo's script: https://github.com/mempo/mempo-deb/tree/master/pack/dpkg that will fetch, build and install reproducible-dpkg locally (no need to upgrade your system wide dpkg) (it will delete our old directory in home without asking)

cd ~
rm -rfi mempo-deb/
git clone https://github.com/mempo/mempo-deb
cd mempo-deb/pack/dpkg
./build-and-install-locally.sh

Run kernel compilation

In home directory run (it will delete our old directory in home without asking)

cd ~
rm -rf deterministic-kernel/ ; git clone https://github.com/mempo/deterministic-kernel.git && cd deterministic-kernel/  && bash run.sh

press ENTER to confirm (e.g. the download) and then kernel should build :)

/!\ (Or better execute this by hand and check sha1sum of git version)

How this works

Following fixes are applied:

  • (TODO update this doc)
  • (./) set env options like USER, HOST

  • (./) faketime

  • (./) replace TIME with given time in kernel sources (maybe not needed since faketime, however it's cleaner solution to the problem)

  • (./) overlay fix-md5sums-sort.patch

  • (./) overlay fix-remove-debuglink.patch (probably not needed because of pu/reproducible_builds debhelper)

  • (./) overlay fix-kpkg-gz.patch (probably not needed because of pu/reproducible_builds dpkg)

  • (./) overlay fix-deterministic-buildinfo.patch

  • (./) change build-id to build-id=none (not sure if needed, will be tested)

  • (./) tar --mtime and sorting files are managed by reproducible dpkg

  • (./) deterministic ar is set by reproducible dpkg

  • (./) force always same XZ options: CRC and compression in ?our dpkg from Mempo

Trust chain

For ?Mempo-Kernel: obtain the .deb from mempo repository, then check checksum of it with trustworthy people who did build .deb from source and seen it produces same binary as in repository.

  • YOU: check checksum of .deb and look for trusted 3rd party signed message that such .deb is fine.
  • Volunteers: will spend the 2-3 hours needed to build and verify our .deb and then confirm you can trust the .deb with given checksum.

So the full chain trust is:

  • you should install the .deb of Kernel
  • if this is the official Debian Kernel, after Debian will use the script/technique described here, then this .deb is signed by apt-get GPG key so you trust it implicitly - all done. And you are able to repeat the build process to verify if the apt-get GPG key was not compromised.
  • if this is for the ?Mempo-Kernel, then until this flavour is hopefully included in Official Debian as another kernel option, it is not signed by GPG key of Debian official apt-get.

  • in this case, verify if the .deb file you downloaded is signed by security@mempo.org

  • but how do you verify is security@mempo.org is not compromised nor malicious?

  • for this verification, you can build the kernel yourself as described here
  • compare the content of provided and builded .deb with eachother
  • if matching, you can publish that "Kernel .deb file with checksum sha512:........ was unpacked and verified to match results of compilation from source by me" so others can do it easier.

In time, we will upgrade the build script to have the final .deb file identical, then procedure is even faster (just run run.sh and publish sha512 of your .deb).

?Mempo project might release the .deb in own repository for easy installation for people that want this.

Ultimately, Debian.org might one day officially include Mempo-kernel in Debian repository (though, even then checksum based verification will be more secure, in case of ftp master key being compromised).

FAQ

FAQ posts solutions to common questions and problems:

wrong checksum on file (on linux kernel)

probably file was corrupted in the cached download on your hard drive, in ~/Downloads/linux... or in kernel-sources/ where you build SameKernel. Delete this partially download file, and script will re-download. -or- in rare cases it could mean network download error, or actual attack on you (DNS spoof/network takeover - and sending malicious file instead), in such case back up the file and report people you trust / security researchers

wrong checksum on file (other file, included in SameKernel)
files corrupted on disk, or mistake in our script/sources listing. Contact us
error unknown option '-' to gzip

error caused by unknown problem, seen by us 1 time so far, please tell us on IRC if you see it too

wrong dpkg version
do as the error message says.
can not run as root
do as the error message says.
wrong username
do as the error message says.
wrong directory
do as the error message says.

Test

Tests - please test this script and report any problems to ?Mempo and on IRC #mempo also add here to wiki in /Test

CategoryKernel