Testing the SameKernel builds. Please help if you can, write to us on #mempo (?Mempo)

Tests

Here we paste research data.

v0.1.25-rc2 buildA plus

Authors: vyrly, rfree, tefnoo, +
Computers: different machines
Options: all options as recommended (same dir and user, using our correct dpkg etc)

Image .deb was always the same for image, but not for headers.
It is probably the [[SameKernel#bug2]].
On few computers the build was not possible [[SameKernel#bug1]].

Even though verification is not always working, it works usually and therefore this is the first version that we recommend to use. 

Usage: 2 computers running it, works fine (actually it was perhaps one more git commit above, but just doc changed).

v0.1.23-rc1 buildA

Testing across several machines

Authors: vyrly, rfree, +
Kernel version: 3.2.54
Kernel deterministic patches: the ones included by Mempo
Kernel extra patches: grsecurity patch
Kernel was built: 2 times per machines
Computers: different machines
Directory: same user - kernelbuild, in ~/deterministic-kernel/
Fakedate: yes
pu/reproducible_builds dpkg: yes
Build date: 2014-02-03

Results:

+ iris, freedompals, vyrly(H) - same checksum:
6d25a76112e4e37e72702bb1fe45c78f7744f30f  ./freedompals/linux-image-3.2.54-grsec-mempo.good.0.1.23_01_amd64.deb
6d25a76112e4e37e72702bb1fe45c78f7744f30f  ./iris/linux-image-3.2.54-grsec-mempo.good.0.1.23_01_amd64.deb

- rfree-B - different checksum, other XZ header/tailer in data.tar.xz (but the .tar uncompressed is same) - so XZ works differently on this machine!

- pinky (new Wheezy) - compilation failed related to gzip packing of files at end of image

v0.1.22-rc10 buildA

Test with pu/reproducible branch of dpkg

Authors: vyrly
Kernel version: 3.2.54
Kernel deterministic patches: the ones included by Mempo
Kernel extra patches: grsecurity patch
Kernel was built: 2 times
Computers: different machines
Directory: same user - kernelbuild2
Fakedate: yes
pu/reproducible_builds dpkg: yes
System 1: Debian GNU/Linux 7.3
System 2: Debian GNU/Linux 7.2
Build date: 2014-01-31

Results:

All *.deb files have the same checksum.

System 1:
aa231c8140d76646ec6af40057a36d83e03159b3  linux-headers-3.2.54-grsec-mempo.good.0.1.22_09_amd64.deb
ba715a4c3e638fc2c3b7b3a41e13759a0c258220  linux-image-3.2.54-grsec-mempo.good.0.1.22_09_amd64.deb

System 2:
aa231c8140d76646ec6af40057a36d83e03159b3  linux-headers-3.2.54-grsec-mempo.good.0.1.22_09_amd64.deb
ba715a4c3e638fc2c3b7b3a41e13759a0c258220  linux-image-3.2.54-grsec-mempo.good.0.1.22_09_amd64.deb

v0.1.22-rc8 buildA

Testing the new tar with new --sort-input

Tar: 028c6cb79b162ccd747a48eba75bedf2a67bff64 (had bug /!\ )

Same as: v0.1.22-rc5

dpkg-deb: building package `linux-image-3.2.54-grsec-mempo.good.0.1.22' in `../linux-image-3.2.54-grsec-mempo.good.0.1.22_01_amd64.deb'.
*** glibc detected *** /home/kernelbuild/.local/usr/bin/tar: corrupted double-linked list: 0x00000000017dead0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x20400084d76]
/lib/x86_64-linux-gnu/libc.so.6(+0x7c971)[0x2040008a971]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0xf0)[0x2040008ac60]
/home/kernelbuild/.local/usr/bin/tar[0x437401]
/home/kernelbuild/.local/usr/bin/tar[0x436bc9]
/home/kernelbuild/.local/usr/bin/tar[0x40cc27]
/home/kernelbuild/.local/usr/bin/tar[0x40d3f1]
/home/kernelbuild/.local/usr/bin/tar[0x40cffd]
/home/kernelbuild/.local/usr/bin/tar[0x40e1bb]
/home/kernelbuild/.local/usr/bin/tar[0x405d45]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x2040002cead]
/home/kernelbuild/.local/usr/bin/tar[0x405f85]

(╯'□')╯ ︵ ︵ ︵ ┻━┻

v0.1.22-rc5 buildA

Build version: https://github.com/mempo/deterministic-kernel/tree/v0.1.22-rc5

Authors: rfree, vyrly
Kernel version: 3.2.54
Kernel deterministic patches: the ones included by Mempo
Kernel extra patches: grsecurity patch
Kernel was built: 2x2 times
Computer: 2 times
Directory: same on computers (same user?)
Fakedate: yes
Dpkg: yes
Tar: tar with --fakedate
System: build on Debian 7 amd64
Build date: 2014-01-15

Results:
.deb - differed
.deb unpacked just ar - same on computer, different between computers(need sort?)
.deb unpacked fully - will be the same then ?

709508d8014a8098b2de497a3aad56e7972915f9  1/data.tar.xz
709508d8014a8098b2de497a3aad56e7972915f9  2/data.tar.xz
332a724f0e4ed93d4c8f07a21c3111d6c40e0402  vyrly1/data.tar.xz
332a724f0e4ed93d4c8f07a21c3111d6c40e0402  vyrly2/data.tar.xz

v0.1.22-rc1 buildA

Build version: https://github.com/mempo/deterministic-kernel/tree/v0.1.22-rc1

test in progress, using new dpkg and --build-id=none - solved the vmlinux, now issues remain with mostly packing
Authors: rfree
Kernel version: 3.2.54

v0.1.21-rc1 buildA

Build version: https://github.com/mempo/deterministic-kernel/tree/v0.1.21-rc1

Authors: members of Mempo project
Kernel version: 3.2.54
Kernel deterministic patches: custom patch to remove __TIME__ and __DATE__ 
Kernel extra patches: grsecurity patch
Kernel was built: 2 times.
Computer: built on same computer each time.
Directory: built in same directory path each time.
Fakedate: yes, using faketime from env.sh in our script
Dpkg: not fixed (regular version from Debian 7)
System: build on Debian 7 amd64, gcc version 4.7.2-5 linux kernel 3.10.9-xxxx-grs-ipv6-64 (OVH server); 
Build date: 2014-01-07
Machine name: (m1)
Settings: HOST=? USER="kernelbuild", PATH="/home/kernelbuild/" 

Results:
vmlinuz - different
other things not tested, should be as in last tests but .gz fixed

On this machine the file vmlinux (we check the unpacked file right after built, not vmlinuz) differs, around address ffffffff8141c167, that is in section ffffffff8141c000 <__start_notes> Used objdump -d vmlinux on the builded kernel file from run.sh.

See this file: vmlinuz-ovh-difference-notes.txt difference between 2 compilations is eg:

 ffffffff8141c162:      00 00                   add    %al,(%rax)
 ffffffff8141c164:      47                      rex.RXB
 ffffffff8141c165:      4e 55                   rex.WRX push %rbp
-ffffffff8141c167:      00 64 ce fe             add    %ah,-0x2(%rsi,%rcx,8)
-ffffffff8141c16b:      0d 9c ac 80 f6          or     $0xf680ac9c,%eax
-ffffffff8141c170:      44 93                   rex.R xchg %eax,%ebx
-ffffffff8141c172:      50                      push   %rax
-ffffffff8141c173:      09 78 a3                or     %edi,-0x5d(%rax)
-ffffffff8141c176:      72 2c                   jb     ffffffff8141c1a4 <__start___ex_table+0x24>
-ffffffff8141c178:      02 46 d5                add    -0x2b(%rsi),%al
-ffffffff8141c17b:      06                      (bad)  
+ffffffff8141c167:      00 6c 94 92             add    %ch,-0x6e(%rsp,%rdx,4)
+ffffffff8141c16b:      49 c4                   rex.WB (bad) 
+ffffffff8141c16d:      db 48 1e                fisttpl 0x1e(%rax)
+ffffffff8141c170:      ff                      (bad)  
+ffffffff8141c171:      f9                      stc    
+ffffffff8141c172:      ee                      out    %al,(%dx)
+ffffffff8141c173:      63 7e 7a                movslq 0x7a(%rsi),%edi
+ffffffff8141c176:      b0 fa                   mov    $0xfa,%al
+ffffffff8141c178:      7a c7                   jp     ffffffff8141c141 <__start_notes+0x141>
+ffffffff8141c17a:      38 2a                   cmp    %ch,(%rdx)

Test20131212

Authors: members of Mempo project
Build version: d51deed258 (https://github.com/mempo/deterministic-kernel/blob/master/run.sh)
Kernel version: 3.2.53
Kernel deterministic patches: custom patch to remove __TIME__ and __DATE__ 
Kernel extra patches: grsecurity patch
Kernel was built: 2 times.
Computer: built on same computer each time.
Directory: built in same directory path each time.
Fakedate: yes, using faketime 2013-11-24 20:32:00 ?
Dpkg: not fixed (regular version from Debian 7)
System: build on Debian 7.1 amd64, gcc version 4.7.2-1, linux kernel 3.2.46; 
Build date: 2013-12-12
Machine name: (t/wb)
Settings: HOST="paleale" USER="kernelbuild", PATH="/home/kernelbuild/" 

Results:
.o - all identical
.ko - all inentical
vmlinuz - identical
.deb - different
.gz - different (wasn't unpacked)
Symlinks - ommited

List of different files:

linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/Buildinfo.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/Changes.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/buildinfo.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/changelog.Debian.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/conf.vars.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/debian.README.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/ChangeLog.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/header_postinst.d/link.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/header_postrm.d/link.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/header_prerm.d/link.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postinst.d/force-build-link.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postinst.d/grub_conf.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postinst.d/initramfs.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postinst.d/symlink_hook.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postinst.d/yaird.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postrm.d/force-build-link.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postrm.d/grub_rm.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postrm.d/initramfs.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/kernel/postrm.d/yaird.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/etc/sample.kernel-img.conf.gz
linux-image-3.2.53-grsec-mempo.good.0.1.18_01_amd64/usr/share/doc/linux-image-3.2.53-grsec-mempo.good.0.1.18/examples/sample.module.control.gz

Disabled grsecurity option CONFIG_PAX_LANTENT_ENTROPY makes *.ko files deterministic - these files compiled twice have the same checksums. This option generates some entropy during boot.
Now we working with *.gz files - now their checksums are different, but contain files with the same checksums.

Test20131127

Authors: members of Mempo project
Kernel version: 3.2.52 
Kernel deterministic patches: custom patch to remove __TIME__ and __DATE__ 
Kernel extra patches: grsecurity patch
Kernel was built: 2 times.
Build tool: using mempo script .
Computer: built on same computer each time.
Directory: built in same directory path each time.
Fakedate: yes, using faketime 2013-11-24 20:32:00
Dpkg: not fixed (regular version from Debian 7)
System: build on Debian 7.1 amd64, gcc version 4.7.2-1, linux kernel 3.2.46; 
Build date: 2013-11-27
Machine name: (t/wb)
Settings: HOST="workbench" USER="gb-kernel-builder", PATH="/home/gb-kernel-builder/kernel-build/" 

Results:
.o - all identical
.ko - all inentical 
vmlinuz - identical
.deb - different
.gz - different (but content seems identical when unpacked. watch out for external symlinks)

Disabled grsecurity option CONFIG_PAX_LANTENT_ENTROPY makes *.ko files deterministic - these files compiled twice have the same checksums. This option generates some entropy during boot.
Now we working with *.gz files - now their checksums are different, but contain files with the same checksums.

Test20131029

Authors: members of Mempo project
Kernel version: 3.2.51 
Kernel deterministic patches: custom patch to remove __TIME__ and __DATE__ 
Kernel extra patches: grsecurity patch
Kernel was built: 2 times.
Build tool: using mempo script.
Computer: built on same computer each time.
Directory: built in same directory path each time.
Fakedate: yes, using faketime 2013-10-19 12:58:00
Dpkg: not fixed (regular version from Debian 7)
System: build on Debian 7.1 amd64, gcc version 4.7.2-1, linux kernel 3.2.46; 
Build date: 2013-10-29
Machine name: (t/wb)
Settings: HOST="workbench" USER="gb-kernel-builder", PATH="/home/gb-kernel-builder/kernel-build/" (afair)

Results:
.o - all identical
.ko - some match, not all (22909 files are the same and 2539 are different) (SOLVED)
vmlinuz - different
.deb - different

List of checksums:

Expamle different .ko files:

Example with differences dissassembled .ko files:

We use to disassemble "objdump -d" command.

CategoryKernel