distcc
''distcc'' is a tool to distribute compilation of C and C++ files over multiple computers. distcc is useful for speeding up builds which are carried out make-kpkg, unfortunately make-kpkg can make using and debugging distcc difficult.
distcc over ssh
distcc may use it's own native networking support (which requires a trusted network, and may not be desirable for security reasons), or support operation over ssh. Since the use of ssh under make-kpkg may also present problems, this guide explicitly covers distcc-over-ssh.
Basic setup
To configure a single remote host for testing purposes only:
- Install distcc on both machines.
- Ensure password-less ssh remote command execution is configured (using ssh key-based authentication).
Create links for gcc and g++ to distcc:
mkdir -p $HOME/bin ln -s `which distcc` $HOME/bin/gcc ln -s `which distcc` $HOME/bin/g++
Having set this up, start a new shell and verify that these commands have preference e.g. using which gcc.
Alternatively, if you'd prefer to add these links to an alternative directory to more-easily facilitate turning the use of distcc on and off by prefixing PATH=/path/to/distcc-link-directory:$PATH to individual build invocations.
Set the following environmental variable:
export DISTCC_HOSTS='remoteuser@compilevolunteer'
where remoteuser and compilevolunteer correspond to appropriate user and host names in your build environment.
Debug Tips
Assuming you are trying to build with this command:
$ make-kpkg --rootcmd fakeroot -j 6 --append-to-version -my-extra-kernel-version-string --revision 42 --initrd kernel-image
... the following will provide verbose debug output:
$ DISTCC_SSH=$HOME/bin/debugssh KBUILD_VERBOSE=1 DISTCC_VERBOSE=1 make-kpkg --rootcmd fakeroot -j 6 --append-to-version -my-extra-kernel-version-string --revision 42 --initrd kernel-image
debugssh wrapper (place in ~/bin/):
... see ~/ssh-debuglogs for the ssh debug logs from the build. With correct ssh ?ControlMaster operation, you will see one large, and many small log files.
Performance
It is strongly advised to use the ssh ControlMaster functionality to avoid the overhead of setting up and tearing down multiple ssh connections. Add the following to your /etc/ssh/ssh_config or ~/.ssh/config:
Host * # ^^ or just restrict to particular hosts if you prefer... ControlMaster auto ControlPath ~/.ssh/master-%r@%h:%p ControlPersist 8s
SSH Cipher performance - since ssh cipher performance can differ by a large amount depending on the CPU (e.g. hardware-accelerated symmetric ciphers), you may wish to verify both distcc client and server ssh throughput - e.g. using the script given in this ssh benchmark blog post.
- Consider the use of distcc 'pump' mode (see the distcc manual page).
Pitfalls
Use of fakeroot - many online resources advise running make-kpkg under fakeroot. If you do this, then distcc (and in turn ssh) will also be run under fakeroot - this breaks ssh in a few subtle ways (e.g. UID checks for ?ControlMaster operation, location of known hosts and authentication key files). Instead use make-kpkg --rootcmd fakeroot - this restricts the use of the fakeroot wrapper to the package creation steps - which is where it is actually needed.