distcc

''distcc'' is a tool to distribute compilation of C and C++ files over multiple computers. distcc is useful for speeding up builds which are carried out make-kpkg, unfortunately make-kpkg can make using and debugging distcc difficult.

distcc over ssh

distcc may use it's own native networking support (which requires a trusted network, and may not be desirable for security reasons), or support operation over ssh. Since the use of ssh under make-kpkg may also present problems, this guide explicitly covers distcc-over-ssh.

Basic setup

To configure a single remote host for testing purposes only:

Install distcc on both machines.
Ensure password-less ssh remote command execution is configured (using ssh key-based authentication).
Create links for gcc and g++ to distcc:

mkdir -p $HOME/bin
ln -s `which distcc` $HOME/bin/gcc
ln -s `which distcc` $HOME/bin/g++

Having set this up, start a new shell and verify that these commands have preference e.g. using which gcc.

Alternatively, if you'd prefer to add these links to an alternative directory to more-easily facilitate turning the use of distcc on and off by prefixing PATH=/path/to/distcc-link-directory:$PATH to individual build invocations.

Set the following environmental variable:

export DISTCC_HOSTS='remoteuser@compilevolunteer'

where remoteuser and compilevolunteer correspond to appropriate user and host names in your build environment.

Debug Tips

Assuming you are trying to build with this command:

$ make-kpkg --rootcmd fakeroot -j 6 --append-to-version -my-extra-kernel-version-string --revision 42 --initrd kernel-image

... the following will provide verbose debug output:

$ DISTCC_SSH=$HOME/bin/debugssh KBUILD_VERBOSE=1 DISTCC_VERBOSE=1 make-kpkg --rootcmd fakeroot -j 6 --append-to-version -my-extra-kernel-version-string --revision 42 --initrd kernel-image

debugssh wrapper (place in ~/bin/):

   1 #!/bin/sh
   2 mkdir -p $HOME/ssh-debuglogs
   3 /usr/bin/ssh -v -E $HOME/ssh-debuglogs/ssh-debuglog-`date  '+%s.%N'` $@

... see ~/ssh-debuglogs for the ssh debug logs from the build. With correct ssh ?ControlMaster operation, you will see one large, and many small log files.

Performance

It is strongly advised to use the ssh ControlMaster functionality to avoid the overhead of setting up and tearing down multiple ssh connections. Add the following to your /etc/ssh/ssh_config or ~/.ssh/config:

Host *
# ^^ or just restrict to particular hosts if you prefer...
    ControlMaster auto
    ControlPath ~/.ssh/master-%r@%h:%p
    ControlPersist 8s

SSH Cipher performance - since ssh cipher performance can differ by a large amount depending on the CPU (e.g. hardware-accelerated symmetric ciphers), you may wish to verify both distcc client and server ssh throughput - e.g. using the script given in this ssh benchmark blog post.
Consider the use of distcc 'pump' mode (see the distcc manual page).

Pitfalls