On this page is described an easy way to debug Linux kernel issues, by using "git-bisect". The idea of this tool is to track down a particular issue (or regression) by selecting the faulty revision.

/!\ Before starting to bisect, it would be a good idea to use Debian's wayback machine to narrow down the range of Linux kernel versions where the faulty revision may occur so that the amount of compiling is reduced somewhat.

Use cases

Needed tools

For these regression tests, you will need the following packages (to install with your favorite package manager):

git gitk kernel-package fakeroot

Alternatively, just install all the Linux kernel build-dependencies and git:

$ sudo apt install git
$ sudo apt build-dep linux

At times the Linux kernel and the toolchain versions might be out of sync, resulting in lots of strange build failures. In that case, it would be a good idea to use a Debian stable chroot to do the building.

Getting the Linux git source

In this step, we will get a local copy of the whole Linux kernel source code by using Linux's distributed version control system: git.

Move to your source directory

$ cd ~/src

Clone Linus's version (HEAD)

This will download a lot of data and thus may need a lot of time.

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Separate the wheat from the chaff

In this step, we will mark the versions good or bad.

Start the git-bisect process

$ git bisect start

Mark the ''good'' version

For example:

$ git bisect good v2.6.25-rc6 

Mark the ''bad'' version

For example:

$ git bisect bad v2.6.25-rc7 

At this point, this command should answer you something like that :

Bisecting: 182 revisions left to test after this
[2c7871982cf27caaddbaeb7e2121ce1374b520ff] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6.25

The interesting part is the number (in our case, 182) which is the number of Linux kernel revisions that lay between your good and your bad.

Build your Linux kernel

In this step, we will configure the future Linux kernel according to the actual configuration and then compile the new Linux kernel.

Copy a valid configuration

For example, the configuration of the running Linux kernel:

$ cp /boot/config-$(uname -r) .config

Put git commit IDs into versions

This ensures that you can easily map the installed Linux kernel images to a git commit ID. If you do not have much disk space on /boot you should either skip this step or be prepared during testing to remove some Linux kernel images that have already been tested.

$ echo CONFIG_LOCALVERSION_AUTO=y >> .config

Configure the Linux kernel

If you do not know the answer to each question, you can just accept the defaults:

$ yes "" | make oldconfig

If you want to customise the configuration, you can run this instead:

$ make oldconfig

Build the Linux kernel

This will take a lot of time and computation power too. Ensure your system is adequately cooled so it will not overheat.

# Detect the amount of CPU cores you have
$ jobs=$(nproc --all)
# Detect if the Linux kernel version needs fakeroot or not
$ fakeroot=$(grep -q fakeroot scripts/package/Makefile || echo fakeroot)
# Build Debian binary packages of the Linux kernel version
$ $fakeroot make -j$jobs bindeb-pkg

Install the newly created Linux kernel

According to the name printed by the last command.

# apt install ../linux-2.6.25-rc6_2.6.25-rc6-2_amd64.deb

Reboot under newly built Linux kernel

# reboot

Test your issue under this Linux kernel

If the feature you are testing works in the newly booted Linux kernel, mark the commit as good:

$ cd ~/src/linux
$ git bisect good

If it does not work, mark the commit as bad:

$ cd ~/src/linux
$ git bisect bad

These commands will then choose a new commit to be configured, built and tested. Repeat the procedure until git bisect decides one particular commit is at fault. Once the commit has been found you can inspect it to see why it caused the regression and or file a bug about it against the Debian Linux kernel package.