Differences between revisions 2 and 28 (spanning 26 versions)
Revision 2 as of 2008-06-23 16:31:26
Size: 2844
Editor: FranklinPiat
Comment: minor formating.
Revision 28 as of 2021-04-25 04:10:22
Size: 7606
Editor: PaulWise
Comment: add a section about spare systems
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
On this page is described an easy way to debug kernel issues, by using "git-bisect". The idea of this tool is to track down a particular issue (or regression) by selecting the faulty revision.
On this page is described an easy way to debug upstream Linux kernel issues, by using "git bisect". The idea of this tool is to track down a particular issue (or regression) by selecting the faulty revision.

/!\ Before starting to bisect, it would be a good idea to use Debian's wayback machine to [[BisectDebian|narrow down]] the range of [[https://snapshot.debian.org/package/linux/|Linux kernel versions]] where the faulty revision may occur so that the amount of compiling is reduced somewhat.

If the issue does not appear in the upstream Linux kernel, but does appear in the Debian Linux kernel images, please refer to the [[https://kernel-team.pages.debian.net/kernel-handbook/|Debian Linux Kernel Handbook]] for information about how to debug and report the issue.
Line 5: Line 10:
 * Hibernation works under 2.6.25-rc6 but not anymore under 2.6.25-rc7 and I would like to know wich revision caused the regression.
* Hibernation works under 2.6.25-rc6 but not anymore under 2.6.25-rc7 and I would like to know which revision caused the regression.
Line 8: Line 14:
For these regression tests, you will need the following packages (to install with your favorite package manager) :

{{{
 git-core gitk kernel-package fakeroot
}}}

For these regression tests, you will need the following packages (to install with your favorite package manager):

{{{
$ sudo apt install git gitk fakeroot
}}}

Alternatively, just install all the Linux kernel build-dependencies and git:

{{{
$ sudo apt install git
$ sudo apt build-dep linux
}}}

At times the Linux kernel and the toolchain versions might be out of sync, resulting in lots of strange build failures. In that case, it would be a good idea to use a Debian stable chroot to do the building.

It is recommended to have your shell prompt display git repository status information. The git package [[https://git-scm.com/book/en/v2/Appendix-A%3A-Git-in-Other-Environments-Git-in-Bash|includes prompt support for bash and zsh]].
Line 15: Line 33:
In this step, we will get a local copy of the whole Linux kernel source code by using Linux's distributed version control system : git.
In this step, we will get a local copy of the whole Linux kernel source code by using Linux's distributed version control system: git.
Line 18: Line 37:
Line 21: Line 41:
===== Clone Linus's version (HEAD) =====
This will download about 170 MiB and thus need some time. Go take yourself a coffee !
{{{
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

===== Clone Linux git repository =====

This will download a lot of data and thus may need a lot of time.

You can either clone Linus Torvald's main Linux repository, or the repository containing the Linux kernel stable release series. The stable repository contains more commits and is useful for bisecting bugs introduced by stable updates.

{{{
$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
Line 28: Line 54:
In that step, we will ''mark'' the versions ''good'' or ''bad''.
In this step, we will ''mark'' the versions ''good'' or ''bad''.
Line 30: Line 58:
Line 33: Line 62:
Line 34: Line 64:
For example
For example:
Line 38: Line 70:
Line 39: Line 72:
For example
For example:
Line 43: Line 78:
Line 44: Line 80:
Line 48: Line 85:
The interesting part is the number (in our case, '''182''') which is the number of kernel revisions that lay between your ''good'' and your ''bad''.

== Build your kernel ==
In this step, we will configure the future kernel according to the actual configuration and then compile the new kernel.

The interesting part is the number (in our case, '''182''') which is the number of Linux kernel revisions that lay between your ''good'' and your ''bad''.

== Build your Linux kernel ==

In this step, we will configure the future Linux kernel according to the actual configuration and then compile the new Linux kernel.
Line 53: Line 93:
For example
{{{
$ cp /boot/config-2.6.25-rc6-amd64 .config
}}}
===== Configure the kernel =====

For example, the configuration of the running Linux kernel:

{{{
$ cp /boot/config-$(uname -r) .config
}}}

===== Put git commit IDs into versions =====

This ensures that you can easily map the installed Linux kernel images to a git commit ID. If you do not have much disk space on /boot you should either skip this step or be prepared during testing to remove some Linux kernel images that have already been tested.

{{{
$ scripts/config --enable LOCALVERSION_AUTO
}}}

===== Disabling debug information =====

If your bisect won't need to debug the kernel, it is a good idea to disable debug info to reduce the build time, especially for the linux-image-*-dbg package, which will be very large and can take a long time to compress.

{{{
$ scripts/config --disable DEBUG_INFO
}}}

===== Disable Secure Boot =====

If you are using a config from a Linux kernel from Debian or other distro kernel that has SecureBoot enabled using the distro key, then your builds will fail because you don't have a copy of the private distro key. First disable Secure Boot in your UEFI boot setup and then disable it in your Linux kernel config:

{{{
$ scripts/config --disable SYSTEM_TRUSTED_KEYRING
$ scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
}}}

===== Disable Stack Protector =====

If you are using a newer GCC while bisecting older kernel versions, you may need to disable the stack protector during the bisect.

{{{
$ scripts/config --disable STACKPROTECTOR_STRONG
}}}

===== Configure the Linux kernel =====

If you do not know the answer to each question, you can just accept the defaults:

{{{
$ make olddefconfig
}}}

If you want to customise the configuration, you can run this instead:
Line 61: Line 146:
Simply press '''Enter''' at each question.
===== Build the kernel =====
This will take a lot of time and computation power too. Get yourself a pause and get your laptop off your legs.
{{{
$ fakeroot make deb-pkg
}}}
===== Install the newly created kernel =====
(According to the name lastly pointed by latest command.
{{{
# dpkg -i ../linux-2.6.25-rc6_2.6.25-rc6-2_amd64.deb
}}}
===== Update the initramfs =====
The version is probably the ''good'' one (new version of it)
{{{
# update-initramfs -c -k 2.6.25-rc6
}}}
===== Update the boot loader =====
This is for all versions of Grub. Anyone for LILO ?
{{{
# update-grub
}}}

== Reboot under newly built kernel ==
{{{
# reboot
}}}
== Test your issue under this kernel ==

If you want to reduce the build time to just what you need you can use this:

{{{
$ make localmodconfig
}}}

If you want are doing this for a remote host, copy the remote lsmod output locally and use this instead:

{{{
$ make LSMOD=$(pwd)/lsmod localmodconfig
}}}

===== Build the Linux kernel =====

This will take a lot of time and computation power too. Ensure your system is adequately cooled so it will not overheat.

{{{
# Detect the amount of CPU cores you have
$ jobs=$(nproc --all)
# Detect if the Linux kernel version needs fakeroot or not
$ fakeroot=$(grep -q 'Rules-Requires-Root: no' scripts/package/mkdebian || grep -q fakeroot scripts/Makefile.package scripts/package/Makefile || echo fakeroot)
# Build Debian binary packages of the Linux kernel version
# Make sure you include the $ sign before fakeroot on the next line
$ $fakeroot make -j$jobs bindeb-pkg
}}}

===== Transfer the Linux kernel image package =====

If you are building on a different system to the one you are testing on, at this point you should transfer the Linux kernel image package to the system you are testing on.

It is recommended to do the testing in a spare system with the same hardware, or [[SystemVirtualization|virtual machine]] (for bisects of code that is not hardware-specific) to avoid any old fixed Linux kernel bugs affecting your main system, especially old fixed issues that may corrupt your data storage.

===== Install the newly created Linux kernel =====

According to the name printed by the last command.

{{{
$ sudo apt install ../linux-2.6.25-rc6_2.6.25-rc6-2_amd64.deb
}}}

== Reboot under newly built Linux kernel ==

{{{
$ sudo reboot
}}}

== Test your issue under this Linux kernel ==

If the feature you are testing works in the newly booted Linux kernel, mark the commit as good:

{{{
$ cd ~/src/linux
$ git bisect good
}}}

If it does not work, mark the commit as bad:

{{{
$ cd ~/src/linux
$ git bisect bad
}}}

These commands will then choose a new commit to be configured, built and tested. Repeat the procedure until `git bisect` decides one particular commit is at fault. Once the commit has been found you can inspect it to see why it caused the regression and or file a bug about it against the Debian Linux kernel package.

----
CategoryGit

On this page is described an easy way to debug upstream Linux kernel issues, by using "git bisect". The idea of this tool is to track down a particular issue (or regression) by selecting the faulty revision.

/!\ Before starting to bisect, it would be a good idea to use Debian's wayback machine to narrow down the range of Linux kernel versions where the faulty revision may occur so that the amount of compiling is reduced somewhat.

If the issue does not appear in the upstream Linux kernel, but does appear in the Debian Linux kernel images, please refer to the Debian Linux Kernel Handbook for information about how to debug and report the issue.

Use cases

  • Hibernation works under 2.6.25-rc6 but not anymore under 2.6.25-rc7 and I would like to know which revision caused the regression.

Needed tools

For these regression tests, you will need the following packages (to install with your favorite package manager):

$ sudo apt install git gitk fakeroot

Alternatively, just install all the Linux kernel build-dependencies and git:

$ sudo apt install git
$ sudo apt build-dep linux

At times the Linux kernel and the toolchain versions might be out of sync, resulting in lots of strange build failures. In that case, it would be a good idea to use a Debian stable chroot to do the building.

It is recommended to have your shell prompt display git repository status information. The git package includes prompt support for bash and zsh.

Getting the Linux git source

In this step, we will get a local copy of the whole Linux kernel source code by using Linux's distributed version control system: git.

Move to your source directory

$ cd ~/src

Clone Linux git repository

This will download a lot of data and thus may need a lot of time.

You can either clone Linus Torvald's main Linux repository, or the repository containing the Linux kernel stable release series. The stable repository contains more commits and is useful for bisecting bugs introduced by stable updates.

$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/

Separate the wheat from the chaff

In this step, we will mark the versions good or bad.

Start the git-bisect process

$ git bisect start

Mark the ''good'' version

For example:

$ git bisect good v2.6.25-rc6 

Mark the ''bad'' version

For example:

$ git bisect bad v2.6.25-rc7 

At this point, this command should answer you something like that :

Bisecting: 182 revisions left to test after this
[2c7871982cf27caaddbaeb7e2121ce1374b520ff] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6.25

The interesting part is the number (in our case, 182) which is the number of Linux kernel revisions that lay between your good and your bad.

Build your Linux kernel

In this step, we will configure the future Linux kernel according to the actual configuration and then compile the new Linux kernel.

Copy a valid configuration

For example, the configuration of the running Linux kernel:

$ cp /boot/config-$(uname -r) .config

Put git commit IDs into versions

This ensures that you can easily map the installed Linux kernel images to a git commit ID. If you do not have much disk space on /boot you should either skip this step or be prepared during testing to remove some Linux kernel images that have already been tested.

$ scripts/config --enable LOCALVERSION_AUTO

Disabling debug information

If your bisect won't need to debug the kernel, it is a good idea to disable debug info to reduce the build time, especially for the linux-image-*-dbg package, which will be very large and can take a long time to compress.

$ scripts/config --disable DEBUG_INFO

Disable Secure Boot

If you are using a config from a Linux kernel from Debian or other distro kernel that has SecureBoot enabled using the distro key, then your builds will fail because you don't have a copy of the private distro key. First disable Secure Boot in your UEFI boot setup and then disable it in your Linux kernel config:

$ scripts/config --disable SYSTEM_TRUSTED_KEYRING
$ scripts/config --set-str SYSTEM_TRUSTED_KEYS ''

Disable Stack Protector

If you are using a newer GCC while bisecting older kernel versions, you may need to disable the stack protector during the bisect.

$ scripts/config --disable STACKPROTECTOR_STRONG

Configure the Linux kernel

If you do not know the answer to each question, you can just accept the defaults:

$ make olddefconfig

If you want to customise the configuration, you can run this instead:

$ make oldconfig

If you want to reduce the build time to just what you need you can use this:

$ make localmodconfig

If you want are doing this for a remote host, copy the remote lsmod output locally and use this instead:

$ make LSMOD=$(pwd)/lsmod localmodconfig

Build the Linux kernel

This will take a lot of time and computation power too. Ensure your system is adequately cooled so it will not overheat.

# Detect the amount of CPU cores you have
$ jobs=$(nproc --all)
# Detect if the Linux kernel version needs fakeroot or not
$ fakeroot=$(grep -q 'Rules-Requires-Root: no' scripts/package/mkdebian || grep -q fakeroot scripts/Makefile.package scripts/package/Makefile || echo fakeroot)
# Build Debian binary packages of the Linux kernel version
# Make sure you include the $ sign before fakeroot on the next line
$ $fakeroot make -j$jobs bindeb-pkg

Transfer the Linux kernel image package

If you are building on a different system to the one you are testing on, at this point you should transfer the Linux kernel image package to the system you are testing on.

It is recommended to do the testing in a spare system with the same hardware, or virtual machine (for bisects of code that is not hardware-specific) to avoid any old fixed Linux kernel bugs affecting your main system, especially old fixed issues that may corrupt your data storage.

Install the newly created Linux kernel

According to the name printed by the last command.

$ sudo apt install ../linux-2.6.25-rc6_2.6.25-rc6-2_amd64.deb

Reboot under newly built Linux kernel

$ sudo reboot

Test your issue under this Linux kernel

If the feature you are testing works in the newly booted Linux kernel, mark the commit as good:

$ cd ~/src/linux
$ git bisect good

If it does not work, mark the commit as bad:

$ cd ~/src/linux
$ git bisect bad

These commands will then choose a new commit to be configured, built and tested. Repeat the procedure until git bisect decides one particular commit is at fault. Once the commit has been found you can inspect it to see why it caused the regression and or file a bug about it against the Debian Linux kernel package.


CategoryGit