Translation(s): English - Русский


How To Get a Meaningful Backtrace

This page will attempt to explain how to get a meaningful debugging backtrace from a reproducible program crash. For this example, we will install the package with debugging symbols for "hello" or if this is not available rebuild & install the "hello" package so that we keep debugging information.

For reference, a "#" at the beginning of a line means that what follows is a command that has to be executed as root (or sudo). A "$" at the beginning means that what follows is a command that should be executed as your normal user. A "(gdb)" at the beginning means that you should be typing the rest of the command line at the gdb (GNU Debugger) prompt.

# apt install gdb

Installing the debugging symbols

Make sure you have the AutomaticDebugPackages archive listed in your SourcesList:

deb http://deb.debian.org/debian-debug/ $RELEASE-debug main
# for security updates
deb http://deb.debian.org/debian-debug/ $RELEASE-proposed-updates-debug main

So, depending on your release, it might look like this:

deb http://deb.debian.org/debian-debug/ buster-debug main
# for security updates
deb http://deb.debian.org/debian-debug/ buster-proposed-updates-debug main

Other possibilities are:

deb http://deb.debian.org/debian-debug/ testing-debug main
deb http://deb.debian.org/debian-debug/ unstable-debug main
deb http://deb.debian.org/debian-debug/ experimental-debug main

If you are using Debian buster or later, install debian-goodies and run the find-dbgsym-packages script to find which dbgsym packages to install:

$ find-dbgsym-packages /usr/bin/hello 
hello-dbgsym

Otherwise, to see if the package you’re trying to debug includes a -dbgsym or -dbg package (e.g. for the hello source package, a hello-dbgsym or hello-dbg package exists), search for it in your package manager. On the command line you can use the following command.

$ apt-cache search hello dbg
p   hello-dbgsym            - Debug symbols for hello

If that doesn't have your debug package either, you could try to rebuild the package as explained in the next section.

If an appropriate package was found you should just be able to install the appropriate debug package and skip the rebuilding process in the following instructions and go straight to running gdb.

# apt install hello-dbgsym

Rebuilding the package you’re debugging

You may skip this section if you were able to install the necessary debug symbols package(s) from the previous section.

# apt install build-essential fakeroot gdb
# apt build-dep hello

$ DEB_BUILD_OPTIONS="nostrip noopt" apt -b source hello

# apt install ./hello_2.1.1-4_amd64.deb

$ file /usr/bin/hello # output should contain "not stripped"

Running gdb

Batch mode

For including a backtrace in a bug report, you can run this command and then try to reproduce your crash:

$ gdb -batch -n -ex 'set pagination off' -ex run -ex bt -ex 'bt full' -ex 'thread apply all bt full' --args hello

Interactively

Use this method if you need to interactively run gdb.

If the problem seems to be in a major library such as libc6, xlibs, or libgtk2.0-0, you’ll want to install the appropriate -dbg package (e.g. libc6-dbg in the case of libc6) and then run the problematic program again under gdb.

Often, you will see a backtrace where one or more of the top lines is in malloc() or g_malloc(). When this happens, chances are your backtrace isn’t very useful. The easiest way to find some useful information is to set the environment variable MALLOC_CHECK_ to a value of 2. You can do this while running gdb by doing this:

 $ MALLOC_CHECK_=2 gdb hello

Advanced gdb commands

Debugging X Errors

If a GTK program has received an X error; i.e. you see a message of the form:

The program 'preview1' received an X Window System error.

then you can try running the program with --sync, and break on the gdk_x_error function in order to obtain a backtrace, thus:

 (gdb) break gdk_x_error
 (gdb) run --sync

Core dump

For a crashed program, a core dump file may be used to evaluate post-mortem snapshot of the program's state at the time of the crash. You can check resource limitation including the maximum size of core files created by typing "ulimit" on the shell prompt. You may set the core file size to unlimited using "ulimit -c unlimited" and/or with some other configuration files (/etc/security/limits.conf, /usr/lib/sysctl.d/50-coredump.conf, and systemd service files).

The location of core dump file is specified by /proc/sys/kernel/core_pattern. If the systemd-coredump package is installed under systemd, it may look like.

|/lib/systemd/systemd-coredump ...

Then the lz4-compressed core dump file is generated in /var/lib/systemd/coredump upon crash. You can get information on this compressed core file with:

 $ coredumpctl info -1

You can use this compressed core file with gdb by:

 $ coredumpctl gdb -1

See more:

--AriPollak and ?LoicMinier

CategoryDebugging