Commit ab0e44c1 authored by Mauro Carvalho Chehab's avatar Mauro Carvalho Chehab Committed by Jonathan Corbet

bug-hunting.rst: update info about bug hunting

The document shows a really old procedure for bug hunting that
nobody uses anymore. Remove such section, and update the
remaining documentation to reflect the procedures used
currently.
Signed-off-by: default avatarMauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent e7f08ffb
Bug hunting Bug hunting
+++++++++++ +++++++++++
Last updated: 20 December 2005 Last updated: 28 October 2016
Introduction Introduction
============ ============
...@@ -20,120 +20,62 @@ Before you submit a bug report read ...@@ -20,120 +20,62 @@ Before you submit a bug report read
Devices not appearing Devices not appearing
===================== =====================
Often this is caused by udev. Check that first before blaming it on the Often this is caused by udev/systemd. Check that first before blaming it
kernel. on the kernel.
Finding patch that caused a bug Finding patch that caused a bug
=============================== ===============================
Finding using ``git-bisect``
----------------------------
Using the provided tools with ``git`` makes finding bugs easy provided the bug Using the provided tools with ``git`` makes finding bugs easy provided the bug
is reproducible. is reproducible.
Steps to do it: Steps to do it:
- start using git for the kernel source - build the Kernel from its git source
- read the man page for ``git-bisect`` - start bisect with [#f1]_::
- have fun
Finding it the old way
----------------------
[Sat Mar 2 10:32:33 PST 1996 KERNEL_BUG-HOWTO lm@sgi.com (Larry McVoy)]
This is how to track down a bug if you know nothing about kernel hacking.
It's a brute force approach but it works pretty well.
You need:
- A reproducible bug - it has to happen predictably (sorry)
- All the kernel tar files from a revision that worked to the
revision that doesn't
You will then do:
- Rebuild a revision that you believe works, install, and verify that.
- Do a binary search over the kernels to figure out which one
introduced the bug. I.e., suppose 1.3.28 didn't have the bug, but
you know that 1.3.69 does. Pick a kernel in the middle and build
that, like 1.3.50. Build & test; if it works, pick the mid point
between .50 and .69, else the mid point between .28 and .50.
- You'll narrow it down to the kernel that introduced the bug. You
can probably do better than this but it gets tricky.
- Narrow it down to a subdirectory $ git bisect start
- Copy kernel that works into "test". Let's say that 3.62 works, - mark the broken changeset with::
but 3.63 doesn't. So you diff -r those two kernels and come
up with a list of directories that changed. For each of those
directories:
Copy the non-working directory next to the working directory $ git bisect bad [commit]
as "dir.63".
One directory at time, try moving the working directory to
"dir.62" and mv dir.63 dir"time, try::
mv dir dir.62 - mark a changeset where the code is known to work with::
mv dir.63 dir
find dir -name '*.[oa]' -print | xargs rm -f
And then rebuild and retest. Assuming that all related $ git bisect good [commit]
changes were contained in the sub directory, this should
isolate the change to a directory.
Problems: changes in header files may have occurred; I've - rebuild the Kernel and test
found in my case that they were self explanatory - you may - interact with git bisect by using either::
or may not want to give up when that happens.
- Narrow it down to a file $ git bisect good
- You can apply the same technique to each file in the directory, or::
hoping that the changes in that file are self contained.
- Narrow it down to a routine $ git bisect bad
- You can take the old file and the new file and manually create depending if the bug happened on the changeset you're testing
a merged file that has:: - After some interactions, git bisect will give you the changeset that
likely caused the bug.
#ifdef VER62 - For example, if you know that the current version is bad, and version
routine() 4.8 is good, you could do::
{
...
}
#else
routine()
{
...
}
#endif
And then walk through that file, one routine at a time and $ git bisect start
prefix it with:: $ git bisect bad # Current version is bad
$ git bisect good v4.8
#define VER62
/* both routines here */
#undef VER62
Then recompile, retest, move the ifdefs until you find the one .. [#f1] You can, optionally, provide both good and bad arguments at git
that makes the difference. start::
Finally, you take all the info that you have, kernel revisions, bug git bisect start [BAD] [GOOD]
description, the extent to which you have narrowed it down, and pass
that off to whomever you believe is the maintainer of that section.
A post to linux.dev.kernel isn't such a bad idea if you've done some
work to narrow it down.
If you get it down to a routine, you'll probably get a fix in 24 hours. For further references, please read:
My apologies to Linus and the other kernel hackers for describing this - The man page for ``git-bisect``
brute force approach, it's hardly what a kernel hacker would do. However, - `Fighting regressions with git bisect <https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html>`_
it does work and it lets non-hackers help fix bugs. And it is cool - `Fully automated bisecting with "git bisect run" <https://lwn.net/Articles/317154>`_
because Linux snapshots will let you do this - something that you can't - `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_
do with vendor supplied releases.
Fixing the bug Fixing the bug
============== ==============
...@@ -141,13 +83,16 @@ Fixing the bug ...@@ -141,13 +83,16 @@ Fixing the bug
Nobody is going to tell you how to fix bugs. Seriously. You need to work it Nobody is going to tell you how to fix bugs. Seriously. You need to work it
out. But below are some hints on how to use the tools. out. But below are some hints on how to use the tools.
objdump
-------
To debug a kernel, use objdump and look for the hex offset from the crash To debug a kernel, use objdump and look for the hex offset from the crash
output to find the valid line of code/assembler. Without debug symbols, you output to find the valid line of code/assembler. Without debug symbols, you
will see the assembler code for the routine shown, but if your kernel has will see the assembler code for the routine shown, but if your kernel has
debug symbols the C code will also be available. (Debug symbols can be enabled debug symbols the C code will also be available. (Debug symbols can be enabled
in the kernel hacking menu of the menu configuration.) For example:: in the kernel hacking menu of the menu configuration.) For example::
objdump -r -S -l --disassemble net/dccp/ipv4.o $ objdump -r -S -l --disassemble net/dccp/ipv4.o
.. note:: .. note::
...@@ -157,7 +102,7 @@ in the kernel hacking menu of the menu configuration.) For example:: ...@@ -157,7 +102,7 @@ in the kernel hacking menu of the menu configuration.) For example::
If you don't have access to the code you can also debug on some crash dumps If you don't have access to the code you can also debug on some crash dumps
e.g. crash dump output as shown by Dave Miller:: e.g. crash dump output as shown by Dave Miller::
EIP is at ip_queue_xmit+0x14/0x4c0 EIP is at +0x14/0x4c0
... ...
Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00 Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
00 00 55 57 56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08 00 00 55 57 56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
...@@ -185,16 +130,25 @@ e.g. crash dump output as shown by Dave Miller:: ...@@ -185,16 +130,25 @@ e.g. crash dump output as shown by Dave Miller::
mov 0x8(%ebp), %ebx ! %ebx = skb->sk mov 0x8(%ebp), %ebx ! %ebx = skb->sk
mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt
gdb
---
In addition, you can use GDB to figure out the exact file and line In addition, you can use GDB to figure out the exact file and line
number of the OOPS from the ``vmlinux`` file. If you have number of the OOPS from the ``vmlinux`` file.
``CONFIG_DEBUG_INFO`` enabled, you can simply copy the EIP value from the
OOPS:: The usage of gdb requires a kernel compiled with ``CONFIG_DEBUG_INFO``.
This can be set by running::
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
EIP value from the OOPS::
EIP: 0060:[<c021e50e>] Not tainted VLI EIP: 0060:[<c021e50e>] Not tainted VLI
And use GDB to translate that to human-readable form:: And use GDB to translate that to human-readable form::
gdb vmlinux $ gdb vmlinux
(gdb) l *0xc021e50e (gdb) l *0xc021e50e
If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
...@@ -204,14 +158,32 @@ offset from the OOPS:: ...@@ -204,14 +158,32 @@ offset from the OOPS::
And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled:: And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::
make vmlinux $ make vmlinux
gdb vmlinux $ gdb vmlinux
(gdb) l *vt_ioctl+0xda8
0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
288 {
289 struct vc_data *vc = NULL;
290 int ret = 0;
291
292 console_lock();
293 if (VT_BUSY(vc_num))
294 ret = -EBUSY;
295 else if (vc_num)
296 vc = vc_deallocate(vc_num);
297 console_unlock();
or, if you want to be more verbose::
(gdb) p vt_ioctl (gdb) p vt_ioctl
(gdb) l *(0x<address of vt_ioctl> + 0xda8) $1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
(gdb) l *0xae0+0xda8
or, as one command:: You could, instead, use the object file::
(gdb) l *(vt_ioctl + 0xda8) $ make drivers/tty/
$ gdb drivers/tty/vt/vt_ioctl.o
(gdb) l *vt_ioctl+0xda8
If you have a call trace, such as:: If you have a call trace, such as::
...@@ -221,17 +193,11 @@ If you have a call trace, such as:: ...@@ -221,17 +193,11 @@ If you have a call trace, such as::
[<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
... ...
this shows the problem in the :jbd: module. You can load that module in gdb this shows the problem likely in the :jbd: module. You can load that module
and list the relevant code:: in gdb and list the relevant code::
gdb fs/jbd/jbd.ko
(gdb) p log_wait_commit
(gdb) l *(0x<address> + 0xa3)
or::
(gdb) l *(log_wait_commit + 0xa3)
$ gdb fs/jbd/jbd.ko
(gdb) l *log_wait_commit+0xa3
Another very useful option of the Kernel Hacking section in menuconfig is Another very useful option of the Kernel Hacking section in menuconfig is
Debug memory allocations. This will help you see whether data has been Debug memory allocations. This will help you see whether data has been
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment