Debugging a kernel issue

Freebirth Toad · n00b Joined: 17 Feb 2011 Posts: 4

I have encountered a bug with the vanilla sources kernel. Normally, I would go to kernel.org's bugzilla, but it's down. Before I brave the LKML, I thought I'd have a go at it here.

Any kernel newer than 2.6.32 (I have tried 2.6.33, 2.6.36, 2.6.37, 2.6.39, & 3.0.4) causes silent data corruption. I only know because random programs start segfaulting (sometimes even getting booted is impossible). If I md5sum the contents of /usr/bin from a known working kernel, I can reliably get a md5sum failure of at least one file if I check the md5sums from a newer (broken) kernel. I'm using the default kernel configuration with only minimal changes to support my hardware, and the problem persists. There's no oops, or any messages in the logs (other than random daemons segfaulting because corrupted binaries are being loaded).

The symptoms are those of hardware failure, except that my SMART stats are fine (no relocated blocks), and I pass memtest86 running for hours. But most importantly, I have run the old kernel without a crash or any sign of problem for around 95 days. It was suspended to disk for a good portion of that, but this was while I was writing my thesis, and I didn't lose anything. Everything seems rock solid.

If I had to guess, I would say that it has something to do with the rewrite to sata_via portion of libata that got done in 2.6.33. It's probably some sort of weird interaction between my particular drive controller and my particular models of hard drives. Googling has turned up nothing, but I'm sort of at a loss for what I should even be searching for.

My question is procedural: what should I do next to figure this out? How can I get more information about what is occurring?

Jaglover · Posted: Mon Oct 03, 2011 12:22 pm Post subject:

I had similar problem once, power supply was out of specs.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

djinnZ · Posted: Mon Oct 03, 2011 12:49 pm Post subject:

I f the requests of the hardware are more than the capacity of the PSU I have experienced semi-random issues.

Another hardware issue (interrupts and DMA conflicts and so more) especially on the I/O controllers are often reported as memory issues by the kernel.
The only message printed is a memory corruption before the system will reset itself.

I have not found any other solution than waste for the problematic mother board of mine.

In the other case I have solved with a psu upgrade but i am speaking about a box with 11 storage units (6 pata disks + pata dvd + 2 sata + 3 usb + one usb self alimented ) connected and 7 pci devices (2 controller + 3 net + 1 wifi + graphic card) and the only symptom was a semi-random reset (often after I start to move files between different devices in sleep mode).

Not remembere if the 2.6.32 support it but you can think about use it as recover (see in documentation for kdump) to debug after crash.

Thats are my two cents.
_________________
scita et risus abundant in ore stultorum sed etiam semper severi insani sunt :wink:

mala tempora currunt...mater stultorum semper pregna est :evil:

Murpy'sLaw:If anything can go wrong, it will - O'Toole's Corollary:Murphy was an optimist :wink:

Freebirth Toad · n00b Joined: 17 Feb 2011 Posts: 4

djinnZ · Posted: Tue Oct 04, 2011 11:38 am Post subject:

First verify the configuration (no oldconfig) and try to disable the heap randomization (but I suppose you are not try to run legacy binaries) and "uninline functions" in kernel config.

memtest results are not 100% sure, the phisical memory can be damaged even if the tests are negative. Try to swap positions if its possible.
Exclude any hardware issue is important.
On the MB I have wasted memtest has never report anything bugs but recompile gcc tree times was resulting in a broken gcc.
And I have forget than I have two identical abit MB running, now.

Another source of strange issues, strictly software related, can be teh libc and the base runtime components. Have you recompiled it? (make a binary package for backup first)

Also its possible than some files (of librery or binaries) are corrupted. The crash involve only some applications or all the applications than depends from a specific library (or try to access a specific device)?

I suggest you to verify first if they are more common elements in the issues/crashes not only the kernel upgrade.

Not sure but the ksm was introduced with the 2.6.33 as I remember.

As start you can compare the results of a core dump before and at crash of the single program.
_________________
scita et risus abundant in ore stultorum sed etiam semper severi insani sunt :wink:

mala tempora currunt...mater stultorum semper pregna est :evil:

Murpy'sLaw:If anything can go wrong, it will - O'Toole's Corollary:Murphy was an optimist :wink:

Freebirth Toad · n00b Joined: 17 Feb 2011 Posts: 4

Jaglover · Posted: Sun Oct 16, 2011 11:05 pm Post subject:

It may not be that simple. I've to cases for you, in first case everything worked great but ogg encoder produced broken files. Trying to solve this I finally downclocked the CPU and it magically fixed ogg encoding. In second case I had file corruption with an AMD, with an addon SCSI controller. I switched to onboard EIDE - same thing, random file corruption. I replaced the CPU, still all same. The last thing I swapped out was PSU and that fixed it.
My point is newer kernels may have a rewritten driver which exposes your hardware problem.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

djinnZ · Posted: Tue Oct 18, 2011 2:54 pm Post subject:

DirtyHairy · Posted: Thu Oct 20, 2011 8:53 am Post subject:

As you know that 2.6.32 works and 2.6.33 doesn't, you could try bisecting using git to isolate the change which caused the problem. However, I'd also consider the possiblity that you are witnessing a hardware error exposed by the newer kernel code (it could be something which causes the newer version of a driver to misbehave and corrupt memory).