System Instability... (no longer resolved, new crash output)

[lmegliol]

I recently installed Gentoo on a system I've had running Linux for 5 years. It was running RedHat previously, but a hard drive crashed required a fresh Linux install and I went with Gentoo. The only changes between the system before Gentoo and after are that a 3ware SATA RAID controller has been installed and 2GB of RAM added. I ran memtest86 on the RAM before going further with the installation and it checked out.

The system runs Apache2 with PHP5 and mod_perl, and I recently experienced 2 kernel panics in a 6 hour period. I'm nowhere near the system, so I had to have someone take a picture of the console so I could see the errors. Here is a transciption of the errors as I can read them.

[msalerno]

Wow, typing that out warrents a reply.

what is the output of your emerge --info and emerge -pv apache ?

Have you re-emerged apache since you started getting the problems? Has it helped?

[bonbons]

In order to get a more useful trace, could you recompile your kernel with debugging symbols enabled?
From just the addresses it's hard to determine what caused the panic or where it was caused (the addresses to code mapping would at best be possible with the kernel config)

To do so, enable the following:

[lmegliol]

I'll answer the first post first, and then get on top of the kernel and netconsole afterward. (Thanks for the netconsole info.)

[lmegliol]

In which package does one get the nc program? I thought it might be cancd, but that doesn't seem to be the case.

[lmegliol]

Answer to my own question... nc is in net-analyzer/netcat.

[lmegliol]

I'm changing the subject on this one to something much more general.

Here's the deal: As I stated, this system ran fine for 5 years with RedHat. When the hard drive crashed, the system was shipped out to me so I could replace the hard drive. I installed the 3ware controller and drives and started the process of installing Gentoo.

The Gentoo install process was a nightmare. I've installed Gentoo before, and never had these types of problems. The system kept crashing mid-install. At the time I chalked this up to the installation program not being entirely compatible with my hardware. This didn't make much sense to me, but this system had run non-stop for 5 years without any strange crashes, until the hard drive crashed.

I had to restart the install process dozens of times. I suppose this should have been my first indication that something else was going on, but I finally pushed through thinking that once I got it installed I'd recompile the kernel to something a little more stable for my hardware. Before I compiled a new kernel I tried installing some other necessary packages. I got a few random segmentation faults. Some I could duplicate a few times, so I assumed they were code problems, but then they'd work. Strange. But I pushed through thinking that a kernel recompile would fix it.

Once the kernel was recompiled, the system ran beautifully. No other segmentation faults or compilation problems occurred. I figured the problem was fixed. I had no crashes, no problems, nothing. I installed some new RAM and ran memtest86 on it and everything looked great. I gave the system a clean bill of health and shipped it back.

Once it arrived, the system was hooked up, fired up and visitors to my web site starting applying a load to it. 36 hours passed without incident, and I had no reason to believe anything was wrong. Then it crashed with a kernel panic.

I don't have a copy of the messages from that crash. We rebooted the system without trouble and 5 hours later it crashed with the messages I entered above. Reboot, but now things get interesting.

On reboot the first time the reboot hangs somewhere in the middle of starting services. Reboot again. This time I get some drive errors and the boot process stops mid way through. I try to tackle those and reboot again. This time it gets to a prompt, but for some reason the hostname is wrong. It says "none login:". Strange. Reboot again. Works this time.

Then I tried to following recommendation above regarding compiling the kernel with netconsole. The kernel recompiled, but I couldn't get netconsole to work. I realized a couple of mistakes in the recompile, so I recompiled again. I tried to reboot. /sbin/shutdown appeared to no longer be working. The system just sits there. I have someone manually reboot the system. Hangs mid boot again, right near starting eth.lo. Try again. Stops with another drive inconsistency. Probably from the nasty reboots.

Long story short, something is royally messed up here. And it makes no sense to me at all that the system was working fine for 5 years and then the entire thing goes out after a drive crash. I suppose it is possible that the original drive crash was not a drive crash at all, but rather a motherboard problem of some sort.

So basically that's where I am with this. I'm not sure whether I should try replacing all major components one at a time until this thing works, or whether I should scrap the whole system, or what. My first step will be to run memtest86 again. After that... any thoughts?

[lmegliol]

One more thing. I did have one problem when compiling kernels. My system has three network adapters. When I tried to compile the drivers for all three of the adapters into the kernel, the system would not boot. I don't remember the exact problem, but it just wouldn't work. When I compiled them as modules, the system worked fine.

[bonbons]

From your story above there could be some issue with disk access...
If memtest reveals no issue, you may want to do some load-test against your disk drives (check the disks and the controller), like running a benchmark, or doing mixtures of reading and writing from filesystem (extract kernel sources, search for files and such, even "cat /dev/<partition> > /dev/null")

During this process, looking at kernel output may be helpful (dmesg -n
When the system freezes again, you may hopefully see some complaints from the kernel on your console. (with that level any message should also be sent out using netconsole... if you have the opportunity to do so, you may want to experiment with netconsole on your LAN first to get some feeling. As you have many interface on your failing box, be careful selecting the right one for netconsole, you can modprobe and rmmod netconsole to change its ettings)

If possible, monitor temperatures and voltages as well

[lmegliol]

I finally got netconsole working on this system and just started some load testing. Netconsole apparently only works with eth0 on my system, and I was trying to use it over eth2, a back-end network for a few of my servers. I changed it to eth0 and all is well.

So, in an attempt to force this system to crash, I've been doing all-of-the-above recommended methods of putting load on the system. I've got three shell scripts running infinite loops unpacking kernel source, searching for files on the system, and cat'ing /dev/sda7 to /dev/null. The load on the system is around 4.00, and has been that way for the last 6 hours. So far no crashes.

I noticed that the memory is only 75% used during these load tests, and I'm not sure what implications that may have regarding the effectiveness of this load testing.

By the way, this is the same system that I am working to debug in this thread also.

Briefly going under the assumption that because the memory all seems to check out, and because the load testing isn't causing a crash, then the system must be fine... It can't be. I'd say that about 50% of the time, the boot process fails unexpectedly. (I haven't seen this happen since getting netconsole to work, so I can't report on it.) Something strange is up.

I appreciate everyone who has helped me thus far and will report more when I know more. Any new thoughts or tips are greatly appreciated.

Thanks.

[lmegliol]

System stability has improved on this system, though I don't have an exact reason why. There were some memory problems that may have been the source. I removed an older, slower memory chip that may have conflicted with two newer chips.

System is now in production and there have been no crashes since the extra chip was removed. Marking this thread as resolved.

[lmegliol]

I've removed the [solved] off this topic, because the system instability is back. The system ran perfectly for the last week or two. I rebooted earlier today and within an hour the system started crashing.

Luckily I have netconsole working. The kernel was also recompiled as described above. Here is the output from the crash...