anatomy of a crash

mno · Posted: Fri Jan 25, 2008 1:24 am Post subject: anatomy of a crash

Hello everyone,

A few days ago, my server running Gentoo all of a sudden disappeared. I have just been working with on a website on it about an hour ago, and when I went back to continue working on the site an hour later, I noticed that all systems were down. No SSH, FTP, etc, no ping. After a remote power reboot, the server came back up just fine. So, I went in to dig around and try to discover the cause.

I've spent the past oh 2 days digging around log files and running some verifications (such as disk check, etc), but all come back without errors. Although, after the remote reboot, there were a significant amount of blocks that had bad addresses, but those disappeared after a second proper reboot. I assume those were just FOL and occured due to me cutting power in the middle of it doing something.

In any case, I can't discover what went wrong. I have a hunch that the actual server didn't flop, but that it was an error with the networking. Whether on my end or on my colo's end with their switch all of a sudden refusing to communicate. But I can't seem to point my finger on anything. So, my question is, how does one go about disecting what went wrong when the standard places don't show anything as wrong? Just in case, here's my emerge --info.

Thanks in advance,
Max

schachti · Posted: Fri Jan 25, 2008 6:37 am Post subject:

You can use smartmontools to check your harddisk, maybe the harddisk is the reason for the problem. You could also test your cpu using cpuburn (the computer might crash during this test).
_________________
Never argue with an idiot. He brings you down to his level, then beats you with experience.

How-To: Daten verschlüsselt auf DVD speichern.

mno · Posted: Fri Jan 25, 2008 6:38 am Post subject:

The harddisks have already been checked. They're connected to a 3ware RAID controller. I verified the array several times, always was perfectly stable.
_________________
"Hello and goodbye. As always." | You can't use   here?? | Unanswered

pelkeyj · n00b Joined: 04 Nov 2006 Posts: 17

Not for this crash, but for future ones:
If the kernel has the netconsole module, you can have it send console messages (usually including ones during a crash after syslog or hard disks are no longer functional) to another computer via UDP packets.
Documentation is in /usr/src/linux/Documentation/networking/netconsole.txt, and changing the kernel console loglevel is done by modifying /proc/sys/kernel/printk (details in proc(5) man page).

mno · Posted: Fri Jan 25, 2008 9:30 am Post subject:

Interesting! What program would the other computer need to have running to receive the packets, and on what port? Though I assume it's the same (netconsole)? I guess it's all in the docs then. Thanks!
_________________
"Hello and goodbye. As always." | You can't use   here?? | Unanswered

StarDragon · Guru Joined: 19 Jun 2005 Posts: 390 Location: tEXas

Did the box ever bounce? If not, then I suspect it was a switch issue. Maybe checking your logs on the switch box might help.

mno · Posted: Fri Jan 25, 2008 10:13 pm Post subject:

Bounce, what do you mean?
_________________
"Hello and goodbye. As always." | You can't use   here?? | Unanswered