Remote Server Hanging Nightly - High Load / Disk I/O

[Crimjob]

Hey Guys,

I've had a remote server for about 3 years now, but only recently (last 3 nights) has it decided to hang up on me. It's happened the last 3 nights in a row, all within 20 minutes of each other.

The only thing I'm noticing is a Zabbix warning about Disk I/O about 15-20 minutes before the server stops responding, and I've had to do a power cycle to recover each time (I've let it sit for 8+ hours to no avail).

I'm desperately trying to track down what is going on - I've disabled all my cron jobs around that time etc., and I've got extensive logs, all of which show perfectly normal operation until I suddenly lose connection with the server and seem to be able to only recover via rebooting.

I've checked SMARTCTL and the disk is clean. I have not had a chance to check memory (a bit more difficult due to it's remoteness), but I have not received any telltale signs in dmesg or similar that there would be something wrong with the RAM.

My best thing to go off of at this point is some way of manually monitoring the Disk I/O - but I'm not entirely sure how to get that down to a graphable solution.

I'm wondering if anyone else has any ideas that might help me log more data or assist in finding the culprit?

Thanks in advanced for any suggestions
_________________
"Who are you to judge the life I live? I know I'm not perfect and I don't live to be, but before you start pointing fingers... make sure your hands are clean." ~Bob Marley

[jpc22]

You could forward your machine stats with conky over internet.

If you want a good conky config to get started here's mine:
http://pastebin.com/9c1MkCMs

covers cpu/ram/disk/network with both graphics and numbers, it should tell you all about hardware if you can get it running remotely and forward the results.