rc-script disaster...? Possibly CRITICAL, rm -Rf / bug!

[progman32]

...well, I couldn't make this up if I tried. So I sit down this evening to work, and guess what I find: NOTHING.
All (well, most of) my files are gone, school things, programs, Gentoo, everything. Let me explain in more detail:

-I boot up my computer as normal, and notice many, many "command not found" errors in the init scripts.
-Most init scripts fail, and I get a text console as X fails to start. The hostname does not even get set.
-I type 'reboot' (don't ask me why I thought it may fix something, it was like 3AM).
-The init scripts that started stop properly.
--prepare for fail of epic proportions--
-Just before the point where a reboot would physically occur, the HDD light goes on solid. It showed no signs of stopping and after 20-30 seconds, it worried me. What could possibly be in the cache to flush after the system had been barely booted up for 60 seconds?
-I use the 'magic SysRq key' to remount everything RO. At this point, hundreds of messages flash across my screen. To my horror, they were error messages from rm (!) complaining of being unable to remove file foo, foo being each and every file on each and every mounted filesystem. Fearing the worst, I boot into a livecd, and find:
-my /home partition is completely empty (ext3)
-my /mnt/stuff partition is halfway empty (ext3, I interrupted the recursive rm while it was deleting files from here)
-my / partition is a mass of confusion (reiserfs). My /etc directory is gone, but in its place are directories with the following names: options, scheduled, snapshot, softscripts, softscripts.new, started, starting, stopping, and wasinactive. There are also two extra files in the root called softlevel, depcache, and deptree. All the above are empty, except for softlevel, which has "shutdown" as it's contents. Random files are missing.

I'm completely stumped. It's not filesystem corruption, as the mass deletion spanned many partitions of different filesystems types. fsck showed no errors, except for some incomplete unlinks in my /mnt/stuff, which is to be expected since I cut off write access. Shutdown before massive death was clean, no issues whatsoever.

I fear this may have to do with the new baselayout... If this is the case, we are all screwed, heh.

So if I may, I'l just scream WHAT THE HELL JUST HAPPENED GUYS? SERIOUSLY, WTF.

Fortunately, /var survived mostly intact, so I can provide logs. log/messages did not offer anything interesting except for a null pointer error.

I'd post an emerge --info, but I don't even have bash anymore:
# ls /mnt/gentoo/bin/bash
ls: cannot access /mnt/gentoo/bin/bash: No such file or directory

More info:
gentoo-sources-2.6.24-r2
... I can't remember anything else relevant, really. It's kind of, um, gone.
System is completely stable and rock solid, power supply fine, RAM fine, CPU fine, disk fine.
core 2 quad qx6700 running 64-bit gentoo.

I am willing to provide full disk images if necessary. I will try a rebuild-tree on a copy of my root, see what that does.

[schachti]

Maybe someone was able to hack your computer? Did you use some not up-to-date software (older firefox for example)? sshd running and having a user with a weak password? Doing common work (browsing, listening to music etc.) as root?
_________________
Never argue with an idiot. He brings you down to his level, then beats you with experience.

How-To: Daten verschlüsselt auf DVD speichern.

[pathfinder]

what i would have done immediately was to cut off the ethernet connection, the wifi connection, etc, by shutting down the router. Though if it s malevolous, it must be somehting networkless, like some script executed at a given time or a given hour.
Is your disk OK?

Then, well, i hope you have backups.

from a livecd, can t you just see what services are present, and can you try if there s one service pretty weird?
do you have hdparm turned on?


you should try to use ddrescue to back up your data.
maybe a cron / at task you did not know to be there?
It s a little bit bizarre. :S
I l try to think about it!

[pathfinder]

maybe var is intact because protected from rm operations?
did you use overlays? do you remember something installed experimental?

[progman32]

Thanks guys. I'm 99% sure it's not a hack. Everything 100% up to date, logged in as root only for system tasks every once in a while. Passwords 100% strong (random letters and numbers, at least 7 chars long for the shortest one). I am connected to a private network, this machine is not exposed to the net. I personally know all the people that could access this computer through the network, and none is the hacker type at all. Disks are 100% fine. No overlay or experimental software, save the occasional ~amd64 package (no system tools though. Only things such as f-spot and the like). reiserfsck --rebuild-tree did not do anything (tree was intact to begin with anyway). I have sifted through the files still left, and see no trace of strangeness. If this was a hacker attack, why would he remove everything but the logs, and make rc-script related folders in / ? I run 2 public servers, so I can hold my own in security issues (as far as I know), so I really don't think it was a hack.

Thanks. Any ideas?

[NeddySeagoon]

progman32,

Make an image of / and think about the last commands you used as root.
Can you find them with a grep?
You are looking for /root/.bash_history with something in it that you didn't do.

Hopefully, your / (root) partition is fairly small, if not, unpicking it will take a long time.

Do you need to invest that sort of time ?

Lots of people are using the new baselayouts. Thats baselayout2 and OpenRC with no ill effects.
If this was something in the Gentoo distro, we would have head of a lot more cases.

Either you were compromised or something went horribly wrong, just for you.
rm does not delete any data, only the pointers to it, so you still have all the evidence, its just not easy to find.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[progman32]

Right, I was thinking of doing that. My / is only about 10gb, so that's fine. I know this looks like a hack, but the strange files in / just seem out of character for a hack. I'll let you know. It's very strange.

[NeddySeagoon]

progman32,

I agree.
I've seen the remains of a few cracks.

Use hexedit rather than grep for searching. hexedit allows you to copy arbitary sections of the filesystem to a file, after you find something interesting. With grep, you need to specify -A and -B before hand.

Use sleuthkit if you want to spent some time on the investigation.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[jcat]

progman32:

You have my sympathies, this does indeed sound highly unusual, and I really hope for your sake that your computer wasn't compromised. As has already been suggested, dd your root partition and go over it with a fine toothed comb.

I would be interested to know your findings. Good Luck.


Cheers,
jcat