Help!! Corrupt ext3 sperblock on root

[mattbates]

Hi,

I am desperately in need of some help to recover my root partition. Although I have not lost my home partition (luckily), I am currently without my web development files as they were sitting in the /var/www!

I am running Gentoo 1.4 with the 2.6.3 kernel. I have been successfully running it for the last fortnight, with little to no problem, however I switched on this morning and I was presented with an ugly error message:

----
ext3-fs: couldn't mount /dev/hda5 - unsupported optional features (2000200)
----

I initially tried booting with a previous kernel (2.4.21) but the same error occurred again. Therefore I grabbed the Gentoo LiveCD and tried mounting it - again, the same problem.

After carrying out some Google research, I came to the conclusion that the primary superblock may have corrupted. As a result, I have since found the locations of the backup superblocks and tried these with e2fsck (-b x) - this yields a new error, but one I cannot solve:

----
ext3 recovery flag clear, but journal has data
Recovery flag not set in backup superblock, so running journal anyway.
fsck.ext3: unable to set superblock flags on /dev/hda5.
----

To be honest, I have tried absolutely everything. Does anybody have any suggestions?

I would appreciate any help anybody can offer.

[MasterX]

Boot with Live CD,
and then run fcsk.ext? -c /dev/hda5

where ? is 2 or 3 depending on whether you have ext2 or ext3.
The above command will check the partition for bad blocks and will try to repair the damage.

WARNING: Can not guarantee for the outcome. I run it on my system and it fixed the damage without destroying the data.
When it finds something corrupted it will ask you whether you want it to repair it. Be carefull, Especially if /dev/hda5 is your home directory.

[mattbates]

I've just tried it but I receive the following:
----
fsck.ext3: Filesystem revision too high while trying to open /dev/hda5
The filesystem is apparently too high for this version of e2fsck.
(Or the filesystem superblock is corrupt)

The superblock could not be read or does not describe a correct ext2 filesystem...
----

Should I resign myself to the fact that this data may be unrecoverable? I just cannot understand how it has occurred, particularly as I had cleanly shutdown the night before.

I would really appreciate any ideas or feedback.

Thanks in advance,
MB
----

[MasterX]

I found this

I'd start by looking at the partition table.
Use fdisk -l [the letter ell, not the number one -- Heather] to list all the partitions that your Linux system can see. Make sure that the /dev/hda2 really is supposed to be a Linux native partition (that you haven't swap devices and the partition has moved to /dev/hdb2 --- and that it isn't a swap partition or whatever).
It's also possible that you've switched from some autotranslation mode to linear (LBA) or otherwise changed how the system addresses this drive. Normally this shouldn't affect Linux --- but I don't know what sort of situation you have.
The "short read" error causes me to suspect that the partition table is wrong or that you're pointing fsck to the wrong device/slice. That's the error I get if I run debugfs on a directory or file rather than the proper /dev/ node. It's also possible to get this if you've got a partition that's listed as Linux native that has no filesystem yet made on it or when you try the e2fsck -b on an MS-DOS filesystem.
You can try a number of other superblocks (they should be scattered every 8K clusters)
In a particularly bad case you can try mke2fs -S (make superblocks and group descriptors only). This is described in the man page --- and is for "last ditch" efforts only.
If you have a tape drive or a suitably large extra disk drive you can make an "image" backup of this device before you try any other (more radical) attempts at data recovery. You'd just use a command like:
dd if=/dev/hda2 of=/dev/nst0
... or better:
dd if=/dev/hda | buffer /dev/st0
... to backup the entire drive through the "buffer" program to stream all of the data out to your SCSI tape drive.
You can write the image to another block device, such as hdc3 using a command like:
dd if=/dev/hda of=/dev/hdc3
(assuming you have a large enough blank partition on the extra or loaner drive).
You can even send the data to another system with a command like:
dd if=/dev/hda | rsh $othersystem dd of=/dev/hdc3
(or whatever).
The advantage of any of these techniques is that it allows you to experiment with various recovery techniques with less chance of "making it worse" (any time you think you've "made it worse" you use the reverse commands to "start over").
There are a number of hex editors for Linux, some with nice ncurses interfaces. These can allow you to explore a filesystem trying to find out where things are. I haven't played with any of these enough to be any good with them --- and I've never read through the sources to find out where the interesting data structures should be, or what they should look like.
Eventually I'd like to see the Linux programming community produce a set of fs recovery tools to rival the Norton Utilities for DOS (for which I used to be a professional support rep). The first such tool would be one that could scan a raw device, find superblocks and report the information from them.

and this

A filesystem can be corrupted so it can't be mounted, the result of damage to
the filesystem's superblock. Any attempt to mount it will fail.
The filesystem keeps backups of the superblock.
THere are backup copies of the superblock at block offsets 8193, 16385 (8192 x
2 + 1), 24577 (8192 x 3 +1)

you can check for the size of superblocks with dumpe2fs <device> | more
if the size is different use that in the formula above.
mine is Blocks per group: 32768

you can run e2fsck using a copy of the superblock

e2fsck -f -b <offset> <device>

eg. e2fsck -f -b 16385 /dev/hda1

(taken from 'running linux' by Matt Welsh and Lar Kaufmann)

Hope these help you

[mattbates]

Thanks for your advice.

I used the advice from both snippets but unfortunately it does not yield anything new... the same errors are being returned and I am still without my root partition .

I did hope that I would be able to use one of the 'backup' superblocks but I have attempted and failed in using each one. It reports:

----
ext3 recovery flag clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
fsck.ext3: unable to set superblock flags on /dev/hda5.
----

I think I may need to download the new 2004 LiveCD and start again.. however any last-ditch thoughts from anyone would be *really* appreciated.

[MasterX]

I am assuming that you dual boot.
In this case, I will recommend you to go to Windows, install a program like Ext2FS Anywhere 3.0 (I found this on Google, I am sure there are many like this one) that can read ext(2,3) filesystems and hopefully you will be able to copy all your data from /dev/hda5 to a windows' partition. Then, use fdisk to delete the partition and recreate it and finally, copy everything back to it.
It should work, if it does not then I am afraid that you have to install Gentoo from scratch. That does not necessarily mean that you will loose all the data.

[MasterX]

Just a second!!!

/dev/hda5 is not the / filesystem but it is /var filesystem, right?
If I am right, then use LiveCd to boot, mount the / filesystem, edit /etc/fstab and remove the entry for /var.
Then you should boot into Linux.
Now, if you want to fix the filesystem you could try the following:

I just got my blog back after wrestling with a hosed ext3 partition, and winning. Long story short, I put some bad ram into Adrock which quickly caused a kernel panic. Upon removing the bad RAM and rebooting /var was toast, the journal was toast and it looked like the superblocks were a bit charred as well. I had to dig for a while to get this problem solved... so here's the info on the failure and the fix.

/var was unmountable.
I removed /var from fstab and booted the machine on a /var dir off the root so I could work on it remotely.

Here's what happened next. I fsck a variety of ways, fsck.ext3 etc.. each reported the same:
fsck /dev/ida/c0d0p7
fsck 1.27 (8-Mar-2002)
e2fsck 1.27 (8-Mar-2002)
Group descriptors look bad... trying backup blocks...
fsck.ext2: Invalid argument while checking ext3 journal for /var

Tried downgrading to ext2 to bypass the hosed journal issues:

tune2fs -O ^has_journal /dev/ida/c0d0p7
tune2fs 1.27 (8-Mar-2002)
tune2fs: Invalid argument while reading journal inode

No good. Tried mounting as ext2 read only:

mount -t ext2 /dev/ida/c0d0p7 /mnt -o ro
mount: wrong fs type, bad option, bad superblock on /dev/ida/c0d0p7,
or too many mounted file systems

Nada. Tried some more stuff... tune2fs, and debugfs to try to fix / disable the journal, then decided I'd better back up the partition... should have done that first! ;P

e2fsck -c /dev/ida/c0d0p7
e2fsck 1.27 (8-Mar-2002)
Group descriptors look bad... trying backup blocks...
e2fsck: Invalid argument while checking ext3 journal for /var
[root@adrock init.d]# tune2fs -j /dev/ida/c0d0p7
tune2fs 1.27 (8-Mar-2002)
The filesystem already has a journal.
[root@adrock init.d]# debugfs -w /dev/ida/c0d0p7
debugfs 1.27 (8-Mar-2002)
/dev/ida/c0d0p7: Can't read an inode bitmap while reading inode bitmap
debugfs: quit
[root@adrock init.d]# dd if=/dev/ida/c0d0p7 of=/big/var.bak.dd
530368+0 records in

Still hosed, but now I have a backup of the burnination. And now for the FIX!!!

mke2fs -S /dev/ida/c0d0p7
mke2fs 1.27 (8-Mar-2002)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
66528 inodes, 265184 blocks
13259 blocks (5.00%) reserved for the super user
First data block=1
33 block groups
8192 blocks per group, 8192 fragments per group
2016 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185

Writing superblocks and filesystem accounting information: done

That did it! mke2fs re-wrote all of the superblocks and group descriptors and I was able to mount in RO mode and copy off the data. After that I was able to fsck and remount the original parition with no problems!

I found some great info on the RedHat ext3 mailing list archive which led me to this fix. Big thanks to the guys on that list!

Even if the above does not help to recover the data, you still do not have to reinstall Gentoo, just format /dev/hda5 or try to do what I wrote in my previous post.

Good luck

[mattbates]

I've followed the latter advice you gave and it has worked a treat! I can now access the drive and content so I have managed to backup my work to another partition. On running fsck on the partition, it did find a huge number of problems - I think a reinstall may be in order however most important of all I have my data back .

Thank you so much for your help. It is great being able to refer to the Gentoo community for help and support.

[MasterX]

Thank you so much for your help. It is great being able to refer to the Gentoo community for help and support.

You are welcome.
I am glad everything worked out for you.