Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Corrupted ext3 FS after proper reboot
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Tue Oct 21, 2008 6:30 pm    Post subject: Corrupted ext3 FS after proper reboot Reply with quote

Hello,

i need some help fixing my root filesystem - or to try to save as much as possible.

My / was on /dev/hdb5 as ext3 for 4 years now with regular fs checks. (The system is 32Bit x86). Yesterday i restarted the machine from gentoo to win (which went without issues - no umount problems or similar) and after a few hours i tried to go back.

Then i was prompted with a kernel panic saying my root fs is not proper - iirc it was like this
Code:

VFS: Cannot open root device "hdb5" or unknown-block...
Please append a correct "root=" boot option; here are the available partitions:


I tried again with a different kernel - no difference.
So i went to boot a knoppix disk i have lying around to to a fsck.

This prompted me with a message that the superblock couldn't be found so i went to try with a different superblock (i tried to find this with mkfs.ext3 -n). This led me then to:

Code:

fsck.ext3 -b 229376 -B 4096 -f /dev/hdb5
e2fsck 1.40-WIP (14-Nov-2006)
Backing up journal inode block information.

fsck.ext3: Illegal triply indirect block found while reading bad blocks inode
This doesn't bode well, but we'll try to go on...
Pass 1: Checking inodes, blocks, and sizes
Bad block inode has illegal block(s).  Clear<y>? no

Illegal block #1 (17170448) in bad block inode.  IGNORED.
Illegal block #2 (1634755688) in bad block inode.  IGNORED.
Illegal block #5 (17235984) in bad block inode.  IGNORED.
Illegal block #6 (1886678888) in bad block inode.  IGNORED.
Illegal block #7 (6780268) in bad block inode.  IGNORED.
Illegal block #9 (17367060) in bad block inode.  IGNORED.
Illegal block #10 (1634954337) in bad block inode.  IGNORED.
Illegal block #11 (1853190003) in bad block inode.  IGNORED.
Bad block inode has an indirect block (100) that conflicts with
filesystem metadata.  IGNORED.
Illegal block #-1 (138555797) in bad block inode.  IGNORED.
Illegal block #-3 (17235984) in bad block inode.  IGNORED.
Error while iterating over blocks in inode 1: Illegal triply indirect block found
e2fsck: aborted



There i stopped and did a smartctl -t short /dev/hdb and a
backup with dd if=/dev/hdb5 of=/mnt/path/backup/some.img
smartctl (which is also ran automatically in regular intervals shows no issues - if useful i can provide full output)


Afterwards i tried to recover by answering the questions with y - wich bothered me with hundreds of questions and a clean but empty fs with hundreds of lost+found entries.

Ok i played back the dd - but i'm still not closer to my data.
All other partitions on this disk mount fine and show no problems (i backuped them again anyway)

If anyone can give me further hints here - i can't find much with the fsck output regaring the triply indirect block which would help me. A backup is present but quite old (as usual if it's needed).

Thanks in advance,
s|mon

[edit: during dd or fsck there was also no output in dmesg concernig write/failures or other related stuff]
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Tue Oct 21, 2008 6:50 pm    Post subject: Reply with quote

s|mon,

dd will have failed at the first bad block. You need something like ddrescue (no hyphen) to read as much data as possible off a failing drive.

Your best bet now is to recover things from lost and found. Everything there is named after the inode that points to it.
You can identify directories as they will be blue. Look inside each blue file in lost and found and try to copy it recursively somewhere else. If there are no errors, you probably got you r data back.

What size is your dd output file ?
If its the same as the partition, you didn't get any read errors and the filesystem metadata and perhaps your data is corrupt, however, it contains validly written but incorrect data.

Be very careful to mount the damaged partition and the dd image read only for recovery operations.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Tue Oct 21, 2008 6:59 pm    Post subject: Reply with quote

Hi thanks for the advice, but i fear the recover from lost+found might be difficult as i already replayed the dd image :(

So my hope lies on the dd image - which went through without errors (if it would present once a badblock is encountered).
Btw i just started a badblock scan on the device before i read your post.

The size of the dd image is: 26G
26863732224 Oct 20 17:43 root_hdb5_081020.img - which seems to be the complete partition.

cfdisk:
hdb5 Logical Linux 26863.77
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Tue Oct 21, 2008 7:06 pm    Post subject: Reply with quote

s|mon,

dd exits on error and tells how much it copied. Check the file size against the partition size ... you may have it all.

If badblocks finds anything, the drive is scrap.
Drives bigger than about 4Gb are made with spare sectors and spare tracks, which the drive brings into use as it detects a sector is about to fal.
When the spares are all used, bad blocks cannot be hidden from the operating system any longer and the drive is end of life.
Catastropic loss of a sequence of blocks also indicate end of life.

Examine the dd image with sleuthkit. Be sure not to allow any writes.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Tue Oct 21, 2008 7:14 pm    Post subject: Reply with quote

Thanks for the advice.
I'll try to install Sleuthkit within knoppix if possible or fetch another cd-image.
I just edited above post with the sizes of the image and the partition - looks ok.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Tue Oct 21, 2008 7:21 pm    Post subject: Reply with quote

s|mon,

Yep, it looks like dd read it all.

Good luck
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Tue Oct 21, 2008 7:34 pm    Post subject: Reply with quote

Well it looks like i have to study Sleuthkit first as it offers many possibilities and i don't know any of it yet.
At least this brings me some new knowledge then :)

Hints are appreciated nonetheless

edit:
Code:
badblocks -n -s /dev/hdb5
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: done
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Tue Oct 21, 2008 8:59 pm    Post subject: Reply with quote

s|mon,

Its a forensics tool, it will allow yu to find files on a damaged filesystem.
Even if it allows it, never attempt in place data recovery. Find your data and move it out to another drive.

You drive appears ok.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Tue Oct 21, 2008 10:01 pm    Post subject: Reply with quote

Ok it gets more difficult. I made another copy of the dd image and gave autopsy a try. But it is not accepting the image also.
Code:
Warning: The file system type of the volume image file could not be determined.
If this is a disk image file, return to the previous page and change the type.


a simple fsstat also doesn't get me further.
Code:
fsstat -f ext root_hdb5_081020.img
Invalid magic value (not an EXTxFS file system (magic))


I'll try to read more about these tools tomorrow - if i do a quick look with cat into the image i see at least some filenames between unreadable output.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Wed Oct 22, 2008 7:25 pm    Post subject: Reply with quote

s|mon,

Lets try something simpler.

You can tell mount to use an alternate superblock. You said that you determined the location of your alternate superblocks, so you know where they are.

A 26G filesystem will use a 4kB block size on the disk and mount expects to be fed 'blocks' of 1kB.

So, try
Code:
mount -f ext2 -o loop,ro,sb=xxx /path/to/image /mount/point

xxx is the location of the superblock*4. I'm not sure if both tools count from either 0 or 1, so
(superblock-1)*4 and (superblock+1)*4 are also worth trying.
Its ext2, not ext3 as you are mounting read only - no journal replay is possible and you don't want it to happen either.

Read
Code:
man mount
under ext2 for an example
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Wed Oct 22, 2008 8:32 pm    Post subject: Reply with quote

Hi,

this also fails with the following output - i tried different superblock locations and with +/-4 as offset.

Code:

mount  -t ext2 -o loop,ro,sb=917504 /mnt/sda2/Backup/root_hdb5_081020.img /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so


i then mounted it again as loop device and used testdisk to have a look - so e.g. the label is still recognized.
Code:

     Partition               Start        End    Size in sectors
D Linux                    2   2  1  3268   0 60   52468224 [root]


While i was trying to access the backup (which was done with "star -czvp ...") i encounter the next issue:
Code:

gunzip /mnt/sda2/Backup/backup_080803_hdb5.star.gz | star tv
star: Hard EOF on input, first EOF block is missing at -1.
star: 0 blocks + 0 bytes (total of 0 bytes = 0.00k).


Seems not to be my week :/
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Wed Oct 22, 2008 9:08 pm    Post subject: Reply with quote

s|mon,

Is the image your root partition or a whole drive ?
It matters as it changes the offset.

If its a partition, it has no partition table itself, so testdisk found something taht looks like a partition table but isn't.

If its a disk image, you need to allow for the 'unused' space at the start of the drive, thats 62 sectors.
Your first partition starts at sector 63.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Wed Oct 22, 2008 9:22 pm    Post subject: Reply with quote

It's an image of the partition taken with "dd if=/dev/hdb5 of=/mount/imagepath.img"
So i guess no offset is needed.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41328
Location: 56N 3W

PostPosted: Wed Oct 22, 2008 9:28 pm    Post subject: Reply with quote

s|mon,

Correct.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
s|mon
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2004
Posts: 119
Location: Bayern [de]

PostPosted: Wed Oct 22, 2008 9:39 pm    Post subject: Reply with quote

Okay i got at least my backup working - it seems that inspite of the -z specified star didn't compress it.
Uncompression was possible within a new partition with
Code:

cat \mnt\myimage.star | star xv


- i'll now start updateing to get back where i left and try to recover lost files/changes from the image afterwards.
Thanks a lot for your help and if you have further hints on getting data from the image they are still welcome.

greetings, s|mon
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum