bad block, reiserfs, reiserfsck, etc: can't boot

Message

sog · Post by **sog** » Sun Jun 11, 2006 5:36 am

really hoping someone can help here. after a forced hard reset of my Thinkpad x40 (X locked up), i find that i'm no longer able to boot.

after progressing part way through the boot sequence, i'm getting an error that says:

Code: Select all

/etc/init.d/bootmisc: line 61: 3752 segmentation fault   rm -f "${x}"

any thoughts as to how i proceed?

sog · Post by **sog** » Sun Jun 11, 2006 5:50 am

one additional bit of info - it's not a hard drive problem, or at least not one that affects the non-Linux partitions. i can log in successfully to the Windows partition on the same machine.

i'm really hoping someone will say you're not screwed.

/carlito · Post by **/carlito** » Sun Jun 11, 2006 7:11 am

Are you by any chance running reiserfs? I've had the exact same problem yesterday and fixed it by booting my rig with a live cd and running reiserfschk --rebuild-tree.

sog · Post by **sog** » Sun Jun 11, 2006 3:08 pm

i am indeed. let me try doing just that.

sog · Post by **sog** » Sun Jun 11, 2006 4:06 pm

ok, had a bit of a time b/c i somehow did not have reiser4progs installed, but got that in and ran reiserfsck.

unfortunately, i'm getting an error that says:

Partition /dev/hda4 is mounted with write permissions, cannot check it

to mount it, i just booted using the live disk, then did a standard:

Code: Select all

mount /dev/hda4 /mnt/gentoo
mount /dev/hda2 /mnt/gentoo/boot
chroot /mnt/gentoo /bin/bash

what do i need to change to get reiserfsck to work?

Update: nevermind - got it. for those who need the info later, it's

Code: Select all

mount -r /dev/hda4 /mnt/gentoo
mount -r /dev/hda2 /mnt/gentoo/boot
reiserfsck --check /dev/hda4
...

/carlito · Post by **/carlito** » Sun Jun 11, 2006 6:24 pm

Correct me if i'm wrong, but i believe it's not very wise to run reiserfschk on a mounted drive..

Anyway, i hope it worked for!

sog · Post by **sog** » Sun Jun 11, 2006 7:28 pm

unfortunately, that didn't work for me. i ran it, and got the following:

Code: Select all

bread: Cannot read the block (5549459): (Input/Output error).

The problem has occured looks like a hardware problem. If you have bad blocks, we advise you to get a new hard drive, because once you get one bad block that the disk drive internals cannot hide from your sight, the chances of getting more are generally said to become much higher...

If you don't want to follow that follow that advice then if you have just a few bad blocks, try writing to the bad blocks and see if the drive remaps the bad blocks 9that means it takes a block that it has in reserve and allocates it for use for of that bluck number). If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly.

i don't have a spare hard drive handy for this machine, and what's more would like to get access to the information on that partition that's not yet backed up.

can anyone help shed light on the second part of their recommendations? how should i go about writing to the bad block? which reiserutil should i be running with option -B?

any and all help appreciated.

sog · Post by **sog** » Tue Jun 13, 2006 4:30 pm

ok, i've toned down my ambitions: now i'd just like to extract whatever information i can from the corrupted reiser volume.

any idea how i might do that? trying to boot into Linux causes kernel panic, and i can't mount the volume to chroot into it.

the data, however, is clearly still there. the question now is how i get access to it and save what i can before i replace the actual drive.

troymc · Post by **troymc** » Mon Jun 19, 2006 12:19 am

First, never mount a possibly corrupt filesystem. The reason why is that the kernel periodically sync's all drives & that write could corrupt it further. So you don't need to mount the filesystem, just run the tools on the device [like you did - you ran the tool correctly, you just didn't need to mount the device before doing it].

Hopefully you have smartmontools on one of those boot CDs. Try running something like this:

Code: Select all

# smartctl -A /dev/hda
smartctl version 5.36 [x86_64-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   062    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   040    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   229   229   033    Pre-fail  Always       -       1
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       203
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   040    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   095   095   000    Old_age   Always       -       2378
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       73
191 G-Sense_Error_Rate      0x000a   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0012   091   091   000    Old_age   Always       -       98236
194 Temperature_Celsius     0x0002   119   119   000    Old_age   Always       -       46 (Lifetime Min/Max 17/56)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       -       0

What you are looking for here are your "Error_Rate" entries, hopefully they should all be zero. If they are not zero, then this drive is failing. Next check the "Reallocated_Sector_Ct". This is the number of bad spots the hard disk has been able to map around. Hopefully, this should also be zero. If there are only a couple, then the drive is probably still usable, but would need to be constantly monitored. Look at the "Current_Pending_Sector" entry. This shows you how many bad spots the drive has detected, but has not yet been able to map out. Again, this should be zero. If not, then you need to trigger the drive to remap it, or there are so many already mapped out, that there is no more space to map out any more [another sign of a failing drive].

With the device NOT mounted, run reiserfsck --check on the device.

If you get the same errors you got before, then run badblocks. But first, stat -f the device to verify its block size. Most like it will be 4096

Code: Select all

# stat -f /dev/hda4
  File: "/dev/hda4"
    ID: 0        Namelen: 255     Type: tmpfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 257389     Free: 257329     Available: 257329
Inodes: Total: 257389     Free: 256931

Then run badblocks in non-destructive read-write mode, using that block size. This is non-destructive so your data is not damaged, but it is read-write so that we can trigger the drive to map-out any bad sectors it finds. [Bad sectors are only re-mapped on a write()]

Code: Select all

# badblocks -s -n -b 4096 -o /tmp/badblocks.out /dev/hda4

This output file that it creates is a map of badblocks. These are blocks which the drive itself cannot map out, so we need to keep track of them. This is what you give to the other reiserfs utils with the -B option. reiserfsck, mkreiserfs, debugreiserfs all accept badblock maps using the -B option.

Now, you can try running reiserfsck again giving it that badblock map.

Code: Select all

# reiserfsck --check --badblocks /tmp/badblocks.out /dev/hda4

Hopefully, at this point it may return & suggest running reiserfsck --fix-fixable to repair errors. Then you should be able to mount it and recover most of your data.

Anything that was in the area with bad blocks will probably be corrupt.

troymc

Hoshimaru · Post by **Hoshimaru** » Mon Aug 14, 2006 7:08 pm

Interesting information... however, this doesn't work for me. It simply refuses to repair my bad blocks.
First only /dev/sda7 (root) was ruined, but today I also noticed sda8 aka /home is destroyed... sigh... last time I buy a Maxtor hard disk. > 187GB of data is lost, among wich was my 10 year old email archive.

This really sucks... I'm getting afraid of using reiserfs as well now...
* hopes his ext3 laptop will have more luck *

troymc · Post by **troymc** » Mon Aug 14, 2006 8:46 pm

Hoshimaru wrote: Interesting information... however, this doesn't work for me. It simply refuses to repair my bad blocks.

You should give us more info on what is not working. Maybe we could help, or learn what to avoid.

Hoshimaru wrote: last time I buy a Maxtor hard disk.

Yeah, I've given up on Maxtor & Western Digital . Right now I'm sticking w/Seagate or Hitachi.

Hoshimaru wrote: > 187GB of data is lost, among wich was my 10 year old email archive.

What? No backups?

Hoshimaru wrote: This really sucks... I'm getting afraid of using reiserfs as well now...

You can't blame reiserfs for your hardware issues. Or did you have fs problems as well?

troymc

Hoshimaru · Post by **Hoshimaru** » Tue Aug 15, 2006 9:24 am

Problems started after the power got abruptly cut in the entire street a few days ago.
Afterwards, it booted again, did some replay on the journal and continued.

First thing I noticed was Gnome being a little slower than usual to respond. When I reused that PC after a few days, it got worse. After doing shutdown -r now, it couldn't mount the root partition anymore.

With the 2006.0 livecd I did a badblocks check, resulting in a 4K file full of blocknumbers.
reiserfsck suggested to use --rebuild-tree, which I did... I get a huge list of

Code: Select all

ata5: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/4
ata5: status=0x51 {DriverReady SeekComplete Error }
ata5: error0x40 {UncorrectableError }
end_request: I/O error; dev sda, sector 1470544

The last line says:

Code: Select all

bread: Cannot read the block (183802): (Input/output error)
Aborted

First I thought only my / got fubar, but it's also /home. I don't even care to take a look at /boot and swap.

Backups ... I bought a 320 external WD hard disk to back up a lot of data, but I didn't sort out everything on that SATA drive yet. I've always handled that disk with care.

Currently doing a

Code: Select all

dd conv=noerror if=/dev/sda8 of=/mnt/usb/disk.bin

I think I'll get myself a new hard disk Wednesday when I get back to work. Probably another brand.
This Summer hasn't been my lucky time of the year >___<

I hope my WD disk will be safe O.o
I did copy most stuff on DVD too, but they get unreadable after a few years (that's why I got that hard disk).

*sigh* when are they going to make quasi undestructable storage and more importantly one that isn't going to degrade over the years? I still want to see some of my data when I'm 60 ^^'

Gentree · Post by **Gentree** » Wed Aug 16, 2006 10:22 am

sob wrote:ok, had a bit of a time b/c i somehow did not have reiser4progs installed, but got that in and ran reiserfsck.

you really need to slow down and start paying attention. reiser4progs are for.... reiser4 , not reiserfs, so I see no way that you needed that package to run reiserfsck.

However, if you are trying to repair reiserfs with reiser4 tools and running fsck on a mounted device you deserve all corrupt data you can make for your self.

I suspect the majority of what you now have is of your own doing. Sorry , but slow down and find out what you are supposed to do before diving into that sort of task.

Be wiser next time.

@Hoshimaru,
I suggest you check out reiser4. I have most of my system running on R4 and one of the main reasons I like is it's resistance to power outs. I can hit a power failure at any time and know I will come straight back to a good fs and lose no more than my last unflushed change to a particular file.

It's way faster than reiserfs as well.

chiwi · Post by **chiwi** » Wed Sep 13, 2006 11:24 pm

Hi there, i'm having the same issues here.

I've run the badblocks cmd and used the output with reiserfsck. The thing is that it seems that the badblocks are part of the section where the journal is...then it cannot do anything, nor apply the changes.

I've done some searching on google,and found an application called Nucleus Kernel reiserFS, which works fine. At least I can see the files in the partition....the downside is that I only could get a demo, and in the demo, you only can see the files, not copy them to another partition....mmm =/ (anyone has the full version? =) )

So, if the partition "crashes" everytime it tries to read the journal....Is it possible to mount the partition without reading it ? I only want to do it in order to get some data...after i do so, i'll format everything, or get myself a new HD, but i really need to get that info from the HD. The mount man page only talks about replaying the uncommited data, i haven't found the way to disable the journal.

Thanks all

yo.

Gentree · Post by **Gentree** » Wed Sep 13, 2006 11:42 pm

well it wont disable the journalling (AFAIK that's not an option , reiserfs is a journalling fs) but maybe if you turn off the autocheck in /etc/fstab it may have the result you want. End the line for this partition with 0 0

chiwi · Post by **chiwi** » Thu Sep 14, 2006 12:53 am

but the partition is not even being able to mount. It returns input/output error while trying to read the journal...do you think this is gonna work ?

thanks

yo.

chiwi · Post by **chiwi** » Thu Sep 14, 2006 4:45 pm

Gentree wrote:well it wont disable the journalling (AFAIK that's not an option , reiserfs is a journalling fs) but maybe if you turn off the autocheck in /etc/fstab it may have the result you want. End the line for this partition with 0 0

hey, there was no way I could mount that partition....the same problem appears, ....hmmm

any ideas ?

thanks

yo.

Gentree · Post by **Gentree** » Thu Sep 14, 2006 5:35 pm

I must say with a price tag of $459US I may be inclined to look for a less "professional" solution. I find that pretty dishonest for a FS recovery tool.

man mkreiserfs

-j | --journal-device FILE path to separate device to hold journal

man reiserfsck

Expert options:
--no-journal-available do not open nor replay journal
-S | --scan-whole-partition build tree of all blocks of the device

if all else fails ..... RTFM!

I hope that helps, it looks likely.

chiwi · Post by **chiwi** » Thu Sep 14, 2006 5:45 pm

Gentree wrote: man mkreiserfs
-j | --journal-device FILE path to separate device to hold journal
man reiserfsck
Expert options:
--no-journal-available do not open nor replay journal
-S | --scan-whole-partition build tree of all blocks of the device
if all else fails ..... RTFM!

gentree, i've already read TFM,

, that's why i'm here. I've already tried with the --no-journal-available, but it ignores that option and looks for the journal anyway....yeah, true.

and mkreiserfs will wipe out the data, that's not what i want to do now, not yet. I want to get some data from it before.

Thanks.

yo.

Gentree · Post by **Gentree** » Thu Sep 14, 2006 5:53 pm

I've already tried with the --no-journal-available, but it ignores that option and looks for the journal anyway....yeah, true.

well you did not make any referce to that so I thought you;d missed it.

can you dd it somewhere else as was suggested above and try to fsck that?

chiwi · Post by **chiwi** » Thu Sep 14, 2006 6:00 pm

Gentree wrote: well you did not make any referce to that so I thought you;d missed it.

sorry, I've tried many things already... =/

Gentree wrote: can you dd it somewhere else as was suggested above and try to fsck that?

I'm affraid I can't, I don't have space available or another partition where to put the reiserfs partition. I just want to copy some files from the reiserfs partition, that's it.. .=/

The strange thing is that there is a windows app called Nucleus Kernel ReiserFs, and it can actualy see the data, but i cannot copy it because it's a demo version... So, if that application can see the data, why can't another - linux - application do the same ?

thanks

Gentree · Post by **Gentree** » Thu Sep 14, 2006 6:05 pm

well it seems that reiserfsck should be that tool, but you say it's borked.

not surprised, I dont rate reiserfs at all.

see if you can find a bug report.

Gentree · Post by **Gentree** » Thu Sep 14, 2006 7:13 pm

--no-journal-available
This option allows reiserfsck to proceed when the journal device is
not available. This option has no effect when the journal is
located on the main data device.

LOL, it's not supposed to do anything with your partition.

however debugreiserfs may provide the means to recontruct it but either way you are going to need at least as much space again to play with the data and a raw backup would be strongly advisable before you do anything.

So bottom line is you need a new disk, But you know that anyway.

If the data is really important I think you can get it back. Just a case of how much time you are preparted to give.

Next time I suggest smallish partitions , between 4 to 8GB that are nice and easy to back up. I makes everything much more flexible.

chiwi · Post by **chiwi** » Thu Sep 14, 2006 7:22 pm

alright, let's move forward the raw backup. I've got a Thinkpad t42, 20gb with windows (ntfs), 19Gb with gentoo, reiserfs, 100Mb /boot, etx2 and 512Mb Swap partition. All of them Primary partitions. Now, I know that writing is not possible to a ntfs filesystem, then the raw backup to the ntfs partition is impossible. But, i've got another box at home, which has gentoo installed as well... May be I can dd to that box...is that possible ? some way to 'pipe' it the dd output to that box ? may be if I mount a dir with samba, it'll work....

any idea ??

how am I supposed to use the dd command ? Does it output everything to a single file ? how does it work ?

thankssssss

yo.

Gentree · Post by **Gentree** » Thu Sep 14, 2006 8:42 pm

if I follow you it's the thinkpad that has blown out the reiserfs , so you have no gentoo anymore just an XP? Limits the options. Can you boot it off a liveCD?

man dd

it does raw byte by byte data from one device to another. In the case of a borked partition you can clone the mess with

dd if=/dev/<borked> of=/dev/<good unmounted dev>

the target must be free, unmounted and at least as large as the source. Equally src dev must not me mounted.

so the easiest is if you can plug both drives into the same system . If not you may be able to get a device interface over nfs. I'm pretty vague on that so you'll have to follow it up yourself.

If you can get the raw data across it will appear as an identically broken reiserfs to the host system.

Then you are a bit freer to hack it about since it's just a copy.