write-protected SATA device [SOLVED]

Message

pointers · Post by **pointers** » Wed May 03, 2006 8:18 am

Hello,
/dev/sdb1 device in my gentoo is set to "ro" mode for some reason that I couldt find yet. This partition is used for large number of apache log and webalizer statistics. It was working properly until yesterday night. When I try to remount

Code: Select all

#mount -o remount,rw /dev/sdb1
mount: block device /dev/sdb1 is write-protected, mounting read-only

it says it is write-protected. I have stopped logging and try to unmount the file system and it says file system is busy. I have used lsof to find which causes the fs to be busy but nothing uses this filesystem.
Do you have any idea why an ext3 filesystem can put itself into RO mode without my manipulation?

Regards.

PaveQ · Post by **PaveQ** » Wed May 03, 2006 2:26 pm

Generally filesystem's will mount itself read-only when they notice corruption. Maybe your harddisk is failing.

pointers · Post by **pointers** » Wed May 03, 2006 2:42 pm

Thanks for your suggestion PaveQ. I will use another disk and fsck the disk. It is a very new disk. Whenever I work with large amounts of small files, disks fail easier. I dont know why...

Regards.

PaveQ · Post by **PaveQ** » Wed May 03, 2006 2:54 pm

pointers wrote:Thanks for your suggestion PaveQ. I will use another disk and fsck the disk. It is a very new disk. Whenever I work with large amounts of small files, disks fail easier. I dont know why...

Regards.

Actually most probable time for disk to fail is when its brand new. Use badblocks command which physically checks the disk platters, fsck only checks filesystem integrity.

pointers · Post by **pointers** » Wed May 03, 2006 2:56 pm

But as far as I remember fsck has a physical option to check the disk ?

PaveQ · Post by **PaveQ** » Wed May 03, 2006 3:01 pm

pointers wrote:But as far as I remember fsck has a physical option to check the disk ?

Hmm, I think it will actually run badblocks

pointers · Post by **pointers** » Thu May 04, 2006 7:11 am

Hi,
As you said, disk has probably physical problem. Here is the final dmesg output.

Remounting filesystem read-only
EXT3-fs error (device sdb1) in ext3_ordered_commit_write: IO failure
EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb1) in ext3_orphan_del: Journal has aborted

sdb2 partition doesnt have a problem like this. Can this problem be propagated into the second partition? What do you think about this?

I think I have replace this disk soon. In fact it is the second disk for the same type of log storage purpose. If it happens third time, I must find another way to store this millions of files.

Regards.

PaveQ · Post by **PaveQ** » Thu May 04, 2006 7:23 am

pointers wrote:Hi,
As you said, disk has probably physical problem. Here is the final dmesg output.

Remounting filesystem read-only
EXT3-fs error (device sdb1) in ext3_ordered_commit_write: IO failure
EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb1) in ext3_orphan_del: Journal has aborted
sdb2 partition doesnt have a problem like this. Can this problem be propagated into the second partition? What do you think about this?

I think I have replace this disk soon. In fact it is the second disk for the same type of log storage purpose. If it happens third time, I must find another way to store this millions of files.

Regards.

Did you to the badblocks? Do it. Thats definitely enough to be document to get the drive replaced. It might be that only first platter is damaged and the second fs lies mostly in second platter. Still, get that replaced.

pointers · Post by **pointers** » Thu May 04, 2006 7:58 am

Because the server is running now in a production environment I canno shut it down and linux doesnt allow me to unmount this partition. It always says busy. I cannot unmount it to make a badblock check:(

pointers · Post by **pointers** » Fri May 05, 2006 3:12 pm

Hello,
it is strange that badblock displayed nothing but fsck says filesystem has errors. It is fixing errors now. Strange,
I was expecting a hardware error.

PaveQ · Post by **PaveQ** » Fri May 05, 2006 8:37 pm

pointers wrote:Hello,
it is strange that badblock displayed nothing but fsck says filesystem has errors. It is fixing errors now. Strange,
I was expecting a hardware error.

strange indeed. Something must have been fubaring the filesystem then. What filesystem it is?

pointers · Post by **pointers** » Mon May 08, 2006 5:36 am

it is ext3

pointers · Post by **pointers** » Mon May 08, 2006 7:41 am

Here is what I have got from my commands;
As you can see badblocks displayed nothing. (I didnt add any extra option to the badblock command. Then e2fsyck displayed some FS errors.

tux ~ # badblocks /dev/sdb1

tux ~ #
tux ~ #
tux ~ #
tux ~ #
tux ~ #
tux ~ #
tux ~ #
tux ~ # e2fsck /dev/sdb1
e2fsck 1.38 (30-Jun-2005)
LOGDISK: recovering journal
LOGDISK contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 3180826, i_size is 0, should be 2048. Fix<y>? yes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +(25447881--25447936) +(25448142--25448151) -(25450500--25450566)
Fix<y>? yes

Free blocks count wrong for group #3106 (3462, counted=3463).
Fix<y>? yes

Free blocks count wrong (190078090, counted=190078091).
Fix<y>? yes

LOGDISK: ***** FILE SYSTEM WAS MODIFIED *****
LOGDISK: 1854439/30519296 files (8.0% non-contiguous), 54069713/244147804 blocks

PaveQ · Post by **PaveQ** » Mon May 08, 2006 1:10 pm

No idea. Sometimes it's possible to mesh up filesystem with some bad combination of raid, lvm and kernel. Maybe try different fs? I use xfs and it have never let me down.