View previous topic :: View next topic |
Author |
Message |
remix l33t
Joined: 28 Apr 2004 Posts: 797 Location: hawaii
|
Posted: Tue Feb 05, 2013 6:33 am Post subject: how to recover from faulty raid5 array? |
|
|
i had a power outtage, server went down, and hasn't bee nthe same since.
Code: | # mdadm --assemble --scan
mdadm: /dev/sdd1 has no superblock - assembly aborted
mdadm: /dev/sdd2 has no superblock - assembly aborted
mdadm: /dev/sdd3 has no superblock - assembly aborted
mdadm: /dev/sdd5 has no superblock - assembly aborted
mdadm: /dev/sdd6 has no superblock - assembly aborted
mdadm: /dev/sdd7 has no superblock - assembly aborted
|
Code: | # mdadm --misc --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 0.90.00
UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
Creation Time : Sat Mar 6 00:43:43 2010
Raid Level : raid5
Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
Array Size : 102558528 (97.81 GiB 105.02 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 1
Update Time : Mon Feb 4 18:54:39 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : b4dca8e1 - correct
Events : 963131
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 49 3 active sync /dev/sdd1
0 0 8 1 0 active sync /dev/sda1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 49 3 active sync /dev/sdd1 |
Code: | # mdadm --misc --examine /dev/sdd2
/dev/sdd2:
Magic : a92b4efc
Version : 0.90.00
UUID : 64205578:cc795cb2:fbf2ea5a:12bfd232 (local to host OptimusPrime)
Creation Time : Mon Mar 8 22:07:40 2010
Raid Level : raid5
Used Dev Size : 258799040 (246.81 GiB 265.01 GB)
Array Size : 776397120 (740.43 GiB 795.03 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2
Update Time : Mon Feb 4 18:54:39 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : 948c4a52 - correct
Events : 386
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 50 3 active sync /dev/sdd2
0 0 8 2 0 active sync /dev/sda2
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 50 3 active sync /dev/sdd2 |
sdd3, sdd5, sdd6, and sdd7 are all about the same.
Code: | # mdadm --misc --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
Creation Time : Sat Mar 6 00:43:43 2010
Raid Level : raid5
Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
Array Size : 102558528 (97.81 GiB 105.02 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 1
Update Time : Mon Feb 4 18:54:39 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : b4dca8ab - correct
Events : 963131
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 49 3 active sync /dev/sdd1 |
sda2, sda3, sda5, sda6, and sda7 are all about the same.
Code: | # mdadm --misc --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
Creation Time : Sat Mar 6 00:43:43 2010
Raid Level : raid5
Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
Array Size : 102558528 (97.81 GiB 105.02 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 1
Update Time : Sat Feb 2 23:55:04 2013
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : b4da4a83 - correct
Events : 962928
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 17 1 active sync /dev/sdb1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 8 49 3 active sync /dev/sdd1 |
sdb2, sdb3, sdb5, sdb6, and sdb7 are all about the same.
Code: | # mdadm --misc --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
Creation Time : Sat Mar 6 00:43:43 2010
Raid Level : raid5
Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
Array Size : 102558528 (97.81 GiB 105.02 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 1
Update Time : Wed Jan 16 04:47:26 2013
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : b4a5715a - correct
Events : 21
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 33 2 active sync /dev/sdc1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1 |
sdc2, sdc3, sdc5, sdc6, and sdc7 are all about the same, i'm confused to why sdb and sdc are in state "active sync" here.
is there a way to rebuild my raid partitions? _________________ help the needy |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Feb 07, 2013 11:39 pm Post subject: |
|
|
What bothers me is that the array is marked "Clean" which means something was done with the array after the power outage - they should be marked "dirty."
Usually when a disk gets kicked from an array, the kicked disk, if it couldn't be written to at the time of kicking, will still have a superblock on it. If it comes back on a power cycle - that's why sometimes they show up as a dirty disk with all other array members missing.
You may need to recreate the superblock on all disks again, but I don't know how you created the array to begin with. Or it's probably possible to hack the missing disks and fake it so that it thinks it's clean. But since it looks more than one disk was removed from a raid 5, you may be facing data loss even if you could get it to reassemble.
This is one of the reasons why I want to get a UPS for my 4 disk RAID5 :( and RAID is not backup... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
remix l33t
Joined: 28 Apr 2004 Posts: 797 Location: hawaii
|
Posted: Fri Feb 08, 2013 12:29 am Post subject: |
|
|
somehow i got mdadm to see the third disk, so then it assembled. i looked and it seems that the fourth is in fact toast. so i've ordered a replacement, and am slowly moving all my backups onto amazon glacier.
i guess i got lucky this time.
the way i got mdadm to see it is to (i had to repeatedly do this) run mdadm -S /dev/md1 to stop it, then reassemble /dev/md1 again.
or stop all /dev/mdX devices, and the reassemble again.
this process took over 8 hours across 2 days cause i'm not very knowledgeable or skilled in this. _________________ help the needy |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|