Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
how to recover from faulty raid5 array?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
remix
l33t
l33t


Joined: 28 Apr 2004
Posts: 797
Location: hawaii

PostPosted: Tue Feb 05, 2013 6:33 am    Post subject: how to recover from faulty raid5 array? Reply with quote

i had a power outtage, server went down, and hasn't bee nthe same since.

Code:
# mdadm --assemble --scan
mdadm: /dev/sdd1 has no superblock - assembly aborted
mdadm: /dev/sdd2 has no superblock - assembly aborted
mdadm: /dev/sdd3 has no superblock - assembly aborted
mdadm: /dev/sdd5 has no superblock - assembly aborted
mdadm: /dev/sdd6 has no superblock - assembly aborted
mdadm: /dev/sdd7 has no superblock - assembly aborted


Code:
# mdadm --misc --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
  Creation Time : Sat Mar  6 00:43:43 2010
     Raid Level : raid5
  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
     Array Size : 102558528 (97.81 GiB 105.02 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Mon Feb  4 18:54:39 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : b4dca8e1 - correct
         Events : 963131

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       49        3      active sync   /dev/sdd1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       49        3      active sync   /dev/sdd1


Code:
# mdadm --misc --examine /dev/sdd2
/dev/sdd2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 64205578:cc795cb2:fbf2ea5a:12bfd232 (local to host OptimusPrime)
  Creation Time : Mon Mar  8 22:07:40 2010
     Raid Level : raid5
  Used Dev Size : 258799040 (246.81 GiB 265.01 GB)
     Array Size : 776397120 (740.43 GiB 795.03 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2

    Update Time : Mon Feb  4 18:54:39 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 948c4a52 - correct
         Events : 386

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       50        3      active sync   /dev/sdd2

   0     0       8        2        0      active sync   /dev/sda2
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       50        3      active sync   /dev/sdd2



sdd3, sdd5, sdd6, and sdd7 are all about the same.



Code:
# mdadm --misc --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
  Creation Time : Sat Mar  6 00:43:43 2010
     Raid Level : raid5
  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
     Array Size : 102558528 (97.81 GiB 105.02 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Mon Feb  4 18:54:39 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : b4dca8ab - correct
         Events : 963131

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       49        3      active sync   /dev/sdd1


sda2, sda3, sda5, sda6, and sda7 are all about the same.


Code:
# mdadm --misc --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
  Creation Time : Sat Mar  6 00:43:43 2010
     Raid Level : raid5
  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
     Array Size : 102558528 (97.81 GiB 105.02 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Sat Feb  2 23:55:04 2013
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0
       Checksum : b4da4a83 - correct
         Events : 962928

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       0        0        2      faulty removed
   3     3       8       49        3      active sync   /dev/sdd1


sdb2, sdb3, sdb5, sdb6, and sdb7 are all about the same.

Code:
# mdadm --misc --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e
  Creation Time : Sat Mar  6 00:43:43 2010
     Raid Level : raid5
  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)
     Array Size : 102558528 (97.81 GiB 105.02 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Wed Jan 16 04:47:26 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : b4a5715a - correct
         Events : 21

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       33        2      active sync   /dev/sdc1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1


sdc2, sdc3, sdc5, sdc6, and sdc7 are all about the same, i'm confused to why sdb and sdc are in state "active sync" here.


is there a way to rebuild my raid partitions?
_________________
help the needy
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Thu Feb 07, 2013 11:39 pm    Post subject: Reply with quote

What bothers me is that the array is marked "Clean" which means something was done with the array after the power outage - they should be marked "dirty."

Usually when a disk gets kicked from an array, the kicked disk, if it couldn't be written to at the time of kicking, will still have a superblock on it. If it comes back on a power cycle - that's why sometimes they show up as a dirty disk with all other array members missing.

You may need to recreate the superblock on all disks again, but I don't know how you created the array to begin with. Or it's probably possible to hack the missing disks and fake it so that it thinks it's clean. But since it looks more than one disk was removed from a raid 5, you may be facing data loss even if you could get it to reassemble.

This is one of the reasons why I want to get a UPS for my 4 disk RAID5 :( and RAID is not backup...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
remix
l33t
l33t


Joined: 28 Apr 2004
Posts: 797
Location: hawaii

PostPosted: Fri Feb 08, 2013 12:29 am    Post subject: Reply with quote

somehow i got mdadm to see the third disk, so then it assembled. i looked and it seems that the fourth is in fact toast. so i've ordered a replacement, and am slowly moving all my backups onto amazon glacier.

i guess i got lucky this time.

the way i got mdadm to see it is to (i had to repeatedly do this) run mdadm -S /dev/md1 to stop it, then reassemble /dev/md1 again.
or stop all /dev/mdX devices, and the reassemble again.
this process took over 8 hours across 2 days cause i'm not very knowledgeable or skilled in this.
_________________
help the needy
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum