Gentoo Forums :: View topic - [SOLVED] raid10 array suddenly drives into spares.

[SOLVED] raid10 array suddenly drives into spares.

View unanswered posts
View posts from last 24 hours

Gentoo Forums Forum Index

Kernel & Hardware

View previous topic :: View next topic

Author

Message

likewhoa
l33t
l33t

Joined: 04 Oct 2006
Posts: 728
Location: Brooklyn, New York

Posted: Fri Apr 27, 2012 9:03 pm Post subject: [SOLVED] raid10 array suddenly drives into spares.

I have just run into a seriously odd issue with mdadm and I am not sure what is going on. Below is some information about my current situation. The array in question is /dev/md1 and devices from /dev/sd[abcdefgh]3

Code:

> mdadm -Esv
ARRAY /dev/md/1 metadata=1.0 num-devices=0 UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
spares=7 devices=/dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdb3,/dev/sda3,/dev/sdc3
ARRAY /dev/md/1 level=raid10 metadata=1.0 num-devices=8 UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
devices=/dev/sdh3

Code:

> mdadm --version
mdadm - v3.2.3 - 23rd December 2011

Code:

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f013

Device Boot Start End Blocks Id System
/dev/sda1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sda2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sda3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

Device Boot Start End Blocks Id System
/dev/sdb1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdb2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdb3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7d10530f

Device Boot Start End Blocks Id System
/dev/sdc1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdc2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdc3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x81a213ab

Device Boot Start End Blocks Id System
/dev/sdd1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdd2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdd3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sde: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00e

Device Boot Start End Blocks Id System
/dev/sde1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sde2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sde3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4644f00c

Device Boot Start End Blocks Id System
/dev/sdf1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdf2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdf3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdg: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x327d8d82

Device Boot Start End Blocks Id System
/dev/sdg1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdg2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdg3 73779200 976773119 451496960 fd Linux raid autodetect

Disk /dev/sdh: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e460f

Device Boot Start End Blocks Id System
/dev/sdh1 2048 2099199 1048576 82 Linux swap / Solaris
/dev/sdh2 2099200 73779199 35840000 fd Linux raid autodetect
/dev/sdh3 73779200 976773119 451496960 fd Linux raid autodetect

Code:

> mdadm -E /dev/sd[abcdefgh]3
/dev/sda3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : 332e805b:b3b16368:3bce5c58:e8ee799f

Update Time : Fri Apr 27 16:10:05 2012
Checksum : b3463d4f - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : 8669f047:4d4dbb90:39cc0da1:920cec84

Update Time : Fri Apr 27 16:10:05 2012
Checksum : f6312fe1 - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : ad80f917:9e6d8299:d919e462:667e6a32

Update Time : Fri Apr 27 16:10:05 2012
Checksum : 3e5626ce - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : f45ef9ec:b7729bf6:31eab8e9:f997f154

Update Time : Fri Apr 27 16:10:05 2012
Checksum : 19caf41e - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sde3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : f9d52087:4a3030c3:fcfb04e2:e9b588f1

Update Time : Fri Apr 27 16:10:05 2012
Checksum : 15685871 - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : 20b45977:15cb995a:683ddd07:1bfda597

Update Time : Fri Apr 27 16:10:05 2012
Checksum : 69025a00 - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : -unknown-
Raid Devices : 0

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : active
Device UUID : 7689dd7c:da80570e:37b3250d:3aaa1b01

Update Time : Fri Apr 27 16:10:05 2012
Checksum : 91020805 - correct
Events : 1

Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 828ed03d:0c28afda:4a636e88:7b29ec9f
Name : Darkside:1 (local to host Darkside)
Creation Time : Sun Aug 15 21:12:34 2010
Raid Level : raid10
Raid Devices : 8

Avail Dev Size : 902993648 (430.58 GiB 462.33 GB)
Array Size : 3611971584 (1722.32 GiB 1849.33 GB)
Used Dev Size : 902992896 (430.58 GiB 462.33 GB)
Super Offset : 902993904 sectors
State : clean
Device UUID : 00565578:e2eaaba3:f1eae17c:f474ee8d

Update Time : Wed Apr 25 17:22:58 2012
Checksum : 1e7c3692 - correct
Events : 82942

Layout : far=2
Chunk Size : 256K

Device Role : Active device 0
Array State : AAAAAAAA ('A' == active, '.' == missing)

Code:

> uname -r
3.3.3-gentoo

I am currently scrambling to find a solution so I figure I leave this here to get some more feedback and hopefully get my data back which I do have backups for but it's more rewarding to troubleshoot issues like this and solve them.

Thanks in advanced.
likewhoa

Last edited by likewhoa on Tue May 01, 2012 4:39 pm; edited 1 time in total

likewhoa
l33t
l33t

Joined: 04 Oct 2006
Posts: 728
Location: Brooklyn, New York

Posted: Sat Apr 28, 2012 7:25 pm Post subject:

so this is one nasty bug I ran into early which was back ported to -stable kernels 3.3. Anyways, what it did was to remove key data from the metadata. It makes the
devices look like a spare for an array of unknown level with no devices and no chunk size, and the fix was to recreate the array with the same order as before but since I created this array back in 2009 when I didn't think to keep the command handy for such an event. I had to perform a series of commands to figure out the correct mappings of partitions.

The following is what I ran to obtain these matches.

Code:

#!/bin/bash
for i in a b c d e f g h
do for j in a b c d e f g h
do if [ $i = $j ] ; then continue; fi
if cmp --ignore-initial=0:231166181376 --bytes=231166181376 /dev/sd${i}3 /dev/sd${j}3
then echo /dev/sd${i}3 and /dev/sd${j}3 seem to match
fi
done
done

which produced an output like:

Quote:

/dev/sda3 /dev/sdb3 differ: byte 1, line 1
/dev/sda3 /dev/sdc3 differ: byte 262145, line 2
/dev/sda3 /dev/sdd3 differ: byte 1, line 1
/dev/sda3 /dev/sde3 differ: byte 1, line 1
/dev/sda3 and /dev/sdf3 seem to match
/dev/sda3 /dev/sdg3 differ: byte 262145, line 2
/dev/sda3 /dev/sdh3 differ: byte 1, line 1
/dev/sdb3 /dev/sda3 differ: byte 1, line 1
/dev/sdb3 /dev/sdc3 differ: byte 1, line 1
/dev/sdb3 /dev/sdd3 differ: byte 1, line 1
/dev/sdb3 and /dev/sde3 seem to match
/dev/sdb3 /dev/sdf3 differ: byte 1, line 1
/dev/sdb3 /dev/sdg3 differ: byte 1, line 1
/dev/sdb3 /dev/sdh3 differ: byte 1, line 1
/dev/sdc3 /dev/sda3 differ: byte 262145, line 2
/dev/sdc3 /dev/sdb3 differ: byte 1, line 1
/dev/sdc3 /dev/sdd3 differ: byte 1, line 1
/dev/sdc3 /dev/sde3 differ: byte 1, line 1
/dev/sdc3 /dev/sdf3 differ: byte 262145, line 2
/dev/sdc3 and /dev/sdg3 seem to match
/dev/sdc3 /dev/sdh3 differ: byte 1, line 1
/dev/sdd3 /dev/sda3 differ: byte 1, line 1
/dev/sdd3 /dev/sdb3 differ: byte 1, line 1
/dev/sdd3 /dev/sdc3 differ: byte 1, line 1
/dev/sdd3 /dev/sde3 differ: byte 1, line 1
/dev/sdd3 /dev/sdf3 differ: byte 1, line 1
/dev/sdd3 /dev/sdg3 differ: byte 1, line 1
/dev/sdd3 and /dev/sdh3 seem to match
/dev/sde3 /dev/sda3 differ: byte 262145, line 2
/dev/sde3 /dev/sdb3 differ: byte 1, line 1
/dev/sde3 and /dev/sdc3 seem to match
/dev/sde3 /dev/sdd3 differ: byte 1, line 1
/dev/sde3 /dev/sdf3 differ: byte 262145, line 2
/dev/sde3 /dev/sdg3 differ: byte 262145, line 2
/dev/sde3 /dev/sdh3 differ: byte 1, line 1
/dev/sdf3 /dev/sda3 differ: byte 1, line 1
/dev/sdf3 /dev/sdb3 differ: byte 1, line 1
/dev/sdf3 /dev/sdc3 differ: byte 1, line 1
/dev/sdf3 and /dev/sdd3 seem to match
/dev/sdf3 /dev/sde3 differ: byte 1, line 1
/dev/sdf3 /dev/sdg3 differ: byte 1, line 1
/dev/sdf3 /dev/sdh3 differ: byte 1, line 1
/dev/sdg3 and /dev/sda3 seem to match
/dev/sdg3 /dev/sdb3 differ: byte 1, line 1
/dev/sdg3 /dev/sdc3 differ: byte 262145, line 2
/dev/sdg3 /dev/sdd3 differ: byte 1, line 1
/dev/sdg3 /dev/sde3 differ: byte 1, line 1
/dev/sdg3 /dev/sdf3 differ: byte 262145, line 2
/dev/sdg3 /dev/sdh3 differ: byte 1, line 1
/dev/sdh3 /dev/sda3 differ: byte 1, line 1
/dev/sdh3 and /dev/sdb3 seem to match
/dev/sdh3 /dev/sdc3 differ: byte 1, line 1
/dev/sdh3 /dev/sdd3 differ: byte 1, line 1
/dev/sdh3 /dev/sde3 differ: byte 1, line 1
/dev/sdh3 /dev/sdf3 differ: byte 1, line 1
/dev/sdh3 /dev/sdg3 differ: byte 1, line 1

now I had some pairs and with 2 attempts at 'mdadm --create ...' the magic order came out to be.

Code:

mdadm --create /dev/md1 --metadata=1.0 -l10 -n8 --chunk=256 --layout=f2 --assume-clean /dev/sdh3 /dev/sdb3 /dev/sde3 /dev/sdc3 /dev/sdg3 /dev/sda3 /dev/sdf3 /dev/sdd3

So recovery was a success \o/ much credit goes to Neil Brown from linux-raid ml for help on this issue.

SOLVED

DaggyStyle
Advocate

Joined: 22 Mar 2006
Posts: 4735

Posted: Sun Apr 29, 2012 6:53 am Post subject:

please modify the title so it will reflect the status of this issue. _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein ProjectFootball

Display posts from previous:

	Gentoo Forums Forum Index Kernel & Hardware	All times are GMT
Page 1 of 1

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Copyright 2001-2013 Gentoo Foundation, Inc. Designed by Kyle Manna © 2003; Style derived from original subSilver theme. | Hosting by Gossamer Threads Inc. © | Powered by phpBB 2.0.23-gentoo-p9 © 2001, 2002 phpBB Group