mdadm won't start array automatically

Message

redwood · Post by **redwood** » Tue Feb 25, 2020 7:24 pm

We had a power failure and the system itself down when the battery of the SmartUPS ran down after an hour.

After starting the system, my soft (mdadm) RAID1 (mirror) drives didn't start properly.

I have 3 WesternDigital 2-T drives in the system:

Code: Select all

/dev/sda: ATA       WDC WD2000FYYZ-0  1K04
/dev/sdb: ATA       WDC WD2000FYYZ-0  1K04
/dev/sdc: ATA       WDC WD2000FYYZ-0  1K04

They're all in good shape:

Code: Select all

apps /var/log # for f in /dev/sd{a,b,c}; do echo $f; smartctl -i $f; smartctl -H $f; done
/dev/sda
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2000FYYZ-01UL1B3
Serial Number:    WD-WMC1P0F42F5Y
LU WWN Device Id: 5 0014ee 059796fbc
Firmware Version: 01.01K04
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Feb 25 14:04:47 2020 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

/dev/sdb
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2000FYYZ-01UL1B1
Serial Number:    WD-WCC1PLEN7LEP
LU WWN Device Id: 5 0014ee 2b5253de4
Firmware Version: 01.01K02
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Feb 25 14:04:47 2020 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

/dev/sdc
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2000FYYZ-01UL1B1
Serial Number:    WD-WCC1PLEN7TSH
LU WWN Device Id: 5 0014ee 2b52539a9
Firmware Version: 01.01K02
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Feb 25 14:04:47 2020 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.27-gentoo-r1] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

I originally set up mirror RAID1 as follows:

Code: Select all

# md0 -- /boot sda1, sdb1, sdc1
# md1 -- /     sda2, sdb2, sdc2(spare)
# md4 -- LVM   sda3, sdb3, sdc3(spare)
# /dev/sdd 4Gb backup
# mdadm --examine --brief --scan --config=partition
ARRAY /dev/md/0  metadata=1.0 UUID=bef11d03:c109d8c6:27cfea12:e81de4ab name=apps:0
ARRAY /dev/md/1  metadata=1.2 UUID=62912fbe:63505255:33686c48:55f3919f name=apps:1 spares=1
ARRAY /dev/md/4  metadata=1.2 UUID=e75144aa:42bc2701:7b0363ba:e528bb48 name=apps:4 spares=1

After booting now, mdadm won't use /dev/sda or /dev/sdb and instead assembles the arrays with just /dev/sdc[1,2,3]:

Code: Select all

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] [linear] [multipath] 
md0 : active raid1 sdc1[5]
      102336 blocks super 1.0 [3/1] [U__]
      
md4 : active raid1 sdc3[4]
      1950134144 blocks super 1.2 [2/1] [U_]
      
md1 : active raid1 sdc2[3]
      3143616 blocks super 1.2 [2/1] [U_]
      
unused devices: <none>

I can add back the missing drive partitions without issue:

Code: Select all

mdadm /dev/md0 -a /dev/sda1 /dev/sdb1
mdadm: re-added /dev/sda1
mdadm: added /dev/sdb1

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] [linear] [multipath] 
md0 : active raid1 sdb1[4] sda1[3] sdc1[5]
      102336 blocks super 1.0 [3/3] [UUU]
      
md4 : active raid1 sdc3[4]
      1950134144 blocks super 1.2 [2/1] [U_]
      
md1 : active raid1 sdc2[3]
      3143616 blocks super 1.2 [2/1] [U_]
      
unused devices: <none>

OK so far, but now stopping /dev/md0 and re-starting it (assembling from /dev/sd[abc]1 ):

Code: Select all

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm -A --verbose /dev/md0 
mdadm: looking for devices for /dev/md0
mdadm: No super block found on /dev/sdc3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc3
mdadm: No super block found on /dev/sdc2 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc2
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: added /dev/sdc1 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 1 drive (out of 3).

Not sure what the "no uptodate device for slot" message means,
but from the above output it looks like mdadm didn't find/try /dev/sd{a,b}1
I've also gone ahead and tried failing the devices and re-adding:

Code: Select all

mdadm /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm /dev/md0 --re-add /dev/sdb1 --re-add /dev/sdc1

But when I stop array /dev/md0 and restart, it still just starts up with /dev/sdc1 instead of all three /dev/sd[abc]1

Post by **NeddySeagoon** » Tue Feb 25, 2020 10:34 pm

redwood,

First, have no confidence in

Code: Select all

 === START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

You need to read the paramater table you get with the -a or -x options to smartctl.

What does

Code: Select all

mdadm -E /dev/sd[abc]1

have to say about your /boot raid set.

We can poke at

Code: Select all

mdadm -E /dev/sd[abc]2

and

Code: Select all

mdadm -E /dev/sd[abc]3

but the method is the same.

I guess its too late to say assemble the raid in read only mode.

redwood · Post by **redwood** » Wed Feb 26, 2020 2:45 pm

Looks good to me:

Code: Select all

# mdadm -E /dev/sd[a,b,c]1

/dev/sda1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : bef11d03:c109d8c6:27cfea12:e81de4ab
           Name : apps:0  (local to host apps)
  Creation Time : Wed Jul  2 22:32:52 2014
     Raid Level : raid1
   Raid Devices : 3

 Avail Dev Size : 204768 (99.98 MiB 104.84 MB)
     Array Size : 102336 (99.94 MiB 104.79 MB)
  Used Dev Size : 204672 (99.94 MiB 104.79 MB)
   Super Offset : 204784 sectors
   Unused Space : before=0 sectors, after=104 sectors
          State : clean
    Device UUID : fb908a0d:11efc048:eea6f588:e5fd2ba7

    Update Time : Tue Feb 25 16:07:58 2020
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : ddecacfe - correct
         Events : 675


   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : bef11d03:c109d8c6:27cfea12:e81de4ab
           Name : apps:0  (local to host apps)
  Creation Time : Wed Jul  2 22:32:52 2014
     Raid Level : raid1
   Raid Devices : 3

 Avail Dev Size : 204768 (99.98 MiB 104.84 MB)
     Array Size : 102336 (99.94 MiB 104.79 MB)
  Used Dev Size : 204672 (99.94 MiB 104.79 MB)
   Super Offset : 204784 sectors
   Unused Space : before=0 sectors, after=104 sectors
          State : clean
    Device UUID : 9557a0f3:8dd0eb5c:0d3746ac:4e47e4d7

    Update Time : Tue Feb 25 16:07:58 2020
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 2c362e9f - correct
         Events : 675


   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : bef11d03:c109d8c6:27cfea12:e81de4ab
           Name : apps:0  (local to host apps)
  Creation Time : Wed Jul  2 22:32:52 2014
     Raid Level : raid1
   Raid Devices : 3

 Avail Dev Size : 204768 (99.98 MiB 104.84 MB)
     Array Size : 102336 (99.94 MiB 104.79 MB)
  Used Dev Size : 204672 (99.94 MiB 104.79 MB)
   Super Offset : 204784 sectors
   Unused Space : before=0 sectors, after=104 sectors
          State : clean
    Device UUID : ff092c52:85a0cd5f:e11bcc55:3ed1f8a8

    Update Time : Tue Feb 25 16:07:58 2020
  Bad Block Log : 512 entries available at offset -8 sectors
       Checksum : 83e1fc5 - correct
         Events : 675


   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

Right now I'm doing the following on the live system to get the RAID1 disks back to how I set them up with /dev/sd[a,b] as the set and /dev/sdc as the spare:

Code: Select all

mdadm /dev/md1 -a /dev/sda2
then wait for sync to finish, then fail sdc 

and replace with sdb
mdadm /dev/md1 --fail /dev/sdc2 --remove /dev/sdc2

then add back sdc as spare
mdadm /dev/md1 -a /dev/sdb2 

Then do same thing for /dev/md4

I've burnt a systemrescue-6.0.7 CD, if I need to reassemble the raid sets without booting into the live system.
The system is working fine AFAIK, but it's annoying to say the least, to be booting from a single spare disk rather than the mirror set.
Kind of defeats the purpose of having RAID1 and the resync of the 2T disks takes hours. Had thought about trying out btrfs instead of ext4
as I think that btrfs doesn't need to resync unused blocks in a raid mirror. OTOH, ext4 has worked fine for me for a long, long time on this server
so no real need to reinstall to btrfs mirror drives for just a lot of extra work.

Have had the system shutdown itself from a power failure before and on reboot have a disk kicked out as dirty, but the system has always just
booted from the other disk in the set and has never kicked both mirrors out and then booted from the spare. Very weird.

Post by **NeddySeagoon** » Wed Feb 26, 2020 6:15 pm

redwood,

As you say, that all looks good.

Jaglover · Post by **Jaglover** » Wed Feb 26, 2020 6:29 pm

The last question remains, why didn't your UPS shut down your box?