Some mirrors fail to assemble on boot

[ese002]

I replaced a failed disk. I booted the surviving disk as sda, partitioned sdb and reassembled the raid.
I then swapped the cables and rebooted.

When the system came up (sda is the new disk), some arrays assembled. Others did not. Those that failed to assemble came up just sda*. That is, only the new disk, not the old one.

I re-assembled the offending arrays. On the next boot: same thing. Same volumes assembled. Same volumes failed to assemble.

crab etc # dmesg |grep md

[ 2.167506] md: raid0 personality registered for level 0
[ 2.176187] md: raid1 personality registered for level 1
[ 2.924889] md: Waiting for all devices to be available before autodetect
[ 2.933752] md: If you don't use raid, use raid=noautodetect
[ 2.965162] md: Autodetecting RAID arrays.
[ 3.001610] md: invalid raid superblock magic on sda9
[ 3.010385] md: sda9 does not have a valid v0.90 superblock, not importing!
[ 3.062272] md: Scanned 9 and added 8 devices.
[ 3.071024] md: autorun ...
[ 3.079617] md: considering sdb3 ...
[ 3.088176] md: adding sdb3 ...
[ 3.096749] md: sdb1 has different UUID to sdb3
[ 3.105303] md: sda10 has different UUID to sdb3
[ 3.113807] md: sda8 has different UUID to sdb3
[ 3.122291] md: sda7 has different UUID to sdb3
[ 3.130612] md: sda6 has different UUID to sdb3
[ 3.138790] md: adding sda3 ...
[ 3.146967] md: sda1 has different UUID to sdb3
[ 3.155371] md: created md3
[ 3.163479] md: bind<sda3>
[ 3.171526] md: bind<sdb3>
[ 3.179517] md: running: <sdb3><sda3>
[ 3.195739] md/raid1:md3: active with 2 out of 2 mirrors
[ 3.203772] md3: detected capacity change from 0 to 50001346560
[ 3.211807] md: considering sdb1 ...
[ 3.219727] md: adding sdb1 ...
[ 3.227556] md: sda10 has different UUID to sdb1
[ 3.235395] md: sda8 has different UUID to sdb1
[ 3.243226] md: sda7 has different UUID to sdb1
[ 3.250987] md: sda6 has different UUID to sdb1
[ 3.258687] md: adding sda1 ...
[ 3.266462] md: created md1
[ 3.273900] md: bind<sda1>
[ 3.281313] md: bind<sdb1>
[ 3.288653] md: running: <sdb1><sda1>
[ 3.296064] md/raid1:md1: active with 2 out of 2 mirrors
[ 3.303386] md1: detected capacity change from 0 to 32768000
[ 3.310674] md: considering sda10 ...
[ 3.317846] md: adding sda10 ...
[ 3.324895] md: sda8 has different UUID to sda10
[ 3.331942] md: sda7 has different UUID to sda10
[ 3.338925] md: sda6 has different UUID to sda10
[ 3.346056] md: created md10
[ 3.352905] md: bind<sda10>
[ 3.359707] md: running: <sda10>
[ 3.366549] md/raid1:md10: active with 1 out of 2 mirrors
[ 3.373222] md10: detected capacity change from 0 to 199997521920
[ 3.379845] md: considering sda8 ...
[ 3.386235] md: adding sda8 ...
[ 3.392412] md: sda7 has different UUID to sda8
[ 3.398614] md: sda6 has different UUID to sda8
[ 3.404904] md: created md8
[ 3.410845] md: bind<sda8>
[ 3.422722] md: running: <sda8>
[ 3.428752] md/raid1:md8: active with 1 out of 2 mirrors
[ 3.434730] md8: detected capacity change from 0 to 10001842176
[ 3.440769] md: considering sda7 ...
[ 3.446673] md: adding sda7 ...
[ 3.452456] md: sda6 has different UUID to sda7
[ 3.458428] md: created md7
[ 3.464138] md: bind<sda7>
[ 3.469757] md: running: <sda7>
[ 3.475461] md/raid1:md7: active with 1 out of 2 mirrors
[ 3.481059] md7: detected capacity change from 0 to 10001842176
[ 3.486738] md: considering sda6 ...
[ 3.492244] md: adding sda6 ...
[ 3.497787] md: created md6
[ 3.503107] md: bind<sda6>
[ 3.508380] md: running: <sda6>
[ 3.513717] md/raid1:md6: active with 1 out of 2 mirrors
[ 3.518950] md6: detected capacity change from 0 to 10001842176
[ 3.524222] md: ... autorun DONE.
[ 3.551173] md3: unknown partition table
[ 3.601071] EXT3-fs (md3): mounted filesystem with writeback data mode
[ 6.867182] md7: unknown partition table
[ 6.891494] md6: unknown partition table
[ 6.902622] md1: unknown partition table
[ 6.928435] md10: unknown partition table
[ 6.937242] md8: unknown partition table
[ 7.754573] md: bind<sdc1>
[ 7.792388] md: bind<sde1>
[ 7.816807] md/raid0:md11: looking at sde1
[ 7.816813] md/raid0:md11: comparing sde1(390716672) with sde1(390716672)
[ 7.816817] md/raid0:md11: END
[ 7.816819] md/raid0:md11: ==> UNIQUE
[ 7.816822] md/raid0:md11: 1 zones
[ 7.816825] md/raid0:md11: looking at sdc1
[ 7.816828] md/raid0:md11: comparing sdc1(398283264) with sde1(390716672)
[ 7.816831] md/raid0:md11: NOT EQUAL
[ 7.816834] md/raid0:md11: comparing sdc1(398283264) with sdc1(398283264)
[ 7.816838] md/raid0:md11: END
[ 7.816840] md/raid0:md11: ==> UNIQUE
[ 7.816842] md/raid0:md11: 2 zones
[ 7.816844] md/raid0:md11: FINAL 2 zones
[ 7.816851] md/raid0:md11: zone 1
[ 7.816853] md/raid0:md11: checking sde1 ... nope.
[ 7.816856] md/raid0:md11: checking sdc1 ... contained as device 0
[ 7.816860] md/raid0:md11: (398283264) is smallest!.
[ 7.816863] md/raid0:md11: zone->nb_dev: 1, sectors: 7566592
[ 7.816866] md/raid0:md11: current zone start: 398283264
[ 7.816868] md/raid0:md11: done.
[ 7.816872] md/raid0:md11: md_size is 788999936 sectors.
[ 7.816874] ******* md11 configuration *********
[ 7.816911] md11: detected capacity change from 0 to 403967967232
[ 7.829044] md11: unknown partition table
[ 7.861917] md: bind<sdb5>
[ 8.005361] md: bind<sda5>
[ 8.016935] md/raid1:md5: active with 2 out of 2 mirrors
[ 8.016974] md5: detected capacity change from 0 to 1998585856
[ 8.062509] md5: unknown partition table
[ 12.479306] EXT3-fs (md3): using internal journal
[ 12.723315] EXT3-fs (md6): using internal journal
[ 12.723321] EXT3-fs (md6): mounted filesystem with writeback data mode
[ 12.831224] EXT3-fs (md7): using internal journal
[ 12.831229] EXT3-fs (md7): mounted filesystem with writeback data mode
[ 12.940049] EXT3-fs (md8): using internal journal
[ 12.940054] EXT3-fs (md8): mounted filesystem with writeback data mode
[ 13.024937] EXT3-fs (md10): using internal journal
[ 13.024943] EXT3-fs (md10): mounted filesystem with writeback data mode
[ 13.049296] XFS (md11): Mounting Filesystem
[ 13.273112] XFS (md11): Ending clean mount

sda9 and sdb9 are not raid volumes. Never had been. They mount fine.
md11 is a raid0 of two physically separate disks unrelated to sda or sdb. Works fine.

I thought maybe the UUID had changed. But when I run mdadm --examine --scan, I get the same UUID's as in my long working mdadm.conf

crab etc # cat mdadm.conf
AUTO -all
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=1b988090:fb5536ec:25d03197:8f16cd56
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=6e357e0d:d0a58339:c50cf1e1:8711e7b4
ARRAY /dev/md5 level=raid1 num-devices=2 UUID=d447d1d9:9aa4683f:e8190025:7b99ad16
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=785d5e85:758933bc:644de436:36df9e00
ARRAY /dev/md7 level=raid1 num-devices=2 UUID=e9d36152:70f1f16a:fccbe436:3229c404
ARRAY /dev/md8 level=raid1 num-devices=2 UUID=23a55e83:e29c2d2f:4846f882:bcc67290
#ARRAY /dev/md9 level=raid1 num-devices=2 UUID=cf21b733:f44fe6ea:77c04c05:e565de93
ARRAY /dev/md10 level=raid1 num-devices=2 UUID=e7306cd9:d7de3f12:1714a1b9:f3a21ec8
ARRAY /dev/md11 level=raid0 num-devices=2 UUID=c6f86aa9:d0c8e92c:b006d13c:98a0e4df

My kernel version is linux-2.6.39-gentoo-r3
I have not updated my kernel since the old raid was working.

The working arrays are md1, md3, and md5: boot, root, and swap. Boot and root are primary partitions. The rest are logical.


crab etc # cat sda.part
Partition Table for /dev/sda

---Starting---- ----Ending----- Start Number of
# Flags Head Sect Cyl ID Head Sect Cyl Sector Sectors
-- ----- ---- ---- ----- ---- ---- ---- ----- ----------- -----------
1 0x80 1 1 0 0xFD 254 63 7 63 128457
2 0x00 0 1 8 0x07 254 63 1312 128520 20964825
3 0x00 0 1 1313 0xFD 254 63 7391 21093345 97659135
4 0x00 0 1 7392 0x05 254 63 84228 118752480 1234386405
5 0x00 1 1 7392 0x82 254 63 7634 63 3903732
6 0x00 1 1 7635 0xFD 254 63 8850 63 19534977
7 0x00 1 1 8851 0xFD 254 63 10066 63 19534977
8 0x00 1 1 10067 0xFD 254 63 11282 63 19534977
9 0x00 1 1 11283 0xFD 254 63 59913 63 781256952
10 0x00 1 1 59914 0xFD 254 63 84228 63 390620412
crab etc # cat sdb.part
Partition Table for /dev/sdb

---Starting---- ----Ending----- Start Number of
# Flags Head Sect Cyl ID Head Sect Cyl Sector Sectors
-- ----- ---- ---- ----- ---- ---- ---- ----- ----------- -----------
1 0x80 1 1 0 0xFD 254 63 3 63 64197
2 0x00 0 1 4 0x07 254 63 1308 64260 20964825
3 0x00 0 1 1309 0xFD 254 63 7387 21029085 97659135
4 0x00 0 1 7388 0x05 254 63 84224 118688220 1234386405
5 0x00 1 1 7388 0x83 254 63 7630 63 3903732
6 0x00 1 1 7631 0x83 254 63 8846 63 19534977
7 0x00 1 1 8847 0x83 254 63 10062 63 19534977
8 0x00 1 1 10063 0x83 254 63 11278 63 19534977
9 0x00 1 1 11279 0x83 254 63 59909 63 781256952
10 0x00 1 1 59910 0x83 254 63 84224 63 390620412

sda1 is larger than sdb1. That is deliberate. I have in mind to increase the size of /boot once I get everything working and have some time to do a restructure. Could this be an issue? md3 and md5 come after and they assemble fine.

I run stable and everything is current.