| View previous topic :: View next topic |
| Author |
Message |
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1095
Location: Edinburgh, UK
|
 Posted: Sat Nov 30, 2013 12:17 pm Post subject: mdadm/RAID1 auto-rebuild |
 |
|
Hello,
I was trying to set up a system with a raid1 volume (for some semi-precious data - but before you say it, yes there are also backups), and I want to make the failover process as simple as possible, to wit: when the ArrayDegraded message is received, shut the machine down, replace the faulty drive, boot back up and the rebuild begins.
The drives I'm using are USB thumbdrives, the idea being that they're cheap/quick to source a replacement, and can be stored away securely when the machine's not in use. What I've learned so far is that this means the array must use partitions, not just raw drives, in case a replacement drive is a bit too small. This way I can make the raid partition on each drive a bit smaller than the published capacity (100MB smaller is suggested) so any drive in the same "size-class" (eg 8GB) can reliably carry a partition of the correct size.
That means, though, that a step in the process has to be added: the new drive has to have its partition map overwritten (with a clone of the other drive/s) before it can join the array. (I think it might also need a new unique ID to be created in some manner to prevent confusion, but I haven't gotten as far as investigating this yet.)
Can anyone suggest a sane way of going about this? As I read it, if I add a POLICY line to mdadm.conf that should take care of auto-adding the new drive, but then again this may not be what I want if I have to prep the disk first. This type of hot-swapping seems like it would be a fairly standard requirement, but it's pretty confusing as to how it should work. Any wizards in the house? |
|
| Back to top |
|
 |
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2475
Location: Germany
|
| Posted: Sat Nov 30, 2013 2:30 pm Post subject: Re: mdadm/RAID1 auto-rebuild |
|
|
| Havin_it wrote: | | shut the machine down, replace the faulty drive |
You can do it without the shutdown, if you have a hot plug bay (or otherwise easy to reach HDD in your case).
| Havin_it wrote: | | The drives I'm using are USB thumbdrives |
USB usually sucks for RAID (shared bandwidth), but no shutdown necessary there either. Just plug the new one in and go.
| Havin_it wrote: | | What I've learned so far is that this means the array must use partitions, not just raw drives, in case a replacement drive is a bit too small. |
That's not true; even without partitions, you can tell mdadm not to use the full size; and even if you notice drive a bit too small afterwards, you could still shrink the filesystem & shrink the RAID. It's not the end of the world.
For fully automated rebuilds - I can't really help you, never done it this way; how to avoid some other drive being inadvertently overwritten by sync because it was confused with a replacement drive? Doing it manually seems a lot safer IMO, provided you know what you're about. |
|
| Back to top |
|
|
Havin_it
Veteran
Joined: 17 Jul 2005
Posts: 1095
Location: Edinburgh, UK
|
| Posted: Sat Nov 30, 2013 7:00 pm Post subject: Re: mdadm/RAID1 auto-rebuild |
|
|
| frostschutz wrote: | | Havin_it wrote: | | shut the machine down, replace the faulty drive |
You can do it without the shutdown, if you have a hot plug bay (or otherwise easy to reach HDD in your case). |
Yeah that was my initial plan, unfortunately it's a Raspberry Pi with the thumbdrives directly attached, and hotplugging makes it panic 
(And a powered hub is not in the [micro]budget.)
TBH I'm happier signing off on this project if I know it will shut itself down in the event of failure, so the end-users don't just ignore the warnings. (I'm writing the monitoring script for that as I speak.)
| frostschutz wrote: | | Havin_it wrote: | | The drives I'm using are USB thumbdrives |
USB usually sucks for RAID (shared bandwidth), but no shutdown necessary there either. Just plug the new one in and go. |
It's not great I know, but it's a fairly low I/O appliance so it's tolerable.
| frostschutz wrote: | | Havin_it wrote: | | What I've learned so far is that this means the array must use partitions, not just raw drives, in case a replacement drive is a bit too small. |
That's not true; even without partitions, you can tell mdadm not to use the full size; and even if you notice drive a bit too small afterwards, you could still shrink the filesystem & shrink the RAID. It's not the end of the world. |
I should really read the manpage more thoroughly 
I'll give this some testing, but sounds good. AIUI this should also mean a removed disk could be read independently, which would be a bonus.
| frostschutz wrote: | | For fully automated rebuilds - I can't really help you, never done it this way; how to avoid some other drive being inadvertently overwritten by sync because it was confused with a replacement drive? Doing it manually seems a lot safer IMO, provided you know what you're about. |
Knowing what I'm about gives me the confidence to automate it 
Seriously, I know what you mean but with two slots and mdstat output it's only ever going to be sda or sdb that's borked, and I can supplement mdadm's email with a more idiot-proof instruction to the device's users as to which drive to replace before they turn it back on. That's a lot less risk than telling them to SSH in and do X and Y... |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2475
Location: Germany
|
| Posted: Sun Dec 01, 2013 8:12 pm Post subject: |
|
|
You have a special use case then - go for it and share your solution with us  |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
