convert RAID-5 to RAID-6 with kernel>=2.6.31 & mdadm>=3.1

dogshu

Hi all! The newest versions of the linux kernel and mdadm support converting between RAID modes:
http://lwn.net/Articles/358682/

After a recent disk failure and subsequent wait for a replacement disk from newegg led me to go almost a week with no redundancy on my RAID-5 array, I decided to convert to RAID-6. Here is my story, which is based on this detailed post by Neil Brown, the mdadm maintainer:
http://neil.brown.name/blog/20090817000931

I am converting a 7 disk RAID-5 array to an 8 disk RAID-6 array. First of all I made sure I was running the right kernel and mdadm version. I compulsively upgrade my kernel, so I was already running 2.6.32.3. Gentoo is still on mdadm 3.0, so I typed "=sys-fs/mdadm-3.1.1" into /etc/portage/package.keywords/mdadm and emerged the new version.

To add the 8th disk to my existing RAID-5 array, I ran:
mdadm --add /dev/md3 /dev/sdh3

this added /dev/sdh3 as a hot spare. Then, to convert this to a RAID-6 array, I ran:
mdadm --grow /dev/md3 --level=6 --raid-devices=8 --backup-file=/nfs/media/tmp/md3.backup

I had tried to run the --grow command without the --backup-file argument, as Neil's post seems to say that a backup file is not necessary when a hot spare is present. But mdadm wasn't having it, it told me:
mdadm level of /dev/md3 changed to raid6
mdadm: /dev/md3: Cannot grow - need backup-file
mdadm: aborting level change

With the --backup-file argument everything seems to be working fine. Here's the relevant part of my /proc/mdstat:

xibo · Posted: Mon Jan 11, 2010 9:19 pm Post subject:

Hi

It shouldn't be that slow. I converted from raid5 to taid6 when going from 5 disks to 6, each disk had 1TB and for nowadays average specs ( hitachi ultrastar ak10000(a) ), and it was reshaping at about 10-12MB/s per disk... took about a day in time and a processor core in load.

Newer kernels have mdraid multithreading support, though i haven't tryed that so far.

EDIT: if backing up all data on the raid before leveling it is an option, you might want to backup anything and tweak a bit on the chunk/block sizes used by mdraid - the performance gain can be quite noticable

Mad Merlin · Veteran Joined: 09 May 2005 Posts: 1155

drescherjm · Posted: Mon Jan 31, 2011 11:48 pm Post subject:

I am getting even worse raid reshape conversion from raid 5 to 6 on a fast machine with a ~50GB array that normally resyncs in less than 5 minutes..

drescherjm · Posted: Sun Mar 06, 2011 4:02 pm Post subject:

BTW, the big drive took a few days with a regular hard drive as the location for the backup.

From this and other testing I believe if you want fast reshape you must increase the size of the array otherwise you will need a backup-file. It also appears that the entire array gets reshaped by copying the entire array through this backup-file. An SSD would have probably sped this process up by at least 10x but this would have used quite a few write/erase cycles on the SSD to copy TB of data.
_________________
John

My gentoo overlay
Instructons for overlay