Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
mdadm raid1 with more then 2 disks?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
tgh
Apprentice
Apprentice


Joined: 05 Oct 2005
Posts: 222

PostPosted: Thu Nov 02, 2006 7:35 pm    Post subject: mdadm raid1 with more then 2 disks? Reply with quote

Here's a question that I can't quite squeeze out of Google.

If I have 3 disks, I have 2 options for setting up a RAID1 array.

1) Disk A+B mirrored, with disk C serving as a hot spare

mdadm --create /dev/md0 --level=raid1 --raid-devices=2 --spare-devices=1 /dev/sda1 /dev/sdb1 /dev/sdc1

2) Mirror A+B+C by making all of them raid devices

mdadm --create /dev/md0 --level=raid1 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

Now for the question. Does option #2 really work? Does mdadm mirror the contents across all 3 devices at the same time? Meaning that I won't have a "rebuild window"? What are the downsides? Can I do this with 7 devices?

A good example of why I'd want to do this is with a system that has 7 devices. I have a 128MB /boot partition on all 7 hard drives, the BIOS is allowed to boot from any of the 7 drives, and I want to make sure that /boot is always in sync across all 7 drives.
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Nov 02, 2006 9:39 pm    Post subject: Reply with quote

er, um raid 1 across 7 drives is a little overkill...

I would do:
Code:

commonly called raid10:
                 raid 0
    /               |            \
   |                |              |
2disks raid1  2disks raid1  2disks raid1
then have a spare in case one dies.

But yes the raid1 should work across as many drives as you want (to a certain extent)
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands

PostPosted: Thu Nov 02, 2006 10:03 pm    Post subject: Reply with quote

in your case, i'd rather do a setup like this:
sd?1: /boot, synchronised using the kernel yoke driver
sd?2: raid5 array.

that way you'll loose a lot less memory (raid 1 would mean that the size of all disks is that of a single disk). using yoke to synchronise the /boot partitions of the disks means that all of them will have the exact same state, so you won't run into potential weirdness with raid-metadata.

EDIT: raid10 as vibrokatana suggested is, of course, also a very good option. i just find it a little harder to maintain, but that's up to the admin to decide... and we're all about choice, ain't we? ;)
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )


Last edited by mdeininger on Thu Nov 02, 2006 10:06 pm; edited 1 time in total
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Nov 02, 2006 10:04 pm    Post subject: Reply with quote

raid5 is rather cpu intensive, depends on their workload.
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
vputz
Guru
Guru


Joined: 16 Mar 2005
Posts: 310
Location: Oxford, England

PostPosted: Thu Nov 02, 2006 10:13 pm    Post subject: Reply with quote

Does depend on the workload, but it's worth investigating (I had a little cruddy raid-5 array--36 GB total!--on an old dual pentium-500, and it trucked along just fine for the most part. Now asking it to read an encrypted partition off the raid-5 array did stress it a bit, but for regular use it was pretty darn good).
Back to top
View user's profile Send private message
Kenji Miyamoto
Veteran
Veteran


Joined: 28 May 2005
Posts: 1452
Location: Looking over your shoulder.

PostPosted: Thu Nov 02, 2006 10:27 pm    Post subject: Reply with quote

vibrokatana wrote:
raid5 is rather cpu intensive, depends on their workload.
Hardware of software RAID?
_________________
[ Kawa-kun, new and improved!! ]

Alex Libman seems to be more of an anarchist than a libertarian.
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands

PostPosted: Thu Nov 02, 2006 10:35 pm    Post subject: Reply with quote

Kenji Miyamoto wrote:
vibrokatana wrote:
raid5 is rather cpu intensive, depends on their workload.
Hardware of software RAID?

software, but hardware raid seems to be even slower... at least the benchmarks tend to say so.

yes, raid5 can be slower, but that's only for writing, isn't it? reading should theoretically be faster, aye?
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Nov 02, 2006 10:37 pm    Post subject: Reply with quote

mostly yes, but their is some md5 calculations in raid5 that slows it down a little.
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands

PostPosted: Thu Nov 02, 2006 10:41 pm    Post subject: Reply with quote

vibrokatana wrote:
mostly yes, but their is some md5 calculations in raid5 that slows it down a little.
ah i see. i had only thought of the XOR when writing, which shouldn't impact reading at all (unless a disk is broken/has to be reconstructed).
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Nov 02, 2006 10:42 pm    Post subject: Reply with quote

For reading the raid1 would theoretically be faster, usually around 10% or so because it doesnt have to do the calculations. But it obviously would be much less space efficient.
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands

PostPosted: Thu Nov 02, 2006 10:46 pm    Post subject: Reply with quote

vibrokatana wrote:
For reading the raid1 would theoretically be faster, usually around 10% or so because it doesnt have to do the calculations. But it obviously would be much less space efficient.
indeed, that is why i tend to suggest raid5 with large disk arrays -- i mean, gotta make some use of those terabytes of potential storage, don'tcha? ;)
depends on what you need though. i usually sacrifice the 10% of speed for the ~20% storage space (did i calculate that right?)
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Nov 02, 2006 11:08 pm    Post subject: Reply with quote

raid 1 is just the size of one drive, raid 5 is (size of disk)*(n-1) where n is the number of disks
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands

PostPosted: Thu Nov 02, 2006 11:10 pm    Post subject: Reply with quote

vibrokatana wrote:
raid 1 is just the size of one drive, raid 5 is (size of disk)*(n-1) where n is the number of disks
ah yes... i had calculated with 1/3rd of the space for overhead for whatever reason.

suppose i better get abed for the night ;)
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
tgh
Apprentice
Apprentice


Joined: 05 Oct 2005
Posts: 222

PostPosted: Thu Nov 02, 2006 11:28 pm    Post subject: Reply with quote

RAID5 isn't preferable for what I'm doing... here's the full disk layout currently:

Code:
livecd ~ # cat /proc/mdstat
Personalities : [raid1] [raid10]
md4 : active raid10 sdg4[6](S) sdf4[5] sde4[4] sdd4[3] sdc4[2] sdb4[1] sda4[0]
      2167545792 blocks 32K chunks 2 near-copies [6/6] [UUUUUU]
      [===================>.]  resync = 98.7% (2141124512/2167545792) finish=3.7min speed=117115K/sec
     
md3 : active raid1 sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
      2008000 blocks [7/7] [UUUUUUU]
     
md2 : active raid1 sdg2[2] sdf2[1] sde2[0]
      7911936 blocks [3/3] [UUU]
     
md1 : active raid1 sdd2[3] sdc2[2] sdb2[1] sda2[0]
      7911936 blocks [4/4] [UUUU]
     
md0 : active raid1 sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      136448 blocks [7/7] [UUUUUUU]
     
unused devices: <none>
livecd ~ #


md0 - this is a 128MB RAID1 set with 7 members, spread across the 7 hard drives in the unit. It's the /boot partition. By making it RAID1 with 7 members, I never have to worry that grub will be trying to boot an outdated /boot partition (which can happen if you do RAID1+hotspares).

md1 - this is a 4-disk RAID1 for the primary OS root partition. I could do 2-disk RAID1 with 2 hot-spares, but it's just as easy to make all 4 disks active and avoid the dangerous period when a disk fails and the hot-spare is still rebuilding.

md2 - this 3-disk RAID1 is the backup OS root partition, mostly a clone of the primary OS root partition and is used to boot the server when the primary OS gets hosed (unlikely since the primary OS is going to be a Xen Dom0 hypervisor)

md3 - a 7-disk RAID1 set for the 2GB swap partition used by either the primary or backup OS. Again, it could have been setup as a 2-disk RAID1 with 5 hot-spares, but it looks like you can just as easily make all 7 disks "hot" with the possibility that mdadm will spread the load better.

md4 - This is a 6-disk RAID10 with a single hot-spare for speed. Net capacity is around 2TB and will be managed using LVM. RAID10 has a much faster rebuild time then RAID5 when a disk fails (so a smaller vulnerability window) and performance doesn't take a dive when a disk fails.

The reason for RAID10 instead of RAID5 is that RAID10 performs better under heavy loads with lots of random seeks then RAID5. Since this server will be running multiple Xen guest domains, I want a solid performance floor that I can count on. Because I'll be running multiple Xen servers on top of this hardware (an X2 4200+ w/ 4GB ECC), paranoia is the name of the game. This box will replace the multiple servers that I'm running in my home office and allow me to consolidate to a single server but with the flexibility of multiple.

Down the road I'll be adding a 2nd set of 7 drives to the system for another 6-disk RAID10 w/ hot-spare. Hopefully the 1.5TB drives (or larger) will be out by then so that I can make that a 4TB net capacity expansion.

So back to my original question. It looks like I can do RAID1 with more then 2 disks at the same time and I'm not seeing a downside. Or is there something I don't know about mdadm in this particular configuration? I ran into an issue with a 5-disk server where grub was booting an outdated /boot partition because that specific partition was a hot-spare of md0 and not a live copy of md0.
Back to top
View user's profile Send private message
tgh
Apprentice
Apprentice


Joined: 05 Oct 2005
Posts: 222

PostPosted: Thu Nov 02, 2006 11:34 pm    Post subject: Reply with quote

FYI, I'll post some performance numbers here.

livecd ~ # hdparm -tT /dev/sde

/dev/sde:
Timing cached reads: 4464 MB in 2.00 seconds = 2231.22 MB/sec
Timing buffered disk reads: 224 MB in 3.02 seconds = 74.21 MB/sec
livecd ~ # hdparm -tT /dev/sdf

All of the individual disks (sda to sdg) return 74-75MB/sec numbers.

The RAID1 arrays (md0-md3) all returned 74-75MB/sec numbers.

The 6-disk RAID10 array returns numbers in the 220MB/sec range (3x an individual disk).

Bonnie results after I build the machine (I expect low-end numbers for random seeks to be around 30-40MB/s).

Code:
scratch # bonnie -s 16384 -m raid10
File './Bonnie.22549', size: 17179869184
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
raid10   16384 58407 94.9 191667 75.6 75826 31.4 55190 83.0 173645 46.0 181.6  1.3


Here's bonnie for a 4-disk RAID10 (similar hardware).

Code:
bacula # bonnie -s 16384 -m raid10
File './Bonnie.9603', size: 17179869184
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
raid10   16384 63571 88.5 98304 14.5 32493  1.7 55463 65.2 136108  1.7 115.0  0.0
Back to top
View user's profile Send private message
tgh
Apprentice
Apprentice


Joined: 05 Oct 2005
Posts: 222

PostPosted: Sat Nov 04, 2006 12:23 pm    Post subject: Reply with quote

And doing an 11GB move between 2 LVM partitions on the 6-disk RAID10...

6m39.284s - 11375983498 bytes = 27.2 MB/s

Not the best score even for RAID10, but that is probably because the drives are 7200 RPM 750GB instead of smaller and faster 10k or 15k SATA/SCSI drives. I'm not sure how well a 6-disk RAID5 array would've handled it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum