Software RAID 0 HOW-TO

[cosjef]

If you like speed, but don't like spending money, software RAID 0 is for you!
Generally, RAID (Redundant Array of Inexpensive Disks) is found on expensive
corporate servers, where performance and reliability matter most. Such servers
generally use SCSI RAID cards that allow RAID modes 0-5 on SCSI disks.

For the home user, SCSI drives and SCSI RAID cards are often overkill, not to
mention very expensive. The alternative is IDE RAID, which requires either
an add-on RAID controller card (approx. $100 retail), or an integrated
motherboard RAID controller. The latter usually tends to be a form of software
RAID (that only works in Windows), not true hardware raid.

There are multiple modes of RAID, each with tradeoffs between speed and data
redundancy. The highest-performing mode is RAID 0, which evenly distributes disk
writes and reads across all member disks of the RAID array. While this mode
offers zero redundancy, it boosts the speed of two drives working together to
greatly outperform a single drive acting alone.

This tutorial will show you how to create a high-performance RAID-0 setup using
two IDE hard drives and NO hardware RAID controller. The Linux community has
created a form of software RAID that is available for free, and is not tied to any
specific vendor hardware. This should result in a huge jump in I/O performance
on your system.

The price of this performance boost is some additional CPU usage; however
most Gentoo systems are built on modern CPU's of a gigahertz or more,
which is more than enough power to drive the array without impacting system performance.

You'll need two IDE hard drives of equal (or almost equal) size. It does not
matter if the drives come from different hardware vendors, but they must be as
close to the same size as possible. (Note: if the size of each drive
differs substantially, the RAID array will shrink to be double the size of the
smallest drive in the array, regardless of whether one drive is larger.)

This document is meant to be used in tandem with the Gentoo Linux Installation
Instruction document. You will need to swap between that document and this one
during installation, paying attention to where the steps differ.
(Note: I owe a great deal of credit for this document to Chris Atwood,
whose RAID-1 directions got me started.)

This tutorial assumes you have both drives connected to your IDE controller. The
ideal setup for maximum performance is for each drive to be set as "master" and
connected to a separate IDE channel. If you have other devices (such as a CDROM)
as slave drives on the same channel, you may experience some performance degradation.

Let's walk through it:

1) Boot the Gentoo LiveCD.
2) Load the module for multi-device support:
# modprobe md
3) Configure networking
4) Set your system date/time.
5) Filesystems, partitions and block devices step:

Follow the Gentoo guidelines of creating /boot, swap, and / partitions, but
create IDENTICAL partitions of the SAME SIZE on BOTH drives. Each physical drive
should look identical to the other, in terms of partitioning. When choosing
the partition type, be certain to select type 83 Linux for the /boot partition.
DO NOT make it a RAID partition. Make the swap partition a type 82 swap, and
make the large / partition type FD (Linux RAID autodetect).

(Note: A software RAID array is not bootable, so you need to use a separate
/boot partition that is not part of the array)

6) Create an /etc/raidtab file. This file maps virtual RAID drives to physical
partitions, and is required for your array to function. If your drives are of
different sizes (which is not recommended), the smaller of the two drives should
be raid-disk 0 in this file.

# / partition
raiddev /dev/md0 # raid device name
raid-level 0 # raid 0
nr-raid-disks 2 # number of disks in the array
chunk-size 32 # stripe size in kilobytes
persistent-superblock 1
device /dev/hda3 # device that comprises the raid array
raid-disk 0 # disk positing index in array
device /dev/hdc3 # device that comprises the raid array
raid-disk 1 # disk position index in array


For more information on the parameters of this configuration file, see the man
page here:
http://leaf.sourceforge.net/devel/cstei ... .5.man.htm

7) Creating filesystems step:

Run the mkdraid command to create the RAID device

# mkraid --really-force /dev/md0
(ignore the scary warnings here)

Type the command cat /proc/mdstat to verify the success of this operation:

# cat /proc/mdstat
Personalities : [raid0]
read_ahead 1024 sectors
md0 : active raid0 ide/host0/bus1/target0/lun0/part3[1]

ide/host0/bus0/target0/lun0/part3[0]
74100352 blocks 32k chunks

This command will show you detail on your RAID drive.

Next create standard EXT3 filesystems for the /boot and RAID virtual drive

partitions:

# mke2fs -j /dev/md0
# mke2fs -j /dev/hda1

Create the swap space on each physical drive and turn it on:

# mkswap /dev/hda2
# swapon /dev/hda2

# mkswap /dev/hdc2
# swapon /dev/hdc2

Note, swap space will not be RAID'ed, but used efficiently by the kernel
via an entry we will make in /etc/fstab.

Mount partitions step:

# mount /dev/md0 /mnt/gentoo
# mkdir /mnt/gentoo/boot
# mount /dev/hda1 /mnt/gentoo/boot

9) Stage tarballs and chroot step:

Copy the /etc/raidtab file your created into the Gentoo chroot:
# cp /etc/raidtab /mnt/gentoo/etc/raidtab

10) Follow all usual Gentoo bootstrapping steps until you come to the Installing
the kernel and a System Logger stage.

In addition to what is normally needed in the kernel by Gentoo, you must compile
in support for RAID devices and raid level 0. Do NOT compile them as modules.
These settings can be found under:

Multi-device support (RAID and LVM)" section. In this section, enable the
following:

Multiple devices driver support (RAID and LVM)
RAID support
RAID-0 (striping) mode

Follow all normal kernel compilation steps.

11) Modifying /etc/fstab for your machine step:

You must alter the /etc/fstab file to account for the RAID virtual devices,
rather than the usual method of specifying physical devices. Your fstab should
look like the one below:

/dev/hda1 /boot ext2 noatime 1 2
/dev/md2 / ext3 noatime 0 1
/dev/hda2 swap swap defaults,pri=1 0 0
/dev/hdc2 swap swap defaults,pri=1 0 0
/dev/cdroms/cdrom0 /mnt/cdrom iso9660 noauto,ro 0 0

Note that the two swap partions have the "pri=1" flag set. Setting a
priority of 1 tells the kernel to use each swap space with equal priority, and
to balance the load between them on a round-robin basis.

Continue following all standard instructions for Gentoo install.

12) The Configure a Bootloader section:

The grub.conf file needs to look like this:

default 0
timeout 10
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title=Gentoo Linux RAID
root (hd0,0)
kernel (hd0,0)/boot/bzImage root=/dev/md0

Note that we specify the kernel root as the RAID multiple device.

13) Your are done!! Reboot and you will be logged into your new RAID 0 system.
To verify everything is working, issue the "cat /proc/mdstat" command again, and
review the information provided.

You may now use your RAID'ed filesystem like any other standard Linux filesystem. Have fun with your "reborn" system!!

[green sun]

Very nice, but how is it significantly different from this?
How to do a gentoo install on a software RAID

On this tho, here's a question:
I have set up raid, gotten up to installing grub & I have a powerloss. When I get back (reboot off of livecd) & modprobe md, it works, but I can not then

Code: Select all

mount /dev/md2 /mnt/gentoo

to process with chroot. Obviously, since I only need to install grub, I dont want to mkraid again... any ideas on how to proceed?

[cosjef]

Thanks. The primary difference is that Chris' tutorial is for RAID1, not RAID0. He also left out a few steps and a good deal of detail that I tried to add.

As to your problem, check the /etc/ directory in your chroot and the LIVECD root directory. If the raidtab file is not in both places, you'll need to recreate it, and copy it back into chroot. MD depends on that file being in place to map out the virtual RAID array.

[Lovechild]

I found that for some reason my 2.5/2.6 kernels disliked /dev/mdX and instead wanted /dev/md/X.

Anyone care to test this, and elaborate on why this is?

[lurid]

I don't really know much about RAID, so excuse my ignorance. Heres my situation. My 40 gig drive died and I'm currently making due with two old drives. Ones an 8 gig ATA33 and the other is an 9 gig ATA66. Obviously these are really slow as compared to the 40 gig ATA100. What I'm wondering is what kind of speed increase are we talking here and whether this might be an option for me since I can't really afford a replacement drive. I'm assuming the speed of the RAID will be as fast as the slowest drive, which being ATA33 is pretty slow. Current hdparm stats are

Code: Select all

root@virulent lurid # hdparm -tT /dev/hda3
 
/dev/hda3:
 Timing buffer-cache reads:   128 MB in  1.03 seconds =123.67 MB/sec
 Timing buffered disk reads:  64 MB in  5.71 seconds = 11.22 MB/sec

root@virulent lurid # hdparm -tT /dev/hdc1
 
/dev/hdc1:
 Timing buffer-cache reads:   128 MB in  0.99 seconds =128.64 MB/sec
 Timing buffered disk reads:  64 MB in  6.48 seconds =  9.88 MB/sec

So my question is basically what kind of speed up am I looking at and whether or not doing this on two slow drives will give me the speed/performance of a newer faster drive. The only thing that might be a downside is the lack of space. I'd be going from 16 gigs, down to 8.

[green sun]

Thanks. The primary difference is that Chris' tutorial is for RAID1, not RAID0. He also left out a few steps and a good deal of detail that I tried to add.

As to your problem, check the /etc/ directory in your chroot and the LIVECD root directory. If the raidtab file is not in both places, you'll need to recreate it, and copy it back into chroot. MD depends on that file being in place to map out the virtual RAID array.

Right.. no criticism intended... Im setting up RAID1, so I'll focus on his post.

As for my problem, I ended up reinstalling the machine (err.. Im reinstalling now). I was getting a kernel panic, most likely because I didnt have the correct things added to the kernel. I'm reloading & being more careful

BTW, both HOW-TOs are very helpful. Thanks.

[Lovechild]

Also, a correction

A software RAID is perfectly bootable, _as long_ as it's RAID1.

I have my 2 40gb hds hooked up in a RAID0 setup:
Where I have 2x50 megs set aside for RAID1 /boot.

[cosjef]

Here's my un-RAID'ed boot partition

# hdparm -tT /dev/hda1

/dev/hda1:
Timing buffer-cache reads: 128 MB in 0.92 seconds =139.13 MB/sec
Timing buffered disk reads: 64 MB in 1.77 seconds = 36.06 MB/sec

Here's the RAID-0 array, which uses the same disk:

# hdparm -tT /dev/md0

/dev/md0:
Timing buffer-cache reads: 128 MB in 0.84 seconds =151.48 MB/sec
Timing buffered disk reads: 64 MB in 1.25 seconds = 51.41 MB/sec

[lurid]

Hm. So you're getting about a 20mb/sec increase in performance it seems. Hows that 'feel' to you? Noticably faster? I would think that with my drives being as slow as they are, a 20mb boost would certainly be noticable. Kinda like how back in the old days overclocking a 200mhz to 266/300 made a huge difference, but now a days, its hardly noticed at all.

Eh maybe I should give it a shot, just for the hell of it.
Btw, you mentioned using ext3 as the file system. Can this be done with ReiserFS as well? It seems to make a major difference in the speed of my disk.

[cosjef]

The system definitely feels more responsive; things seem to "snap in" very quickly. Disk I/O is one place you'll definitely see a performance boost. Since its free to setup and you already have the drives, why not give it a whirl? You get a cheap performance boost, and you'll learn something along the way!

I'm certain you can use Reiser FS instead of EXT3 on the RAID array, which should also help performance. You might want to do some research on the proper RAID chunk size for a REISER filesystem, as I seem to recall REISER being optimized for smaller file sizes.

[taskara]

raid howtos and info are always welcome

[green sun]

The system definitely feels more responsive; things seem to "snap in" very quickly. Disk I/O is one place you'll definitely see a performance boost. Since its free to setup and you already have the drives, why not give it a whirl? You get a cheap performance boost, and you'll learn something along the way!

I'm certain you can use Reiser FS instead of EXT3 on the RAID array, which should also help performance. You might want to do some research on the proper RAID chunk size for a REISER filesystem, as I seem to recall REISER being optimized for smaller file sizes.

FYI, I run Reiser on a RAID1 machine. Gentoo backup DNS server. If Reiser won't work due to chunk size, etc, you can try XFS (I never have), which allows you to customize settings quite a bit with the mkxfs command.... but this is straying OT

[lurid]

I found this document describing raid0 with reiserfs, its pretty old though. They suggest a chunk size of 64. Its a rather detailed peice and they are using reiserfs in it, so I guess I should set it up like that. I think I'll give it a shot as soon as I backup some stuff.

I'll report back with my experience, good or bad.

[taskara]

what is the best chunk size to use in conjunction with reiserfs ?

[lurid]

Ok.. problems.

Install went fine, I have the raid0 setup going and its fast.. I was pretty happy until I rebooted. On boot, fsck wants to check the filesystem.. evidently, md0 doesn't look like a proper reiserfs filesystem to it, so it says it can't open the partition. Then tells me I can log in as root, or CTRL+D to reboot. Logging in as root lets me into the system, but its all mounted read-only.

So I boot the LiveCD but I can't get it to mount the raid patition(s). Says it needs me to specify the filesystem type, which is reiser, but doesn't look like one. So I'm basically locked out of my system.

Erm.. any advice? First, how can I get the raid array to mount with the LiveCD. Second, how can I get either fsck to not attempt to check / on start up, or to get it to recognise the raid?

I have 0 0 as the 5th and 6th entry in my fstab for my / partition..

[lurid]

Sorry about the second post.. I'm doing this from lynx off the LiveCD, so bare with me.

Alright, I've got the raiddisk mounted. I'm using the rc3 LiveCD, so it might be different with this one, but I just had to recreate the /etc/raidtab file, then zap /etc/init.d/md and evms, then restart both.

Mount it with: mount /dev/evms/md/md0 /mnt/gentoo

Not to hard. However, I still can't boot normally. Since it was fsck that was causing the problems, I renamed /sbin/fsck.reiserfs and tried it that way. Made it through the boot sequence, but a login prompt never appeared. I'm still at a complete loss.

[taskara]

did u emerge reiserfsprogs ?

[lurid]

Yes, I did.

The problem is that the scripts checkroot and checkfs mount / readonly then have fsck look them over, before remounting read-write. For whatever reason, fsck.reiserfs is unable to open the raid array and it drops to a shell. Its got to be an issue with fsck.reiserfs because I'm posting this on the mounted and chroot'ed raid and it all works fine.

[lurid]

Alright, I figured it out. Sorry for all the posts.

The answer was right in front of me. The problem is what Lovechild said, the file system wasn't recognised because /dev/md0 doesn't exist. devfs rearanges everything. Its not a kernel issue, which from Lovechilds post I thought it was, its a devfs issue. I'm running 2.4.20-ck6 so I didn't pay attention to what he said.. but it appies for me too. I need to put /dev/md/0 in fstab, not /dev/md0.

So I'm all set. Watching X compile right now. I did a hdparm test and was pretty impressed with the results. I'll paste them later since I'm still using lynx.

[taskara]

cool

[cosjef]

Lurid--glad you got it working. Looks like a subtle nuance in /etc/fstab for a REISER filesystem. Can you post your hdparm results?

[lurid]

Yup, got it working great. Bogged my brain for a few hours there, but thats a good thing in a way.

If you remember, I posted my old hdparm stats as this:

Code: Select all

 Timing buffer-cache reads:   128 MB in  0.99 seconds =128.64 MB/sec
Timing buffered disk reads:  64 MB in  6.48 seconds =  9.88 MB/sec

The new stats are:

Code: Select all

root@virulent lurid # hdparm -tT /dev/md/0
 
/dev/md/0:
 Timing buffer-cache reads:   128 MB in  0.88 seconds =146.29 MB/sec
 Timing buffered disk reads:  64 MB in  2.87 seconds = 22.30 MB/sec

While that might not seem fast to anyone with newer faster drives, look at the difference in seconds. 6.48 seconds, down to 2.87 seconds. Thats huge and I can certainly feel it.. and all it cost me were a few tense hours as I fumbled around trying to figure some stuff out and a reinstall of Gentoo. You can't beat that deal with a stick. When I do eventually replace these slow drives, instead of getting an 80 gig, I'm gonna get 2 40's and do this again. Hell, I might even get my gf a second 20 gig for her machine and raid her up too.

I'm very impressed. Thanks for the how-to.

[Moled]

yum

just set this up

Code: Select all

/dev/md0:
 Timing buffer-cache reads:   3756 MB in  2.00 seconds = 1878.00 MB/sec
 Timing buffered disk reads:  328 MB in  3.00 seconds = 109.33 MB/sec

[puddpunk]

yum

just set this up

Code: Select all

/dev/md0:
 Timing buffer-cache reads:   3756 MB in  2.00 seconds = 1878.00 MB/sec
 Timing buffered disk reads:  328 MB in  3.00 seconds = 109.33 MB/sec

Nice! What the hell kinda HDD cache you got there?

I'm running 2 x 40Gb Seagate 7200 RPM drives in raid0 (with a raid1 boot ). Here are my results, single drive, then raid:

Code: Select all

# hdparm -tT /dev/hda
/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.45 seconds =284.44 MB/sec
 Timing buffered disk reads:  64 MB in  1.14 seconds = 56.14 MB/sec

# hdparm -tT /dev/md1
/dev/md1:
 Timing buffer-cache reads:   128 MB in  0.45 seconds =284.44 MB/sec
 Timing buffered disk reads:  64 MB in  0.64 seconds =100.00 MB/sec

Thats a pretty tidy increase Almost double, about a 90% increase!

BTW, I'm not sure why the buffer-cache reads are really included in these tests as they don't test disk speed at all. The buffer-cache reads will be the same in or out of raid.

[taskara]

hey moled

WOW what the hell kinda computer are you using!?

are you using linux software raid or the silicon "hardware" raid?

As per your suggestion on another thread, I used

hdparm -d1 -X66 -A1 -c1 -u1 -m16 -a64

on my hdd's and it seems stable atm.

however here are my screaming hdparm results!

RAID:

/dev/md5:
Timing buffer-cache reads: 128 MB in 0.23 seconds =556.52 MB/sec
Timing buffered disk reads: 64 MB in 2.03 seconds = 31.53 MB/sec

NORAID:

bash-2.05b# hdparm -tT /dev/hde

/dev/hde:
Timing buffer-cache reads: 128 MB in 0.24 seconds =533.33 MB/sec
Timing buffered disk reads: 64 MB in 2.96 seconds = 21.62 MB/sec

I know this is only with UDMA2, but u seem to get such a high score there... what's your secret!

If you can help, here's the info of my drives:

bash-2.05b# hdparm -i /dev/hde

/dev/hde:

Model=ST380013AS, FwRev=3.05, SerialNo=3JV32V8W
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=8192kB, MaxMultSect=16, MultSect=16
CurCHS=65535/1/63, CurSects=4128705, LBA=yes, LBAsects=156301488
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 *udma2
AdvancedPM=no WriteCache=enabled
Drive conforms to: ATA/ATAPI-6 T13 1410D revision 2: 1 2 3 4 5 6

/dev/hdg is the same

bash-2.05b# hdparm -i /dev/hde

/dev/hde:

Model=ST380013AS, FwRev=3.05, SerialNo=3JV32V8W
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=8192kB, MaxMultSect=16, MultSect=16
CurCHS=65535/1/63, CurSects=4128705, LBA=yes, LBAsects=156301488
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 *udma2
AdvancedPM=no WriteCache=enabled
Drive conforms to: ATA/ATAPI-6 T13 1410D revision 2: 1 2 3 4 5 6

as u can see they are running udma66

hey puddpunk - what chipset are u running?