Hardware RAID 5 (3Ware 9500S-12) [SOLVED]

Shazer · Last edited by Shazer on Sat Mar 04, 2006 1:09 pm; edited 1 time in total

I'm trying to setup a hardware raid 5 with 12 400GB drives. Here what I have setup:

* The RAID is built
* Compiled the 3w9XXX as a module
* Module loads without any issues.

I get the following:

Shazer · Posted: Sat Mar 04, 2006 5:49 am Post subject: Drivers

Could this be a driver issue? I found this btw:

http://www.3ware.com/support/download_9.3.0.3.asp?SNO=702

markkuk · Guru Joined: 29 Nov 2002 Posts: 446

You can't use fdisk on drives (or RAID arrays) larger than 2TB. You must use parted and create a GPT partition table (disk label in parted-speak). Note that you can't boot from a GPT drive.

Shazer · Posted: Sat Mar 04, 2006 1:12 pm Post subject:

bombcar · Guru Joined: 08 Apr 2003 Posts: 453 Location: Wisconsin

Bleh. I first discovered that all was not happy in 2.6 land with support for volumes over 2 TB when I first built my home server with a 2.5 TB array. I eventually determined that I needed to use a GPT partition table, as the standard MSDOS partition table that fdisk uses is limited to 2 terabytes.

And then I learned that GRUB doesn't support GPT partition tables, so I couldn't boot until I got lilo installed. And that's when I discovered that the GPT partition table support isn't compiled in the kernel by default. But I got it working, and all was happy.

Then I decided to rebuild my server using some new 500 GB drives instead of the pile of 250 GB drives I had been using before. As I only had 800 GB of data, I decided to get 3 Seagate 500 GB SATA drives, and build a RAID 5 out of them, and since I'm using a 3ware 9550S-12, I'd be able to easily add drives in the future and expand all the way to 6 TB. I was right, except for the "easily" part.

So I got my drives in, and kicked on of the 250 GB drives out of the current array so I'd have a port free. I copied all my OS data and most of my other data to the 500 GB, setup as a single disk, backed up the remaining data that wouldn't fit to tape, chrooted to the new disk, ran lilo, and rebooted, changed the 3ware controller to boot off the new disk first, and all was happy.

Now, even though the single disk was only 500 GB (and so it would have been able to use a normal fdisk MSDOS partition table), I used a GPT partition table so that I wouldn't have a problem as the array went past 2 TB in the future.

Everything booted perfectly, so I shut down the old array, and installed the other two 500 GB drives, and instructed tw_cli to convert my single drive unit into a 3 drive RAID 5 unit with a stripe size of 256K. After about 20 hours, the conversion was done, and I had a 1000 GB usable array. Rebooting Linux made the kernel see the new size of the array, and that's when the fun began.

Now I'm using xfs on this array, which is a good, reliable filesystem with built in online expansion capabilities. You can increase the size of the partition that an xfs filesystem is on, and the use xfs_growfs to tell the filesystem to expand to the partition size. So I thought it would be easy to tell parted (which as far as I can tell is the only program for Linux that can handle GPT partition tables) to extend the main partition to the end of the newly enlarged "disk."

No such luck. First of all, parted won't let you edit a partition that is mounted, so I had to boot of CD. No big problem.

But after booting from CD I couldn't get parted to use the resize command, as it didn't know how to extend the xfs filesystem. I was pissed. Stupid program, I'm going to extend the filesystem myself! Just move the end of the partition table and let me deal with it. But no go.

So my next idea was to "delete" the existing partiton (/dev/sda2), and recreate it with the exact same beginning and make it go all the way to the end. No dice, I get some cryptic error message about how that is not an operation that parted wants to do. I finally determined that I had to create an entire new GPT disklabel, and then create the partitions as they were before, but extended.

After forgetting to run lilo, rebooting, seeing that for some reason the kernel on my CD didn't see the GPT partitions, running parted and "naming" one partition and exiting, which caused the kernel to see the GPT partitions, chrooting, running lilo, and rebooting, and I was done! My 500 GB disk had expanded to 1000 GB, and I knew how to do it in the future.

Silly parted.