File server ponderings

[addeman]

Hello,

my current fileserver is getting low on space, so this summer i'll be redoing it.

The current setup is a 1.2 TB lvm consisting of 3 320 GB sata disks, 1 250 GB ide and 1 160 GB ide disk. The sata controller card is a Promise SATA 300 TX4 and the ide controller is a Promise IDE TX4.

My plan is to buy 5 more 320 GB disks and an additional Promise SATA 300 TX4 controller. But this time around i want some kind of data security, so i'm thinking raid 5. My problem: what file system to go with? I really like SUN's ZFS, but that's not really available in linux ( i know about fuse, yes) so i have a couple of options:

1. Use some opensolaris version. This is not really an option as there is no support for the promise controller. And i'm not really comfortable with solaris administration yet.

2. Use FreeBSD. Has support for the controller. Not really comfortable with it either. But once installed, it'll probably just sit there.

3. Run linux+xen and export all disks to the domU (which would probably be opensolaris). I guess this is the geekiest way of doing it, but what about performance? And i still have to admin either a opensolaris or FreeBSD system.

4. Run linux software raid 5 with perhaps xfs or jfs.

There are some implementation issuses: I have to start the new array with 4 new disks and transfer as much as possible from the existing setup so i can use some pvmove to free the 3 sata disks in the running array. Then i can move the 3 old disks and controllercard the new new server and get my 8 disk array fully functional.

How do I disassemble my current diskdrive correctly? As I understand it JFS (which i run today) can't be shrunken, so do i need to get some temp storage for the rest of the data on the array?

What have i missed? Which option do you think i should go with? Anyone with similar experience?

[alex.blackbit]

is it the built-in capability of software raid in zfs that attracts you? you did not make that very clear.
one option that is possible too is to use a hardware raid controller.
this of course implies higher costs but brings better performance too.
i personally like the 3ware raid controllers very much.
there is a really cool filesystem coming up for linux too that has built-in raid magic, btrfs, but it is still in an early stage of development.

[addeman]

Yes, the built-in software raid is certainly attractive but not the only 'winning feature' of zfs in my mind. If i go with zfs i'll definetly use it.

A hardware raid controller is unfortunatly to expensive for me, thats the downside of being a student And the speed is not my biggest concern, especially since my network bandwith is the bottleneck.

I read a little about btrfs, but afaik it's nowhere near production quality yet, which zfs is.

Best regards
addeman

[gentoo-dev]

4. Run linux software raid 5 with perhaps xfs or jfs.

or simply ext3

[addeman]

4. Run linux software raid 5 with perhaps xfs or jfs.

or simply ext3

Why would i want to go with ext3?

It's certainly the easiest to set up alternative 4 for me. But at least xfs is pretty sensitive to powerloss. And neither xfs or jfs supports shrinking of fs, which is nice.

On a personal note i feel like linux in terms of filesystems is falling behind other unix like OSes. There probably is a good reason that both FreeBSD and Apple decided to support ZFS. And since fbsd has it, it shouldn't be too long before other *BSDs have it too. Well i guess the grass is always greener on the other side

/addeman

[Akkara]

There are some implementation issuses: I have to start the new array with 4 new disks and transfer as much as possible from the existing setup so i can use some pvmove to free the 3 sata disks in the running array.

Interesting problem, how to do it without extra temporary storage. Here's one way. (Which upon re-reading, seems like a complicated way. Maybe there's a better answer.)

Make a raid5 with 4 of the new disks (3 + 1 redundancy). Copy over as much as will fit from the original filesystem to this 4-disk raid. Hopefully you can get 900+ GB onto it. Copy the remaining 1.2TB - 900GB = 300GB to the fifth new 320GB disk.

Checksums, etc., to make sure it all got over OK.

Then tear apart the old LVM and build a new LVM using just the 250 and 160 GB disks. Copy the contents of that last solo 320GB drive onto this LVM.

Then build another 4-disk raid5 using the 3 old 320GB disks and the remaining new 320GB one.

LVM the two raids together and extend the filesystem to cover both.

Finish by copying the data from the 250+160GB LVM, onto this larger LVM'ed array.

The downside is you end up with 2 raids that are LVM'ed into a bigger pool, and two redundancy drives without the benefit of tolerating two arbitrary failures (only one failure in each array).

Whew! Maybe there's a better way to do all this. Do you have a friend with a spare drive you can borrow?

Edit: simplified a step

[addeman]

...
Whew! Maybe there's a better way to do all this. Do you have a friend with a spare drive you can borrow?

Edit: simplified a step

I think i can find a friend with some storage. Maybe i can scrape that space together myself, i have a few other computers, so that is not something i really worry about. I appreciate your solution, it was elegant. Maybe it could be turned into:
Start as before, build a new array of the 4 new disks, but with a little twist

Code: Select all

mdadm --create --verbose /dev/md1 --level=5 --raid-devices=8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 missing missing missing missing

then move ~900GB from the old array to the new and the rest the the fifth new disk. Tear the old array down and add the 3 old drives to the array with

Code: Select all

mdadm --manage --add /dev/md1 /dev/sde1 /dev/sdf1 /dev/sdg1

and then copy the data from the fifth disk and the add that to. Of course there would be some lvm magic in there and some resizing of filesytems, which is a bit scary... Maybe i should just skip the lvm part, i should be able to live without it.

Well turns out that a raid5 can't be created with more than 1 missing device, so the above will not work. But this would have a chance i think:

Code: Select all

mdadm --create --verbose /dev/md1 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

Copy the remaining ~300GB to some spare storage. Tear the old array down and then

Code: Select all

mdadm --add /dev/md1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 
mdadm --grow /dev/md1 --raid-devices=8

The add command basicly adds those disks as spares and then the grow converts them to 'the real thing'

/addeman

[HeissFuss]

You can have 1 missing drive in a raid 5 and run in degraded mode. So, you can create the new raid5 with the four new drives with one missing to get the full initial size of 4x320GB.

Code: Select all

mdadm --create --verbose /dev/md1 --level=5 --raid-devices=5 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 missing

After you've copied the original LVM contents to the new raid, you'll need to add one of the old 320GB drives and wait for it to sync before adding the other two drives and growing.

[addeman]

Ahh, thats smart Should have thougt of that. Thanks for the tip!

One thing i actually thougt of was to set virtualbox or vmware to do some testing. I tried virtualbox, but it has its limits, i could only export 3 drives to the vm ( couldn't find the sata controller option anywhere) so maybe i have to go with vmware after all. But i guess thats really another topic.

So everyone is going for the 'use linux software raid 5' option?

Best regards
addeman