Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Howto Install Gentoo 2006.0 on Software RAID
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
markir
n00b
n00b


Joined: 22 Apr 2006
Posts: 6
Location: New Zealand

PostPosted: Sun Apr 23, 2006 12:06 am    Post subject: Howto Install Gentoo 2006.0 on Software RAID Reply with quote

Introduction

I recently installed Gentoo onto a completely software raid system.

There are a number of helpful articles on the web that discuss this very setup (e.g: http://www.fatofthelan.com/articles/articles.php?pid=19 and https://forums.gentoo.org/viewtopic-t-8813-highlight-raid.html), however they typically do not cover a complete installation.

So, in the hope that it might be useful, here is a complete installation starting from live cd.

The particular hardware used (a Promise TX4000) shows the disks as SCSI - in fact they were not (Maxtor 6E040L0), but with the obvious substitution, these instructions should be helpful for those cards that show ATA/SATA disks as they really are.

Thanks to those who provided feedback - some of this has been incorporated into the latest update!

Pre-Install Steps

1. Obtain the live cd.
2. Boot system from the live cd.


Install Steps

1. Setup networking on live cdrom system

Code:
(root) # ifconfig eth0 inet my.host.ip.addr netmask my.net.mask.addr
(root) # echo "nameserver name.server.ip.addr" > /etc/resolv.conf
(root) # route add default gw my.gw.ip.addr


2. Prepare disks

Layout the 4 disks identically with fdisk, setting the partition types
all to 'Linux RAID autodetect'.

Code:
/dev/sd*

1 128M      (boot)     (type fd)
2 1G        (swap)     (type fd)
3 128M      (/)        (type fd)
4 extended
5 1G        (/tmp)     (type fd)
6 1G        (/var)     (type fd)
7 5G        (/usr)     (type fd)
8 5G        (/home)    (type fd)
9 rest!     (/data0)   (type fd)

e.g:

Code:
Disk /dev/sda: 41.1 GB, 41110142976 bytes
255 heads, 63 sectors/track, 4998 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          17      136521   fd  Linux raid autodetect
/dev/sda2              18         142     1004062+  fd  Linux raid autodetect
/dev/sda3             143         159      136552+  fd  Linux raid autodetect
/dev/sda4             160        4998    38869267+   5  Extended
/dev/sda5             160         284     1004031   fd  Linux raid autodetect
/dev/sda6             285         409     1004031   fd  Linux raid autodetect
/dev/sda7             410        1032     5004216   fd  Linux raid autodetect
/dev/sda8            1033        1655     5004216   fd  Linux raid autodetect
/dev/sda9            1656        4998    26852616   fd  Linux raid autodetect


There is a little subtlety here - the partition to be used as /boot is going to be RAID1, whereas the remainder are to be RAID0. So with the exception of the to-be boot, they are all sized 1/4 of the final assembled size!

3. Setup raid devices

Code:
(root) # modprobe raid0
(root) # modprobe raid1
(root) # mknod /dev/md0 b 9 0
(root) # mknod /dev/md1 b 9 1
(root) # mknod /dev/md2 b 9 2
(root) # mknod /dev/md3 b 9 3
(root) # mknod /dev/md4 b 9 4
(root) # mknod /dev/md5 b 9 5
(root) # mknod /dev/md6 b 9 6
(root) # mknod /dev/md7 b 9 7

(root) # mdadm --create /dev/md0 --level=1 --chunk=256 --raid-devices=4 /dev/sda
1 /dev/sdb1 /dev/sdc1 /dev/sdd1
(root) # mdadm --create /dev/md1 --level=0 --chunk=256 --raid-devices=4 /dev/sda
2 /dev/sdb2 /dev/sdc2 /dev/sdd2
(root) # mdadm --create /dev/md2 --level=0 --chunk=256 --raid-devices=4 /dev/sda
3 /dev/sdb3 /dev/sdc3 /dev/sdd3
(root) # mdadm --create /dev/md3 --level=0 --chunk=256 --raid-devices=4 /dev/sda
5 /dev/sdb5 /dev/sdc5 /dev/sdd5
(root) # mdadm --create /dev/md4 --level=0 --chunk=256 --raid-devices=4 /dev/sda
6 /dev/sdb6 /dev/sdc6 /dev/sdd6
(root) # mdadm --create /dev/md5 --level=0 --chunk=256 --raid-devices=4 /dev/sda
7 /dev/sdb7 /dev/sdc7 /dev/sdd7
(root) # mdadm --create /dev/md6 --level=0 --chunk=256 --raid-devices=4 /dev/sda
8 /dev/sdb8 /dev/sdc8 /dev/sdd8
(root) # mdadm --create /dev/md7 --level=0 --chunk=256 --raid-devices=4 /dev/sda
9 /dev/sdb9 /dev/sdc9 /dev/sdd9

(root) # echo "DEVICE /dev/sd[abcd][12356789]" >> /etc/mdadm.conf
(root) # mdadm --detail --scan >> /etc/mdadm.conf


4. Create and mount filesystems

Code:
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md0
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md2
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md3
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md4
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md5
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md6
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md7
(root) # mkswap /dev/md1

(root) # swapon /dev/md1
(root) # mount /dev/md2 /mnt/gentoo
(root) # mkdir /mnt/gentoo/boot
(root) # mkdir /mnt/gentoo/tmp
(root) # mkdir /mnt/gentoo/var
(root) # mkdir /mnt/gentoo/usr
(root) # mkdir /mnt/gentoo/home
(root) # mkdir /mnt/gentoo/data0       # (don't use, but create the mountpoint)

(root) # mount /dev/md0 /mnt/gentoo/boot
(root) # mount /dev/md3 /mnt/gentoo/tmp
(root) # mount /dev/md4 /mnt/gentoo/var
(root) # mount /dev/md5 /mnt/gentoo/usr
(root) # mount /dev/md6 /mnt/gentoo/home


5. Install distribution and portage snapshot

Code:
(root) # date 060112002006             # (as probably in UTC, so back it off)

(root) # cd /home/gentoo
(root) # wget http://mirrors.acm.cs.rpi.edu/gentoo/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2
(root) # cd /mnt/gentoo
(root) # tar -xvjpf /home/gentoo/stage3-i686-2006.0.tar.bz2
(root) # tar -xvjf /mnt/cdrom/snapshots/portage-20060123.tar.bz2 -C /mnt/gentoo/usr
(root) # mirrorselect -i -o >> /mnt/gentoo/etc/make.conf


6. Chroot system

Code:
(root) # cp /etc/mdadm.conf /mnt/gentoo/etc/mdadm.conf
(root) # cp /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
(root) # mount -t proc none /mnt/gentoo/proc
(root) # mount -o bind /dev /mnt/gentoo/dev

(root) # chroot /mnt/gentoo /bin/bash
(root) # env-update
(root) # source /etc/profile
(root) # export PS1="(chroot) $PS1"

(root) # echo "USE=\"-gtk -gnome qt kde dvd alsa cdr\"" >>/etc/make.conf

(root) # emerge --sync                 # get latest portage updates.



7. Kernel configuration using genkernel

Code:
(root) # cp  /usr/share/zoneinfo/<yourzonefile> /etc/localtime
(root) # USE="-doc symlink" emerge gentoo-sources
(root) # emerge genkernel
(root) # zcat /proc/config.gz > /usr/share/genkernel/x86/kernel-config-2.6
(root) # genkernel --menuconfig all    # include xfs, RAID, not just as modules!
                                       # and maybe SCSI low level drivers
                                       # e.g. SCSI_PROMISE_SATA in this case


8. Install Ebuilds

Code:
(root) # emerge coldplug
(root) # emerge syslog-ng

(root) # emerge xfsprogs
(root) # emerge xfsdump
(root) # emerge mdadm
(root) # emerge portage-utils
(root) # emerge vim
(root) # emerge lilo


9. Configure system

Code:
(root) # vi /etc/fstab
(root) # cat /etc/fstab|grep -v '^#'

/dev/md0                /boot           xfs     defaults,noatime        1 2
/dev/md2                /               xfs     noatime                 0 1
/dev/md3                /tmp            xfs     noatime                 0 2
/dev/md4                /var            xfs     noatime                 0 2
/dev/md5                /usr            xfs     noatime                 0 2
/dev/md6                /home           xfs     noatime                 0 2
/dev/md7                /data0          xfs     noatime                 0 2
/dev/md1                none            swap    sw                      0 0
/dev/cdrom             /mnt/cdrom   iso9660 noauto,ro               0 0

proc                    /proc           proc    defaults                0 0

shm                     /dev/shm        tmpfs   nodev,nosuid,noexec     0 0

(root) # vi /etc/conf.d/hostname
(root) # vi /etc/conf.d/domainname
(root) # rc-update add domainname default

(root) # echo "config_eth0=( \"my.host.ip.addr netmask my.net.mask.addr brd my.net.bcast.addr\" )" >> /etc/conf.d/net
(root) # echo "routes_eth0=( \"default gw my.gw.ip.addr\" )" >> /etc/conf.d/net
(root) # rc-update add net.eth0 default
(root) # cat /etc/hosts|grep -v "^#"

127.0.0.1       localhost myhostname.mydomain myhostname

(root) # vi /etc/rc.conf
(root) # vi /etc/conf.d/clock   # set clock to 'local' if not using UTC.

(root) # rc-update add coldplug boot
(root) # rc-update add syslog-ng default
(root) # rc-update add sshd default


10. Set root password

Code:
(root) # passwd   # need to set root passwd!
(root) # pwconv   # sync passwd and shadow


11. Configure bootloader

Code:
(root) # vi /etc/lilo.conf
(root) # cat /etc/lilo.conf

boot=/dev/md0
raid-extra-boot=/dev/sda,/dev/sdb
map=/boot/map
install=/boot/boot.b

prompt
timeout=300

image=/boot/kernel-genkernel-x86-2.6.16-gentoo-r7
  root=/dev/md2
  read-only
  label=Gentoo

(root) # /sbin/lilo -F          # carps about overwriting filesystem (ignore).

12. Reboot

Code:
(root) # exit   # leave chrooted system
(root) # cd
(root) # umount /mnt/gentoo/boot /mnt/gentoo/tmp /mnt/gentoo/var
(root) # umount /mnt/gentoo/usr /mnt/gentoo/home
(root) # umount /mnt/gentoo/dev /mnt/gentoo/proc /mnt/gentoo
(root) # reboot


Notes

Initial performance with common defaults (md chunk-size 32 and calling mkfs.xfs without any options) was very poor. Changing the RAID chunk and xfs stripe unit to 256K improved performance considerably (e.g. from about 50Mb/s -> 215 Mb/s for sequential 8K reads- which seems pretty good).

The filesystem choice of xfs over say the more usual ext3 was driven by the intended use as a database server and a general preference for xfs.

It should be noted that it is possible to avoid the use of extended partitions and gain additional fexlibility by using LVM2. I did not choose to do this, as I like the simplicity of the current setup and also was after as much performance as I could get (I have not specifically checked LVM2, but many LVM implementations add a noticable performance hit.)


Last edited by markir on Thu Jun 08, 2006 2:23 am; edited 6 times in total
Back to top
View user's profile Send private message
nixnut
Administrator
Administrator


Joined: 09 Apr 2004
Posts: 10973
Location: the dutch mountains

PostPosted: Sun Apr 23, 2006 7:46 pm    Post subject: Reply with quote

Moved from Installing Gentoo to Documentation, Tips & Tricks.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered

talk is cheap. supply exceeds demand
Back to top
View user's profile Send private message
neysx
Retired Dev
Retired Dev


Joined: 27 Jan 2003
Posts: 795

PostPosted: Mon Apr 24, 2006 8:15 am    Post subject: Reply with quote

Or simply the draft http://gentoo.neysx.org/mystuff/gentoo-x86+raid+lvm2-quickinstall.xml that I still need to copy to our web site. Based on a reviewed http://gentoo.neysx.org/mystuff/gentoo-x86-quickinstall.xml.

Hth
Back to top
View user's profile Send private message
markir
n00b
n00b


Joined: 22 Apr 2006
Posts: 6
Location: New Zealand

PostPosted: Mon Apr 24, 2006 9:30 am    Post subject: Reply with quote

Quote:
Or simply the draft http://gentoo.neysx.org/mystuff/gentoo-x86+raid+lvm2-quickinstall.xml that I still need to copy to our web site. Based on a reviewed http://gentoo.neysx.org/mystuff/gentoo-x86-quickinstall.xml.


Using mdadm instead of raidtools is nice! Having said that - the raidtab file good, as it is super obvious about how to set the important parameters (like chunksize in particular). Given how much difference it makes to performance, that's a good thing!

On that note, did you do any performance testing with ext3 and 64K chunksize? I'd be interested in seeing some numbers - especially if you have got a 4 disk configuration lying around! (something like 'iozone -t 1 -r 8k -s 8192m -i 0 -i 1 -i 2').
Back to top
View user's profile Send private message
Hun73r
n00b
n00b


Joined: 28 Jun 2004
Posts: 43

PostPosted: Mon Apr 24, 2006 8:42 pm    Post subject: Reply with quote

I have given up on using software raid on gentoo i cant get it to work enyway i get the same error every time
Back to top
View user's profile Send private message
markir
n00b
n00b


Joined: 22 Apr 2006
Posts: 6
Location: New Zealand

PostPosted: Tue Apr 25, 2006 12:05 am    Post subject: Reply with quote

If you have eny energy left for another go, these instructions have worked for me for several installs on different hardware, so there is a good chance that they'll work for yours!
Back to top
View user's profile Send private message
Hun73r
n00b
n00b


Joined: 28 Jun 2004
Posts: 43

PostPosted: Tue Apr 25, 2006 12:36 am    Post subject: Reply with quote

Nope aint gona do it one more time i did it about 30 times now useing meny difrend guids from forum
Back to top
View user's profile Send private message
ben-xo
n00b
n00b


Joined: 13 Dec 2004
Posts: 38

PostPosted: Tue Apr 25, 2006 5:12 pm    Post subject: Reply with quote

Hun73r wrote:
Nope aint gona do it one more time i did it about 30 times now useing meny difrend guids from forum


At a guess, I'd say that you probably typo'd something. Better luck next time.
_________________
Ben XO
Back to top
View user's profile Send private message
odioworks_com
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia

PostPosted: Sat May 27, 2006 3:12 am    Post subject: Reply with quote

nice! I'm following this tutorial right now, and so far so good.

A couple comments:
For those of you who get the following error (like I did):
Code:
cannot determine md version: no MD device file in /dev

do this and try again:
Quote:
cd /dev ; MAKEDEV md


Also, you don't need a local machine to download the stage 3, simply run a command like:
Quote:
wget http://mirrors.acm.cs.rpi.edu/gentoo/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2

in the /home/gentoo directory. You can do the same for the portage snapshot (which wasn't on my install disk):
Quote:
wget http://mirrors.acm.cs.rpi.edu/gentoo/releases/snapshots/2006.0/portage-20060123.tar.bz2



Thanks for the great howto!
Back to top
View user's profile Send private message
odioworks_com
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia

PostPosted: Sat May 27, 2006 7:49 am    Post subject: Reply with quote

problem booting....

hey guys - for some reason I've had a problem booting to the root directory. During boot I get these errors:
Code:
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ReiserFS: md3: warning: sh-2006: read_super_block: bread failed (dev md3, block 2, size 4096)
ReiserFS: md3: warning: sh-2006: read_super_block: bread failed (dev md3, block 16, size 4096)
EXT3-fs: unable to read superblock
EXT2-fs: unable to read superblock
SQUASHFS error: sb_bread failed reading block 0x0
FAT: unable to read boot sector
VFS: Cannot open root device "903" or unknown-block(9,3)
Please append a correct "root=" boot option
Kernel panic - not sysncing: VFS: Unable to mount root fs on unknown-block(9,3)


Any ideas? I've been working on this all night (went through the tutorial twice), and can't figure it out. Not sure why md3 (which is my root directory) is listed above as having a size of 4096, it should be around 140GB...

This is my lilo.conf:
Code:
boot=/dev/md1
raid-extra-boot=/dev/sda,/dev/sdb
map=/boot/map
install=/boot/boot.b

prompt
timeout=30

image=/boot/kernel-genkernel-x86-2.6.15-gentoo-r5
  root=/dev/md3
  read-only
  label=Gentoo

(md1 is the boot partition, md3 the root)
Back to top
View user's profile Send private message
neysx
Retired Dev
Retired Dev


Joined: 27 Jan 2003
Posts: 795

PostPosted: Sat May 27, 2006 8:34 am    Post subject: Reply with quote

odioworks_com wrote:
nice! I'm following this tutorial right now, and so far so good.

A couple comments:
For those of you who get the following error (like I did):
Quote:
cannot determine md version: no MD device file in /dev

do this and try again:
Code:
cd /dev ; MAKEDEV md
Hence the following commands
Code:
livecd ~ # mknod /dev/md1 b 9 1
livecd ~ # mknod /dev/md3 b 9 3
livecd ~ # mknod /dev/md4 b 9 4

odioworks_com wrote:
During boot I get these errors:
Code:
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
No arrays have been detected. Look no further than that.
Did you set the partition type to FD?
Back to top
View user's profile Send private message
odioworks_com
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia

PostPosted: Sat May 27, 2006 2:48 pm    Post subject: Reply with quote

neysx wrote:
odioworks_com wrote:
nice! I'm following this tutorial right now, and so far so good.

A couple comments:
For those of you who get the following error (like I did):
Quote:
cannot determine md version: no MD device file in /dev

do this and try again:
Code:
cd /dev ; MAKEDEV md
Hence the following commands
Code:
livecd ~ # mknod /dev/md1 b 9 1
livecd ~ # mknod /dev/md3 b 9 3
livecd ~ # mknod /dev/md4 b 9 4

I get the above error even after running the mknod commands. MAKEDEV md seems to solve it though

Quote:
odioworks_com wrote:
During boot I get these errors:
Code:
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
No arrays have been detected. Look no further than that.
Did you set the partition type to FD?

Yah, I set the partition type to fd. I was also sure to compile support for the raid devices & xfs, as well as emerge raidtools & mdadm

Not sure what's going on because the raid arrays seem to work find when I boot from the live CD, configure & mount them manually.

Should I be posting this in another forum? Let me know...
Back to top
View user's profile Send private message
odioworks_com
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia

PostPosted: Sat May 27, 2006 7:07 pm    Post subject: Reply with quote

FYI I moved my question to kernel & hardware since this isn't a support forum...
Back to top
View user's profile Send private message
markir
n00b
n00b


Joined: 22 Apr 2006
Posts: 6
Location: New Zealand

PostPosted: Wed May 31, 2006 2:10 am    Post subject: Going to rebuild and update HOWTO Reply with quote

I'm going to be rebuilding this server (one of the disks is getting high numbers of offline sectors, so am replacing it). This means I can see if I can reproduce some of the errors you guys are seeing (I'll make some comments if I do). In addition I'm going to use mdadm instead of raidtools.
Back to top
View user's profile Send private message
jonas_larson
n00b
n00b


Joined: 11 Nov 2004
Posts: 1

PostPosted: Mon Jun 05, 2006 11:56 am    Post subject: Nice but not flexible Reply with quote

Hi,

Very nice!

I would recommend any user to get lvm2 into the picture as well. That way it will be much more flexible. Resizing (online with ext2online) and adding new disks if needed to existing mount-points.

What you need is 2 meta-devices...

md0 = your /boot (ext2/3)
md1 = your 1:st physical volume


my2cents...

//jonas
Back to top
View user's profile Send private message
markir
n00b
n00b


Joined: 22 Apr 2006
Posts: 6
Location: New Zealand

PostPosted: Mon Jun 05, 2006 11:36 pm    Post subject: Reply with quote

Hmm - I quite liked the simplicity of the current method, sure lvm2 will provide more flexibility - particularly resizing volumes etc, but having said that, I can count on my hand the number of times I have actually used those features.

Do you know if there is much of a performance hit with lvm2? - the current setup is blindingly fast, it would be a shame to lose that....

cheers

Mark
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum