| View previous topic :: View next topic |
| Author |
Message |
markir
n00b
Joined: 22 Apr 2006
Posts: 6
Location: New Zealand
|
 Posted: Sun Apr 23, 2006 12:06 am Post subject: Howto Install Gentoo 2006.0 on Software RAID |
 |
|
Introduction
I recently installed Gentoo onto a completely software raid system.
There are a number of helpful articles on the web that discuss this very setup (e.g: http://www.fatofthelan.com/articles/articles.php?pid=19 and https://forums.gentoo.org/viewtopic-t-8813-highlight-raid.html), however they typically do not cover a complete installation.
So, in the hope that it might be useful, here is a complete installation starting from live cd.
The particular hardware used (a Promise TX4000) shows the disks as SCSI - in fact they were not (Maxtor 6E040L0), but with the obvious substitution, these instructions should be helpful for those cards that show ATA/SATA disks as they really are.
Thanks to those who provided feedback - some of this has been incorporated into the latest update!
Pre-Install Steps
1. Obtain the live cd.
2. Boot system from the live cd.
Install Steps
1. Setup networking on live cdrom system
| Code: | (root) # ifconfig eth0 inet my.host.ip.addr netmask my.net.mask.addr
(root) # echo "nameserver name.server.ip.addr" > /etc/resolv.conf
(root) # route add default gw my.gw.ip.addr |
2. Prepare disks
Layout the 4 disks identically with fdisk, setting the partition types
all to 'Linux RAID autodetect'.
| Code: | /dev/sd*
1 128M (boot) (type fd)
2 1G (swap) (type fd)
3 128M (/) (type fd)
4 extended
5 1G (/tmp) (type fd)
6 1G (/var) (type fd)
7 5G (/usr) (type fd)
8 5G (/home) (type fd)
9 rest! (/data0) (type fd)
|
e.g:
| Code: | Disk /dev/sda: 41.1 GB, 41110142976 bytes
255 heads, 63 sectors/track, 4998 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 17 136521 fd Linux raid autodetect
/dev/sda2 18 142 1004062+ fd Linux raid autodetect
/dev/sda3 143 159 136552+ fd Linux raid autodetect
/dev/sda4 160 4998 38869267+ 5 Extended
/dev/sda5 160 284 1004031 fd Linux raid autodetect
/dev/sda6 285 409 1004031 fd Linux raid autodetect
/dev/sda7 410 1032 5004216 fd Linux raid autodetect
/dev/sda8 1033 1655 5004216 fd Linux raid autodetect
/dev/sda9 1656 4998 26852616 fd Linux raid autodetect
|
There is a little subtlety here - the partition to be used as /boot is going to be RAID1, whereas the remainder are to be RAID0. So with the exception of the to-be boot, they are all sized 1/4 of the final assembled size!
3. Setup raid devices
| Code: | (root) # modprobe raid0
(root) # modprobe raid1
(root) # mknod /dev/md0 b 9 0
(root) # mknod /dev/md1 b 9 1
(root) # mknod /dev/md2 b 9 2
(root) # mknod /dev/md3 b 9 3
(root) # mknod /dev/md4 b 9 4
(root) # mknod /dev/md5 b 9 5
(root) # mknod /dev/md6 b 9 6
(root) # mknod /dev/md7 b 9 7
(root) # mdadm --create /dev/md0 --level=1 --chunk=256 --raid-devices=4 /dev/sda
1 /dev/sdb1 /dev/sdc1 /dev/sdd1
(root) # mdadm --create /dev/md1 --level=0 --chunk=256 --raid-devices=4 /dev/sda
2 /dev/sdb2 /dev/sdc2 /dev/sdd2
(root) # mdadm --create /dev/md2 --level=0 --chunk=256 --raid-devices=4 /dev/sda
3 /dev/sdb3 /dev/sdc3 /dev/sdd3
(root) # mdadm --create /dev/md3 --level=0 --chunk=256 --raid-devices=4 /dev/sda
5 /dev/sdb5 /dev/sdc5 /dev/sdd5
(root) # mdadm --create /dev/md4 --level=0 --chunk=256 --raid-devices=4 /dev/sda
6 /dev/sdb6 /dev/sdc6 /dev/sdd6
(root) # mdadm --create /dev/md5 --level=0 --chunk=256 --raid-devices=4 /dev/sda
7 /dev/sdb7 /dev/sdc7 /dev/sdd7
(root) # mdadm --create /dev/md6 --level=0 --chunk=256 --raid-devices=4 /dev/sda
8 /dev/sdb8 /dev/sdc8 /dev/sdd8
(root) # mdadm --create /dev/md7 --level=0 --chunk=256 --raid-devices=4 /dev/sda
9 /dev/sdb9 /dev/sdc9 /dev/sdd9
(root) # echo "DEVICE /dev/sd[abcd][12356789]" >> /etc/mdadm.conf
(root) # mdadm --detail --scan >> /etc/mdadm.conf |
4. Create and mount filesystems
| Code: | (root) # mkfs.xfs -d su=256k,sw=2 /dev/md0
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md2
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md3
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md4
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md5
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md6
(root) # mkfs.xfs -d su=256k,sw=2 /dev/md7
(root) # mkswap /dev/md1
(root) # swapon /dev/md1
(root) # mount /dev/md2 /mnt/gentoo
(root) # mkdir /mnt/gentoo/boot
(root) # mkdir /mnt/gentoo/tmp
(root) # mkdir /mnt/gentoo/var
(root) # mkdir /mnt/gentoo/usr
(root) # mkdir /mnt/gentoo/home
(root) # mkdir /mnt/gentoo/data0 # (don't use, but create the mountpoint)
(root) # mount /dev/md0 /mnt/gentoo/boot
(root) # mount /dev/md3 /mnt/gentoo/tmp
(root) # mount /dev/md4 /mnt/gentoo/var
(root) # mount /dev/md5 /mnt/gentoo/usr
(root) # mount /dev/md6 /mnt/gentoo/home |
5. Install distribution and portage snapshot
| Code: | (root) # date 060112002006 # (as probably in UTC, so back it off)
(root) # cd /home/gentoo
(root) # wget http://mirrors.acm.cs.rpi.edu/gentoo/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2
(root) # cd /mnt/gentoo
(root) # tar -xvjpf /home/gentoo/stage3-i686-2006.0.tar.bz2
(root) # tar -xvjf /mnt/cdrom/snapshots/portage-20060123.tar.bz2 -C /mnt/gentoo/usr
(root) # mirrorselect -i -o >> /mnt/gentoo/etc/make.conf |
6. Chroot system
| Code: | (root) # cp /etc/mdadm.conf /mnt/gentoo/etc/mdadm.conf
(root) # cp /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
(root) # mount -t proc none /mnt/gentoo/proc
(root) # mount -o bind /dev /mnt/gentoo/dev
(root) # chroot /mnt/gentoo /bin/bash
(root) # env-update
(root) # source /etc/profile
(root) # export PS1="(chroot) $PS1"
(root) # echo "USE=\"-gtk -gnome qt kde dvd alsa cdr\"" >>/etc/make.conf
(root) # emerge --sync # get latest portage updates. |
7. Kernel configuration using genkernel
| Code: | (root) # cp /usr/share/zoneinfo/<yourzonefile> /etc/localtime
(root) # USE="-doc symlink" emerge gentoo-sources
(root) # emerge genkernel
(root) # zcat /proc/config.gz > /usr/share/genkernel/x86/kernel-config-2.6
(root) # genkernel --menuconfig all # include xfs, RAID, not just as modules!
# and maybe SCSI low level drivers
# e.g. SCSI_PROMISE_SATA in this case |
8. Install Ebuilds
| Code: | (root) # emerge coldplug
(root) # emerge syslog-ng
(root) # emerge xfsprogs
(root) # emerge xfsdump
(root) # emerge mdadm
(root) # emerge portage-utils
(root) # emerge vim
(root) # emerge lilo |
9. Configure system
| Code: | (root) # vi /etc/fstab
(root) # cat /etc/fstab|grep -v '^#'
/dev/md0 /boot xfs defaults,noatime 1 2
/dev/md2 / xfs noatime 0 1
/dev/md3 /tmp xfs noatime 0 2
/dev/md4 /var xfs noatime 0 2
/dev/md5 /usr xfs noatime 0 2
/dev/md6 /home xfs noatime 0 2
/dev/md7 /data0 xfs noatime 0 2
/dev/md1 none swap sw 0 0
/dev/cdrom /mnt/cdrom iso9660 noauto,ro 0 0
proc /proc proc defaults 0 0
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0
(root) # vi /etc/conf.d/hostname
(root) # vi /etc/conf.d/domainname
(root) # rc-update add domainname default
(root) # echo "config_eth0=( \"my.host.ip.addr netmask my.net.mask.addr brd my.net.bcast.addr\" )" >> /etc/conf.d/net
(root) # echo "routes_eth0=( \"default gw my.gw.ip.addr\" )" >> /etc/conf.d/net
(root) # rc-update add net.eth0 default
(root) # cat /etc/hosts|grep -v "^#"
127.0.0.1 localhost myhostname.mydomain myhostname
(root) # vi /etc/rc.conf
(root) # vi /etc/conf.d/clock # set clock to 'local' if not using UTC.
(root) # rc-update add coldplug boot
(root) # rc-update add syslog-ng default
(root) # rc-update add sshd default |
10. Set root password
| Code: | (root) # passwd # need to set root passwd!
(root) # pwconv # sync passwd and shadow
|
11. Configure bootloader
| Code: | (root) # vi /etc/lilo.conf
(root) # cat /etc/lilo.conf
boot=/dev/md0
raid-extra-boot=/dev/sda,/dev/sdb
map=/boot/map
install=/boot/boot.b
prompt
timeout=300
image=/boot/kernel-genkernel-x86-2.6.16-gentoo-r7
root=/dev/md2
read-only
label=Gentoo
(root) # /sbin/lilo -F # carps about overwriting filesystem (ignore).
|
12. Reboot
| Code: | (root) # exit # leave chrooted system
(root) # cd
(root) # umount /mnt/gentoo/boot /mnt/gentoo/tmp /mnt/gentoo/var
(root) # umount /mnt/gentoo/usr /mnt/gentoo/home
(root) # umount /mnt/gentoo/dev /mnt/gentoo/proc /mnt/gentoo
(root) # reboot |
Notes
Initial performance with common defaults (md chunk-size 32 and calling mkfs.xfs without any options) was very poor. Changing the RAID chunk and xfs stripe unit to 256K improved performance considerably (e.g. from about 50Mb/s -> 215 Mb/s for sequential 8K reads- which seems pretty good).
The filesystem choice of xfs over say the more usual ext3 was driven by the intended use as a database server and a general preference for xfs.
It should be noted that it is possible to avoid the use of extended partitions and gain additional fexlibility by using LVM2. I did not choose to do this, as I like the simplicity of the current setup and also was after as much performance as I could get (I have not specifically checked LVM2, but many LVM implementations add a noticable performance hit.)
Last edited by markir on Thu Jun 08, 2006 2:23 am; edited 6 times in total |
|
| Back to top |
|
 |
nixnut
Bodhisattva
Joined: 09 Apr 2004
Posts: 10974
Location: the dutch mountains
|
| Posted: Sun Apr 23, 2006 7:46 pm Post subject: |
|
|
Moved from Installing Gentoo to Documentation, Tips & Tricks.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
talk is cheap. supply exceeds demand |
|
| Back to top |
|
|
neysx
Retired Dev
Joined: 27 Jan 2003
Posts: 795
|
|
| Back to top |
|
|
markir
n00b
Joined: 22 Apr 2006
Posts: 6
Location: New Zealand
|
| Posted: Mon Apr 24, 2006 9:30 am Post subject: |
|
|
Using mdadm instead of raidtools is nice! Having said that - the raidtab file good, as it is super obvious about how to set the important parameters (like chunksize in particular). Given how much difference it makes to performance, that's a good thing!
On that note, did you do any performance testing with ext3 and 64K chunksize? I'd be interested in seeing some numbers - especially if you have got a 4 disk configuration lying around! (something like 'iozone -t 1 -r 8k -s 8192m -i 0 -i 1 -i 2'). |
|
| Back to top |
|
|
Hun73r
n00b
Joined: 28 Jun 2004
Posts: 49
|
| Posted: Mon Apr 24, 2006 8:42 pm Post subject: |
|
|
| I have given up on using software raid on gentoo i cant get it to work enyway i get the same error every time |
|
| Back to top |
|
|
markir
n00b
Joined: 22 Apr 2006
Posts: 6
Location: New Zealand
|
| Posted: Tue Apr 25, 2006 12:05 am Post subject: |
|
|
| If you have eny energy left for another go, these instructions have worked for me for several installs on different hardware, so there is a good chance that they'll work for yours! |
|
| Back to top |
|
|
Hun73r
n00b
Joined: 28 Jun 2004
Posts: 49
|
| Posted: Tue Apr 25, 2006 12:36 am Post subject: |
|
|
| Nope aint gona do it one more time i did it about 30 times now useing meny difrend guids from forum |
|
| Back to top |
|
|
ben-xo
n00b
Joined: 13 Dec 2004
Posts: 38
|
| Posted: Tue Apr 25, 2006 5:12 pm Post subject: |
|
|
| Hun73r wrote: | | Nope aint gona do it one more time i did it about 30 times now useing meny difrend guids from forum |
At a guess, I'd say that you probably typo'd something. Better luck next time.
_________________
Ben XO |
|
| Back to top |
|
|
odioworks_com
Tux's lil' helper
Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia
|
| Posted: Sat May 27, 2006 3:12 am Post subject: |
|
|
nice! I'm following this tutorial right now, and so far so good.
A couple comments:
For those of you who get the following error (like I did):
| Code: | cannot determine md version: no MD device file in /dev
|
do this and try again:
| Quote: | | cd /dev ; MAKEDEV md |
Also, you don't need a local machine to download the stage 3, simply run a command like:
in the /home/gentoo directory. You can do the same for the portage snapshot (which wasn't on my install disk):
Thanks for the great howto! |
|
| Back to top |
|
|
odioworks_com
Tux's lil' helper
Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia
|
| Posted: Sat May 27, 2006 7:49 am Post subject: |
|
|
problem booting....
hey guys - for some reason I've had a problem booting to the root directory. During boot I get these errors:
| Code: | md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ReiserFS: md3: warning: sh-2006: read_super_block: bread failed (dev md3, block 2, size 4096)
ReiserFS: md3: warning: sh-2006: read_super_block: bread failed (dev md3, block 16, size 4096)
EXT3-fs: unable to read superblock
EXT2-fs: unable to read superblock
SQUASHFS error: sb_bread failed reading block 0x0
FAT: unable to read boot sector
VFS: Cannot open root device "903" or unknown-block(9,3)
Please append a correct "root=" boot option
Kernel panic - not sysncing: VFS: Unable to mount root fs on unknown-block(9,3) |
Any ideas? I've been working on this all night (went through the tutorial twice), and can't figure it out. Not sure why md3 (which is my root directory) is listed above as having a size of 4096, it should be around 140GB...
This is my lilo.conf:
| Code: | boot=/dev/md1
raid-extra-boot=/dev/sda,/dev/sdb
map=/boot/map
install=/boot/boot.b
prompt
timeout=30
image=/boot/kernel-genkernel-x86-2.6.15-gentoo-r5
root=/dev/md3
read-only
label=Gentoo |
(md1 is the boot partition, md3 the root) |
|
| Back to top |
|
|
neysx
Retired Dev
Joined: 27 Jan 2003
Posts: 795
|
| Posted: Sat May 27, 2006 8:34 am Post subject: |
|
|
| odioworks_com wrote: | nice! I'm following this tutorial right now, and so far so good.
A couple comments:
For those of you who get the following error (like I did):
| Quote: | | cannot determine md version: no MD device file in /dev |
do this and try again:
| Code: | | cd /dev ; MAKEDEV md |
|
Hence the following commands | Code: | livecd ~ # mknod /dev/md1 b 9 1
livecd ~ # mknod /dev/md3 b 9 3
livecd ~ # mknod /dev/md4 b 9 4 |
| odioworks_com wrote: | During boot I get these errors: | Code: | md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE. |
|
No arrays have been detected. Look no further than that.
Did you set the partition type to FD? |
|
| Back to top |
|
|
odioworks_com
Tux's lil' helper
Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia
|
| Posted: Sat May 27, 2006 2:48 pm Post subject: |
|
|
| neysx wrote: | | odioworks_com wrote: | nice! I'm following this tutorial right now, and so far so good.
A couple comments:
For those of you who get the following error (like I did):
| Quote: | | cannot determine md version: no MD device file in /dev |
do this and try again:
| Code: | | cd /dev ; MAKEDEV md |
|
Hence the following commands | Code: | livecd ~ # mknod /dev/md1 b 9 1
livecd ~ # mknod /dev/md3 b 9 3
livecd ~ # mknod /dev/md4 b 9 4 |
|
I get the above error even after running the mknod commands. MAKEDEV md seems to solve it though
| Quote: | | odioworks_com wrote: | During boot I get these errors: | Code: | md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE. |
|
No arrays have been detected. Look no further than that.
Did you set the partition type to FD? |
Yah, I set the partition type to fd. I was also sure to compile support for the raid devices & xfs, as well as emerge raidtools & mdadm
Not sure what's going on because the raid arrays seem to work find when I boot from the live CD, configure & mount them manually.
Should I be posting this in another forum? Let me know... |
|
| Back to top |
|
|
odioworks_com
Tux's lil' helper
Joined: 23 Jan 2005
Posts: 82
Location: Charlottesville, Virginia
|
| Posted: Sat May 27, 2006 7:07 pm Post subject: |
|
|
| FYI I moved my question to kernel & hardware since this isn't a support forum... |
|
| Back to top |
|
|
markir
n00b
Joined: 22 Apr 2006
Posts: 6
Location: New Zealand
|
| Posted: Wed May 31, 2006 2:10 am Post subject: Going to rebuild and update HOWTO |
|
|
| I'm going to be rebuilding this server (one of the disks is getting high numbers of offline sectors, so am replacing it). This means I can see if I can reproduce some of the errors you guys are seeing (I'll make some comments if I do). In addition I'm going to use mdadm instead of raidtools. |
|
| Back to top |
|
|
jonas_larson
n00b
Joined: 11 Nov 2004
Posts: 1
|
| Posted: Mon Jun 05, 2006 11:56 am Post subject: Nice but not flexible |
|
|
Hi,
Very nice!
I would recommend any user to get lvm2 into the picture as well. That way it will be much more flexible. Resizing (online with ext2online) and adding new disks if needed to existing mount-points.
What you need is 2 meta-devices...
md0 = your /boot (ext2/3)
md1 = your 1:st physical volume
my2cents...
//jonas |
|
| Back to top |
|
|
markir
n00b
Joined: 22 Apr 2006
Posts: 6
Location: New Zealand
|
| Posted: Mon Jun 05, 2006 11:36 pm Post subject: |
|
|
Hmm - I quite liked the simplicity of the current method, sure lvm2 will provide more flexibility - particularly resizing volumes etc, but having said that, I can count on my hand the number of times I have actually used those features.
Do you know if there is much of a performance hit with lvm2? - the current setup is blindingly fast, it would be a shame to lose that....
cheers
Mark |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
