root on hardware RAID(scsi) with initrd

[conchyliferous]

Hey, this whole scenario might be tricky to explain but I have fought with it some days now so I will try.

I have three disks in my box. One 140GB IDE and two RAIDed 120GB SATA disks.

I have run gentoo for a long time now only on my IDE drive, not even used the RAID disks.

Since the RAID disks are so much faster than the single IDE one, I would like to install gentoo on them and throw away my slow disk.

This should be kind of easy I though. I'm using a RAID/SATA card from Highpoint and I have compiled the driver module from their site and that works. The module is called 'hpt374.ko'.

In my current configuration when i do a 'insmod hpt374.ko', it works like expected. I get some sda devices from it and I can mount them without problems.

When I knew that worked i begun to try get a bootable root on the RAID. First of all I partitioned the RAID into a boot, swap and root partition.

Since BIOS uses interrupt on SCSI (RAID) exactly the same way as it does on ordinary IDE it's no problems to get Grub use the boot partition on the RAID. The tricky part, for me anyway, is to get the Linux kernel to find it's way on the RAID. So the boot partition on the RAID is setup and working but if I try to go ahead and load the kernel I get 'Kernel panic: bla bla' because the RAID driver isn't loaded into kernel yet (since the kernel isn't loaded ). So for this I have to use initrd, a RAM-disk that loads before the kernel.

As I see it, it's when making this initrd image i get problems. I have spent some days now googeling for information how to do this the right way, but there isn't much to go on and most of the information is outdated or unprecise. But I did find how to make and mount the initrd image and some information on what files and directorys that should be in it. Additional to this I want to include and load my 'hpt374.ko' module so that the rest of the kernel can be loaded later. And for the 'hpt374.ko' module I guess I need to have SCSI support and SCSI disk support (RAID=>SCSI). Therefore I need to load 'scsi_mod.ko' and 'sd_mod.ko'. And after that I need to get ext3-support so that I can mount the filesystem, for that I need to load 'jbd.ko' and 'ext3.ko'.

A made the initrd and put the files needed in it. The following is a list of files and directory that I have:

• .
  ./lib
  ./lib/scsi_mod.ko
  ./lib/sd_mod.ko
  ./lib/hpt374.ko
  ./lib/jbd.ko
  ./lib/ext3.ko
  ./bin
  ./bin/nash
  ./bin/insmod
  ./sbin
  ./sbin/bin
  ./sbin/modprobe
  ./etc
  ./dev
  ./dev/console
  ./dev/null
  ./dev/ram
  ./dev/systty
  ./dev/tty1
  ./dev/tty2
  ./dev/tty3
  ./dev/tty4
  ./loopfs
  ./proc
  ./sysroot
  ./linuxrc

This image gets loaded into the memory before the kernel gets loaded. To make this useful the 'linuxrc' script is run and things should get prepared before the real kernel is loaded. In my case I must have kernel-support for the disks that the kernel is loaded from.

This is my linuxrc script:

Code: Select all

echo "Loading scsi_mod module"
insmod /lib/scsi_mod.ko
echo "Loading sd_mod module"
insmod /lib/sd_mod.ko
echo "Loading RocketRAID module"
insmod /lib/hpt374.ko
echo "Loading jbd module"
insmod /lib/jbd.ko
echo "Loading ext3 module"
insmod /lib/ext3.ko
mount -t proc /proc /proc
echo Mounting /proc filesystem
echo Creating root device
mkrootdev /dev/root
echo 0x0100 > /proc/sys/kernel/real-root-dev
umount /proc
echo Mounting root filesystem
mount --ro -t ext3 /dev/root /sysroot
pivot_root /sysroot /sysroot/initrd

(I know there is a program called mkinitrd that is supposed to make an initrd for you. But I didn't get that to work and I felt I got more control this way)

Now I should boot this up. But before that I need to make changes in my grub.conf and tell grub to load the image.
This is my grub.conf:

Code: Select all

timeout 43

default 0

title  GNU/Linux
root (hd0,0)
kernel /bzImage ro root=/dev/sda3 noapic
initrd /initrd.img

The initrd.img is in place in the boot partition and all that. So let's reboot!
I get trough the bootloader alright and grub reads the initrd image and starts executing the linuxrc script. But already at 'insmod /lib/scsi_mod.ko' I get an error. The error output rushes over the screen very fast and it's hard to read what it's all about, but it's practically alot of hex code and stuff and in the end I get the message: ERROR: /bin/insmod exited abnormally!
But the boot continues and I get the same error at every other module except 'hpt374.ko' that, for some reason, seam to load correct and I can see that it gets the devices sda1, sda2 and sda3. And i can see that the total size of the disk is 240GB (and that's correct, it's two RAID 120GB disks). But still I got errors on the other modules so I get a kernel panic when root is going to mount.

So the question is. Why can't I insmod the other modules? This could maybe have something to do with the kernel version the modules are compiled for. But how could that be? They are all compiled for 2.6.11-gentoo-r6 and that's also the version I'm running. Please help me with this or give me other ways of doing this if they exists.

Sorry for the essay, but I wanted this to be useful for as many as possible because it's hard to find information on this.

[kabage]

your initrd commands need libs too (unless you compiled static) like insmod modprobe

I personally would build all those modules into the kernel not as modules (build-in) then you just need
to map the devices

dmsetup (from device-mapper application ) device mapper need to be enabled in kernel also
dmraid comes to mind

(their should be a highpoint module in the kernel if not there some howtos on sticking it in there)

Now your on the right track heres a link to the best initrd (from scratch making I've seen)

http://forums.gentoo.org/viewtopic-t-24 ... art-0.html

go to creating the initrd section

oh that reminds me in the grub.conf you will need root=/dev/ram0 that is your initial ram disk starting point not the harddisk your harddisk partition would be

real_root=/dev/sda3 or whatever and dont forget init=/linuxrc

Heres mine (I'm using dmraid thats why the weird device on my harddrive -- If your dualbooting thats the only way to go and I believe its compatible with highpoint *some of them anyway)

Code: Select all

title=X-Dream Gentoo v7 new init UDEV
root=(hd0,7)
kernel /xdream-v7 root=/dev/ram0 real_root=/dev/mapper/pdc_ciiaceccai7 init=/linuxrc gentoo=udev
initrd /xdreaminit

[cummings66]

First off I think you're making it too hard on yourself. You don't need to be so complex. Things are so much easier if you just say no to genkernel.

Compile a new kernel, and in that kernel make sure you have the driver you need, do not make it a module.

Now once that's done, and I hope you didn't use genkernel for it, do it like this;

make menuconfig and select what you need.
make && make modules_install

mount /boot
cd arch/your type/boot might be something like cd /usr/src/linux/arch/i386/boot/ for example
cp bzImage /boot/kernel-2.6.11-gentoo-r9 (or whatever version you use)
edit the grub.conf as I have it here for example and then run grub
I do it manually per the install manual and keep the old kernel around so that if I screwed up something I can reboot into a working system instead of chrooting into it.

Here's a simple grub.conf that works from my scsi array. Since you said yours uses scsi device names I think it would work ok for you as is. You've got to use the hd0,0 bit because that's what grub expects, and later you use the root with the scsi device name that you actually used in fdisk for your root partition.

default 0
timeout 10
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title=Gentoo Linux (kernel-2.6.11-r9)
root (hd0,0)
kernel (hd0,0)/boot/kernel-2.6.11-gentoo-r9 root=/dev/sda3 splash=verbose

title=Gentoo Linux (kernel-2.6.11-r8)
root (hd0,0)
kernel (hd0,0)/boot/kernel-2.6.11-gentoo-r8 root=/dev/sda3 splash=verbose

[conchyliferous]

I have made some misunderstandings on this that's for sure.

I still don't know exactly in what order everything gets loaded.
First I thought that the initrd image was loaded first of all and if I wanted to load a 3rd part module in it I also needed all dependencies for that module. But that doesn't seam to be the case according to some googling.

The second misunderstanding was about how hardware raid works. The thing is that as soon as I have loaded the Highpoint module and got the sda devices I can fully consider the raid as a scsi disk. My box does not know the difference either. So searching for "root scsi initrd" instead if "root raid initrd" helps a lot. In fact almost everything you find if searching the net for linux and raid is regarding software raid.

I don't think I can make use of cummings66's solution because I need to load my module from Highpoint and I can't compile that into kernel. For that I need the initrd image, but as you say, my initial solution is unnecessary complex.

One more thing, the Highpoint thing you can compile into kernel is not the same module that I need. That is something else.

My second try:
What I did was that I didn't care about loading dependencies for the 'hpt374.ko' module from Highpoint. And the second thing I did was to read the manual for 'mkinitrd'.

So now I got (like before) ext3, scsi_mod and sd_mod compiled into kernel but I will not include and load them in the initrd. (I guess the error I got before when insmoding the modules had to do with that I had the modules compiled in kernel and loaded them with insmod at the same time)

And now I make my image with mkinitrd.

Code: Select all

mkinitrd --with=hpt374 --fstab etc/fstab --builtin=scsi_mod --builtin=sd_mod --builtin=jbd --builtin=ext3  initrd.img 2.6.11-gentoo-r6

Where the fstab contains the following:

Code: Select all

#SCSI (RAID)
/dev/sda1      /boot   ext3        defaults,noatime    1 2
/dev/sda2      none    swap        sw                  0 0
/dev/sda3      /       ext3        defaults,noatime    0 1

#IDE . doesn't matter anymore. Just put it somewhere. Ignore old swap for now
/dev/hda1       /IDE/boot   ext2        defaults,noatime    1 2
/dev/hda3       /IDE/       ext2        defaults,noatime    0 1

#cd and floppy
/dev/hdc        /mnt/dvdrw  auto        noauto,ro,user  0 0
/dev/hdd        /mnt/dvdr   auto        noauto,ro,user  0 0
/dev/fd0        /mnt/floppy auto        noauto          0 0

none            /proc       proc        defaults        0 0
none            /dev/shm    tmpfs       defaults        0 0

My /etc/modules.conf contains, among a lot of other things, the following line:

Code: Select all

alias scsi_hostadapter hpt374

I can manually open the initrd file and check it's doing what it's supposed to do.
The image contains the hpt374.ko file and the linuxrc file is like this:

Code: Select all

#!/bin/nash

echo "Loading hpt374.ko module"
insmod /lib/hpt374.ko
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo Creating block devices
mkdevices /dev
echo Creating root device
mkrootdev /dev/root
echo 0x0100 > /proc/sys/kernel/real-root-dev
echo Mounting root filesystem
mount -o defaults,noatime --ro -t ext3 /dev/root /sysroot
pivot_root /sysroot /sysroot/initrd
umount /initrd/proc

And that looks alright, and since 'mkinitrd' did it and I haven't done it myself it should be right.

I put this image in my /boot and configure my grub.conf like this:

Code: Select all

timeout 63

default 0

title  GNU/Linux
root (hd0,0)
kernel /bzImage ro root=/dev/sda3 noapic
initrd /initrd.img

When I was going to reboot now I was optimistic, but of course I got an error on boot. Took a picture of the error message and here it comes:

.
.
.
RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem).
Red Hat nash version 3.5.7 starting
Loading hpt374.ko module
hpt374: module license ´Proprietary´ taints kernel.
RocketRAID 154x/1640 SATA Controller driver
Version 2.13, Compiled May 9 2005 18:44:05
usb 4-1: new full speed USB device using uhci_hcd and adress 2
RAID5 write-back enabled
ACPI: PCI interrupt 0000:02:03.0[A] -> GSI 11 (level, low) -> IRQ 11
ACPI: PCI interrupt 0000:02:03.1[A] -> GSI 11 (level, low) -> IRQ 11
scsi0 : hpt374
Vendor: HPT3xx Model: RAID 0 Array Rev: 3.0
Type: Direct-Access ANSI SCSI revision: 00
SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
sda: sda1 sda2 sda3
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Mounting /proc filesystem
Creating block devices
Creating root device
Mounting root filesystem
mount: error 6 mounting ext3
pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2
umount /initrd/proc failed: 2
Freeing unused kernel memory: 216k freed
Kernel panic - not syncing: No init found. Try passing init= option to kernel.

As you see the image gets loaded and the linuxrc script executed. I'm not really sure about what that usb thing is doing there. The hpt374 seam to load alright but the root device cannot open.
Please help!

[irondog]

Why don'y you replace:

Code: Select all

mkrootdev /dev/root
echo 0x0100 > /proc/sys/kernel/real-root-dev
echo Mounting root filesystem
mount -o defaults,noatime --ro -t ext3 /dev/root /sysroot 

By:

Code: Select all

mknod sda3 b 8 3
echo Mounting root filesystem
mount --ro -t ext3 /dev/sda3 /sysroot

[conchyliferous]

I tried that but got another kernel panic:

.
.
.
Mounting /proc filesystem
Creating block devices
Creating root device
Mounting root filesystem
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2
umount /initrd/proc failed: 2
VFS: Cannot open root device "sda3" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

The rest of it is like before.
Can't say I understand this. Is this better or worse? Any other suggestions?

[Unther]

It's an improvement, it's the pivot_root command failing not the mount command.

try replacing

Code: Select all

pivot_root /sysroot /sysroot/initrd 

with

Code: Select all

cd /sysroot
pivot_root . initrd

At one point I did this for an install to a USB device. I just found that this was the way the pivot_root man page recommended, and it wroked for me.

You should probably change your grub config from

Code: Select all

kernel /bzImage ro root=/dev/sda3 noapic 

to

Code: Select all

kernel /bzImage ro root=/dev/ram0 noapic 

[conchyliferous]

Well that didn't make it better I'm afraid.

Just changing the 'pivot_root' thing resulted in this:

Code: Select all

.
.
.
EXT3-fs: mounted filesystem with ordered data mode.
ERROR: failed in exec of cd
pivotroot: pivot_root(.,initrd) failed: 2
umount /initrd/proc failed: 2
ERROR: cd exited abnormally!
pivotroot: pivot_root(.,initrd) failed: 2
umount /initrd/proc failed: 2
VFS: Cannot open root device "sda3" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) 

Then changing the kernel root option also had an even worse result:

Code: Select all

.
.
.
RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 216k freed
Kernel panic - not syncing: No init found. Try passing init= option to kernel. 

Looks like linuxrc isn't read at all! Suggestions?

[Unther]

My bad, getting careless, and making assumptions from my setup.

That should have read

Code: Select all

kernel /bzImage ro root=/dev/ram0 noapic init=linuxrc

for the grub enetry.

You also need a directory to drop the old root into when you pivot_root in the real root filesystem, I called mine initrd. Create that.

You said you were able to get into the initrd file, what does it contain (Which executables)?

[conchyliferous]

That didn't work either. Still got that 'cd' problem but I also got another interesting error.

Check the following output, it's the same till the "mount /proc" thing:

Code: Select all

.
.
.
RAMDISK: Compressed image found at block 0 
VFS: Mounted root (ext2 filesystem). 
Red Hat nash version 3.5.7 starting 
Loading hpt374.ko module 
hpt374: module license ´Proprietary´ taints kernel. 
RocketRAID 154x/1640 SATA Controller driver 
Version 2.13, Compiled May 9 2005 18:44:05 
usb 4-1: new full speed USB device using uhci_hcd and adress 2 
RAID5 write-back enabled 
ACPI: PCI interrupt 0000:02:03.0[A] -> GSI 11 (level, low) -> IRQ 11 
ACPI: PCI interrupt 0000:02:03.1[A] -> GSI 11 (level, low) -> IRQ 11 
scsi0 : hpt374 
Vendor: HPT3xx Model: RAID 0 Array Rev: 3.0 
Type: Direct-Access ANSI SCSI revision: 00 
SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB) 
sda: asking for cache data failed 
sda: assuming drive cache: write through 
SCSI device sda: 468883200 512-byte hdwr sectors (240068 MB) 
sda: asking for cache data failed 
sda: assuming drive cache: write through 
sda: sda1 sda2 sda3 
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Mounting /proc filesystem
Creating block devices
failed to create /dev/sda
failed to create /dev/sda1
failed to create /dev/sda2
failed to create /dev/sda3
Creating root device
mknod: failed to create sda3: 30
Mounting root filesystem
mount: error 2 mounting ext3
ERROR: failed in exec of cd 
pivotroot: pivot_root(.,initrd) failed: 2 
umount /initrd/proc failed: 2 
ERROR: cd exited abnormally! 
pivotroot: pivot_root(.,initrd) failed: 2 
umount /initrd/proc failed: 2 
Kernel panic - not syncing: Attempted to kill init!

The output is like before
It looks like sda gets initialized alrigt but later it fails to load
And that cd problem is still there... Maybe that other way works since mkinitrd made it that way?

The little tux in me is about to give up and the big bad windows monster is thinking about how easy this is in windows where you just have to press F4 in the windows installation-program...
Please save me!

[cummings66]

Try this in your grub.

Enter "pci=noacpi" or "acpi=off", in lower case.

[conchyliferous]

I'm sorry, can't do that.
Gave up the whole 'boot from raid just like scsi'-idea, put far to many days on that. Put an old fashion IDE into the box and put / on that while /usr, /var, /opt, /tmp and /home on the scsi(raid). That way I still get the speed where I want it and have no problems booting.
But it would still be great if someone could point out to me what's the problem here. I will check in on this thread now and then and if someone says something that sounds lika a solution I could use that next time. Otherwise it would be good for others with the same problem...

[cummings66]

Well, I did a google search and a guy had the exact same problem you did. He added those lines and it worked for him, thus my passing it along.

Sometimes things like this are tough to setup. I used a second computer to test things like this and when it's working I backup the system and restore it on the hardware I want.

[micah_death]

I don't know if this 'caused a problem for you but when I compiled my raid stuff in my kernel, it reordered my devices according to the lspci list order =(
(all of mine are ide)

/dev/hda became /dev/hde and so on...
I did have a fix for it: (/etc/lilo.conf)

Code: Select all

image=/boot/bzImage
  label=gentooraid
  read-only
  root=/dev/hdb3
  append="ide4=0x9800,0x9c02,5 ide5=0xa000,0xa402,5 ide6=0xac00,0xb002,11 ide7=0xb400,0xb802,11"

Don't know if it will help, but it fixed me =)

[volumen1]

Based on this error:

Code: Select all

pivotroot: pivot_root(.,initrd) failed: 2
umount /initrd/proc failed: 2 

I think you should do this:

Code: Select all

mkdir /initrd

That is, just make an empty /initrd directory off of your root. I was getting that same error and that fixed me right up. I got the idea from here, http://gentoo-wiki.com/HOWTO_Install_Ge ... our_initrd