[SOLVED] AACRAID kernel panic and other driver issues

[abduct]

Hello everyone. I am trying to install gentoo onto my IBM x3650 7979 M1 server and I have a bootable OS already installed but there are a few problems due to drivers. All drivers work on the system rescue cd so it is narrowed down to me missing some settings or firmware blobs.

The first problem is AACRAID (which system rescue cd loads fine) causes the 4.1.7-hardened-r1 kernel to panic during boot, and without it I cannot see my two raid volumes. One kernel panic says something along the lines of "VFS: unable to mount root fs on unknown block(...)" and another kernel panic that doesn't state much information besides some addresses such as 0xffffffffffffffff. When removing the driver from the kernel all kernel panics disapeer and the system boots fine, but gentoo does not see any of the drives.

When enabling I simply used the search function for "AACRAID" and enabled the only option returned. Is there more that has to be done? Is there a firmware blob I have to compile into the kernel maybe?

My next problem is that I can not get the RADEON DRM drivers to work on my kernel even though they work fine in the system rescue CD. I tried going through dmesg and it says it loaded the /lib/radeon/R100_cp.bin file (when booting from system rescue cd), but when i tried copying the file to my drive and recompiling the kernel to use RADEON and set the firmware blobs/root directory I get a blank screen right after the first few lines of the init process (roughly right when it starts loading the drivers).

This isn't crucial since VESAFB works and I have grub2 setting the resolution fine, but it would be nice to have the ATI ES1000 card working fully with proper drivers.

Lastly I have two QLogic QLE2460 fiber PCI cards which I have tried compiling the drivers for but they complain about needing firmware. They link http://ldriver.qlogic.com/firmware/ this as the site but I am wondering which firmware binary I should use. I assume the ql2400_fw.bin would be it correct? These are not needed and will likely never be used, but having them in a working state would be nice in case I ever install the server into a fiber fed data center.

Here are the current drivers I have compiled that are in use (if you can't tell I have loadable module support disabled so everything must be compiled in):

Code: Select all

 ~ # lspci -vk | grep Kernel | sort -u
lspci: Unable to load libkmod resources: error -12
        Kernel driver in use: ata_piix
        Kernel driver in use: bnx2
        Kernel driver in use: ehci-pci
        Kernel driver in use: i5000_edac
        Kernel driver in use: i801_smbus
        Kernel driver in use: ioatdma
        Kernel driver in use: lpc_ich
        Kernel driver in use: pcieport
        Kernel driver in use: qla2xxx
        Kernel driver in use: uhci_hcd

Here is a list of all the hardware in the server:

Code: Select all

~ # lspci
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev b1)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev b1)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev b1)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev b1)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev b1)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev b1)
00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA Engine (rev b1)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09)
00:1c.1 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 2 (rev 09)
00:1d.0 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09)
00:1d.1 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09)
00:1d.2 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09)
00:1d.3 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09)
00:1d.7 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller (rev 09)
00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller (rev 09)
01:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
04:00.0 RAID bus controller: Adaptec AAC-RAID (Rocket) (rev 02)
05:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
1a:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
1a:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01)
1b:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01)
1b:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01)
1c:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
24:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

And here is the list of hardware for use on search sites:

Code: Select all

~ # lspci -n
00:00.0 0600: 8086:25d8 (rev b1)
00:02.0 0604: 8086:25e2 (rev b1)
00:03.0 0604: 8086:25e3 (rev b1)
00:04.0 0604: 8086:25f8 (rev b1)
00:05.0 0604: 8086:25e5 (rev b1)
00:06.0 0604: 8086:25f9 (rev b1)
00:07.0 0604: 8086:25e7 (rev b1)
00:08.0 0880: 8086:1a38 (rev b1)
00:10.0 0600: 8086:25f0 (rev b1)
00:10.1 0600: 8086:25f0 (rev b1)
00:10.2 0600: 8086:25f0 (rev b1)
00:11.0 0600: 8086:25f1 (rev b1)
00:13.0 0600: 8086:25f3 (rev b1)
00:15.0 0600: 8086:25f5 (rev b1)
00:16.0 0600: 8086:25f6 (rev b1)
00:1c.0 0604: 8086:2690 (rev 09)
00:1c.1 0604: 8086:2692 (rev 09)
00:1d.0 0c03: 8086:2688 (rev 09)
00:1d.1 0c03: 8086:2689 (rev 09)
00:1d.2 0c03: 8086:268a (rev 09)
00:1d.3 0c03: 8086:268b (rev 09)
00:1d.7 0c03: 8086:268c (rev 09)
00:1e.0 0604: 8086:244e (rev d9)
00:1f.0 0601: 8086:2670 (rev 09)
00:1f.2 0101: 8086:2680 (rev 09)
00:1f.3 0c05: 8086:269b (rev 09)
01:06.0 0300: 1002:515e (rev 02)
02:00.0 0604: 1166:0103 (rev c3)
03:00.0 0200: 14e4:164c (rev 12)
04:00.0 0104: 9005:0286 (rev 02)
05:00.0 0604: 1166:0103 (rev c3)
06:00.0 0200: 14e4:164c (rev 12)
1a:00.0 0604: 8086:3500 (rev 01)
1a:00.3 0604: 8086:350c (rev 01)
1b:00.0 0604: 8086:3510 (rev 01)
1b:01.0 0604: 8086:3514 (rev 01)
1c:00.0 0c04: 1077:2432 (rev 03)
24:00.0 0c04: 1077:2432 (rev 03)

Any help is appreciated. Getting AACRAID working is a must priority since I can not use any of my drives without it.

Thanks!

[Keruskerfuerst]

1. Detailed hardware info.
2. Which version of Systemrescuecd?
3. Which Stage3 tarball are you using?
4. Which version of the ATI driver?
5. It might be better to use the non hardend version

[abduct]

1. Detailed hardware info.

• Ram: 64GB DDR2 ECC
  CPU: 2x Intel xeon E5405 quadcore 64bit 2ghz
  Hardware Raid: ServeRAID-8k
  Hard Drives: 1x 160gb SSD (internal boot drive labeled as "master" in bios), 2x 2TB Toshiba Raid0, 2x Misc 160GB drives Volume raid in hotswap bays.
  Fiber Cards: 2x QLogic QLE2460
  Server: IBM x3650 7979 M1
  Misc: Dual gigabit NICs and some other stuff that are currently working which can be found in the first post.

Anything else you need just ask.

2. Which version of Systemrescuecd?

SystemRescueCd-x86-4.6.1 or SystemRescueCd-x86-4.5.2, I believe it is the newer version I have burnt as I have recently re-downloaded a copy.

3. Which Stage3 tarball are you using?

stage3-amd64-hardened-20151203.tar.bz2

4. Which version of the ATI driver?

I am not sure which version systemrescuecd loads, but I used this link to attempt to enable it: https://wiki.gentoo.org/wiki/Radeon#Kernel (also enabling fb support). I used the R100_cp.bin file which systemrescuecd supposedly loaded (as stated in dmesg while booting the cd) when trying to get the driver to work.

5. It might be better to use the non hardend version

I haven't had much troubles with it on any of my other desktops/laptops/servers, although none of them use hardware raid.

[abduct]

https://bbs.archlinux.org/viewtopic.php?id=124288
https://www.linuxquestions.org/question ... -a-935355/

These two articles have very similar or almost exactly the same symptoms that I have.

How do I set the pci=nocrs option with grub2? Do I add it to the GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub? Do I have to separate the entries via a comma or a space?

My current config option looks like so:

Code: Select all

# Append parameters to the linux kernel command line for non-recovery entries
GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=0"

Also do I only need to compile the AACRAID driver or is there also other modules needed?

I have tried setting this option before but it did not work although it did not work at the time (either I did not edit the default grub correctly when I rebuilt the config, or I didn't compile all the needed drivers/modules).

Thanks for any help.

edit:: Also when looking at the grub.cfg grub2 generates there is double spaces on this line:

Code: Select all

linux   /kernel-4.1.7-r1 root=/dev/sda3 ro  net.ifnames=0

Where it appended my GRUB_CMDLINE_LINUX_DEFAULT option. Is this correct for the config or was this a generation error via the application?

[krinn]

The first problem is AACRAID (which system rescue cd loads fine) causes the 4.1.7-hardened-r1 kernel to panic during boot, and without it I cannot see my two raid volumes. One kernel panic says something along the lines of "VFS: unable to mount root fs on unknown block(...)" and another kernel panic that doesn't state much information besides some addresses such as 0xffffffffffffffff. When removing the driver from the kernel all kernel panics disapeer and the system boots fine, but gentoo does not see any of the drives.

When you don't have aacraid, you're are able to boot : so your boot disk isn't using the raid arrays
When you do have aacraid, you're unable to boot with unable to mount rootfs, why? because the way you add the driver makes linux detect the drives prior your other controller and the drives naming change, and what was sda, is now sdc (sda and sdb taken by raid volume).
I have done a thread to explains drives naming, and glad for you, i use aacraid in my examples, have a look : https://forums.gentoo.org/viewtopic-t-1007788.html

[abduct]

Thanks Krinn for the forum post. You are correct in assuming that my boot drive is independent of the raid array so this does make sense up to the part in which I get the 0xFFFFFFFFFFFFFFFF error (not sure if it is related or not).

When configuring grub2 may I use the /dev/disk/by-id names for the drives and partitions so that it uses a fixed drive name or is there another way to do that?

Tomorrow I am going to attempt to recompile my kernel with AACRAID enabled again and to try to insert PCI=NOCRS to see if I can get it working again.

[krinn]

For your case, it's pretty easy:
- build your controller driver as included in kernel (in order to boot, for most people controller are made by intel)
- build your aacraid as a module.

so intel load first, take sda... and aacraid loaded last take remain drives name for its arrays.

[NeddySeagoon]

abduct,

Use root=PARTUUID= ....

Code: Select all

$ /sbin/blkid
/dev/sda1: UUID="69792596-0a01-4fee-8844-75b202cb5293" TYPE="ext2" PARTLABEL="Boot" PARTUUID="3b122731-6db2-4b49-b5db-b3cb3b7ef308"

That works without an initrd. I'm not sure if the quotes are required.

With an initrd to provide the mount command root=LABEL=... and root=UUID= ... work too.
LABEL in this case is the filesystem label, not the PARTLABEL above.

[abduct]

Thanks NeddySeagoon.

I have managed to get past both kernel panics. The first one regarding the unknown blocks was solved via using

Code: Select all

root=PARTUUID="..."

inside the grub config (replacing all instances of /dev/sda3). Likewise I had to update /etc/fstab to also use these PARTUUIDs in order for it to mount swap and root.

For the AACRAID kernel panic, appending pci=nocrs to the grub boot flags it passes to the kernel seemed to have solved the issue. When I first tried I tried to add a space in between grub options, but it appears you have to use a comma to separate them.

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=0,pci=nocrs"

------------------------------

I have also managed (I believe) to get my QLogic fiber pci cards working, as they are not complaining about firmware any more, although I am not sure how I would verify that they work as I don't own any hardware that uses fiber.

I managed to fix this by downloading the firmware blob to /lib/firmware on the boot drive, and then enabled/set

Code: Select all

Device Drivers  --->
    Generic Driver Options  --->
        -*- Userspace firmware loading support
        [*] Include in-kernel firmware blobs in kernel binary
            (ql2400_fw.bin) External firmware blobs to build into kernel binary
            (/lib/firmware) Firmware blobs root directory

------------------------------

I am going to try a bit now on getting the RADEON DRM open source driver working as per https://wiki.gentoo.org/wiki/Radeon to attempt to get the

Code: Select all

01:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
        Subsystem: IBM ES1000

device working. Last time I tried I ended up getting black screens and the system was halting during boot, but I believe most of this may be due to the AACRAID drivers panicking the kernel in the first place.

------------------------------

New question: Does anyone know how to get grub2-mkconfig to supply the PARTUUID or other UUID schemes automatically, or do I have to hand edit the generated grub.cfg by hand each time I create a new config?

I saw this option:

Code: Select all

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to kernel
#GRUB_DISABLE_LINUX_UUID=true

Although it has always been commented out and not once have I seen it try to supply the UUID for the drive.

It's not a major deal, but it would be nice if there was a way, or a way to specifically set the root= directive within the config.

Thanks!

[NeddySeagoon]

abduct,

Likewise I had to update /etc/fstab to also use these PARTUUIDs in order for it to mount swap and root.

I'm surprised PARTUUID worked in fstab.

fstab cannot used to mount root. Its located on the root filesystem, which can't be read until root is mounted, so it can't be used to find root.
That's why your need a root= on the kernel command line, to break the circular dependency, to find root.
However, using /dev/sda3 for root, in fstab, when root is not /dev/sda3 is a problem. During the startup process, fstab is consulted to determine the root file system type to run the filesystem check. Horrible things happen if that check fails for any reason, as it may if root moves around and fstab does not keep up.

[abduct]

Yea I didn't know either that you could use PARTUUID in fstab either until reading on the arch wiki.

I gave up on getting RADEONFB working with the server. I can get it to load and confirmed it via lspci -k but the monitor is always blank no matter what resolution I have grub2 pass to it.

Although I am fine with using VESAFB since it works at 1024x768x32 which is good enough for emergencies in which I need to connect a monitor to the server.

Thanks for all your help everyone, I believe all my problems have been solved.