Kernel panic - VFS: Unable to mount root fs on unknown-block

[FarkR]

Let me start off by saying that I'm aware of the documentation for this error on the wiki.
But nothing on this page is of any use to me since as far as I can tell, my initramfs is configured correctly.

Please take note of the following output before making a diagnosis

Here is the output given by

Code: Select all

efiboomgr -v

Code: Select all

BootCurrent: 001B
Timeout: 2 seconds
BootOrder: 0002,0007,0006,0005,0019,0000,0010,0011,0012,0013,0017,0018,001A,001B,001C,001D,001E,0023,0001
Boot0000* rEFInd boot manager	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\refind\refind_x64.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...\................
Boot0001* Gentoo	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\Gentoo\vmlinuz-4.14.65-gentoo.efi)i.n.i.t.r.d.=.'.\.i.n.i.t.r.a.m.f.s.-.g.e.n.k.e.r.n.e.l.-.a.m.d.6.4.-.4...1.6...6.5.-.g.e.n.t.o.o...i.m.g.'.
Boot0002* Gentoo Linux 4.14 Dracut	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\Boot\bootx64.efi)initrd=\initramfs-4.14.70-std531-amd64.img
Boot0005* Gentoo Linux 4.14	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\Boot\bootx64.efi)initrd=\initramfs-genkernel-x86_64-4.14.65-gentoo
Boot0006* rEFInd	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\refind\refind_x64.efi)
Boot0007* Windows	HD(2,GPT,1fb5ae89-6746-4252-9ef5-4d69ee8dc23b,0xfa000,0x32000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)
Boot0010  Setup	FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu	FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen	FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics	FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Startup Interrupt Menu	FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0015  Rescue and Recovery	FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0016  MEBx Hot Key	FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0017* USB CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot0018* USB FDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot0019* NVMe0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001A* ATA HDD0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f600)
Boot001B* USB HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot001C* PCI LAN	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot001D  Other CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)
Boot001E  Other HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)
Boot001F* USBR BOOT CDROM	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,1)
Boot0020* USBR BOOT Floppy	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,0)
Boot0021* ATA HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0022* ATAPI CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)
Boot0023* PCI LAN	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)

note how I have two boot entries for Gentoo, one called "Gentoo Linux 4.14" that uses the initramfs generated by

Code: Select all

genkernel --install --no-ramdisk-modules initramfs

and the other, "Gentoo Linux 4.14 Dracut" which uses the initramfs generated by

Code: Select all

dracut --no-kernel --force

both of which are located in the ESP, evidenced by

Code: Select all

/boot/
├── config-4.14.65-gentoo
├── EFI
│   ├── Boot
│   │   └── bootx64.efi
│   └── Gentoo
│       ├── bootx64.efi
│       ├── initramfs-4.14.65-gentoo.img
│       ├── initramfs-genkernel-x86_64-4.14.65-gentoo
│       └── vmlinuz-4.14.65-gentoo.efi
├── initramfs-4.14.70-std531-amd64.img
├── initramfs-genkernel-x86_64-4.14.65-gentoo
├── initramfs-genkernel-x86_64-4.14.65-gentoo.old
├── System.map-4.14.65-gentoo
└── vmlinuz-4.14.65-gentoo

3 directories, 11 files

Boot/bootx64.efi and Gentoo/bootx64.efi are copies of the same file

Now here's the contents of my fstab by which the kernel and initramfs were compiled

Code: Select all

UUID=647D-9C90		/boot		vfat		noauto,noatime	1 2
UUID=d906f773-6de8-4e0d-8202-8a78d1eab7d3	/		ext4		noatime		0 1
/swapfile		none		swap		defaults	0 0

And yes, I verified that the vfat ESP was mounted on /boot/
It might help to have some context; this is not the first time I compiled my kernel, I've been using this Gentoo system for a few months now. This kernel panic occured AFTER I tried moving my EFISTUB onto the ESP. originally, I had my EFISTUB entirely on my root fs in the /boot/ folder. But since the ThinkPad T480s UEFI can't read ext4 I used rEFInd to boot my system. But rEFInd randomly broke, so here I am.
I suspect it's either an issue with how my kernel modules are configured so that my initramfs aren't handling them properly, or my computer's UEFI is causing issues which maybe broke rEFInd in the first place. I'm not an expert on any of this so I don't assume I did everything right.

[eccerr0r]

Would be helpful to know a bit more context around the panic -- did it load the initramfs? Did it mount and use the initramfs?

One thing that looks a tad bit suspicious is that your direct EFIbootmgr settings do not include a root filesystem... but I've not tried installing without grub2, so not sure what to extrapolate from this information.

[FarkR]

Would be helpful to know a bit more context around the panic -- did it load the initramfs? Did it mount and use the initramfs?

how would I check this?

One thing that looks a tad bit suspicious is that your direct EFIbootmgr settings do not include a root filesystem... but I've not tried installing without grub2, so not sure what to extrapolate from this information.

didn't make a difference for me. even with a rootfs= variable I got the same kernel panic screen.

[eccerr0r]

A screenshot would be helpful. Or if it has some dialog during boot, then it could be initramfs that loaded up. In any case if your initramfs didn't load, you should check your kernel options for initramfs handling; if it did load, then you should be able to break into a shell to inspect your devices.

Did you build your drivers into the kernel or as modules?

What happened to your refind?

[FarkR]

A screenshot would be helpful. Or if it has some dialog during boot, then it could be initramfs that loaded up. In any case if your initramfs didn't load, you should check your kernel options for initramfs handling; if it did load, then you should be able to break into a shell to inspect your devices.

Did you build your drivers into the kernel or as modules?

What happened to your refind?

here's a photo of the kernel screen I keep getting: https://i.imgur.com/0C3ERGx.jpg
I'm guessing you mean my fs drivers? The ext and vfat were both enabled as modules.
Also no clue what happened to my rEFInd, just woke up one morning and it suddenly decided to get stuck on the "Initializing..." screen. I've scoured the internet about it but I don't think anyone else in history has had my issue ¯\_(ツ)_/¯

[eccerr0r]

Hmm...looks like your initramfs did not get loaded, or very unlikely that it got loaded (then again, when you select an EFI boot option it should start with loading the kernel and then immediately afterwards it should say that it's loading the initramfs image). Without a root= and the initramfs not helping locate a root, it would explain why it's conking out...

I vaguely recall an issue with initramfs and efi ... ugh... someone remind me what it was. There was some known problem with some EFI implementations and command line argument.

My current suggestion is to either build all your drivers into the kernel, and/or build your initramfs into your kernel (there's a kernel config option to embed the initramfs image.)

As for refind, interesting....sure you don't have hardware problems? a block device flaking out?

(Yes I'm trying hard not to suggest installing grub2...)

[NeddySeagoon]

FarkR,

The interesting part of the image is

Code: Select all

unknown-block(0,0)

Your boot loader (if any) loads the kernel, that bit works or the kernel could not panic but unknown-block(0,0) tells that the kernel cannot see any block devices.
The

Code: Select all

Modules linkeh in: 

list is empty, which suggests that the initrd didn't load, or if it loaded, the kernel didn't find it, or even you don't have initrd support configured in your kernel.

What boot loader are your using?

Code: Select all

genkernel --install --no-ramdisk-modules initramfs

Seems wrong. Your root filesystem driver is configured as a module, so it must be in the intrd or its not possible to mount root.

It looks like your SCSI stack is also configured as a modules from the unknown-block(0,0).

[FarkR]

Hmm...looks like your initramfs did not get loaded, or very unlikely that it got loaded (then again, when you select an EFI boot option it should start with loading the kernel and then immediately afterwards it should say that it's loading the initramfs image). Without a root= and the initramfs not helping locate a root, it would explain why it's conking out...

I vaguely recall an issue with initramfs and efi ... ugh... someone remind me what it was. There was some known problem with some EFI implementations and command line argument.

My current suggestion is to either build all your drivers into the kernel, and/or build your initramfs into your kernel (there's a kernel config option to embed the initramfs image.)

As for refind, interesting....sure you don't have hardware problems? a block device flaking out?

(Yes I'm trying hard not to suggest installing grub2...)

I tried building my drivers into the kernel (via =y in .config) and still got the same empty

Code: Select all

Modules linkeh in:

in my kernel panic. How would I embed my initramfs if I used genkernel to make the initramfs but used make to compile the kernel? I'm aware of the

Code: Select all

--integrated-initramfs

flag for genkernel but I'm not sure if that would work for my setup.

What boot loader are your using?

none. see the directory tree of my ESP in OP. bootx64.efi is just a copy of vmlinuz.

In the meantime, I'm probably going to try setting up grub2 to see what happens.

[NeddySeagoon]

FarkR,

Make the initrd first and put it somewhere the kernel build can pick it up.
Configure the kernel to include the file.

This will end badly if the initrd must contain kernel modules.

[FarkR]

FarkR,

Make the initrd first and put it somewhere the kernel build can pick it up.
Configure the kernel to include the file.

This will end badly if the initrd must contain kernel modules.

Thanks, but does that make it imperative that I use the

Code: Select all

--no-ramdisk-modules

flag? I recompiled my initramfs without it in between posts to see what would happen (got same error)

[FarkR]

Okay, so ironically with grub2 the initramfs does appear to load, but now it can't load the root filesystem: https://i.imgur.com/MWIepTq.jpg (you can also see me screwing around with busybox, peek blkid)
The UUID is definitely right (especially since grub generated the grub.cfg in the first place), and my ext drivers are enabled in .config, so... I'd post proof but not if I don't have to, since I'd have to reboot for it.

As for refind, interesting....sure you don't have hardware problems? a block device flaking out?

I'm beginning think you're right. But still, it doesn't explain why Windows 10 boots just fine.
Judging by the grub2 output I'm guessing there's something I forgot to enable that would help me see my block devices. Is there some obscure Thinkpad t480s driver I don't know about (or even a basic obvious one)?

[NeddySeagoon]

FarkR,

If you use --no-ramdisk-modules, your kernel must be built with everything needed to drive your hardware and filesystems configured as <*>, or built in, as it will not be possible to load any kernel modules until. root is mounted.
You may find

Code: Select all

/usr/src/linux/Documentation/efi-stub.txt

worth a read. It tells how to use an efi stub with an external initrd.

If you build your kernel, then your initrd (including modules) then build the kernel again, make won't actually change anything. It will just include your initrd.
I would not be surprised to find that there is a script somewhere that lets you change the initrd embedded in the kernel.

Your image contains the line

Code: Select all

!! Could not find the root block device in .

which is very telling.
The list just before the final full stop is a list of every block device the kernel can see. In your case, its empty, so the kernel cannot see any block devices at all.
This suggests that your SCSI stack is incomplete, incorrect or set as <M> when you need <*>

I don't see any initramfs or initrd files listed in your image.

Please post the output of

Code: Select all

lspci -knn

so we can see your hardware and put your kenel .config file onto a pastebin site, so we can match your kernel and hardware.
You will find wgetpaste very useful.

[eccerr0r]

Yeah, with grub2 it's now clear that your disk subsystem modules were not included in the initramfs, cannot be loaded by initramfs, or not built into the kernel.

For informational purposes only, if you're interested - it's confusing that the handoffs cause drivers to be invalidated:

EFI can only read the ESP (EFI System Partition) and other FAT-formatted disks.
EFI can only load EFI applications: a kernel with an EFI stub, window's loader, grub2, refind, etc. Some EFI can direct the Linux kernel EFI stub to load initramfs using EFI directives, and EFI boot menu implementations seem to have problems directing the kernel to load initramfs -- this may be the problem you're seeing. Typically it will always work from the EFI shell. Note that Windows does not use an initramfs and hence the menu bug was likely poorly tested by your manufacturer.

GRUB2 and refind have their own capability of reading ext2 partitions, but cannot write to them. They also have a driver to read the ESP and FAT partitions. They can load the initramfs without the kernel's help, using their own drivers.

At this point, once kernel is running with initramfs loaded, it's up to the kernel to do everything else. Grub2/rEFInd and EFI itself cannot further help the kernel find anything else. If the kernel doesn't have the drivers built in, it's up to the instructions and contents of the initramfs to load drivers and find your root disk.

Hope this clarifies what needs to be done at this point to get a booting system (and the value of using grub2/refind instead of just using EFI menu boot directly).