(Solved)Kernel Panic-not syncing:VFS:Unable to mount root fs

[ultimeight]

Hey folks, hope everyone's doing great. As the title of the post suggests, I am in a pickle right now. This is my first "Kernel Panic". I was not able to put the entire error message in the title of the post due to the limit but here's the whole thing:

---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Now I am trying to resolve this issue by using a Linux Mint live environment. Read this and after mounting the root partition and checking out the kernel configuration, I see that support for Ext4 is built into the kernel itself. So, I moved to the next thing, which was to check if the block-ids were somehow messed up, I think there's no issue over there either, but as I am a newb, here's grub.cfg, fdisk -l and the output of blkid is as follows:

Code: Select all

/dev/nvme0n1p3: UUID="36cbcca8-5d46-492b-a319-8790b027851f" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5b03a4a8-df5f-6c42-a5ea-6bc03cc3909e"
/dev/nvme0n1p1: UUID="8AFC-79D4" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="7d0b4168-639e-0a4e-ba5b-ca6f2ec12775"
/dev/nvme0n1p2: UUID="38244e92-59dc-4aed-baef-09a28c23fa49" TYPE="swap" PARTUUID="a8821742-0bec-8a4b-bde9-5735c0dc5494"
/dev/sdb1: LABEL="Ventoy" UUID="1712-9171" BLOCK_SIZE="512" TYPE="exfat" PTTYPE="dos" PARTUUID="391ba92f-01"
/dev/loop0: TYPE="squashfs"
/dev/dm-0: BLOCK_SIZE="2048" UUID="2022-12-17-20-57-31-00" LABEL="Linux Mint 21.1 MATE 64-bit" TYPE="iso9660" PTUUID="3f309abc-7fd8-484e-ad0c-f7cac0efb2ed" PTTYPE="gpt"
/dev/sda: UUID="ce6da296-4ea1-458f-8764-11a3f7484991" BLOCK_SIZE="4096" TYPE="ext4"

Please let me know if you need anything else and thank you very much for reading my post.

[pietinger]

unknown-block(0,0) means your kernel is not able to access the disk at all (If you would have a problem with a specific partition you would have numbers instead 0,0)

Your kernel needs static <*> these modules:

- scsi,
- probably ahci, or/and
- probably nvme,
- probably module for gpt disks
- module for your filesystem (you said it is ext4)
- maybe another module for your pci controller

see more here: viewtopic-p-8740900.html#8740900
and here: https://wiki.gentoo.org/wiki/User:Pieti ... Must_Haves

OR ... give us your kernel .config; together with the output of "lspci -k"

[pietinger]

Sorry ... you have given us your kernel config already ...

If you want boot from a NVMe you must change this:

Code: Select all

CONFIG_NVME_CORE=m

If you see module "vmd" in your "lspci -k" you must (also) change this:

Code: Select all

CONFIG_VMD=m

... to static == <*>

[ultimeight]

Morning everyone, and thank you very much for your patience.

Sorry ... you have given us your kernel config already ...

No need to apologize, no worries taken


After searching for the string "NVME_CORE" I get the following output in menuconfig:

Code: Select all

 Symbol: NVME_CORE [=y]                                                                                                                                                                │  
  │ Type  : tristate                                                                                                                                                                      │  
  │ Defined at drivers/nvme/host/Kconfig:2                                                                                                                                                │  
  │ Selects: BLK_DEV_INTEGRITY_T10 [=y]                                                                                                                                                   │  
  │ Selected by [y]:                                                                                                                                                                      │  
  │   - BLK_DEV_NVME [=y] && PCI [=y] && BLOCK [=y]                                                                                                                                       │  
  │ Selected by [m]:                                                                                                                                                                      │  
  │   - NVME_FABRICS [=m]                                                                                                                                                                 │  
  │ Selected by [n]:                                                                                                                                                                      │  
  │   - NVME_APPLE [=n] && OF [=n] && BLOCK [=y] && APPLE_RTKIT [=n] && APPLE_SART [=n] && (ARCH_APPLE || COMPILE_TEST [=n]) 

Please note, this is after I changed Device Drivers --> NVMe Support --> NVM Express block device from M to *, I couldn't find the exact location of CONFIG_NVME_CORE, but after enabling NVM Express block device, I can see that NVME_CORE has changed from m to y. How to compile the kernel after this? Just run make && make modules_install followed by make install? I am using a distribution kernel and not a custom one.

Output of lspci -k when chrooted into the gentoo partition:

Code: Select all

pcilib: Cannot open /proc/bus/pci
lspci: Option ecam.addrs was not specified and ACPI MCFG table cannot be found.

Output of lspci -k when not chrooted (no "vmd")in the gentoo partition: https://0x0.st/H1eV.txt

Output of make && make modules_install:

Code: Select all

make[1]: *** No rule to make target 'arch/x86/entry/syscalls/syscall_32.tbl', needed by 'arch/x86/include/generated/uapi/asm/unistd_32.h'.  Stop.
make: *** [arch/x86/Makefile:242: archheaders] Error 2

Output of make install:

Code: Select all

  INSTALL /boot
run-parts: executing /etc/kernel/postinst.d/91-grub-mkconfig.install 6.1.31-gentoo-dist /boot/vmlinuz-6.1.31-gentoo-dist
 * Generating new GRUB config as /boot/grub/grub.cfg
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
 * grub-mkconfig failed

PS: Sorry for the garbled up output of menuconfig, it looked fine in the preview, let me know if it is not clear.

[NeddySeagoon]

ultimeight,

The distribution kernel has to work almost everywhere, so its provided with a matching initramfs, which will take care of things being set to =m that are needed to mount root.
You are not using that initramfs. If you were, you would get a rescue shell instead of a kernel panic when something went wrong.

An initramfs is a temporary root filesystem in a file. It contains all the things needed to mount root and a rescue shell for when things go wrong.

Cannot open /proc/bus/pci

usually means that /proc is not mounted inside the chroot.
That makes me ask about /dev and /sys too.

Hint ...

/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).

It sounds like you have all the bits but gore some reason your initrd/initramfs is not being used.
What does

Code: Select all

ls -l /boot 

show for your Gentoo /boot?

[ultimeight]

Hi NeddySeagoon,

The contents of my boot partition for Gentoo are as follows:

Code: Select all

total 245381
drwxr-xr-x 3 root root      512 May  3 13:59 EFI
-rwxr-xr-x 1 root root  5830100 May  3 12:04 System.map-6.1.24-gentoo-dist
-rwxr-xr-x 1 root root  5831436 May 24 15:25 System.map-6.1.27-gentoo-dist
-rwxr-xr-x 1 root root  5834426 Jul  4 19:47 System.map-6.1.31-gentoo-dist
-rwxr-xr-x 1 root root  5834426 Jun 20 10:29 System.map-6.1.31-gentoo-dist.old
-rwxr-xr-x 1 root root   249343 May  3 12:04 config-6.1.24-gentoo-dist
-rwxr-xr-x 1 root root   249312 May 24 15:25 config-6.1.27-gentoo-dist
-rwxr-xr-x 1 root root   249358 Jul  4 19:47 config-6.1.31-gentoo-dist
-rwxr-xr-x 1 root root   249361 Jun 20 10:29 config-6.1.31-gentoo-dist.old
drwxr-xr-x 6 root root     1024 Jul  4 16:23 grub
-rwxr-xr-x 1 root root 29364098 May  3 12:04 initramfs-6.1.24-gentoo-dist.img
-rwxr-xr-x 1 root root 48641237 May 24 15:25 initramfs-6.1.27-gentoo-dist.img
-rwxr-xr-x 1 root root 14532608 Jul  4 19:47 initramfs-6.1.31-gentoo-dist.img
-rwxr-xr-x 1 root root 71234323 Jun 20 10:29 initramfs-6.1.31-gentoo-dist.img.old
-rwxr-xr-x 1 root root 14156800 Jun 20 10:42 intel-uc.img
-rwxr-xr-x 1 root root 12248192 May  3 12:04 vmlinuz-6.1.24-gentoo-dist
-rwxr-xr-x 1 root root 12249984 May 24 15:25 vmlinuz-6.1.27-gentoo-dist
-rwxr-xr-x 1 root root 12255040 Jul  4 19:47 vmlinuz-6.1.31-gentoo-dist
-rwxr-xr-x 1 root root 12255040 Jun 20 10:29 vmlinuz-6.1.31-gentoo-dist.old

[pietinger]

I am using a distribution kernel and not a custom one.

Oh ... I thought you did a manual kernel configuration. Now we have a different situation:

As @Neddy already said you should never have the problem of a kernel panic with the distribution kernel, because in this kernel (almost) everything is integrated as module ... BUT ... these modules must be loaded by the initramfs (before kernel tries to mount root partition). FOR THIS a bootloader (e.g. grub) MUST tell the kernel where to find this initramfs ... and here comes grub and/or UEFI ...

It would be very helpful if you tell us what you did before. For example: If you have tried to change booting from grub to UEFI boot then you surely miss the kernel command line parameter (pointing to initramfs) in your UEFI entry; see more here: https://wiki.gentoo.org/wiki/Handbook:A ... efibootmgr

If you have changed something (bad) in grub then you surely miss now this parameter. Maybe you will need only a new grub-mkconfig ?

[ultimeight]

I did a system update just before facing this issue, I should have mentioned that earlier. Ran the following commands in the same order;


doas emaint sync -a

doas emerge -avuDN @world

doas emerge --depclean

doas eclean-dist -d


Haven't made any changes to grub.

[pietinger]

How do you boot ? With grub or UEFI (or another bootmanager/bootloader) ?
If grub, have you done a" grub-mkconfig -o /boot/grub/grub.cfg"
(-> https://wiki.gentoo.org/wiki/GRUB#Insta ... new_kernel )

[ultimeight]

I use grub. And to answer your last question, no, I haven't done any changes neither have I ran the mkconfig command.

[pietinger]

[...] neither have I ran the mkconfig command.

Please just do it (pls dont forget to mount /boot before).

[ultimeight]

Can you specify the mountpoints for / and boot partitions please?

[pietinger]

Can you specify the mountpoints for / and boot partitions please?

Boot with our Gentoo minimal CD (or Live) and do the same steps as described in our AMD64 handbook:

https://wiki.gentoo.org/wiki/Handbook:A ... _partition
https://wiki.gentoo.org/wiki/Handbook:A ... ilesystems

For example (if you dont use systemd):

Code: Select all

- boot with amd64-minimal CD
# clear
# mount /dev/sdXY /mnt/gentoo
# cd /mnt/gentoo
# mount -t proc /proc /mnt/gentoo/proc
# mount --rbind /sys /mnt/gentoo/sys
# mount --rbind /dev /mnt/gentoo/dev
# mount --bind /run /mnt/gentoo/run
# chroot /mnt/gentoo /bin/bash
# . /etc/profile
# mount /dev/sdXZ /boot
# grub-mkconfig -o /boot/grub/grub.cfg

[ultimeight]

After running grub-mkconfig -o /boot/grub/grub.cfg

Code: Select all

Generating grub configuration file ...
cat: write error: No space left on device

[pietinger]

Maybe the reason ...

Why not deleting some old kernels you dont need anymore ? (rm /boot/*6.1.24*)


P.S.: ... and try your mkconfig again please ...

[ultimeight]

I was able to run grub-mkconfig successfully after removing old kernels, then I restarted the system, however, I am still being welcomed by the same kernel panic message as before.

[pietinger]

Maybe your initramfs is broken (due to your missing space) ... look to the file size:

Code: Select all

-rwxr-xr-x 1 root root 14532608 Jul  4 19:47 initramfs-6.1.31-gentoo-dist.img
-rwxr-xr-x 1 root root 71234323 Jun 20 10:29 initramfs-6.1.31-gentoo-dist.img.old

Try to get enough available space and re-install (=emerge) the dist-kernel (and do a mkconfig afterwards; this is always recommended with new kernels).

[ultimeight]

The issue has been resolved
Re-emerging the dist-kernel did the job.
Followed these steps:

Code: Select all

- boot with amd64-minimal CD
# clear
# mount /dev/sdXY /mnt/gentoo
# cd /mnt/gentoo
# mount -t proc /proc /mnt/gentoo/proc
# mount --rbind /sys /mnt/gentoo/sys
# mount --rbind /dev /mnt/gentoo/dev
# mount --bind /run /mnt/gentoo/run
# chroot /mnt/gentoo /bin/bash
# . /etc/profile
# mount /dev/sdXZ /boot
# emerge -a sys-kernel/gentoo-kernel
# emerge --prune sys-kernel/gentoo-kernel
# emerge --depclean
# grub-mkconfig -o /boot/grub/grub.cfg
- reboot

Don't forget to connect to the internet after the initial boot up of amd64-minimal CD, I used wpa_supplicant to set up mine.
However, I still wonder what led to this issue in the first place. Thank you pietinger and NeddySeagoon for all the invaluable suggestions and help.

[pietinger]

However, I still wonder what led to this issue in the first place.

I think the initial reason was there was not enough space left on your /boot and the emerge of a new kernel could not make a correct initramfs (maybe look now to the filesize of your new initramfs).

Thank you pietinger and NeddySeagoon for all the invaluable suggestions and help.

You are very Welcome !

Have fun with Gentoo !

[ultimeight]

Should I periodically cleanup /boot? Is that something that I should do?

[pietinger]

Should I periodically cleanup /boot? Is that something that I should do?

YES

(it is something you *must* do )

[ultimeight]

All right, once again thanks a lot pietinger. Closing this thread now.