grub: kernel not found after system, kernel update [Solved]

Message

yarug · Post by **yarug** » Mon Dec 23, 2019 5:10 pm

I have a home router configured and had to replace a hard disk (RAID 1) which prompted me to make sure the system was up to date.

First off after replacing the hard disk (this was before updating) Grub couldn't find the root file system so I had to manually enter /dev/sda4 at the Grub prompt that was shown with the error after which booting continued.

I then linked the system to the latest profile (17.1, also following the profile update instruction), did a world update, and created a new genkernel.

It was after upgrading genkernel (to 4.0.1) and creating a new (gen)kernel/initramfs and subsequently updating the Grub configuration file with grub-mkconfig that I cannot get the system to boot anymore.

Grub loads showing the new kernel entry, but it immediately fails with:

Loading Linux 4.19.86-gentoo-x86_64
error: file '/vmlinuz-4.19.86-gentoo-x86_64' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

I have tried to pinpoint the problem, but cannot figure this out.

- grub-mkconfig worked and installed the kernel and initramfs in /boot
- my partition layout is very simple and didn't change, /dev/sda4 for root and /dev/sda2 for boot
- I tried regenerating grub.cfg with GRUB_DISABLE_LINUX_UUID=true (as I suspect UUIDs have something to do with it)
- changing root= kernel parameter to use /dev/sda4 instead of UUID does not help (this did work as a workaround after the hard disk swap and before the upgrade)

Does anyone know how I can fix this?

Thanks in advance.

yarug · Post by **yarug** » Mon Dec 23, 2019 8:30 pm

This is the Grub entry:

Code: Select all

    load_video
	if [ "x$grub_platform" = xefi ]; then
		set gfxpayload=keep
	fi
	insmod gzio
	insmod part_gpt
	insmod ext2
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  c1f44c7c-ad69-49dc-9959-cfc16eb0e295
	else
	  search --no-floppy --fs-uuid --set=root c1f44c7c-ad69-49dc-9959-cfc16eb0e295
	fi
	echo	'Loading Linux 4.19.86-gentoo-x86_64 ...'
	linux	 /vmlinuz-4.19.86-gentoo-x86_64 root=UUID=363a15fb-07b4-4f5a-b8b9-20e422e4e0fc ro
    echo	'Loading initial ramdisk ...'
	initrd	  /initramfs-4.19.86-gentoo-x86_64.img

GDH-gentoo · Post by **GDH-gentoo** » Mon Dec 23, 2019 9:58 pm

yarug wrote:This is the Grub entry:

Code: Select all

[...]
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  c1f44c7c-ad69-49dc-9959-cfc16eb0e295
	else
	  search --no-floppy --fs-uuid --set=root c1f44c7c-ad69-49dc-9959-cfc16eb0e295
[...]

Is c1f44c7c-ad69-49dc-9959-cfc16eb0e295 the UUID of /dev/sda2? Does ls /boot show that file vmlinuz-4.19.86-gentoo-x86_64 is present?

yarug · Post by **yarug** » Mon Dec 23, 2019 10:18 pm

GDH-gentoo wrote:Is c1f44c7c-ad69-49dc-9959-cfc16eb0e295 the UUID of /dev/sda2? Does ls /boot show that file vmlinuz-4.19.86-gentoo-x86_64 is present?

Thanks for the reply GDH-gentoo.

Yes, c1f44c7c-ad69-49dc-9959-cfc16eb0e295 corresponds to /dev/sda2 (my boot partition) and /boot contains the files vmlinuz-4.19.86-gentoo-x86_64 and initramfs-4.19.86-gentoo-x86_64.img.

GDH-gentoo · Post by **GDH-gentoo** » Tue Dec 24, 2019 1:03 am

yarug wrote:Yes, c1f44c7c-ad69-49dc-9959-cfc16eb0e295 corresponds to /dev/sda2 (my boot partition) and /boot contains the files vmlinuz-4.19.86-gentoo-x86_64 and initramfs-4.19.86-gentoo-x86_64.img.

Hm, it should just work then. Reboot the computer, and when GRUB's menu is showing, instead of selecting an option, press 'c' to get to the grub> prompt. Then use the ls -lh (hd0,gpt2)/ command to ask GRUB to display the contents of what should be /dev/sda2. Do you see those files in the output?

You can go back to the menu by pressing ESC after the test.

yarug · Post by **yarug** » Tue Dec 24, 2019 9:06 am

Oh, this is weird.

ls -lh (hd0,gpt2) from the Grub command prompt lists my old kernels. These are from before the system/kernel update.

Is my new hard drive in the RAID 1 array somehow not syncing??

So Grub sees one hard drive of the array, but when I boot with a live/rescue USB it sees the other?

Is this why "# blkid" when booted with the live/rescue USB outputs both /dev/sda1-4 as /dev/sdb1-4 (but for example /dev/sda4 and /dev/sdb4 have the same UUID and the same PARTUUID)?

ps. apologies for the late response, I am on a west Europe time zone.

yarug · Post by **yarug** » Tue Dec 24, 2019 11:28 am

To clarify:

I deleted the files listed in (hd0,gpt2)/ earlier and you don't see those in a chrooted live/rescue session (/boot mounted on /dev/sda2).

The new kernel (4.19.86) is not there, but it is there in a chrooted live/rescue session (while the old kernels are not).

In both cases (Grub's screen and chrooted live/rescue session) it seems to pick up the latest /boot/grub/grub.cfg (with 4.19.86 kernel).

DONAHUE · Post by **DONAHUE** » Tue Dec 24, 2019 2:03 pm

possible that a separate boot or EFI system partition was or was not mounted at the time the new kerenl was compiled and installed or when grub -mkconfig was run?? usually ls -al run with separate /boot not mounted should show empty except . and .. and show kernels and asystem maps and configs and grub directory when boot partition is mounted. umount the boot partition then mount it somewhere else like /mnt or /mnt/boot and then mv all files in the boot directory to the /mnt location where the boot partition is mounted then umount from /mnt abd mount boot partition to the empty boot directory/mountpoint ls /boot check all is well, rerun grub-mkinstall and grub-mkconfig -o /boot/grub

yarug · Post by **yarug** » Tue Dec 24, 2019 10:49 pm

Hi DONAHUE. Thanks for your support. See my answers below.

possible that a separate boot or EFI system partition was or was not mounted at the time the new kerenl was compiled and installed or when grub -mkconfig was run??

Not that I can think of.

usually ls -al run with separate /boot not mounted should show empty except . and .. and show kernels and asystem maps and configs and grub directory when boot partition is mounted

Correct. When I enter the chrooted environment before mounting /dev/sda2 to /boot, /boot is empty. After mounting /boot it lists the kernel files:

Code: Select all

(chroot) 
00:39 0 livecd / # ls -al /boot
total 8
drwxr-xr-x  2 root root 4096 Oct 16  2016 .
drwxr-xr-x 21 root root 4096 Dec 23 16:36 ..
-rw-r--r--  1 root root    0 Oct 16  2016 .keep
(chroot) 
00:39 0 livecd / # mount /dev/sda2 /boot
(chroot) 
00:40 0 livecd / # ls -al /boot
total 17594
drwxr-xr-x  4 root root    1024 Dec 23 20:20 .
drwxr-xr-x 21 root root    4096 Dec 23 16:36 ..
-rw-r--r--  1 root root       0 Oct 16  2016 .keep
-rw-r--r--  1 root root 3309939 Dec 23 20:00 System.map-4.19.86-gentoo-x86_64
drwxr-xr-x  6 root root    1024 Dec 23 20:21 grub
-rw-r--r--  1 root root 8070776 Dec 23 20:17 initramfs-4.19.86-gentoo-x86_64.img
drwx------  2 root root   12288 Sep 12  2013 lost+found
-rw-r--r--  1 root root 6541984 Dec 23 20:00 vmlinuz-4.19.86-gentoo-x86_64

umount the boot partition then mount it somewhere else like /mnt or /mnt/boot and then mv all files in the boot directory to the /mnt location where the boot partition is mounted then umount from /mnt abd mount boot partition to the empty boot directory/mountpoint ls /boot check all is well

I'm not sure I follow. There are no files in the /boot directory if /boot is not mounted (/dev/sda2).

yarug · Post by **yarug** » Wed Dec 25, 2019 10:28 am

Some further investigation. I started up again with a gentoo live USB.

If I mount /dev/sda2 to /mnt/gentoo I see my new kernel (4.19.86)

Code: Select all

livecd ~ # mount /dev/sda2 /mnt/gentoo
livecd ~ # ls -al /mnt/gentoo/
total 12946
drwxr-xr-x 4 root root    1024 Dec 25 00:52 .
drwxr-xr-x 6 root root     120 Dec 24 17:22 ..
-rw-r--r-- 1 root root       0 Oct 16  2016 .keep
-rw-r--r-- 1 root root 2961468 Dec 25 00:38 System.map-4.19.86-gentoo-x86_64
drwxr-xr-x 6 root root    1024 Dec 25 00:52 grub
-rw-r--r-- 1 root root 4954620 Dec 25 00:46 initramfs-4.19.86-gentoo-x86_64.img
drwx------ 2 root root   12288 Sep 12  2013 lost+found
-rw-r--r-- 1 root root 5268016 Dec 25 00:38 vmlinuz-4.19.86-gentoo-x86_64

However, if I mount /dev/sdb2 to /mnt/gentoo I see the list of old kernels (before system/kernel update) and this is probably what Grub sees when I boot up.

Code: Select all

livecd ~ # umount /dev/sda2
livecd ~ # mount /dev/sdb2 /mnt/gentoo
livecd ~ # ls -al /mnt/gentoo/
total 59726
drwxr-xr-x 4 root root    1024 Oct 16  2016 .
drwxr-xr-x 6 root root     120 Dec 24 17:22 ..
-rw-r--r-- 1 root root       0 Oct 16  2016 .keep
-rw-r--r-- 1 root root 2101526 Apr 16  2014 System.map-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 2193027 Jan 24  2015 System.map-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 2192880 Jan 31  2015 System.map-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 2264577 Jan  3  2016 System.map-genkernel-x86_64-4.1.12-gentoo
drwxr-xr-x 4 root root    1024 Jan  3  2016 grub
-rw-r--r-- 1 root root 9234244 Apr 17  2014 initramfs-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 9312112 Jan 24  2015 initramfs-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 9308860 Jan 31  2015 initramfs-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 9426612 Jan  3  2016 initramfs-genkernel-x86_64-4.1.12-gentoo
-rw-r--r-- 1 root root 3600880 Apr 16  2014 kernel-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 3704384 Jan 24  2015 kernel-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 3704224 Jan 31  2015 kernel-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 3840832 Jan  3  2016 kernel-genkernel-x86_64-4.1.12-gentoo
drwx------ 2 root root   12288 Sep 12  2013 lost+found

The weird thing is though that /dev/sda2(/grub.grub.cfg) contains also the corresponding (new) Grub config and that /dev/sdb2(/grub.grub.cfg) contains the old Grub config. But when booting, Grub displays the new config (from /dev/sda2) and the old kernel list (/dev/sdb2)

Also I can mount /dev/sda4, but not /dev/sdb4:

Code: Select all

livecd ~ # mount /dev/sda4 /mnt/gentoo
livecd ~ # umount /dev/sda4
livecd ~ # mount /dev/sdb4 /mnt/gentoo
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/sdb4, missing codepage or helper program, or other error.

This is the output from parted for bot /dev/sda and /dev/sdb:

Code: Select all

livecd ~ # parted -a optimal /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: ATA MB1000GCWCV (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name    Flags
 1      1049kB  34.6MB  33.6MB                  grub    bios_grub
 2      34.6MB  303MB   268MB   ext2            boot    msftdata
 3      303MB   4598MB  4295MB  linux-swap(v1)  swap    msftdata
 4      4598MB  1000GB  996GB   ext4            rootfs  msftdata

(parted) q                                                                
livecd ~ # parted -a optimal /dev/sdb
GNU Parted 3.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK                                                             
Model: ATA MB1000GCWCV (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name    Flags
 1      1049kB  34.6MB  33.6MB                  grub    bios_grub
 2      34.6MB  303MB   268MB   ext2            boot    msftdata
 3      303MB   4598MB  4295MB  linux-swap(v1)  swap    msftdata
 4      4598MB  1000GB  996GB   ext4            rootfs  msftdata

(parted) q

And the output from blkid:

Code: Select all

livecd ~ # blkid
/dev/loop0: TYPE="squashfs"
/dev/sda1: PARTLABEL="grub" PARTUUID="f07b1822-c457-44f1-bcfb-9a75a15a4648"
/dev/sda2: UUID="c1f44c7c-ad69-49dc-9959-cfc16eb0e295" TYPE="ext2" PARTLABEL="boot" PARTUUID="fb2eb17c-f0a6-4799-b1e0-c856cc4caec9"
/dev/sda3: UUID="b68d0122-6c9e-44de-8ae3-3e0f38a9fb10" TYPE="swap" PARTLABEL="swap" PARTUUID="44fd6afc-6c84-4ee5-95c8-692f3c6b8423"
/dev/sda4: UUID="363a15fb-07b4-4f5a-b8b9-20e422e4e0fc" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="02b72bbd-cec7-4f99-b8c0-245a2bdabe8a"
/dev/sdb1: PARTLABEL="grub" PARTUUID="f07b1822-c457-44f1-bcfb-9a75a15a4648"
/dev/sdb2: UUID="c1f44c7c-ad69-49dc-9959-cfc16eb0e295" TYPE="ext2" PARTLABEL="boot" PARTUUID="fb2eb17c-f0a6-4799-b1e0-c856cc4caec9"
/dev/sdb3: UUID="b68d0122-6c9e-44de-8ae3-3e0f38a9fb10" TYPE="swap" PARTLABEL="swap" PARTUUID="44fd6afc-6c84-4ee5-95c8-692f3c6b8423"
/dev/sdb4: UUID="363a15fb-07b4-4f5a-b8b9-20e422e4e0fc" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="02b72bbd-cec7-4f99-b8c0-245a2bdabe8a"
/dev/sdc1: UUID="2019-12-23-03-48-36-03" LABEL="Gentoo amd64 20191222T214502Z" TYPE="iso9660" PTUUID="5cd52f0e" PTTYPE="dos" PARTUUID="5cd52f0e-01"
/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="GENTOOLIVE" LABEL="GENTOOLIVE" UUID="913D-5710" TYPE="vfat" PARTUUID="5cd52f0e-02"

So, the questions are

a) why do I see both /dev/sda and /dev/sdb? There only should be /dev/sda as per the logical volume definition in my hardware raid setup.
b) why does grub see the new configuration file but the old kernel list (some sort of mix from /dev/sda and /dev/sdb)?

I could just copy everything from /dev/sda2 to /dev/sdb2 and try to reboot but I'm worried that I'm in some corrupted drive state and that my RAID (1) is not functioning properly.

DONAHUE · Post by **DONAHUE** » Wed Dec 25, 2019 2:31 pm

blklid says there is no raid

you wrote:had to replace a hard disk (RAID 1)

apparently array was not rebuilt after the disk replacement
addressing a raid by a name that the kernel assigns to a disc device seems likely to just address the device and not the raid
a decade or two ago ran software raid1 on most of two disks but not on sda1,sda2, legacy grub did not do raid at the time so sda1 sdb1 were for boot. installed to one and copied to the other to maintain redundancy.

Tony0945 · Post by **Tony0945** » Wed Dec 25, 2019 3:18 pm

Sorry I thought this was different thread. Erased my answer.

yarug · Post by **yarug** » Wed Dec 25, 2019 3:49 pm

Thanks DONAHUE. I appreciate you answering on Christmas Day. I'll investigate the B120i controller some more. From what I saw so far was that everything was reported as normal and there was a logical drive (disk name /dev/sda) created consisting of 2 hard drives, status was reported as OK, RAID Level as RAID 1. I might just take out the new drive and boot up with just the 1 original drive, see if everything is in place that way (fingers crossed) and recreate the array.

DONAHUE wrote: blklid says there is no raid
you wrote:had to replace a hard disk (RAID 1)

apparently array was not rebuilt after the disk replacement

Is it normal blkid shows the same UUID/PARTUUID for the 2 drives (in my case)? I would assume if it where 2 seperate drives the IDs would be different?

DONAHUE · Post by **DONAHUE** » Wed Dec 25, 2019 5:55 pm

you wrote:in my hardware raid setup.
and
Is it normal blkid shows the same UUID/PARTUUID for the 2 drives (in my case)? I would assume if it where 2 seperate drives the IDs would be different?

Cloning UUID/PARTUUID is possible but seems to violate the spirit and intent of Universally Unique IDentifier. HP does do strange things ...
Every linux raid I have dealt with (2 or 3 max) has been a different device with different device names from its component disks or partitions.
B120i controller linux stuff here https://access.redhat.com/articles/118133 & https://support.hpe.com/hpsc/doc/public ... -c03871499
B120i controller gentoo google search finds results in forum

yarug · Post by **yarug** » Thu Dec 26, 2019 10:33 am

From some further reading into the B120i controller (including this forum) I understand now that:

a) the hardware is not fully supported outside Red Hat / SUSE (proprietary drivers)
b) seems to be not full/real RAID in any case (see also this post viewtopic-p-7965638.html#7965638)

I have been running this server for years in the understanding that it was (and probably lucky that it was the drive in Bay 2 that failed and not the drive in Bay 1, but I'll find that out soon).

What I will do is remove the new hard drive, boot up with only the older drive, reconfigure the HP controller to disable RAID and configure the system with software raid (mdadm).