Kernel Hangs at Loading

[jyoung]

Hi Folks,

After upgrading from a 4.12.12 kernel, the new kernel (4.13.15, 4.14.8-r1) hangs while grub is loading it. All I get this message:
Loading Linux 4.13.15-gentoo ...

Grub is detecting it, since if I move or delete the kernel grub complains that it can't find it. I've tried numerous kernel options (make oldconfig from the working kernel's config, genkernel, etc.), but the result is the same. This problem is particularly hard to debug since the result is a non-working system. Some web searching turns up folk with similar problems, but usually grub complains with some kind of error message (can't find the kernel, can't load the kernel, etc.). Here's, it's just silent.

Any ideas?

[Hu]

First, check that the kernel is supposed to print output: no command line options silenced it, that it is writing to the correct output device, etc.

Second, try to find the origin of the failure. Set aside your known-good 4.12.12 kernel. Clean your kernel build area and build a new 4.12.12 kernel from the same configuration, to rule out toolchain changes that may have broken something. If the newly built 4.12.12 also works, then we can assume a kernel source code change is your problem. If the old 4.12.12 works and the new one fails, we can assume there is a toolchain problem.

The rest of this post is written on the assumption it is a source change, not a toolchain change. If the toolchain is implicated, stop here and post back with your findings. Otherwise, read on.

Kernel 4.12.x went up to 4.12.14 before being retired. You could test later 4.12.x kernels in the hope that one of them is broken. If so, it will be comparatively easy to find the bad patch since only a few hundred commits are in question. If all 4.12.x work, and no 4.13.x work, then finding the bad commit is much more tedious. In either case, you will use git bisect to test intermediate kernels to find the first that fails to boot. If 4.12.14 is bad, you can probably find it in ~log2(78) steps. If 4.12.14 is good and 4.13 is bad, you may need ~log2(14150) steps.

[NeddySeagoon]

jyoung,

[jyoung]

The clean 4.12.12 doesn't have this issue, and neither does the 4.12.14 kernel. I think this rules out a toolchain problem, and points toward a kernel source change between 4.12 and 4.13. To confirm, I'll attempt to compile an earlier 4.13.

I'm also interested in the idea of booting off an uncompressed kernel. When I simply copy vmlinux to /boot (along with the other kernels), grub-mkconfig doesn't seem to detected it. Should I modify grub.cfg for this experiment (despite the warning)?

[NeddySeagoon]

jyoung,

Read the bottom of grub.cfg.

The warning is because manual edits will be removed when grub.cfg is regenerated.
Don't break any of your your working grub.cfg entries.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[jyoung]

I guess I'd never looked at the bottom of grub.cfg! I've setup a manual config file in custom.cfg; grub now detects the uncompressed kernel, but I'm getting
"error: invalid magic number"

Below is the contents of custom.cfg. I've mostly copied the setup from the menu entries in grub.cfg


menuentry 'uncompressed' {
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 39ec98c1-c234-4ee3-bb13-6d0d5f84b1be
else
search --no-floppy --fs-uuid --set=root 39ec98c1-c234-4ee3-bb13-6d0d5f84b1be
fi
echo "Loading linux..."
linux /boot/vmlinux root=/dev/nvme0n1p4 ro rootfstype=ext4
}

One the other front, the 4.13.5 kernel (which doesn't work) is the lowest 4.13 kernel available.

[jyoung]

I also get the magic number error if I try to boot off the compressed kernel using the custom menu option.

[Jaglover]

Did you run make clean before you ran make? Maybe you should.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

[ipic]

[jyoung]

Just to be sure, I ran make clean in 4.13.5 and rebuilt it. The results were unchanged.

My system is setup with /boot/grub/efi on a separate partition, but /boot/grub on the root partition. So, I'm definitely not running out of space for the new kernels.

[xpxp2002]

I'm also experiencing this issue. Upgrading from 4.12.12 to 4.14.8-r1. Used oldconfig to bring .config current. Booting off of 4.14.8-r1 freezes at the loading line out of Grub. 4.12.12 works just fine.

I think this new 4.14 branch still has some issues.

[xpxp2002]

Been working on this for hours since Monday. Just figured it out...for my system, at least.

Try setting CONFIG_PGTABLE_LEVELS=4 if it is set to 5.

[jyoung]

xpxp2002, CONFIG_PGTABLE_LEVELS=4 in my .config.

It looks like it's set by arch, without a prompt in menuconfig. I tried setting to to 5 manually (just to see what would happen), but 'make' immediately rewrote .config with 4.

[Jaglover]

This may be at least partially a gcc-6 problem. Have you tried with gcc-7.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

[NeddySeagoon]

jyoung,

A word of advice on editing the kernel .config by hand. Don't.

Its very easy to end up with an illegal .config that produces a horribly broken kernel.
Then its difficult to diagnose because nobody has seen anything like it before.
The problem stems from a single menuconfig entry flipping lots of .config file flags.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[jyoung]

Yeah, I would never try this on a kernel that I actually needed ... I'm still running off 4.12.12 while I work on 4.14.8-r1.

I'm using gcc-7.2.0; in fact, that's the only one I'm seeing in gcc-config -l.

[xpxp2002]

[NeddySeagoon]

jyoung,

It looks like 4.14 is more trouble than its worth
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[jyoung]

I'm also on amd64.

This problem also occurred with 4.13.5, but I see that that's already been removed.

Shall we mark this thread as solved? It's not really solved ... I'm certainly willing to try again and report back once a 4.15 kernel is released.

[Hu]

Why wait? 4.15 is already up to -rc5. Linus usually releases around -rc7 or -rc8, depending on how he feels about overall quality. You might not want to stay on a -rcN kernel for daily work, but for a quick test, it's probably safe.

[NeddySeagoon]

Hu,

I've been using it since 4.15.0-rc1 as it has a new amdgpu driver.
I'm on rc-4 now and I can say that it seems to work for me.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[jyoung]

I've justed tried 4.15_rc5, and the result is the same. The boot sequence stuck at "Loading Linux " bit.

You folks are getting your kernels from git-sources, yes?

[NeddySeagoon]

jyoung,

I fetched the kernel from kernel.org but git-sources is the Gentoo way to do the same thing.

Lets start from the very beginning. Post your

[jyoung]

Okay, here's a link to my .config

https://pastebin.com/4d4T1pms

This is a fresh .config, generated by running make menuconfig and then exiting without making any changes. I've also attempted using make oldconfig off the 4.12.12 kernel.

My drive is NVME, so it uses EFI. Also, instead of the partitions being name /dev/sda#, they're /dev/nvme0n1p#

/dev/nvme0n1p1 boot bios partition
/dev/nvme0n1p2 boot partition, mounted at /boot/grub/efi
/dev/nvme0n1p3 swap
/dev/nvme0n1p4 root partition
/dev/nvme0n1p5 home partition

Curiously, running df tells me that the boot and home partitions are mounted as expected, but the root path (/) corresponds to /dev/root instead of /dev/nvmen1p4.

On significant issue is that I had to enable the NVME items in the kernel in order to properly load the drive. I doubt that is the problem here for two reasons: 1) I've also tried enabling them in 4.14, and the problem remained, and make oldconfig from 4.12.12 would have enabled them too, and 2) before I enabled the proper NVME drivers in 4.12.12, the kernel loaded itself and started the boot process, failing at a later step, while here it's not even loading. All that said, I'd be happy to mess around with the NVME stuff some more if you folks think it's likely.

Running

[NeddySeagoon]

jyoung,

Here's how booting works. It solves the problem of loading an operating system from the block device without being able to read the filesystem on the block device.
This is for BIOS, not EUFI, but the problems are the same EUFI can read exactly one filesystem - vfat, So everything needed to get started has to be there.

The BIOS can read exactly one disk block. That's LBA 0. When it starts, it does all the POST checks, sets up the hardware, loads LBA 0 into RAM and jumps to its start address.
LBA 0 contains at most 446 bytes of code. All it can do is make BIOS calls to load some more disk blocks into RAM ... and jump to the start address.
So we have a chain of loaders, each more capable than the last. Eventually, grub gets loaded, by reading the filesystem its on and shows you a menu.
When you make your choice, Grub shows the message about loading the kernel and if you have an initrd, about loading the intrd.
You don't report the initrd message - so lets assume you don't have one.

Grub exits by jumping to the kernel start address. The kernel binary is all alone at this time, it can't load any modules until root is mounted as they are in /lib/modules

The kernel decompresses itself and as it starts, it puts a message on the display that you don't see. Its a good idea to have all the modules used in lspci configured as <*> in the kernel.