Debugging kernel boot?

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

I don't think I'm getting as far as Open-RC - I get the linux penguins along the top, with [timestamp] message displayed.

Anyway, the kernel appears to be getting hung, without actually going into a kernel panic. Is there a way to debug the actual kernel boot process? I find a lot about debugging Open-RC, little about the kernel's boot...

eccerr0r · Posted: Sun Nov 19, 2017 6:09 am Post subject:

Knowing what were the last things that it printed before it hangs would be useful. A picture would help if it's difficult to type what it wrote.

Yes, it would be helpful to know if it started openRC or did it not even get that far, did it free unused kernel memory (usually one of the last things it does before handing off control to init)?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Sun Nov 19, 2017 9:55 am Post subject:

uberDoward,

Its possible to configure the kernel with no console. Everything still works but there is nothing on the screen.
From memory, you don't even get the tux icons.

If openrc is being started, you may be able to log in via ssh, if thats set up.

Then there is a serial console, if your system has a real serial port and you have a way to connect to it.
There is also console over network.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

bunder · Bodhisattva Joined: 10 Apr 2004 Posts: 5934

Is it possible to get early printk, then have it switch to a non-functional framebuffer? If that's possible, that could in theory give the appearance of a hung boot with penguins.

With no framebuffer, you'd just get a black screen and no penguins.

NeddySeagoon · Posted: Sun Nov 19, 2017 11:55 am Post subject:

bunder,

Thats possible with the text console build in then switching to a broken framebuffer built as a module.
However - no penguins.

Its also possible to have several framebuffers configured the preferred one could be broken.
Being lazy, I have vesafb and amdgpudrmfb.

They both work and the switch is clearly visible. Both during the boot process and in dmesg.

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

Using ASPEED, no modules (going for a very lean monolithic kernel). Framebuffer appears to switch over - now you'e got me wondering if I forgot to put in the console LOL, let me check that.

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

I'll get a picture shortly. TTY is compiled into the kernel, let me check the framebuffers enabled.... are there specific options I need to have enabled?

eccerr0r · Posted: Sun Nov 19, 2017 5:48 pm Post subject:

I'm confused, you did say the penguins showed up as well as the [0.2512512] blahblah timestamped kernel messages?

I wasn't sure if you were asking a general question or trying to fix your specific box... If you did see the penguins and kernel messages, at least there's some clues to figure out what's going on.

On the other hand, black screen boots with no penguins are most annoying to debug.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Sun Nov 19, 2017 6:00 pm Post subject:

uberDoward,

Make friends with wgetpaste.

Put your lspci output in a post together with a link to your grub.cfg, or whatever your boot loader config file is and a link to your kernel .config file.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

NeddySeagoon, that's freaking awesome!

config: https://paste.pound-python.org/show/7eCSAtiIBY3XYcLrhLUY/
grub.cfg: https://paste.pound-python.org/show/P69GrwzUOs0R54dWhAOY/

Picture of boot: https://imagebin.ca/v/3ht5U779oAg3

NeddySeagoon · Posted: Mon Nov 20, 2017 11:22 am Post subject:

uberDoward,

What we know so far ...
Grub loads the kernel and the kernel starts with a framebuffer console but we don't know which one.
UVESA is no longer complete, so Its not that one.

The help text on

eccerr0r · Posted: Mon Nov 20, 2017 5:53 pm Post subject:

Since the min installer CD booted( ? - what boot media did you use? ) using that kernel's .config as a starting point would be helpful. Agreed your lspci info would be useful.

Other key things that are "interesting": It took almost 20 seconds for things to settle down. A lot of these messages may have shown up asynchronously.

Some things I'd try to help debug(but not "fix"): Removing USB drivers (so we don't see all the USB async stuff at the expense of no keyboard/mouse, but at least less things will scroll off). Does shift-pageup scroll back show anything interesting (which won't work if your usb/keyboard isn't compiled, but, hey...)

Does /dev/sde4 sound like the proper root disk?

(Incidentally, I hate the penguins. I never compile that in because it hides 3-4 lines of screen real estate for kernel boot debug :-)

)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Mon Nov 20, 2017 6:12 pm Post subject:

eccerr0r,

There is also framebuffer rotate to get more lines on the screen.

The kernel normally mounts root before USB is initalised, so I think that root is mounted but it might still be read only.

-- edit --

We can try Interactive mode for openrc too.
Edit /etc/rc.conf
Find the line that says

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

lspci -nnk output:
https://paste.pound-python.org/show/OzuhINTSLRkDa5SYIt4z/

Let me recompile without the penguins, lol - I've tried interactive mode, but to no avail.

/dev/sde4 is rootfs, /dev/sde2 is boot. /dev/sde is the OS drive (32GB ssd)

I'll pull out the USB stuff, see if anything else helpful comes up

NeddySeagoon · Posted: Mon Nov 20, 2017 11:55 pm Post subject:

uberDoward,

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

Console grep from LiveCD: https://paste.pound-python.org/show/wiZ4B8lXGzUipTJnRiud/

Very interesting - let me remove the AST module, and see if there's a vesafb I don't have set up in the kernel...

*edit*

Latest kernel .config : https://paste.pound-python.org/show/Vb7OLaGwv2LfxgsAxbtc/

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

Ok, weird. No more high res VESA, but got a kernel panic (VFS: unable to mount root fs on unknown block(8,6

)

Looking into it, noticed that hitting grub's command line, everything was pointing @ hd4,gpt2. ls hd0,gpt2, however was the correct one. Now, why would the BIOS decide to change hd0? No idea, I didn't change anything in there (I have first HDD boot as my /dev/sde 32GB SSD).

So manually editing grub via 'e' to point to (hd0,gpt4) and /dev/sda4 for the kernel command line yields: https://imagebin.ca/v/3i0X9x50yDLx

eccerr0r · Posted: Tue Nov 21, 2017 2:47 am Post subject:

One thing that gets tripped up frequently is that the (hdX,Y) in grub has no relationship to /dev/sdX in Linux. Ideally they map directly but no. Also the (hdX,Y) do nothing for the kernel, it's only for grub to locate boot images (kernel, initramfs).

Anyway, interesting, so if you do have root=/dev/sde4 it behaves differently than if you have root=/dev/sda4? That would imply the kernel did end up switching over to init?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Tue Nov 21, 2017 10:54 am Post subject:

uberDoward,

Grub sees block devices as enumerated by the motherboard firmware.

The kernel sees them in PCI bus order as it scans the PCI bus but its not that simple either.
'Built in' are always enumerated before modules.
It gets worse. The kernel may use several threads for PCI enumeration, so there is a possibility of a race.
The race can change the HDD order from kernel build to kernel build or if you are really unlucky, from boot to boot.

The point is that there is no deliberate correlation between what grub sees and the kernel sees.

Use root=PARTUUID=<your_root_PARTUUID> in place of root=/dev/sd..

blkid will show all the PARTUUIDs. Google for the exact syntax.
It won't matter what device root is on, the kernel will find it.

The same thing will upset /etc/fstab. You can use UUID or PARTUUID there, so that dynamic device renaming is harmless.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

So how do I make the UUID stick to a grub-mkconfig?

Note also, no initramfs here on startup. I've just tried modifying /boot/grub/grub.cfg by hand, though, so let's see what happens, lol

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

So I got it to boot by setting /dev/sdg4 after editing the grub menu ('e' @ grub entry).

Something somewhere is happily screwing up my boot device names. I couldn't even boot with root=PARTUUID=<partuuid> so I'm really at a loss.

eccerr0r · Posted: Wed Nov 22, 2017 6:36 am Post subject:

When you made grub-mkconfig it generated the /dev/sdXX and not PARTUUIDs? Changed /etc/default/grub ?

I suppose if you have a lot of hard drives and they may change around randomly, it might be worth it to make sure PARTUUID works, or use initramfs that supports UUID.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Wed Nov 22, 2017 11:46 am Post subject:

uberDoward,

The kernel understands PARTUUID - its a property of the partition.
To use UUID, which is a property of a filesystem, you must use an initrd that contains the userspace mount binary.

uberDoward · n00b Joined: 09 Jun 2011 Posts: 46

Yeah, it refused to work via UUID.

I'm not using an initramfs - I thought the kernel should boot anyway.

For now, though, it's fixed. I re-did the grub-mkconfig -o /boot/grub/grub.cfg after booting the manually altered grub to get into my Gentoo system, and all is good now.

Very odd behavior - wish I had the time to figure out what went wrong and fix it, though.

NeddySeagoon · Posted: Thu Nov 23, 2017 8:09 pm Post subject:

uberDoward,

You write UUID - which cannot work, instead of PARTUUID.
Did you test with UUID or PARTUUID in grub.cfg?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.