[SOLVED] Ryzen 5 3600 system randomly refuses to boot

[Amity88]

I have a Ryzen 5 3600 system that randomly freezes at early boot, right after grub selection and at the message saying "Loading linux kernel 5.15.59". When this happens, ctrl+alt+del doesn't work and I have to use the reset button. After one or multiple hard resets, it will boot.

Since this happens so early in the bootup sequence, I'm unable to get any logs for clues. fTPM is disabled too.

Could you guys help me root cause the issue and hopefully fix it?

I'm not sure if it's a hardware issue because once it boots, I never experienced a freeze. It sleeps-wakes successfuly, I've done multiple emerge -e system without any issues.

[mike155]

Please post your kernel config.

[Amity88]

Sure, here is a pastebin of the kernel config

This system uses an Nvidia 710 GPU for display but I doubt it's the graphics drivers because it's so early in the boot and I still see the text while it stays hanged.

[pietinger]

It is highly recommended to load a microcode for your CPU (as you already did for your graphics adapter; just add microcode files also):
https://wiki.gentoo.org/wiki/AMD_microc ... _kernel.29

If your problems will exist furthermore you could try to set "generic CPU" instead of MK8:

Code: Select all

CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set

Only for cleaning - it should not cause your problem:

This you dont have:

Code: Select all

CONFIG_X86_INTEL_PSTATE=y

Here you should decide which one do you have (you will see it in "lspci -k"):

Code: Select all

#
# PCI controller drivers
#
CONFIG_VMD=y
 
#
# DesignWare PCI Core Support
#
CONFIG_PCIE_DW=y
CONFIG_PCIE_DW_HOST=y
CONFIG_PCIE_DW_PLAT=y
CONFIG_PCIE_DW_PLAT_HOST=y
CONFIG_PCI_MESON=m
# end of DesignWare PCI Core Support

(There are many other mnodules you dont need, but this should not cause your problem)

[NeddySeagoon]

Amity88,

dmesg may be useful too.
Ideally, when the but fails but that will need a real serial console and you won't have one of those, so we can start with any dmesg you can get.

[Amity88]

@NeddySeagoon,
Hey there! Your presence helps with my morale lol

Here's a dmesg pastebin (working boot, uptime ) Isn't dmesg pointless in this case? cause I'd only see the ones for the booting case. You know, interestingly the grub code runs without hanging, else the grub menu wouldn't have worked.

@pietinger,
Currently the microcode is loaded from initramfs. Are you suggesting that I put in in the kernel itself? Will try that.

I've removed

Code: Select all

CONFIG_X86_INTEL_PSTATE

based on your suggestion and DesignWare PCIe stuff because

Code: Select all

lspci -k 

doesn't list the latter. Left in a lot of extra drivers in case I missed something, didn't think they'd be harmful. Why'd you narrow down on the DW and VMD config options? do they cause conflicts usually?

Here's the pastebin of the lspci command!

[toralf]

What takes so long here ?

Code: Select all

0x000c0000-0x000dffff window]
[    6.677485] caller _nv000720rm+0x1ad/0x200 [nvidia] mapping multiple BARs
[    9.364488] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx
[    9.364501] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready
[   41.515915] elogind-daemon[2511]: Removed session 1.
[   41.533422] elogind-daemon[2511]: New session 2 of user amith.

[mike155]

This looks wrong:

Code: Select all

[    0.000000] random: crng init done     # too early!

[    0.000910] ------------[ cut here ]------------     # exception
[    0.000987] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/rdrand.c:61 x86_init_rdrand.cold+0x0/0x21
[    0.001096] Modules linked in:
[    0.001169] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.59-gentoo #12
[    0.001254] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A320M-HDV, BIOS P5.90 07/03/2019
[    0.001360] RIP: 0010:x86_init_rdrand.cold+0x0/0x21
[    0.001440] Code: 48 c7 c7 6d 7e 03 9e 5d e9 06 76 00 00 48 c7 c7 b8 7c 03 9e e8 fa 75 00 00 48 c7 c7 18 7d 03 9e e8 ee 75 00 00 e9 77 ae d3 fe <0f> 0b 48 c7 c7 b0 7e 03 9e e9 db 75 00 00 48 c7 c7 7a 7e 03 9e c6
[    0.001587] RSP: 0000:ffffffff9e403e88 EFLAGS: 00010246
[    0.001667] RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffffffffffffffff
[    0.001752] RDX: 000000000000000a RSI: 0000000000000000 RDI: 0000000000000000
[    0.004116] RBP: ffffffff9e6da2e0 R08: ffffffff9e51eec8 R09: 00000000ffffdfff
[    0.004697] R10: ffffffff9e43eee0 R11: ffffffff9e43eee0 R12: ffffffff9e6da2e0
[    0.004782] R13: 0000000000000247 R14: ffffffff9e414118 R15: 0000000000000000
[    0.004867] FS:  0000000000000000(0000) GS:ffff8fb2d6c00000(0000) knlGS:0000000000000000
[    0.004969] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.005050] CR2: ffff8fb2df1ff000 CR3: 0000000018c0c000 CR4: 0000000000350eb0
[    0.005135] Call Trace:
[    0.005207]  <TASK>
[    0.005276]  identify_cpu+0x2ec/0x5b0
[    0.005354]  identify_boot_cpu+0xc/0x8f
[    0.005432]  check_bugs+0x21/0xd9f
[    0.005507]  ? __get_locked_pte+0xaa/0x120
[    0.005585]  start_kernel+0x622/0x65a
[    0.005661]  secondary_startup_64_no_verify+0xc2/0xcb
[    0.005742]  </TASK>
[    0.005812] ---[ end trace 7b7ed6a708978f8c ]---

[    0.005889] RDRAND gives funky smelling output, might consider not using it by booting with "nordrand"

In your kernel config, I see:

Code: Select all

CONFIG_RANDOM_TRUST_CPU=y
CONFIG_RANDOM_TRUST_BOOTLOADER=y

CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
# CONFIG_HW_RANDOM_INTEL is not set
# CONFIG_HW_RANDOM_AMD is not set
# CONFIG_HW_RANDOM_BA431 is not set
CONFIG_HW_RANDOM_VIA=y
# CONFIG_HW_RANDOM_VIRTIO is not set
# CONFIG_HW_RANDOM_XIPHERA is not set
# CONFIG_APPLICOM is not set

You may want to

1. disable CONFIG_HW_RANDOM_VIA
2. enable CONFIG_HW_RANDOM_AMD

If that doesn't help, then boot with "nordrand", as suggest above.

Messages about a failing random number generator should be taken seriously. The issue should be fixed.

[Amity88]

@toraif

What takes so long here ?

I think it's just a delay with logging into the gui session (as in I didn't login immediately after the system booted). In the recent boot, that's absent:

Code: Select all

[    9.065368] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx
[    9.065380] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready
[   14.080448] random: crng init done
[   14.080452] random: 62 urandom warning(s) missed due to ratelimiting

@mike155

Thanks for pointing that out! looks like on my Ryzen CPU the rdrand instruction doesn't work correctly, always returning a 0xFFFF FFFF. Disabling rdrand removed this dependency, I need to remove it from CPU OPTS too.

Here is a link to the article describing the Ryzen rdrand issue

Back to the issue, I've done the following and it's still there:

• passed the nordrand option to the kernel boot (the dmesg errors went away)
• disabled CONFIG_ARCH_RANDOM to get rid of rdrand usage
• disabled CONFIG_HW_RANDOM_VIA
• enabled CONFIG_HW_RANDOM_AMD
• disabled CONFIG_RANDOM_TRUST_BOOTLOADER

I noticed that the chance of the hang issue happening is much higher if I press enter right after the grub boot menu comes. On the other hand, if I wait for a few seconds or let the menu timeout, it boots without a hang much more often.

Do you think this is a clue suggesting that it has something to do with the entrophy in the system? (related to rdrand )

Here are the latest pastebins:

• dmesg
• kernel config

[NeddySeagoon]

Amity88,

I noticed that the chance of the hang issue happening is much higher if I press enter right after the grub boot menu comes. On the other hand, if I wait for a few seconds or let the menu timeout, it boots without a hang much more often.

That's odd. At that time its just Grub and the hardware and the hardware should be settled. As the kernel is not loaded, that's in the clear.
The kernel is full of junk but that's OK.

Code: Select all

CONFIG_PCCARD=y
CONFIG_PCMCIA=y
CONFIG_CARDBUS=y

PCMCIA was the original go faster than USB interface on laptops. It was a folded up PCI slot. Then came PCCARD which was the same thing but folded differently.
They have been gone a long time. Its just kernel bloat.

Code: Select all

CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
CONFIG_USB_UHCI_HCD=y

CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
CONFIG_USB_EHCI_FSL=m

That's USB1 and USB2 root hub support. Its likely that you only have USB 3 root hubs in your system, so USB 1 and USB 2 support is not required.

Are you sure the system is freezing as you don't have any text console drivers configured?
Ahhh

Code: Select all

CONFIG_DRM_SIMPLEDRM=y

is a known source of problems.
Turn that off.
Turm on

Code: Select all

# CONFIG_FB_VESA is not set
# CONFIG_FB_EFI is not set

instead.
The kernel will probably pick FB_EFI

If that theory is correct, you can test like this.
Trigger a 'failed' boot. Give the system time to start.
Log in 'blind' and start the GUI.

If the GUI starts, your console is broken. If not, one of the 'blind' steps may have been incorrect.
If ssh is set up, even with no console, a ssh login would still work.

That is, there is nothing on the console until the GUI starts.

[logrusx]

Have you tested your memory? If not - run at least several rounds of memtest86+

Regards,
Georgi

[Amity88]

Amity88,

I noticed that the chance of the hang issue happening is much higher if I press enter right after the grub boot menu comes. On the other hand, if I wait for a few seconds or let the menu timeout, it boots without a hang much more often.

That's odd. At that time its just Grub and the hardware and the hardware should be settled. As the kernel is not loaded, that's in the clear.

Incidentally, with more cold reboots, it seems like the hang happens in the bootloader itself. I'd scroll down the menu selection and it'd hang. It beeps halfway (as in it barely starts to beep but goes silent immeditely). Tested this with both grub and lilo.

It's almost as if there is some kind of settling time after a cold boot. I wonder if it's an electrical issue, but then why is it so stable once it boots?

I had changed the processor from an A8-9600 to a Ryzen 5 3600. Ever since that, after a power up, it takes a few seconds before the bios graphics show up.

Are you sure the system is freezing as you don't have any text console drivers configured?

Yup, cause when it hangs, it's very early in the boot process before the kernel even decompresses. It's a proper hang without it responding to ctrl-alt-del (this seems to cause a restart WHEN I screwed up with the display drivers, during test)

I've taken your other suggestions to turn of those extra options but that didn't help. Also getting rid of the simpledrm causes the console to go blank after boot. FB_VESA and FB_EFI were enabled. BTW, I'm using BIOS, not UEFI, isn't FB_EFI useless in this case?

also tried blind login + startx, didn't do anything.

@logrusk
Nope but let me go do that.

[NeddySeagoon]

Amity88,

Correct. You won't be able to use EFI Framebuffer without EFI firmware.
VESA should still work though.

Those kernel options we just examples of kernel clutter. I was not expecting any change to the booting behaviour.

[Amity88]

NeddySeagoon,

Looks like it's due to some weird interaction / handoff between the bios (legacy) on this motherboard (ASrock A320M-HDV) and the bootloader. (lilo, grub) Here are my latest observations

Observations:

• The hang occurs even before the bootloader loads up the kernel. It would sometimes hang when I scroll the kernel selection menu
• Both LILO and Grub show similar behaviour.
• The hang even effects the gentoo livecd (randomly of course)
• Incidentally, when booting in UEFI, the problem doesn't happen. ( this was tested initially with the livecd)

Current situation:

The Gentoo install was migrated to GPT / UEFI and so far so good, the hang doesn't happen any more. Seems like UEFI solved it but I'm gonna observe for a few more days before marking this thread as solved.

[pietinger]

Amity88,

I am very glad that it will work now with UEFI boot. Sorry, for being so late to the party, but you had two questions I have to answer:

Currently the microcode is loaded from initramfs. Are you suggesting that I put in in the kernel itself? Will try that.

It is not a must and in mostly every case there is no problem if CPU microcode will be loaded later. But of course: best is to load as soon as possible; this is only possible with built-in (look at the timestamp in "dmesg" => 0.000 )

Why'd you narrow down on the DW and VMD config options? do they cause conflicts usually?

These are drivers for PCI ... and ... NO, this should not cause any conflicts, but these drivers have to determine that they are useless (=not needed) and I had some thoughts on timing problems (a second arguement: If something does not exist it can make no troubles )

[NeddySeagoon]

Amity88,

In the early days of (U)EFI, it wan buggy broken and horrible.
As time passed, (U)EFI got better and the legacy BIOS became buggy broken and horrible. :)

[Amity88]

pietinger,

No worries! you and everyone in this thread have been extremely helpful. Thank you

I should work on de-bloating the kernel. This install was rushed through, so I just blindly enabled a lot of things to make it work quickly, with the aim of trimming later on from a high-res interface. As for the microcode, I do have a follow up question. Currently the microcode is compiled into the kernel, in addition to being present in the initramfs BUT the dmesg shows that it's being loaded later, after initramfs. Do you have any idea on why this might be so?

The dmesg
The kernel config

[NeddySeagoon]

Amity88,

There is only early microcode updating now.
When it does anything you get "microcode updated early to new patch_level ... " in dmesg.

Your dmesg contains

Code: Select all

[    1.258907] microcode: CPU0: patch_level=0x0870100a

Is 0x0870100a the latest microcode for your CPU?
If so there may have been nothing to do.

The CPU has micorcode programmed into it at manufacture. If it didn't it wouldn't work. This permanent microcode is slow.
It needs to operate at full CPU core clock, so its copied into the microcode RAM in the CPU. The CPU does this as a part of its initialisation, before it fetches the first instruction.
Once the BIOS starts, it can update the CPU firmware. A current BIOS will be provided with a current firmware at no extra cost, :)

The kernel can update the firmware too. Updating the firmware is tricky. It involves changing the code that describes to the CPU how to execute each instruction.
There is only one copy of microcode for all of your cores, so its far easier to do when only one core is running.
Think about Core War and not being hit.

In short, it looks like the kernel is reporting the microcode patch level but not changing it.
That's fine if its already correct.

[pietinger]

No worries! you and everyone in this thread have been extremely helpful. Thank you

You are very welcome, Amity88 !

... and it was very useful to post your kernel config again with output of dmesg, because you always should read dmesg very carefully ... look here:

Code: Select all

[    0.000000] smpboot: 32 Processors exceeds NR_CPUS limit of 15
[    0.000000] smpboot: Allowing 15 CPUs, 3 hotplug CPUs

Now you know what you have to do with this kernel setting:

Code: Select all

CONFIG_NR_CPUS=15

Your other question was answered already by @Neddy ... but you should know what to do:

In dmesg you will find:

Code: Select all

[    0.131586] smpboot: CPU0: AMD Ryzen 5 3600 6-Core Processor (family: 0x17, model: 0x71, stepping: 0x0)

and with help of https://wiki.gentoo.org/wiki/AMD_microc ... _kernel.29 you now know, you need only: amd/amd_sev_fam17h_model0xh.sbin
So, yes, you can cleanup here:

Code: Select all

 CONFIG_EXTRA_FIRMWARE="amd-ucode/microcode_amd_fam17h.bin amd-ucode/microcode_amd.bin amd-ucode/microcode_amd_fam15h.bin amd-ucode/microcode_amd_fam16h.bin amd/amd_sev_fam17h_model0xh.sbin amd/amd_sev_fam17h_model3xh.sbin amd-ucode/microcode_amd_fam19h.bin amd/amd_sev_fam19h_model0xh.sbin"

Have fun with gentoo

[Goverp]

... Looks like it's due to some weird interaction / handoff between the bios (legacy) on this motherboard (ASrock A320M-HDV) and the bootloader. (lilo, grub) ...

At the considerable risk of muddying the waters, have you updated/considered updating your motherboard's BIOS? My desktop uses a Ryzen 5 3900, and the ASUS motherboard has had perhaps 10 updates since its launch in about 2019; looking at the various comments (albeit on gamer forums) it was more like late Beta code

[Amity88]

Goverp,

It's a logical question given the latest observations. I have updated the BIOS but not to the latest version. The CPU on this machine was upgraded from an A8-9600 to a Ryzen 5 3600, based on the CPU support list here, the latter is supported from BIOS version 5.90.

At the same time, the description of the BIOS updates here, state that versions 6.00 and above aren't recommended for the older A8-9600 cpus. There is also a chance that a downgrade isn't possible. I wanted to keep my option open to use the old cpu again on this motherboard if needed. So the BIOS wasn't updated beyond 5.9

I suppose the key takeaway here is that it's a good idea to buy a mainboard from a vendor that has a history of good BIOS support among other things.