[SOLVED] Enabling KVM for AMD leads to boot hang

[MadWhiteCat]

Hello everyone,

I've been following Gentoo QEMU setup instructions.
I enabled "Virtualization -> KVM for AMD processors support" kernel option (I have Ryzen 7 9800X3D), as well as virtualization support (SVM Mode) in BIOS.
After building kernel and rebooting, the system hangs indefinitely at either "Setting system clock using the hardware clock" or "Activating swap devices" - seemingly random on each reboot.

What made no difference:
- Setting "KVM for AMD processors support" as built-in or module
- Enabling other kernel options suggested in the QEMU guide (Intel module and "Useful kernel configuration options")

I also checked that I have IOMMU enabled in BIOS and valid IOMMU groups according to this Arch Wiki guide.

Tried seeing if anything useful appears in /var/log/rc.log (I enabled logging in /etc/rc.conf), but it seems that writing starts after these failure points. The log file wasn't updated on unsuccessful boots - I checked its contents from my Windows installation after a failed Gentoo boot.

What can I do to debug this further? Has anyone seen a similar problem before?

[Anon-E-moose]

Did you set IOMMU_DEFAULT_PASSTHROUGH

[MadWhiteCat]

Tried this now, rebuilt kernel - still got stuck at "Activating swap devices".

[Anon-E-moose]

sounds like not all of the kvm stuff is set or set right.

Best to have kvm as a separate module, but kvm_amd needs to also be set the same way (both modules or both built in)

wgetpaste your .config file and maybe something will show there.

[MadWhiteCat]

Tried building everything as module, no difference.
Here's my .config - https://0x0.st/8SsR.txt

[pietinger]

MadWhiteCat,

first of all: Welcome to Gentoo Forums!

You miss two very important settings - please enable both:

Code: Select all

# CONFIG_X86_AMD_PLATFORM_DEVICE is not set
# CONFIG_IRQ_REMAP is not set

These settings should not be related to your problem but I mention it, because I have seen them:

Code: Select all

1.
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
2.
CONFIG_BLK_DEBUG_FS=y
3.
# CONFIG_LRU_GEN is not set
4.
CONFIG_EXTRA_FIRMWARE=""
5.
CONFIG_SYSFB_SIMPLEFB=y
6.
CONFIG_I2C_I801=y
# CONFIG_PINCTRL is not set

1. see: https://wiki.gentoo.org/wiki/User:Pieti ... q_governor
2. disable it
3. enable it for better performance (see also in linked article from above)
4. If you use an initramfs then it is okay if this is empty
5. Should not be necessary (and makes more problems than it helps; you have FB_EFI and FB_VESA enabled)
6. You have an AMD machine; so you will need another i2c-module (I801 is for Intel machines) ... but ... i2c needs PINCTRL; without it i2c will not work ... if you dont have any i2c devices (or sensors) you can disable all.

[MadWhiteCat]

pietinger, thanks a lot! Enabling those two options was the solution.
I will also check out the rest of the guides you linked for improved kernel settings.

[pietinger]

pietinger, thanks a lot! [...]

You are very Welcome!

Have fun with Gentoo.

[MadWhiteCat]

Update: Sorry for the confusion, it turns out I was loading the old kernel (grub-mkconfig replaced first entry which had new kernel with old one)... The boot hang is still there when I load my latest build.

[pietinger]

[...] The boot hang is still there when I load my latest build.

Please describe exactly where it hangs (and what has already been started before) ... perhaps with a photo.
We would need also the content of /boot/grub/grub.cfg

[MadWhiteCat]

Here's a photo: https://ibb.co/PsSMmtv1
And here is my grub.cfg: https://0x0.st/8Sbn.cfg
"vmlinuz-6.13.6-gentoo-working" is the backup I've been loading instead of "vmlinuz-6.13.6-gentoo", again, sorry about that.

I also tried setting rc_parallel="YES" in rc.conf and got a lot of services timed out waiting for swap:
https://ibb.co/sJyq8z9f
https://ibb.co/fY2Wkwny

Disabling swap service gets the boot stuck on the next step - "Mounting local filesystems".

[Anon-E-moose]

If I read the grub config properly you have

Code: Select all

amd_iommu=on

which is overriding the pass-through option in the config file.
That might be part of the problem.


And have you updated the initrd when you rebuilt the kernel

Code: Select all

/boot/amd-uc.img

[MadWhiteCat]

Removed

Code: Select all

amd_iommu=on

from grub config, hang still occurs.
Checked the file dates,

Code: Select all

/boot/amd-uc.img

is indeed updated along with a newly built kernel.

[Anon-E-moose]

Is this a desktop or laptop, model name/number if you know what it is off hand.

And if kvm is turned off does it boot?

[MadWhiteCat]

Desktop PC. Hardware, if relevant: AMD Ryzen 7 9800X3D, NVIDIA 2080 Ti, MSI MAG X870E TOMAHAWK WIFI Motherboard. Gentoo on SSD.

And yes, simply disabling KVM makes it boot normally without changing any other options.

What's interesting is that I can boot into recovery mode with this kernel and see /dev/kvm present, but then it hangs when I try to reboot: https://ibb.co/PZSKq10j

[Anon-E-moose]

/dev/kvm should not show unless kvm/amd_kvm is loaded.


I see you don't have VFIO set, but you have KVM_VFIO set
Turn on VFIO (under drivers) and then see what happens (set it the same as kvm/kvm_amd, either builtin or module)

[MadWhiteCat]

Enabled "Device Drivers -> VFIO Non-Privileged userspace driver framework", rebuilt, still no luck.

[pietinger]

And yes, simply disabling KVM makes it boot normally without changing any other options.

Okjay. I would like to see this (successful) boot and then please give us the output of:
* dmesg
* lspci -nnk
* the .config of THIS kernel

I am confused because this:

Code: Select all

	linux	/boot/vmlinuz-6.13.6-gentoo-working root=PARTUUID=c3528a00-01 ro verbose amd_iommu=on
	echo	'Loading initial ramdisk ...'
	initrd	/boot/amd-uc.img

means: You dont use any initramfs (amd-uc.img is "only" the micorocode for your AMD CPU; u means µ; -> µc -> microcode)

How you have enabled kvm modules? With an edit of .config? If this is true please read: https://wiki.gentoo.org/wiki/User:Pieti ... it_.config

[MadWhiteCat]

I fixed the issue by doing these steps from https://wiki.gentoo.org/wiki/Kernel/Upgrade -

Code: Select all

make modules_prepare
make
emerge --ask @module-rebuild

Before that, I haven't been doing @module-rebuild. Now the boot continues normally past "Activating swap devices" into the desktop environment and I can confirm that /dev/kvm is present (and QEMU works with KVM).

My @module-rebuild is rebuilding a proprietary NVIDIA driver. I've no idea how this is connected to activating swap/mounting filesystems, if at all, but here we go.

In case anyone finds it useful, I'll leave here my logs from this successful boot:
dmesg: https://0x0.st/8S_1.txt
lspci -nnk: https://0x0.st/8S22.txt
.config: https://0x0.st/8S_e.txt

I edited .config only through "make menuconfig", never manually. And yes, I don't use initramfs.

pietinger, Anon-E-moose, thank you for your time and help, despite the strange solution I learned more about kernel configuration from your advice.

[pietinger]

I fixed the issue [...]

pietinger, Anon-E-moose, thank you for your time and help, despite the strange solution I learned more about kernel configuration from your advice.

Happy to hear that.

Because you delivered the requested data (even though you solved the problem) I took a look at it anyway

Your system log looks very good / correct. You have a very nice machine, which also has a very up-to-date BIOS. So you should not worry if the CPU microcode is not (yet) updated (because it is already up to date). If newer CPU microcodes become available later, the system log will also show this (timestamp: 5.277812). What else you COULD do is disable AMDGPU when using a NVIDIA card (but not really necessary; this is just a flaw).

Code: Select all

[    0.000000] DMI: Micro-Star International Co., Ltd. MS-7E59/MAG X870E TOMAHAWK WIFI (MS-7E59), BIOS 2.A20 12/27/2024
[    0.266816] smpboot: CPU0: AMD Ryzen 7 9800X3D 8-Core Processor (family: 0x1a, model: 0x44, stepping: 0x0)

[    5.277812] microcode: Current revision: 0x0b404023

[   14.389113] amdgpu 0000:78:00.0: ROM [??? 0x00000000 flags 0x20000000]: can't assign; bogus alignment
[   14.389115] amdgpu 0000:78:00.0: amdgpu: Unable to locate a BIOS ROM
[   14.389115] amdgpu 0000:78:00.0: amdgpu: Fatal error during GPU init
[   14.389116] amdgpu 0000:78:00.0: amdgpu: amdgpu: finishing device.
[   14.389125] amdgpu 0000:78:00.0: probe with driver amdgpu failed with error -22

[MadWhiteCat]

Good to hear, thanks for the update!