AMDGPU driver failing on at least one system

[RayDude]

I'm running AMDGPU driver on my laptop (APU=6800H RDNA2), my server (GPU=RX6650XT) and another old laptop that acts as a surfer on the old Samsung TV whose apps are no longer supported.

I play Hogwart's Legacy on the media server and once in a while it crashes and exits X Windows. I figured that was a game bug, but this morning I noticed my laptop which had been up for nine days crashed out to the login screen. I looked in /var/log/messages and found this:

[Goverp]

[RayDude]

[ali3nx]

There is an outstanding bug that reappeared with 6.1+ linux kernels and some models of amd graphics cards that rely on amdgpu graphics drivers that causes the gpu driver to generate a kernel panic. The bug results in an amdgpu driver initialization ring timeout.

I have two RX 5500 cards in different builds and neither system are functionally reliable with a 6.1 kernel. My only reliable resolution was forcing a downgrade to 5.15 LTS linux headers and kernel version. Sometimes the kernel panic is so subtle it's not greatly disruptive to system stability but the amdgpu driver will ultimately be entirely unsable as a result.

here's the link to the kernel bugzilla thread

I added this to package.mask then recompiled the entire system build because major linux-headers changes are always a major build time consistency concern.

[RayDude]

[ali3nx]

[nvaert1986]

I had a very similar issue with the 5.19.x and 6.0 kernels (the exact same symptoms and sporadic though the error message slightly different; mostly the same). This issue disappeared for me with the 6.1 kernel up until 6.1.20 so far (that's the kernel I'm currently using on the system). You could try using the 6.1.20 kernel to see if this resolves the issue. I'll be upgrading to the latest 6.1.x kernel this weekend and I'll let you know if anything unusual happens. I'm using a AMD Radeon RX6750XT on an Intel Core i9-12900K if that matters.

[RayDude]

I upgraded to 6.3.7 and the problem has gotten worse.

I think I may have a clue as to what is sending it over the edge.

I'm running a program called vivado that eats RAM for lunch. Every time I run this thing it takes ram usage from roughly 25% up to around 75% and then when it's done memory drops back to 25%.

After that, twice today, I had failures. One was a video reset and the other was a dead lock of the laptop.

I have no data on the video reset because the system was flaky so I rebooted it. And after reboot the USB mouse and keyboard didn't work so I had to shut down and reboot to get them working again.

I think this is all that applies to the hard crash in /var/log/messages

[logrusx]

[ali3nx]

[RayDude]

[RayDude]

It just crashed hard a few minutes after completing a vivado run, but the driver was able to recover.

For future reference this is kernel 6.3.7-gentoo and video-amdgpu-22.0.0.

Here are the relevant portions of various logs:

dmesg

[logrusx]

[CooSee]

[RayDude]

[ali3nx]

Some gentoo community members are aware I've provided completed gentoo chroot prebuilds for many years and considering the circumstances I felt it could be beneficial or useful for diagnostic testing this kernel bug to provide a gentoo LTS prebuild to attempt to aid affected users.

Perhaps this could be a stage3 build the releng team developers could consider officially providing.

I've started a new systemd desktop merged-usr stage3 chroot build specifically configured for amdgpu support restricted to linux-headers and gentoo-kernel-bin 5.15.x that should be available on my prebuilds webserver directory perhaps within six to twelve hours from this post reply.

The only changes made are some minor use flag alterations and other necessary make.conf configurations, linux-headers and kernel version restrictions and world rebuild followed by completing the xorg-server and xorg--apps package group installations.

Completed 5.15 lts chroot prebuild is available here

100 MBit upload bandwidth speed limit so please try to be courteous of you notice a slower upload speed.
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!

[RayDude]

[ali3nx]

If you wanted to attempt to reproduce the kernel bug with a specifically built 5.15 build without altering your own current build you could configure this new build to be usable from a perhaps either a separate partition on some disk of your choosing or a new btrfs subvolume or zfs dataset.

Similar concept as dual booting multiple os installs. Pooled storage filesystems are fantastic for multi boot installation options once your familiar with the usage concept.

Perhaps it's possible to run this build environment from a chroot but testing the running 5.15 kernel with a consistent system build compiled around the 5.15 linux headers version is a consistent sane testing environment that could be considered valid for diagnostic validation.

My build is a chroot build because all gentoo systems are commonly built using a chroot build environment. My custom xeon powered Larry enjoys working hard to aid fellow gentoo users

It occurred to me i've never seen someone report this ring timeout amdgpu driver bug using a 5.15 kernel nor have I experienced it but the hardware I have available perhaps wouldn't qualify as a large enough sample size to reproduce the amdgpu ring timeout kernel driver bug with linux 5.15

My own systems when running 6.1.x the ring timeout is guaranteed to occur immediately on system boot but never occurs with 5.15.x

Perhaps building your own purpose built 5.15 lts gentoo desktop build is difficult and inconvenient due to personal computing horsepower limitations. This could help you begin using that configuration with the 5.15 linux-headers world package contents consistency rebuild having been completed in advance.
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!

[RayDude]