[Solved] Possible XFS bug in kernel 5.15.32-r1 - how to dbg?

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

I was doing a benchmark of linux file systems and I'm running into some very strange behavior with XFS and kernel 5.15.32-r1. When I try to extract an uncompressed archive on this newly created (empty) XFS file system, after first low tens of gigabytes extracted the process hangs in iowait indefinitely. One CPU core is 100% occupied with iowait, the other CPU core is idle (on 2-core Intel Celeron G1610T).

No other filesystem tested before or after this (ext2, ext3, ext4, reiserfs3, jfs, nilfs2, f2fs, btrfs, zfs) on the same drive has shown this behavior. When I downgraded the kernel to 5.10.109, the XFS started working again. Kernel versions higher than 5.15 seem to be affected, I tried 5.17.1 and 5.17.6, they both hang up after a few minutes.

No error is reported to the system log or to dmesg when the process hangs. No error shows on stdout or stderr of the tar process either.

Next I want to address the elephant in the (other) room. This is not a SMR problem. None of the disks present in the test setup are SMR. All are CMR, and while they certainly are not brand new, they are all in good working condition.

I searched recent kernel bugs related to XFS and only found ones about mounting a corrupted XFS file system. Mine is not corrupted, it is in fact newly created.

Here are some screenshots when it hangs up: dstat, top

Is there any way I can be useful to the community? How can I trace the bug and where can I submit my findings?

UPDATE 2022-05-26: should be solved in future upstream kernel versions.

mike155 · Posted: Tue May 17, 2022 11:17 am Post subject:

Which command do you run to extract the umcompressed archive?

What's the size of the uncompressed archive?

Do I understand correctly that the uncompressed archive is on another volume? And that your computer hangs while the extraction program writes files to the newly created XFS volume?

Which kernels do you use? Vanilla kernels? Or Gentoo kernels?

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

mike155 · Posted: Tue May 17, 2022 11:56 am Post subject:

Retry with a vanilla kernel from https://www.kernel.org. If it happens there too, it might be time to contact the kernel maintainers. Take a look at this document, section "What information should I include when reporting a problem?"

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

mike155 · Posted: Tue May 17, 2022 12:29 pm Post subject:

I think it's a good idea to use a kernel from http://www.kernel.org - or, even better, to clone the stable kernel git repository from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

I compiled the latest kernel from the kernel.org (5.18.0-rc7), rebooted and tried the test again. Shortly after it is stuck in iowait, just 22GiB into the archive. So, the bug is not exclusive to Gentoo.

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

I compiled a more mainstream version of sys-kernel/gentoo-sources-5.15.32-r1 (removed my .config file and let genkernel to fill it with default options) and lo and behold, in this kernel I could not make it go stuck anymore.

The only difference in XFS options were ("my" config on the left, the genkernel default config on the right):

mike155 · Posted: Thu May 19, 2022 9:17 am Post subject:

Section 'What information should I include when reporting a problem?' recommends to run the commands below if the kernel hangs:

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

araxon · Tux's lil' helper Joined: 25 May 2011 Posts: 83

To whom it may concern, this thread continues here: https://bugzilla.kernel.org/show_bug.cgi?id=216007

mike155 · Posted: Sat Jun 04, 2022 7:15 pm Post subject:

A similar bug or the same bug under ext4: https://forums.gentoo.org/viewtopic-t-1151138.html