Bug in Zram

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

Hi

I've encountered a really unpleasant bug that hangs zram. Usually I use a zram drive for portage to build. It uses zstd. Funny thing is, if I try to format that zram device as ext4, portage hangs while untaring archive into it, xfs hangs while configuring or compiling. Not sure why FS hangs from start and why takes a bit longer. It seems am able to trigger it in 6.6 kernel line, but not in 6.1 line. So far, was only able to trigger it on ARM. I do have some x64 systems, but somehow those are not affected by the looks of it. Use the same layout there, but it doesn't trigger on intel machines. Those machines are on 6.6.13. maybe the regression is more recent. Have to test more i guess.

I was assuming I did something wrong in building the kernel for my PIs 5, because with the same config, 6.1.77 zram works as expected, however on 6.6.21 it hangs. Was ok blaming myself or figuring out its a bug somewhere in the raspberry pi part of the kernel, but, I am also trying asahi/gentoo on one of my Apple M1 machines and triggered it again on a fedora kernel that came with vmlinuz-6.6.3-413.asahi.fc39.aarch64+16k.

To trigger it, or test, I just zramctl /dev/zram0 -a zstd -s 16G -t8; mkfs.xfs -L ramfs /dev/zram0; mount /dev/zram0 /ramfs -o noatime,nodiratime,discard; PORTAGE_TMPDIR="/ramfs" PORTAGE_TMPFS="/ramfs" emerge iptraf-ng.

It is most likely a bug that needs to be reported, but before doing that, was going to ask if it happens to anyone else. What versions of kernel. What arch. Might be easier to find it if others also encountered it.

sam_ · Developer Joined: 14 Aug 2020 Posts: 1678

If you have a simple reproducer, I'd just bisect it.

Child_of_Sun_24 · Guru Joined: 28 Jul 2004 Posts: 578

Here on amd64 and kernel 6.8.1-gentoo no problems with zram. But i use btrfs for /var/tmp/portage. I also use zstd compression.

NeddySeagoon · Posted: Mon Mar 18, 2024 12:17 pm Post subject:

axl,

arm or arm64?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

Neddy arm64 both. rpi 5 and apple m1.

NeddySeagoon · Posted: Mon Mar 18, 2024 8:08 pm Post subject:

axl,

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

Thank you for taking the time to test Neddy. Hmmm. That's the stock kernel from github right?

Its not the pause between the commands. I only wrote them here on the forum like that. In real life I write them manually 1 by 1 with 5 - 10 seconds between them. And the whole thing only hangs when portage starts writing in there. I've noticed that it doesn't hang when portage is mkdir .distcc and creates the cpu sockets, but only when it untars archives as portage. I tried to untar manually as root and it didn't hang.

Guess I will have to think of more creative ways to test. I'll try to install the stock kernel from github and see if that triggers it when portage is writing to it. Thanks.

NeddySeagoon · Posted: Mon Mar 18, 2024 8:24 pm Post subject:

axl,

Its a stock binary kernel from the Foundation firmware repo, just like https://wiki.gentoo.org/wiki/Raspberry_Pi_Install_Guide#Installing_the_Raspberry_Pi_Foundation_files
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

well, it seems only zstd is affected. Had a bit of time to test and LZ4 is working just fine. I'll do more testing as soon as I get some more free time.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

Might have spoken too early. Still hangs the same way when trying to do fstrim. And without that, its back to square one. I noticed in 6.6 kernel they have a new option in block devices: ZRAM_MULTI_COMP. aint that. Tried on and off. Have to get myself a proper x86 to test. Sux to test on arm. Asahi only has one kernel, and not exactly sure how to bisect between the two branches of 6.1 and 6.6 that raspberry pi has.Not sure there's a direct correlation of patches between those 2 branches. But I might be wrong. Wouldn't be the first time.

gentoo_ram · Posted: Tue Mar 19, 2024 10:31 pm Post subject:

I'm currently stable on my RPi5 using the raspberry-pi linux kernel sources. /var/tmp/portage is formatted ext4. But I'm using 4k kernel page size. I was getting weird effects with the 16k page size. I think I remember zram hangs when using the 16k. Maybe that's what you are seeing? I figured it had to do with the filesystem being 4k page size when the kernel was 16k page size. If you aren't using 4k page size, try using that instead.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

still weird if you think about it. how come neddy wasn't able to trigger it? he has 16k pages.

meanwhile am noticing that after a few days of usage, even on kernel 6.1, zram drive simply becomes unusable. Was trying to compile clang on one of the new pi5. Had 3 days uptime. At one point, couldn't even get libcxx compiled. That's kinda small. Every time it would fill at least one of the files with garbage. randomly. Works fine if you reset machine or reset the zram device and start over.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

Seems like you were right. 4k pages on 6.6 line seems to work just fine. Thanks for the suggestion.

gentoo_ram · Posted: Thu Mar 21, 2024 9:00 pm Post subject:

That's another person with the same observation I had. At least I know I wasn't crazy! I noticed it pretty quickly after I changed my kernel config to move to 16k pages after running stable for a while at 4k. The page size was the only thing I changed when zram started acting badly.