[tutorial] BadRAM 2025 update

[ecko]

I have been dealing with RAM issues, and I found the existing information online to be slightly outdated, so I decided to write a short tutorial.

Previous references on this forum:

• viewtopic-t-1140476-start-0.html "[SOLVED] qtwebengine fails to build" (2021)
• viewtopic-t-1165113-start-0.html "badmem static allocation of memory hole" (2023)

Symptoms

I had compiler segfault when building in big packages, such as dev-qt/qtwebengine-6.8.2, or other qt or kde applications. With --keep-going in the options and going until the end, I would relaunch it and often it would then be successful, though on very large packages like qtwebengine, or chromium, it would fail again.

Memtest86+

I installed sys-apps/memtest86+-7.20 and rebooted into it. Memtest86+ runs 10 different tests (e.g. simple read/write, block move, modulo 20) and on my machine at 1 GB/min, meaning 2 hours for the 128 GB I have.

Anytime during the test, press <F1><F4><F4> to go to the badram mode, where the output is most usable for the solution. Instead of listing individual failed bytes (which could be thousands), memtest86+ tries to summarize them into ranges using a mask. The output is limited to 10 lines.

Here is the output in my case, copied from a mobile phone picture:

Code: Select all

badram=0x00000010168001b8,0xfffffffffffdb8,
       0x0000001016800538,0xfffffffffffd38,
       0x00000010168009f8,0xfffffffffffff8,
       0x0000001016801178,0xfffffffffff578,
       0x00000010168040b8,0xfffffffffff3f8,
       0x0000001016804638,0xfffffffffffff8,
       0x0000001016804cb8,0xfffffffffffdb8,
       0x0000001016805038,0xfffffffffff978,
       0x00000011af842960,0xfffffffffffff8,
       0x00000011af847ea0,0xfffffffffffff8,

The output is composed of an address and a mask, where the f (the all-ones) correspond to the bits that are common in the range, and the zeros correspond to the bits that can vary.

It can be seen that the addresses are very close and can be summarized by two 64k blocks:

Code: Select all

0x0000001016800000,0xffffffffffff0000
0x00000011af840000,0xffffffffffff0000

We have to exclude at least 4k blocks which is the page size of the kernel, but it is safer to exclude bigger blocks. It is likely that the number failing bytes are going to increase over time near the ones that already failed (and 64k is very small anyway).

GRUB2
We add the ranges to /etc/default/grub then update the grub configuration:

Code: Select all

GRUB_BADRAM=0x0000001016800000,0xffffffffffff0000,0x00000011af840000,0xffffffffffff0000

grub-mkconfig -o /boot/grub/grub.cfg

This calls the script /etc/grub.d/00_header which will copy the GRUB_BADRAM parameter from /etc/default/grub into /boot/grub/grub.cfg :

Code: Select all

…
play 60 800 1
badram 0x0000001016800000,0xffffffffffff0000,0x00000011af840000,0xffffffffffff0000
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Gentoo GNU/Linux, with Linux 6.13.2-gentoo-x86_64' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.13.2-gentoo-x86_64-advanced-36f0c021-f626-4ac4-b1f1-b3dc23d8f85d' {

Reboot
The badram parameter is not a command-line parameter to the linux kernel. Nothing special is not visible in the grub menu, even if using "e" to edit a line.

After reboot, we check that the kernel takes into account the excluded address. We use the number "11af" which is part of the address (we could also check the other range which contains "10168").

Code: Select all

dmesg | grep 11af # to be adapted to each address range
[    0.000000] BIOS-e820: [mem 0x0000001016810000-0x00000011af83ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000011af840000-0x00000011af84ffff] unusable
[    0.000000] BIOS-e820: [mem 0x00000011af850000-0x000000201f2fffff] usable
[    0.000000] reserve setup_data: [mem 0x0000001016810000-0x00000011af83ffff] usable
[    0.000000] reserve setup_data: [mem 0x00000011af840000-0x00000011af84ffff] unusable
[    0.000000] reserve setup_data: [mem 0x00000011af850000-0x000000201f2fffff] usable
[    0.167868]   node   0: [mem 0x0000001016810000-0x00000011af83ffff]
[    0.167869]   node   0: [mem 0x00000011af850000-0x000000201f2fffff]
[    0.173324] PM: hibernation: Registered nosave memory: [mem 0x11af840000-0x11af84ffff]
[    0.679391] e820: reserve RAM buffer [mem 0x11af840000-0x11afffffff]

Kernel early memtest
Another, or complementary solution, is to ask the kernel to perform a memtest at boot.

According to the kernel documentation:

Memtest fills the memory with this pattern, validates memory contents and reserves bad memory regions that are detected.
It will write different patterns and read them back, finding memory elements which are not functional and reserving them.

https://www.kernel.org/doc/html/latest/ ... eters.html

The parameter can be added dynamically at boot by editing the kernel line in GRUB. It can also be added to the GRUB configuration /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="memtest=4"

grub-mkconfig -o /boot/grub/grub.cfg

The memtest is performed early in the boot process and shows no output. On my machine, with memtest=4 (analysing 4 patterns) 128 Gb are analyzed in approximately 90 seconds.
After boot is complete, one can see the result within the first lines of dmesg

Code: Select all

dmesg | less

[    0.000000] early_memtest: # of tests: 4
[    0.000000]   0x0000000000100000 - 0x0000000001000000 pattern aaaaaaaaaaaaaaaa
...
[    0.000000]   0x0000000100000000 - 0x0000001016800000 pattern 5555555555555555
[    0.000000]   5555555555555555 bad mem addr 0x00000001d386fee8 - 0x00000001d386fef0 reserved
[    0.000000]   0x0000001016810000 - 0x00000011af840000 pattern 5555555555555555
...

The blocks which were previously passed by GRUB are not tested as the kernel already ignores them. This enables to detect the additional wrong bytes as they appear.
The user can add a block size at choice in the GRUB configuration; for example to exclude a 64 kb block containing the bad byte: GRUB_BADRAM=0x00000001d3860000,0xffffffffffff0000

[pietinger]

Moved from Kernel & Hardware to Documentation, Tips & Tricks.

[NeddySeagoon]

ecko,

Two things
A memtest failure does not always mean the RAM is faulty.
Often, removing the RAM and refitting it fixes the problem. Its called 'wiping the contacts'.

This information would be better as a Wiki page. It can get lost/forgotten here.

[ecko]

In my case I tested with the 4 DIMMs together (initial configuration), then tested them one by one (I numbered them with a pencil), then two by two, then all 4 again. The results were consistent with 2 particular DIMMs reporting errors, and not the other 2. I did not attempt to clean the contacts though. You're right, reseating the components should be mentioned as part of the procedure.

I will consider creating an account on the wiki and move the howto there so more contributors can improve the instructions.

[anacrolix]

I also went down this rabbit hole just recently.

badmem appears to be a kernel patch set that was supported out of tree until maybe 2009. It was last used by a big distro in 2013 I think.

Kernel devs like Greg KH recommended against it as far back as 2004.

memmap is the modern way to do it. I had to escape the hell out of what I gave GRUB to get it to work. I added to /etc/default/grub something to the effect of GRUB_LINUX_DEFAULT="... memmap=4K\\\$0x12341234". \\\$ goes through the shell that grub-mkconfig uses, and \$ goes from GRUB at boottime and passes in as $. 64bit addresses are supported.

memtest also works, but I had to put it up to 8 before it picked up my bad mem.

I'm not sure if GRUB supports its bad mem option or if it translates that to memmap as appropriate.