Speeding up complilation on Raspberry Pi 5 bare metal

[netsplit]

Obviously it's never gonna be blazing but so far I've noticed:

By default the kernel is in power saving mode. It can be set to ondemand with echo "ondemand" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor. You can also change the default in your kernel config.

MAKEOPTS -ln seems to be overly aggressive compared to amd64. Setting it forced many gcc builds to a single thread for me. Unsetting it seems to gave fixed that. However now it seems I might need to investigate water cooling because it throttles from the heat now lol

[NeddySeagoon]

netsplit,

For cooling you need the official Pi5 fan assisted cooler. That keeps the Pi out of thermal throttling, unless you put the Pi and cooler into a case that restricts the airflow.
With MAKEOPTS="-j4" and --jobs=1 that will let you build Chromium in only 32h.

You could also try cross distcc and ccache. Both have their drawback as they don't work for everything.

-- edit --

Try

Code: Select all

# vcgencmd get_throttled && vcgencmd measure_temp && vcgencmd measure_clock arm
throttled=0x0
temp=65.9'C
frequency(0)=2400030464

to see what's going on and

Code: Select all

# genlop -c

 Currently merging 1 out of 1

 * www-client/chromium-139.0.7258.31 

       current merge time: 2 hours, 20 minutes and 28 seconds.
       ETA: 1 day, 4 hours, 31 minutes and 22 seconds.

That estimate is about 2 hors short.

[netsplit]

netsplit,

For cooling you need the official Pi5 fan assisted cooler. That keeps the Pi out of thermal throttling, unless you put the Pi and cooler into a case that restricts the airflow.
With MAKEOPTS="-j4" and --jobs=1 that will let you build Chromium in only 32h.

When updates finish I'll try building Chromium. A 32 hour build sounds like an experience.

You could also try cross distcc and ccache. Both have their drawback as they don't work for everything.

I tried getting distcc working but ran into problems. Also tried setting up an arm64 vm, and discovered Qemu has some issues with threading. The Pi seems to build fast enough. One
thing that seems to make a difference is the storage medium. It builds a lot faster off an nvme drive than a usb stick.

Try

Code: Select all

# vcgencmd get_throttled && vcgencmd measure_temp && vcgencmd measure_clock arm
throttled=0x0
temp=65.9'C
frequency(0)=2400030464

to see what's going on and

Code: Select all

# genlop -c

Code: Select all

raspberrypi ~ # vcgencmd get_throttled && vcgencmd measure_temp && vcgencmd measure_clock arm
throttled=0x0
temp=67.5'C
frequency(0)=2400037120

Seems like it's doing okay. I don't have the raspberry pi heatsink fan but I have a 3rd party one. At some point it's going to be installed in my car and I live in a sunny place so it'll probably need better cooling, at least when the car starts.

[NeddySeagoon]

netsplit,

Chromium on Pi5 needs both -mcpu and -march unset to avoid build failures.
The bug has been reported to Gentoo but not yet upstream.
It needs more work first.

NVMe is a lot faster than USB. Especially if your USB is bulk mode only.

[netsplit]

netsplit,

Chromium on Pi5 needs both -mcpu and -march unset to avoid build failures.
The bug has been reported to Gentoo but not yet upstream.
It needs more work first.

NVMe is a lot faster than USB. Especially if your USB is bulk mode only.

Thanks for the tips!

I think the storage medium was the biggest surprise, but it makes sense. Even on an NVMe (that I've verified is running at PCIe 3.0) I'm seeing CPU usage percentages are ranging between 25% to 75% per build thread. Assuming the CPU usage percents are accurate, the CPU isn't the current bottleneck. When I'll do the chromium build I'll try setting the build environment to use a ram drive.

[netsplit]

Just a follow up, tmpfs was actually worse. 38.25 hour build. Oddly the build threads had higher CPU use. It seems the RAM file system had more CPU overhead then I would have guessed. I also suspect there was some swap usage which would have negated the point of tmpfs.

Here's the setup:
pi 5, 16gb ram

/etc/fstab

Code: Select all

#size=10G would fail for not enough disk space
tmpfs           /var/tmp/tmpfs  tmpfs   size=102401M,uid=portage,gid=portage,mode=775        0 0

/etc/portage/env/tmpfs.conf

Code: Select all

PORTAGE_TMPDIR="/var/tmp/tmpfs"

/etc/portage/package.env

Code: Select all

www-client/chromium             tmpfs.conf

Code: Select all

[ebuild   R    ] www-client/chromium-138.0.7204.92:0/stable::gentoo  USE="cups hangouts official proprietary-codecs pulseaudio qt6 rar screencast system-harfbuzz system-png system-zstd wayland -X -bindist -bundled-toolchain -custom-cflags -debug -ffmpeg-chromium -gtk4 (-headless) -kerberos -pax-kernel (-pgo) (-selinux) (-system-icu) -test -vaapi (-widevine)" L10N="-af -am -ar -bg -bn -ca -cs -da -de -el -en-GB -es -es-419 -et -fa -fi -fil -fr -gu -he -hi -hr -hu -id -it -ja -kn -ko -lt -lv -ml -mr -ms -nb -nl -pl -pt-BR -pt-PT -ro -ru -sk -sl -sr -sv -sw -ta -te -th -tr -uk -ur -vi -zh-CN -zh-TW" LLVM_SLOT="20 -19" 0 KiB

Results:

Code: Select all

raspberrypi ~ # genlop -t chromium
 * www-client/chromium

     Tue Jul 15 07:07:49 2025 >>> www-client/chromium-138.0.7204.92
       merge time: 1 day, 14 hours, 14 minutes and 10 seconds.

So tmpfs added an extra 6 hours lol.

[NeddySeagoon]

netsplit,

That result does not surprise me. If you have the RAM to build in RAM, the kernel cache will do it anyway.
Building in tmpfs saves writes that will never be read. That may be a good thing for SSDs.
Reads/writes are all DMA, so time savings from not setting up DMA will be too small to measure.

If you have swap, the content of tmpfs can be moved to swap under pressure of RAM.

When you don't have swap, there is no home on disk fbr dynamically allocated RAM, so the kernel has to 'swap' in other ways.
It can drop clean pages, then reload them later. This includes code that it will execute real soon now.
It can write 'dirty' pages out, so that they are clean, then drop/reload them.

All my Pis have either 8G of swap, for RAM <= 8G or 16G on the 16G ones.
It may not be used but it makes it easy to spot when things are being pushed a bit hard.
Then it's time to reduce MAKEOPTS on a per package basis.

[Hu]

Here's the setup:
pi 5, 16gb ram

/etc/fstab

Code: Select all

#size=10G would fail for not enough disk space
tmpfs           /var/tmp/tmpfs  tmpfs   size=102401M,uid=portage,gid=portage,mode=775        0 0

Normalizing that size:

Code: Select all

$ numfmt --to=iec --from=iec 102401M
101G

You told the kernel to allow up to ~101G in the tmpfs, but you only had 16G of real RAM to use, with that split between the tmpfs pages and ordinary usage. As you and Neddy noted, swapping is bad for performance, and the configuration you set here makes it easy to overload the tmpfs to the point that swapping is needed. Typical guidance for C++ heavy programs is to plan for 2GiB RAM per compiler process, so even if your 10G had been sufficient disk space and you had only emerge running, you could not count on using more than ~3 concurrent compiler processes. I'm assuming you bumped to 101G without measuring exactly how much you needed, but even if bumping to 20G had been sufficient, that would have left you with at most negative 2 compilers running, if you wanted to avoid swapping. You need at least positive 1 compilers running to make forward progress, and ideally you want enough compiler processes to saturate every CPU core. Chromium is well known to be huge and slow to build. I think the Pi 5 is just not powerful enough to build Chromium in tmpfs in a reasonable time, and arguably not powerful enough to build Chromium in reasonable time at all.

[netsplit]

netsplit,

That result does not surprise me. If you have the RAM to build in RAM, the kernel cache will do it anyway.
Building in tmpfs saves writes that will never be read. That may be a good thing for SSDs.
Reads/writes are all DMA, so time savings from not setting up DMA will be too small to measure.

If you have swap, the content of tmpfs can be moved to swap under pressure of RAM.

When you don't have swap, there is no home on disk fbr dynamically allocated RAM, so the kernel has to 'swap' in other ways.
It can drop clean pages, then reload them later. This includes code that it will execute real soon now.
It can write 'dirty' pages out, so that they are clean, then drop/reload them.

All my Pis have either 8G of swap, for RAM <= 8G or 16G on the 16G ones.
It may not be used but it makes it easy to spot when things are being pushed a bit hard.
Then it's time to reduce MAKEOPTS on a per package basis.

The swap usage was indeed not surprising when I discovered just how much space Chromium wants to build. Still tried it. Swap had 32GB available so I'm certain it had enough at least. It shouldn't ever need more than 48gb (32+16) total memory. Was mostly just testing things to learn. The actual paging is quite interesting. Thank you for shedding more light on it. I used to believe setting swapiness to 0 was ideal because it'd prevent swap usage. It might have been better on old systems where using the page file would cause freezes, but 0 in hindsight probably wasn't ideal.

Here's the setup:
pi 5, 16gb ram

/etc/fstab

Code: Select all

#size=10G would fail for not enough disk space
tmpfs           /var/tmp/tmpfs  tmpfs   size=102401M,uid=portage,gid=portage,mode=775        0 0

Normalizing that size:

Code: Select all

$ numfmt --to=iec --from=iec 102401M
101G

You told the kernel to allow up to ~101G in the tmpfs, but you only had 16G of real RAM to use, with that split between the tmpfs pages and ordinary usage. As you and Neddy noted, swapping is bad for performance, and the configuration you set here makes it easy to overload the tmpfs to the point that swapping is needed. Typical guidance for C++ heavy programs is to plan for 2GiB RAM per compiler process, so even if your 10G had been sufficient disk space and you had only emerge running, you could not count on using more than ~3 concurrent compiler processes. I'm assuming you bumped to 101G without measuring exactly how much you needed, but even if bumping to 20G had been sufficient, that would have left you with at most negative 2 compilers running, if you wanted to avoid swapping. You need at least positive 1 compilers running to make forward progress, and ideally you want enough compiler processes to saturate every CPU core. Chromium is well known to be huge and slow to build. I think the Pi 5 is just not powerful enough to build Chromium in tmpfs in a reasonable time, and arguably not powerful enough to build Chromium in reasonable time at all.

I meant to do 10 GB + 1 MB because 10G was erroring out with an error of need 10 gigs of disk space, so assumed it was a greater than check (but hindsight perhaps emerge checks for gibibytes, and tmpfs works in gigabytes). Anyway silly math goof aside, lucky for me tmpfs didn't try to take all that, and just took ram needed for actual virtual storage. I was attempting to use ram storage to saturate the CPU cores, because it seemed a single lane PCIe 3.0 bus wasn't enough. The problems you noted prevented that. Thank you for catching my goof with tmpfs.

Anyway I agree with your conclusion. if I ever need Chromium on Raspberry Pi 5 I'll use a bin package.

[thegrind]

I didn't think there is a bin for Chromium.

[s0ulslack1]

I'm curious why you're knowingly mounting it in a horrible spot, I'm assuming to make wiring easier? Stuff that girl out the way and wire up a fan thats switched too!

[netsplit]

I'm curious why you're knowingly mounting it in a horrible spot, I'm assuming to make wiring easier? Stuff that girl out the way and wire up a fan thats switched too!

This pi is the system board for the head unit in my 300,000 mile gen 2 Prius. It's basically a little box I can bring in the house for compiling updates and plug back into my dash for music, navigation, and ODBII. Unfortunately the need for molecularity and portability limits cooling options. Ideally I'd set it up with water cooling.

My interest in compiling Chrome was it seems to be a common mountain to climb and the music app I'm using is Electron based, so compiling Chrome would seem to be a needed step for a truly local build. Fortunately there's a prebuilt arm64 binary for the music app. Which given the build time I'm graciously choosing to use.

Mostly just wanted a head unit I could upgrade myself. Previously had an Android based one, but I find Android very frustratingly limited, and it runs like a dog. Further can't upgrade the hardware. My new build, I could swap out the system board with a new one pretty easy. I got Waydroid working with GPS for Android apps, configured OpenRC to run a bluetooth audio sink service so people can play music off their phone's on bluetooth. Used a USB hub for audio out, and touchscreen input. For audio I used a USB 7.1 audio adapter I connected to a cheap amp which has RCA in and the out connects to the car speakers. Ideally it'd just be a couple of USB cables (hub and power), and HDMI to connect it to my car.. However annoyingly I discovered USB 3 has RF interference issues with GPS, so it needs to be directly connected to one of the Pi's USB 2 ports.

The pi sets in a little storage area above the glovebox to make it accessible but secured. I did most of the work in the summer so was happy to see it didn't have the August overheating issues I feared.

For the touch screen I made a VESA to Prius adapter with pieces from an old Prius radio, and a cutting board I bought at a dollar store. I cut the cutting board to size, attached the Prius mounts and drilled the VESA mounting holes. It just mounts to the frame like the old factory radio did. The monitor is a small HDMI touchscreen.

It's not complete. I still have to figure out the steering wheel controls, make a bezel, see if I can get voice control working, and install a stand by battery so the unit doesn't have to reset when I turn the car off to get gas or something. It's wired to an accessory circuit in the car, you see. A complete powerless interrupts the music and navigation, further it doesn't have CMOS, and it takes way too long for it to fix the clock using GPS time data.

It's a hobby project, so time, and fighting my own profound ignorance to get things working makes it slow going lol. It took many days to figure out why I couldn't get the onscreen keyboard working in KDE for example, Anyway that's my infodump on how it's mounted.