Gentoo on Raspberry Pi B+ and Distcc

[vanjo9800]

Hello, I use Gentoo for more than an year on my laptop and I am pretty familiar with how it works. So I started installing it on a Raspberry Pi Model B+. Everything went fine. The Raspberry booted successfully and I managed to install and configure distcc. However, now when I compile systemd with distcc, I notice something unusual. After my laptop cross-compiles the file, it says in its logs that it had returned it to the raspberry, but the raspberry does not immediately mark it as compiled, but waits for something like 10 seconds and then continues with the next file.

First, is this something normal. If it is not, how can I fix it?

Second, up to now when compiling with distcc, the processors almost never goes beyond 10% and the process happens always on one thread. Can I make that more process are executed at the same time in different threads, like the normal compiling on the laptop, in order to compilation of the packages on the Raspberry Pi to be faster.

Thank you in advance!!!

[axl]

I have a few PI's myself and also use distcc to help them move along. It's not unusual for the B+ model (which is the arm6j, one core, 512 mb ram) to take 10s to send and receive compiled files. Especially if it is using wlan instead of eth network. Either with -j1/-j2.

And as far as I can tell, CPU goes way above 10% while compiling stuff. Well, it uses the cpu to send code, receive code, read from disk, write to disk and still link stuff. So maybe it's not consumed compiling code, but it's still consumed moving that code around, especially if the distcc server do a very fast job. It should never be too idle. If it is, then I suspect the network as a bottleneck.

Hope it helps in anyway.

[NeddySeagoon]

vanjo9800,

Maybe ...

On the Pi, set MAKEOPTS in make.conf to all the cores available for building, including all the helper cores.
You may as well use pump mode too.

On the Pi, be sure localhost does not appear in /etc/distcc/hosts and that it reads something like

Code: Select all

192.168.100.20/8,cpp,lzo [more helpers]

See

Code: Select all

man distcc

The /8 in my example allows the Pi to send 8 compile jobs to the helper at 192.168.100.20.

The poor wee Pi will be busy keeping up with the helper, never mind doing any compiling. Take care not to drive the Pi into swapping.
Do not use WiFi. The Pi will spend all its CPU time doing the WiFi crypto.

How many jobs you can actually build concurrently varies from package to package.
Many packages will cross compile, then you do everything on the helper and run a BINHOST in some manner for the Pi.

[axl]

@NeddySeagoon the b+ has only 1 core. And with only 1 core, he's better off using -j2 and disable helpers. Compressing stuff with 1 core and 2 threads wastes a lot of time that would other wise be gained on a fast eth network.

@vanjo9800 also, as a suggestion since the b+ model only has 512M ram, also it would be a good idea to reduce the video memory to something like 16, if you dont have good reason to keep it high. System becomes crazy slow when it has to swap data. And from my tests, no matter what kind of fast storage i tried to use (swap over nfs/with tmpfs, swap over usb/sata adaptor with ssd hdd, fast usb stick) the system is still crazy slow. So try to use those 512 as efficiently as possible. For instance gcc build better with one thread specifically because of this problem. Other more light programs will work better with 2 threads. But since that only has one core, don't go higher then that. Good luck

[NeddySeagoon]

axl,

Well, I manage with -j8 on a Pi B with 512Mb RAM and one core. Its IO bound as all the IO (except the SDCard) is over a single 400Mbit/sec USB2 port.
I haven't tried any big packages.
Mind you, it does choke on things that won't distribute.

[axl]

well, it chokes on stuff that wont distribute (like gcc or qt stuff) with just 2 threads. Rather logical to choke with 8 threads.

what i did observe, and rather recent, is gcc. took a while to build it, and i did multiple builds on both b+ and and pi2. the one with arm7a, 4 cores and 1Gb ram. gcc has stuff like genattrtab genautomata that hog incredible amounts of ram (way above what b+ has) and get stuck for days if you throw some other threads on top of that. And since gcc doesn't work with distcc no point in having multiple threads. AGAIN, in builds like gcc.

in other stuff, i'll make a few tests with postfix.

First: -j8. compile times are around 3 secs in distcc log. some are bigger (4-5-6 secs), some are smaller. 1-2 secs. but most are around 3-4 secs. seen some very long ones. 17 secs is the biggest i seen. with -j8 cpu had almost 0 idleness according to the /proc/stat file.

total compile time:

real 20m44.202s
user 13m41.640s
sys 4m35.410s

Second -j2. compile times reduced significantly. most of them are under the half second mark. the biggest ones i seen are 1.5 sec. this method also didn't increase the idleness of the cpu according to /proc/stat.

total compile time:

real 19m51.747s
user 13m29.840s
sys 4m31.240s

the reason behind these numbers is very easy. when you have a smp machine, and 2 processes fight for cpu power, then they are moved in 2 different cores. in a unicore environment, the scheduler has to distribute ticks for each process. and when it does that, looses ticks by doing management of resources between them. so, theoretically, if you do 2 tasks in one core, it would take more time to finish them simultaneously rather than in a row. and the more tasks you add to that equation, the more resources you loose managing those multitasks in one core.

however, i suspect it makes sense to always have an extra task. so if you have 1 core, you should have 2 parallel tasks. the reason is that they almost never hog the cpu at the same time. while hogs cpu, the other one closes and prepares. and then they change between them. adding a third however makes 2 of them hog the cpu at the same time and time increases dramatically. even if you have 4 cores, you should have 5 tasks. not 8. that 5th one can change any of the other 4 and it's sufficient to accelerate things.

I understand why intuitively it makes sense to add more threads. Because you would think that PI has to wait for server to reply with compiled code, while the PI stands idle while it could send another package to compile to another server. In a single core environment (like with b+) this never happens unless for a very short time if it would have been build with -j1. But starting with -j2, one distcc cleans up, the other one starts. and it's starting to create 0 idle. this is when it works best. adding anything on top of this, will just waste speed. compression for instance is hogging cpu for no point at all. Again intuitively you think, I compress this data and there will be less data to transmit. but it takes more time to compress/uncompress that data than it would take to wait for it to transmit whole. and it takes more cpu cycles that you dont have doing that.

[NeddySeagoon]

axl,

All that you say is borne out by building things on my Pi3. 1G RAM is not enough to support -j2 with some bigger packages.
Even -j1 can push it into using 250Mb of swap and that's without distcc.

[axl]

that's a totally different monster. the pi3. that is 64 bit, has multiple cores, and 1 gb of ram.

but OP has b+. which is basically a pi 1. and with 1 cores, he is better of with -j2. on pi2 -j5.

and on your pi3, you should use ncpu+1 not ncpu*2. so still -j5 (and disabling compression) would further improve your general speed.

[NeddySeagoon]

axl,

The Pi3 runs out of RAM and into swapping before you can get all four cores busy.
I'm running 64 bit Gentoo on it, so those 64bit pointers won't help.

I suspect that's a feature of gcc. It wasn't such a memory hog four years ago when the Pi was introduced.

[vanjo9800]

Thank you for all your help on that topic!!!

So as a recap:

1) For the Raspberry Pi Model B+ what way would you advice me to compile: without distcc with -j2 directly on the Pi; or with distcc with -j4. For the distcc option, can you explain me how to activate the pump because I cannot find any tutorial in particular. As for the things I want to compile. Most of them are simple, just some drivers. My plan for this Raspberry Pi B+ is to participate in a cluster, and I find Gentoo the best choice for an operating system.

2) I read that you were discussing the Pi 3, and I own one too, so I would like to ask some things about it. In many places, I have read that it would not be beneficial to run a 64-bit operating system on it because of the limited RAM and that the best way now is just to imitate the ARMv7 architecture. Can you tell me what part of this is true? Also, I read that you have managed to run a 64-bit Gentoo on it: can you briefly explain me the whole process of achieving that?

Thank you in advance!!!

[szatox]

1) For the Raspberry Pi Model B+ what way would you advice me to compile: without distcc with -j2 directly on the Pi; or with distcc with -j4.

I've recently put my hands on an old banana (arm7). I had some problems cross-compilling, then tried cross-distcc, cried on performance, and went back to setting up crossdev, properly this time.
After changing profile from the default "embedded" to more reasonable "armv7a" I seem to be more successful, even if some packages require local overlay with modified ebuilds.

For the distcc option, can you explain me how to activate the pump because I cannot find any tutorial in particular.

inform distcc where the remote worker is:
distcc-config --set-hosts <remote_worker>,cpp,lzo
launch emerge:
pump emerge <package>

Pump is a wrapper that will prepare environment and launch the command specified in params.
pump make will usually work too (though some packages do not like it, AFAIK kernel is one of them)

In many places, I have read that it would not be beneficial to run a 64-bit operating system on it because of the limited RAM and that the best way now is just to imitate the ARMv7 architecture.

With what sort of workload? Running a web server: pointless. Heavy number crunching... Well, that could work better*.

*with a board designed for number crunching, like parallella.

[NeddySeagoon]

szatox,

Its easier than that. Let me just say

Code: Select all

man make.conf

See under FEATURES.
What you say is still correct.