Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The Compile 'Cluster'-Raspberry pi
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM
View previous topic :: View next topic  
Author Message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sat Mar 30, 2019 12:18 am    Post subject: The Compile 'Cluster'-Raspberry pi Reply with quote

So in short I have an abundance of pi's, and nothing todo, figured i would recreate a project i did a few years back on the original pis(I used three instead of nine) and see how beneficial it would be to whip them all up in the following configuration:

Currently they are all running the stock raspberry pi 4.19 kernel, though I am hoping to change that soon.

pi3- stock-builder: Handles most of the base system compiling, build package for the rest to update from.
pi3- Nas: Has two 128GB sticks using mdadm to set them in raid 1, NFS exporting packages for binhost and distfile
I believe I read that it may be better to do a full portage export, if anyone can chime in I would be interested, can this be shared by 9 systems at the same time?
pi3- bramble3: nothing special just a builder
pi3- bramble4: nothing special just a builder
pi3+- bramble5: nothing special just a builder
pi3+- bramble6: nothing special just a builder
pi3+- bramble7: nothing special just a builder
pi2- g-pig: test bed
pi2- murmur server

Each is setup as a distcc node and set to use the cluster when compiling.
Is there a way to set distcc to use the hosts in a random order rather then as listed in hosts every time?

As the pi2 are in there you probably guessed I am running this in 32bit mode and compiling for armv7.

I plan to add more duties to the others as i see fit.

I want to move them from the stock kernel to one that is compiled, but wouldn't know where to start or how to automate the roll out process, I figure it could be packaged but again no idea.

I do have a pi0, I may look into seeing if I can get crossdev on each of these to compile for it, mainly to see if I can rather then practicality.

This was my project over the last few days, looking for any suggestions or answers aside from "just use a crossdev/x86" I have had one setup before this was merely using unused hardware.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43439
Location: 56N 3W

PostPosted: Sat Mar 30, 2019 10:42 am    Post subject: Reply with quote

jacealvejetti,

Welcome to Gentoo

Quote:
Is there a way to set distcc to use the hosts in a random order rather t

Yes. read
Code:
man distcc
look out for the special host --randomize

Quote:
I want to move them from the stock kernel to one that is compiled, ...

The first step is to build the Pi Foundation for yourself.

This Wiki page describes the fetch build install process for using a cross compiler. If you want to build on the Pi, the prefix
Code:
ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu-
is not required.

That page is for 64 bit Raspberry pi. Once you have your kernel sources, look in its arch/arm/configs/ for promising defconfig.
There are three that stand out
Code:
bcm2709_defconfig   
bcm2835_defconfig
bcmrpi_defconfig


Do a bit more reading and find out which one you need. Build, install, and run the kernel just as the Pi Foundation provided it.
Once you can do that, your build process is sound. Now you can configure and build your own kernels, knowing that when it breaks, its a configure option.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sat Mar 30, 2019 4:21 pm    Post subject: Reply with quote

Thanks for the welcome Neddy!

I should have read the manual... I wonder if that was there a few years back when i was doing this on the original pi 1bs, I appreciate the --randomize insight implemented that about 5 minutes ago based on your suggestion and its working just as i wanted it to.


I am moving on to the kernel, I think I will make one of the spare "bramble" boxes my kernel builder, installing git now then I will attemp to follow the link you sent.

Thanks for the insights, they are much appreciated!
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43439
Location: 56N 3W

PostPosted: Sat Mar 30, 2019 4:39 pm    Post subject: Reply with quote

jacealvejetti,

The kernel is one of the easiest packages to cross compile.
You will save a lot of time building Pi kernels on, say, an amd64 system.

You can use distcc to build the kernel too. The Pi is not best suited to distcc, as its network port is on USB2.
Its 100Mbit on Pi3 and less and a bit more on Pi3B+ as it has a 1GB network chip. Even there, its on USB2 so 400MBit/sec is doing really well.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sat Mar 30, 2019 7:30 pm    Post subject: Reply with quote

NeddySeagoon,

I do have the ability to setup a crossdev environment, I have reservations as the last time I did this project on the original pis setting up crossdev was a nightmare, currently its not a question of efficiency, more of killing time while learning.

I 'beat' you to the distcc for the kernel though I appreciate it, after googling and a bit of rtfn I found how to invoke distcc along side of make, I am currently building the rpi-5.0.y across the cluster, after about two hours, give or take me not watching it like a hawk, its done. Now to look into installing it not only on the one that compiled it but rolling it to the others.

I want them to be 'self reliant' while this may not be fast eventually I just plan to use them for micro services, ntp, murmur, maybe a backup dns server, I may even make one sit there a "WHOPPER" by playing freeciv against its self.

Again, I appreciate the insights!
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sun Mar 31, 2019 2:31 am    Post subject: Reply with quote

Follow up:

So I went with nfs mounting the linux src directory from the build pi to the others, from there I just followed the post compile/"install" commands on each pi in the cluster, this did indeed work, my only question is in this configuration am I looking at potential problems in the future?

Currently they all mount the nfs as read only as to not allow the rest to tamper with the source.

Now rocking 5.0.3 over the cluster.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43439
Location: 56N 3W

PostPosted: Sun Mar 31, 2019 12:09 pm    Post subject: Reply with quote

jacealvejetti,

With distcc in use /usr/src is only used by by the system doing the build. It pushes out preproceesed sources to helpers.
Making /usr/src read only is fine as long as you set to output directory to some read/write space.

You can also share /usr/portage and /usr/portage/distfiles this way but maybe not /usr/portage/packages unless you maintain /etc/portage in sync across all the Pis.
An nfs mount would fix that.

I'm not a big fan of nfs on Raspberry Pi, it works but it soaks up a lot of the IO bandwidth.
The SDcard interface can run at 50MHz, which is 25MB/sec and its not on the USB2 port. That's a large chunk of the Pis IO bandwidth going to waste.
On the Pi3, the WiFi is on the second mmc card interface, so there is scope for networking there, the down side is that any WiFI crypto is done in the CPU, so its CPU load.

I've not tried a second SD card on the Pi, in place of the Wifi. That would involve the use of a soldering iron but the thought had crossed my mind.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sat Jun 15, 2019 4:12 am    Post subject: Reply with quote

Just following up a few months later, the little cluster is chugging along, it auto updates weekly, both emerge and kernel.

NeddySeagoon, I followed your advice, I made one of the cluster a "nas" and it stores portage, including distfiles, for the cluster.

Another is dedicated to kernel work and hosts it's own builds for the rest to install.

Saddly outside of the learning aspect the one used the most is the murmur pi, though I'm shocked at it's stability.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 577
Location: Romania

PostPosted: Sat Jun 15, 2019 11:55 am    Post subject: Re: The Compile 'Cluster'-Raspberry pi Reply with quote

jacealvejetti wrote:
Is there a way to set distcc to use the hosts in a random order rather then as listed in hosts every time?



I thought about this experiment myself. This and other similar projects that would require something similar.

So, I planned for a DNS solution to this question. One A record with multiple IPs. Something like this:

distcc A $IP1
distcc A $IP2
distcc A $IP3
distcc A $IP4

But now, that I think about it, Neddy's solution might be better because distcc (I think) keeps track of which server is compiling what, and with what I suggested I don't think it can "see" all servers at all times instead relying on DNS to offer the random one. Still, fun to ponder.
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Sat Jun 15, 2019 7:22 pm    Post subject: Reply with quote

Honestly dont know if it keeps track which server is doing the work, I know if a server takes to long to return it will pass the work to another and just disregaurd the work when the first catches up.

I also though about going the DNS route specialy as the "random" server use in distcc leaves something to be desired, as far as i can tell you can have it randomize its hosts, but it only does so on service launch or call, I would love to have it shotgun the cluster everytime a compile piece is pulled, I have noticed during long builds that 2/6 of the ompile cluster remain unused generally.

Though this dosent happen for the linux kernel, probably the biggest things this cluster builds at the moment.

I have come up with a spare pi-3b+ I may make into a desktop and let it rely on the cluster for building, but I also want to move the cluster to aarch64 even it it would leave the pi2s out of operation.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43439
Location: 56N 3W

PostPosted: Sat Jun 15, 2019 8:08 pm    Post subject: Reply with quote

jacealvejetti,

arm64 on a Pi3 is fun. I started that within a month or two of the Pi3 appearing.
Its got a lot easier since then.

I have a public binhost for arm64 Pi3 too. Help yourself if its useful.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 577
Location: Romania

PostPosted: Sat Jun 15, 2019 8:21 pm    Post subject: Reply with quote

I think I said this before in another thread. I'd like to test a cluster of PIs (i'm thinking 4 or 8), to see how they are doing with general C code. Like the kernel. I know it's not a tell_you_all test, but it's a good place to start.

But with a few amendments. First of all, the current "distcc server" I use, is a VM that has crossdev, and toolchains for every arch I own. I think I made a thread about it. Its something of the sort:

Code:
gcc-config -l
 [1] aarch64-unknown-linux-gnu-8.3.0
 [2] aarch64-unknown-linux-gnu-9.1.0 *

 [3] arm-linux-gnueabi-8.3.0
 [4] arm-linux-gnueabi-9.1.0 *

 [5] armv6j-hardfloat-linux-gnueabi-8.3.0
 [6] armv6j-hardfloat-linux-gnueabi-9.1.0 *

 [7] armv7a-hardfloat-linux-gnueabi-8.3.0
 [8] armv7a-hardfloat-linux-gnueabi-9.1.0 *

 [9] i686-pc-linux-gnu-8.3.0
 [10] i686-pc-linux-gnu-9.1.0 *

 [11] powerpc64-unknown-linux-gnu-8.3.0
 [12] powerpc64-unknown-linux-gnu-9.1.0 *

 [13] x86_64-pc-linux-gnu-8.3.0
 [14] x86_64-pc-linux-gnu-9.1.0 *


Each machine I own has one of these vm's. And ofc, I use the DNS method mentioned above. To keep things simple it's compiled for generic x86_64 and therefor works on any amd64 pc. So any and every gentoo I own uses this infrastructure. It's great. I really have nothing to complain. Except one thing... the electric bill.

So I was wondering to myself, how would 4 or 8 PI's would scale against one of my machines. I'm thinking the 3b+, arm64, and ofc you can totally compile the x64_86 toolchain on arm64, and provide distccd for a normal amd64 machine. I don't know if I'm making myself clear. I want to compile amd64 code on arm64. Why? Because I wonder how many PI's you need in order to catch up speed wise a native amd64, and how that scales in terms of consumed electricity. I have a feeling the PI can easily win. Think about it.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43439
Location: 56N 3W

PostPosted: Sat Jun 15, 2019 8:35 pm    Post subject: Reply with quote

axl,

The 1G RAM limit will mean you can only run -j1 for lots of things.
You need a good thermal solution to avoid thermal throttling too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 577
Location: Romania

PostPosted: Sat Jun 15, 2019 8:40 pm    Post subject: Reply with quote

NeddySeagoon wrote:
axl,

The 1G RAM limit will mean you can only run -j1 for lots of things.
You need a good thermal solution to avoid thermal throttling too.



Yeah. I know. It sux that it only has one Gb of RAM. Would have been awesum if it had at least 4. But hey, maybe it's not a good idea now for a PI 3b, but maybe for the next generation of PI. Or the one after that. Or maybe another ARM64. Since ATM money hold me back... I can only fantasize. I guess I could test with my own 3b+ (only have one), but... fantasizing at this point seems fun. Sorry. Had a few beers :) Perhaps I should turn in.
Back to top
View user's profile Send private message
jacealvejetti
n00b
n00b


Joined: 29 Mar 2019
Posts: 13
Location: Michigan, USA

PostPosted: Tue Jun 18, 2019 4:39 pm    Post subject: Reply with quote

NeddySeagoon,

One of these days I am going to have to hop in the cluster and turn them all down to -j1, distcc down to 6 and see if theres a big difference.

I have each of them set to -j4, mind they are all only cli so nothing truly big crosses them.

I have spun up a single aarch64 node and likely over the coming weeks I will slowly migrate the cluster, as I ended up with an extra node, again, It will likely replace the murmur pi.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum