Fun With Distcc

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Edit: Yeah, no, this was dumb. That's not how this works--THAT'S NOT HOW ANY OF THIS WORKS:

Conclusion: I've had best results with setting --randomize and multiplying every host's processor by two (over distcc's recomendations) in /etc/distcc/hosts. It makes a big difference to remember to double-up the number of potential jobs [considering the number of each helper's cores] in your /etc/distcc/hosts file, as well as your make.conf. I'm also not afraid to do emerge -uDNavt @world at the same time between computers, as they're all doing their own things at their own times. It's rare that they clash, the one time being one of them decided to build clang and rust at the same time. All it did was slow things down--in the end there were no errors.

Questions: Does it really matter how many -jobs I set in MAKEOPTS so long as my --load is sane? Couldn't I just throw maximun number of jobs and let the load setting manage it?

I also set EMERGE_DEFAULT_OPTS="--jobs 2 --load-average 0.9". So I'm thinking so long as the load average isn't over 0.9 that it won't try to start emerging another package, but if it does, will it send the same amount of jobs as set in MAKEOPTS for another package? In other words: would the computer then be sending out double the jobs it was sending before? On top of the first emerged/emerging package?

Sometimes I'm just fit-to-be-tied over how intelligent Portage is. I mean, really? Cool!

For as long as I've read (not that long) its been dogma for us to set our -j* settings to double whatever cores we have running if we're using Distcc. Assuming we all have -zeroconf in our make.conf, here's how I go about setting up my build cluster for maximum CPU saturation:

Our own Neddy has stated (paraphrasing) that computers spend most of their time waiting for user input. So in the interest of economizing through-put, I place here my questions on why the man pages of distcc tout such great increases in build-time for our computers, yet my own results (out-of-the-box) have been so thoroughly mediocre.

Setting it up as per the instructions, everything hums along nicely, but I've wondered why (if my cranky old single core) is sending jobs out to all the other computers, it's loaded up so much, yet my other honking fast (for me) quad core is getting in barely a trickle of work to process

Lets assume I have a teensy little single core sending out signals to a hulking quad core. Standard operating procedure would have you set your MAKEOPTS at "-j10" and then walk away happy, but in the distcc man pages it states that:

NeddySeagoon · Posted: Wed Jul 04, 2018 7:38 pm Post subject:

unheatedgarage,

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

That's good to know. I'll keep playing with it and see what works best. Thanks, Neddy!
_________________
I'm not even mad; I'm impressed!

johngalt · Posted: Thu Jul 05, 2018 5:37 am Post subject:

@unheatedgarage - Thanks for this post. My Core i7 965 EE has some work to do for my two pitiful laptops lol.

@Neddy - as always, thanks for your fixes and explanations.
_________________

R0b0t1 · Apprentice Joined: 05 Jun 2008 Posts: 264

What is the average cores to GB of RAM ratio in your cluster? Is there a way to track machine utilization?

I have been wanting to set up a distcc farm on cheap ARM boards instead of buying a server.

NeddySeagoon · Posted: Fri Jul 06, 2018 4:29 pm Post subject:

R0b0t1,

You need about 1GB/core for some C++ code, which is why its not possible to build some things on small ARM borads.
On the other hand, is that very much worst case.
With 96 cores working hard, I've not got over 60G RAM used, which suggests the average RAM/core is much lower.

Lots of small RAM ARM boards won't help you build the things that need 1G RAM per core.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Nevermind! Don't do that! This was a bad idea from the beginning, and I should feel bad for having thought of it--this is why we can't have nice things!

I write this from a computer that's herking-and-jerking from swapping too much. I set both my laptop and my helper to do some building and it just about crashed everything (thank goodness for large swap partitions!), and I have been experiencing these issues just about ever since I made the first post.

HOW EMBARASSING

So it's back to the sane distcc defaults for me--no more overloading everything.

Hope I didn't crash anyone elses computers. :oops:

Happy distributing, everyone!
_________________
I'm not even mad; I'm impressed!

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

In this case, say my helper: tiny-core is set to

MAKEOPTS="-J100 -l1"
EMERGE_DEFAULT_OPTS=" --jobs 100 --load-average 0.9"

It was my hope that Portage would be smart enough to know to balance things based on the "-l" and "--load-average" settings in MAKEOPTS and EMERGE_DEFAULT_OPTS, respectively, but this doesn't seem to be so.

As we speak, tiny-core is stuck grinding into swap trying to emerge dev-libs/popt and sys-libs/glibc at the same time, so it seems likely that emerging more than job at once (depending on the size and intensity of the jobs) will overload the processor, and portage won't be able to keep the number of jobs set in MAKEOPTS to a sane level.

When my desktop quad-core was seizing up earlier, it was recieving jobs from both tiny-core and laptop. Not sure which packages they were emerging at the time, but it was definitely enough to clog up quad's tubes and bring things to a near stand-still.

So, if I were to continue setting both tiny-core and laptop to build at the same time (with both sending jobs to quad-core), it certainly makes sense to keep distcc at its defaults.

But...

Maybe I could keep the not-so-sane settings I was talking about earlier in both tiny-core and laptop's /etc/distcc/hosts, but make sure to not set EMERGE_DEFAULT_OPTS at more than one --job, maybe then that wouldn't completely overrun either quad-core, or tiny-core/laptop.

Note: I'm no longer running localhost in either tiny-core or laptop, and they're just sending jobs to quad-core, not each other.

I like the idea of Portage beginning to process another package while the system is building another (as long as it's not overloaded), that's why I was all for setting more than one job in EMERGE_DEFAULT_OPTS, once I read about it, but if Portage is not meant (or not capable of) dealing with multiple jobs and distcc, then it's all for naught.

Well anyway, it's a blast--having a ball. I'm still on the path to get my computer science degree and become a Gentoo developer, it's just a long (expensive) path. Meanwhile, I'm standing on the shoulders of giants, and I know that you juggernauts are here to help me along.

It's funny how when your Windows system breaks, and you install Ubuntu as a patch (all the while proclaiming you hate computers and just want them to "work"), things flip. You spend your life wrenching on cars and motorcycles, then you go install one system--you do ONE THING, and the next thing you know your path and interests have completely changed.

And, well, here we are.

Thanks, everyone. Thanks everyone so much.
_________________
I'm not even mad; I'm impressed!

NeddySeagoon · Posted: Sat Jul 14, 2018 7:44 am Post subject:

unheatedgarage,

The helpers make.conf is never consulted when its running distcc for helping other systems.
Its distccd listens for incoming jobs and runs them, regardless of make.conf.

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Mr. Seagoon,

Thank you, sir! As always, I place your input on the highest pedistal (you have no idea how much you've helped me over the years).

Yessir, that MAKEOPTS was a typo, rest assured it was correct in the make.conf--thank you very much for your observations!

I've found, using these absurd MAKEOPTS, glibc does not respect the load setting, at least not while it's installing. While it's building all seems all is well, but if I keep MAKEOPTS at -j100 -l1, the installation goes haywire--load spikes, swap starts, it's a mess. But if I set -j1 (like any sane person would do), tiny-core goes ahead and installs it (although it takes a while!).

What I ended up doing was setting a build environment for glibc called sane-makeopts and pointing glibc to it in package.env.

What I'm still curious about, though is: how does Portage treat two packages. How does Portage two-fist two packages, and how does it treat the "-l" setting in MAKEOPTS and the "--load-average" in EMERGE_DEFAULT_OPTS?

What's the magic with parallel builds, Distcc, and Portage?

None of this is urgent--just curious. Should I just get on IRC (it frightens and confuses me) and ask these questions there?

I'm grateful for your time!

Edited for spelling.
_________________
I'm not even mad; I'm impressed!

johngalt · Posted: Sun Jul 15, 2018 5:16 am Post subject:

I asked a similar question in the Installing Gentoo forum regarding MAKEOPTS and EMERGE_DEFAULT_OPTS:

https://forums.gentoo.org/viewtopic-t-1083538.html

I've still not quite got my head wrapped around it, but I've resorted to MAKEOPTS="-j{#cores +1} -l{#cores} for CPU utilization and EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=2 --with-bdeps=y" on my Core i7 965 Extreme (Nehalem, aka 1st gen core). So, in my case, it's -j9 -l8
_________________

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Jeez. I really screwed up by posting this, then, because your thread was pretty much the same thing as mine. Hu always gives great, thorough, answers.

Thanks, Hu!

Thank you for your reply johngalt.

Just discovered this tonight: I've been running --quiet-build for so long, I forgot to watch the output of each package. You know how some packages won't build with distcc-pump enabled? I had created a build environment just for them called disable-pump. In that I had:

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Two of my systems seem to be really, really broken since the python 3_6 upgrade, and I do believe it's because of using distcc in pump mode:

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

Maybe it's from having this in my /etc/fstab?

eccerr0r · Posted: Tue Jul 31, 2018 2:18 pm Post subject:

report the output of

$ df

... it is reporting you're out of space, but don't know where. Appears to be a logfile.

Since tmpfs is cleared on boot, if the problem persisted through reboots I'd be surprised.
You could just clear out the contents of your tmpfs too if needed, especially if it was full of failed builds.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Moderator Joined: 06 Mar 2007 Posts: 21602

For this case, I'd suggest df --o, which will give us both blocks and inodes in a single step. Exhausting either resource leads to ENOSPC, which can confuse users who check their free blocks, find space, and wonder why there is "No space left on device."

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

df --o after doing a reboot:

eccerr0r · Posted: Thu Aug 02, 2018 1:51 pm Post subject:

Is is safe to have a distcc for rust so firefox-60 will build faster?
It would be called distrust after all.

<flees from the thrown rotten tomatoes>
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

eccerr0r · Posted: Fri Aug 03, 2018 3:31 am Post subject:

There's plenty of hearsay everywhere, until there's some actual serious issues I'm not too concerned. I'm still using firefox out of creature of habit, using the ESR.

I suspect that the unkeyworded (versus masked) rust compiler is mostly because there are probably few devs that really know rust and can't quite vouch for it yet. Rust was meant to be more secure than C++ as it has stronger intrinsic pointer checking, but it's still rather new and not as well deployed as other languages, especially interpreted languages...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

There you go again--going waaay over my head, what with your (clearly) deep knowledge of this stuff! I'll be getting there, but in good time...

All in good time...

Actually, I'd feel less offended by the new Firefox if I could have gotten it and Rust to distribute...

Off topic, but I've been waffling about which programming language to learn first, but then it dawned on me that Portage is written in Python. Since this magic that is Portage is so fascinating, and since I'm no longer taking classes (for now), and since there's plenty of free info out there on the web anyway...

*takes a breath*

...and since you good people are probably tired of me pestering you with questions, well...Python it is!
_________________
I'm not even mad; I'm impressed!

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

As was said by Neddy up-thread, and has been said time, and time, and time again, there’s a limit with distcc.

I figure, it was designed for programmers with very specific use cases, with a very specific build cluster that they could fine-tune to their very unique needs. I gather it was never intended for Gentoo users trying to build an entire world—multiple different packages across many different architectures—some packages building parallel nicely, some not—it’s a crazy place here, and that just wasn’t what the designers had in mind?

It’s a testament to the Gentoo developers/users that it works so well!

For sure, going through all the work to set up one or two helpers really isn’t worth it. And, if they’re old/slow enough, will probably slow you down, what with the overhead of running the distcc client. Again, this has all been said before! The only thing that would burn it into me was just trying it out for myself, and believe me, The Electric Company (Hey You Guys!) thanks me for it!

Setting up many more helpers (at one point I had 25 cores working hard) makes a big difference, but even then only some packages would take advantage of that much potential power, while a lot of them would not.

Then comes in the whole Clang/Chromium thing—I spent waaaay too much time trying to get that to distribute. Granted, I’m completely clueless in these matters, but for the life of me I couldn’t find diddly on the internet except for one or two tantalizing glimpses. I’d eat my hat if Chromium wasn’t distributable, so...anyone get any closer to that?

I can see now how, for some packages, there’s a huge benefit to having a large cluster. But, when tiny-core is struggling under its own weight, not to mention distributing jobs to everyone else, there’s only so much that can happen—even when it’s left out of the /etc/distcc/hosts file—tiny-core don’t care if you’ve got all that processing power, it’s got to run its checks first!

Now the benefit of having a binhost becomes glaringly obvious. Now I see the magic of cross-compiling. Now I understand why a lot of you Big Dogs have been working with embedded devices—what with the fact that modern computers have more than enough horsepower to deal with compiling anymore, that’s where all the fun is!

So it’s time to chuck these old boxes in the landfill. It’s time to step up and learn a whole lot more about a whole lot more. It’s time to quit pissing like a puppy. My newest things: learning the intricacies of Shorewall, getting a Grub-free UEFI system, and reading Common Lisp: A Gentle Introduction to Symbolic Computation.

Best wishes to you all.
_________________
I'm not even mad; I'm impressed!

NeddySeagoon · Posted: Thu Sep 06, 2018 5:20 pm Post subject:

unheatedgarage,

Chromium uses clang to build by default. That happened when it broke with gcc.
distcc is a wrapper around gcc, so out of the box it does not distribute for clang.

So your challenge is to fix Chromium to build with gcc or fix distcc to wrap both gcc and clang.

A first step might be to change the Chromium ebuild to use gcc and see what happens.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

hephooey · n00b Joined: 24 Feb 2003 Posts: 74

I have been using distcc to compile chromium for some time, I think the only thing I did is making symlinks of x86_64-pc-linux-gnu-clang and x86_64-pc-linux-gnu-clang++ to distcc under /usr/lib/distcc/bin on the client (The one you want to install chromium), the compiler servers does not need any change (they need to have the matching clang etc installed though). The bundled ffmpeg coming with chromium failed to compile with distcc for me. So I hacked the ebuild to use the system ffmpeg. After that everything just go through smoothly. The last chromium I built was 69.0.3497.57, if you want to test with specific version.

unheatedgarage · n00b Joined: 19 Sep 2016 Posts: 60

hephooey,

That's interesting to hear. Here's what's in my file: