Scaling hardware for a growing project

[Theodor]

Hi everyone
I’ve been running my small dev project on a few recycled Optiplex nodes for a year now, expecting Gentoo’s optimization to keep things snappy as we grew. But recently, our build times have exploded and the power consumption is starting to outweigh the performance benefits of our custom CFLAGS. I tried moving some workloads to a rented VPS to save on local cooling and noise, but the lack of control over the kernel is driving me crazy. I'm no hardware guru, and I've tried distcc to offload the work, but it’s still not the "set and forget" solution I was hoping for to stay productive. I’m at a point where I need to decide if I invest in a serious Threadripper workstation or just move everything to a managed cloud provider. Has anyone else here had to scale a professional setup from "hobby gear" to something more serious without losing their mind (or their budget)?

[pingtoo]

Theodor,

Welcome to Gentoo Forums.

First of all Gentoo do NOT optimize anything, Gentoo offer opportunity to perform optimization if one want to. I just want to be clear for all fellow reader. Sorry for nitpicking.

If you are in the state as your subject suggest, Project out grew current setup capability there is very little can be done do improve. Optimization is actually a trade off exercise, your trade something for somewhere else to have gain. it could be power, it could be storage, it could be times but something usually loose for somewhere else to gain. In IT there is saying "fast, good, cheap" pick two. It mean for example you can have it fast and good, but it ain't cheap, or you can have it fast and cheap but it ain't good.

Now having said that I suggest your start off with some sort of monitoring system that can actually profile where worthy the effort to target for better investment. The usual mind stress actually come from not knowing exact where is the issue. Once you gain confident know where is problem the reset is much easy to address because that is just action to do.

[Banana]

But recently, our build times have exploded and the power consumption is starting to outweigh the performance benefits of our custom CFLAGS.

Just to make it clear: You are talking about the build time of Gentoo/portage and its packages?

I was hoping for to stay productive. I’m at a point where I need to decide if I invest in a serious Threadripper workstation or just move everything to a managed cloud provider.

Keep in mind that those providers to bill by usage (RAM, Disk, CPU etc) resources.

As pingtoo already said, you need to know what is you bottlneck and then you can decide what to do.
Maybe is is only one package? Maybe some strange config which grew over time?

[NeddySeagoon]

Theodor,

Suddenly exploding build times ...

I would guess lack of RAM causing paging/swapping.
As others have said, find out where your bottleneck is and fix that.

Threadrippers may well be wasted. Playing on a 96 core Cavium Thunder (arm64) I found that build systems could not get more than about 30 parallel make jobs in flight concurrently.

distcc is not a silver bullet. Only C and C++ compile jobs are distributed.
Then, not all of them. It also breaks some builds.
The box calling distcc helpers still does the configure and link steps.
Linking can take a lot of RAM. Swapping/paging may get you out of the end but it takes time and does nothing to progess the task at hand.