| View previous topic :: View next topic |
| Author |
Message |
c2
n00b
Joined: 14 Mar 2025
Posts: 37
|
 Posted: Sun Apr 20, 2025 9:10 pm Post subject: increasing cpu util for portage |
 |
|
hello, so basicaly my cpu is not beign utlized for most of package installs, when ever i am compling a large package like llvm or the kerrnel util is high but for all the smaller packages like kde-plasma-meta utli is low, i assume beuase it is doing IO stuff that it has to do with every package but sense it's doing that why not start another package sense it's using an extreemly small amount of cpu util.
https://drive.google.com/file/d/14xDiK8D_cOF4a14XyeGww8RnO0VQ72yY/view?usp=sharing |
|
| Back to top |
|
 |
bstaletic
Guru
Joined: 05 Apr 2014
Posts: 506
|
| Posted: Mon Apr 21, 2025 12:54 am Post subject: |
|
|
emerge supports --jobs and --load-average switches. See:- man 1 emerge - for the switches
- man 5 make.conf - for MAKEOPTS mentioned in the previous man page.
|
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5669
Location: Bavaria
|
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Mon Apr 21, 2025 7:49 am Post subject: |
|
|
| bstaletic wrote: | emerge supports --jobs and --load-average switches. See:- man 1 emerge - for the switches
- man 5 make.conf - for MAKEOPTS mentioned in the previous man page.
|
Even with more than one emerge jobs, there may be and often are paths in the dependency graph that have no other parallel paths and the dependencies can only be satisfied by building the packages on that path one by one.
Increasing the number of emerge jobs will only work if there are parallel paths. Then they can be processed in parallel. There can be only as many parallel jobs as the parallel paths are.
Of course at different stages there can be different number of parallel paths, but if you see that only one package at a time is being built that's the case at the current stage of processing the dependency graph.
Also one needs to be careful not to start additional jobs while a big package is being built. And since bigger packages take more time, the probability that more than one of them start in parallel increases.
Thus I advice against increasing the number of jobs by default. You can always inspect emerge output and decide if you'll pass -jobs to emerge. Limiting average load does not guarantee the compile jobs already started will not overload the system.
Best Regards,
Georgi |
|
| Back to top |
|
|
Slippery Jim
Apprentice
Joined: 08 Jan 2005
Posts: 295
|
| Posted: Mon Apr 21, 2025 12:40 pm Post subject: |
|
|
As logrusx said, portage will restrict parallel builds when it has to in order to satisfy dependencies. That being said, I don't see any reason to default to non-parallel processing. I have an 8 core CPU that can do 2 threads per core, so I set the following in my make.conf:
| Code: | MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4" |
The MAKEOPTS settings tell make that it can parallelize its own processes.
The EMERGE_DEFAULT_OPTS settings tell portage that it can do the same.
BTW, you can leave the number off the -j option, and if you skip the -l14.4, the tools will keep spawning jobs until they run out of jobs, or until you run out of memory and the OOM comes to visit. I think that's the danger logusrx was referring to wrt large builds. I had exactly that problem a couple of years ago, even with 64GiB of RAM, and I figured out it was my naked -j option only by accident.
And you need the -l option to limit TOTAL the number of threads, since the maximum number of jobs running is the product of the two -j options (from both emerge and make). A good rule of thumb is to set the load average to 90% of your -j number. That leaves a bit of juice free to keep your user interface responsive. That's where the -l14.4 comes from. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Mon Apr 21, 2025 1:15 pm Post subject: |
|
|
| Slippery Jim wrote: | As logrusx said, portage will restrict parallel builds when it has to in order to satisfy dependencies. That being said, I don't see any reason to default to non-parallel processing. I have an 8 core CPU that can do 2 threads per core, so I set the following in my make.conf:
| Code: | MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4" |
The MAKEOPTS settings tell make that it can parallelize its own processes.
The EMERGE_DEFAULT_OPTS settings tell portage that it can do the same.
BTW, you can leave the number off the -j option, and if you skip the -l14.4, the tools will keep spawning jobs until they run out of jobs, or until you run out of memory and the OOM comes to visit. I think that's the danger logusrx was referring to wrt large builds. |
That's right.
| Slippery Jim wrote: | I had exactly that problem a couple of years ago, even with 64GiB of RAM, and I figured out it was my naked -j option only by accident.
And you need the -l option to limit TOTAL the number of threads, since the maximum number of jobs running is the product of the two -j options (from both emerge and make). A good rule of thumb is to set the load average to 90% of your -j number. That leaves a bit of juice free to keep your user interface responsive. That's where the -l14.4 comes from. |
That's not guaranteed to limit the already running jobs from increasing the load, only emerge and the build system (not all build systems respect limiting the number of threads) to spawn new ones if the load is above the limit. But again, the already running threads can increase the total load well above the threshold and render your system unresponsive. Once you get there your only option is the power button /they don't add reset button anymore :( /
Best Regards,
Georgi |
|
| Back to top |
|
|
Slippery Jim
Apprentice
Joined: 08 Jan 2005
Posts: 295
|
| Posted: Mon Apr 21, 2025 3:48 pm Post subject: |
|
|
| logrusx wrote: |
That's not guaranteed to limit the already running jobs from increasing the load, only emerge and the build system (not all build systems respect limiting the number of threads) to spawn new ones if the load is above the limit. But again, the already running threads can increase the total load well above the threshold and render your system unresponsive. Once you get there your only option is the power button /they don't add reset button anymore  /
Best Regards,
Georgi |
Ok, so what about somehow reserving a core and some RAM exclusively for running the house, so to speak? Shirley, there's a mechanism for that somewhere... |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Mon Apr 21, 2025 4:32 pm Post subject: |
|
|
Are you regularly running so large updates that that would matter?
For 7 years running Gentoo exclusively, I haven't noticed much of a need to have parallel emerge jobs. Especially once you build your system. Surely I went through a period of having such ideas and that's how I learned the details I'm sharing, maybe more but I can't recall all of them. But at the end of the day I don't think anybody needs parallel emerge jobs.
Back when I had slow hardware it was impossible and now I don't need it. Even when migrating from 17.1 to 23.0, that large empty tree emerge, bay majority of packages waited for some other. Also there was this /var/db/pkg lock which prevented merging more than one package at a time and that was slowing down even the binary packages. Afair 6 and a half hours for close to 1600 packages, 2/3 of them binary.
| Slippery Jim wrote: |
Ok, so what about somehow reserving a core and some RAM exclusively for running the house, so to speak? Shirley, there's a mechanism for that somewhere... |
Surely there are(meaning I believe so but don't know them), but how far are you willing to go?
Best Regards,
Georgi |
|
| Back to top |
|
|
Slippery Jim
Apprentice
Joined: 08 Jan 2005
Posts: 295
|
| Posted: Tue Apr 22, 2025 3:08 pm Post subject: |
|
|
| logrusx wrote: |
Are you regularly running so large updates that that would matter?
|
I tend to periodically rebuild @world, and I also like to test ebuilds for broken deps by building them into an empty target ROOT. If I'm testing with varying USE flag settings, then multiple build iterations can become quite expensive time-wise. So I like to maximize the usage of my hardware resources as much as possible.
| Quote: | | Slippery Jim wrote: |
Ok, so what about somehow reserving a core and some RAM exclusively for running the house, so to speak? Shirley, there's a mechanism for that somewhere... |
Surely there are(meaning I believe so but don't know them), but how far are you willing to go?
Best Regards,
Georgi |
Well, it's not really a burden. I like hot-rodding Gentoo. I don't see any downside to setting up my system to use my hardware to its full potential, and the upside, apart from the obvious productivity gain, is that I learn more about how everything works. It's the itch I like to scratch, I guess.
EDIT: Oh yeah, and you were supposed to say "and don't call me Shirley.". I left it wide open... |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Tue Apr 22, 2025 4:58 pm Post subject: |
|
|
| Slippery Jim wrote: | | logrusx wrote: |
Are you regularly running so large updates that that would matter?
|
I tend to periodically rebuild @world, |
That's pretty useless these days.
| Slippery Jim wrote: | | and I also like to test ebuilds for broken deps by building them into an empty target ROOT. If I'm testing with varying USE flag settings, then multiple build iterations can become quite expensive time-wise. So I like to maximize the usage of my hardware resources as much as possible. |
.
I'm not sure what you mean by broken deps, but what's widely understood by that hasn't happen in ages. Last time it was due to broken glibc and I don't remember if I deliberately added it to package.accept_keywords to avoid rebuilds as I was using pretty old and slow CPU.
| Slippery Jim wrote: | | Slippery Jim wrote: |
Ok, so what about somehow reserving a core and some RAM exclusively for running the house, so to speak? Shirley, there's a mechanism for that somewhere...
| logrusx wrote: | Surely there are(meaning I believe so but don't know them), but how far are you willing to go?
Best Regards,
Georgi |
|
Well, it's not really a burden. I like hot-rodding Gentoo. I don't see any downside to setting up my system to use my hardware to its full potential |
It's only possible what nature has allowed us (and we've discovered the limits of course). As mentioned above, when you encounter a single path connecting two subgraphs of the dependency graph, portage has no choice but build the packages on it sequentially. And there aren't that many parallel paths in general. And the more you approach the top of the graph (or the bottom, depending on the representation), the fewer the parallel paths become.
You're not winning that much either way. The packages that benefits multiple emerge jobs tend to be the small ones and the ones that are broken by that are the bigger ones which take time in order of magnitude bigger than the combined times of the small ones.
Best Regards,
Georgi |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3650
|
| Posted: Tue Apr 22, 2025 6:15 pm Post subject: |
|
|
| Quote: | | but for all the smaller packages like kde-plasma-meta utli is low, i assume beuase it is doing IO stuff that it has to do with every package but sense it's doing that why not start another package sense it's using an extreemly small amount of cpu util. |
Looks like emerge has some new relatively new FEATURE you might find interesting. Things that caught my eye:
merge-wait (enabled by default, slows things down for sake of consistency)
parallel-install (disabled by default, should speed things up).
Check out man make.conf for details before you decide to risk your foot.
Oh, for testing dependencies, you might also consider enabling building (feature) and reusing (command line switch?) binary packages too, along with binpkg multi-instance (feature).
| Quote: | | I'm not sure what you mean by broken deps, but what's widely understood by that hasn't happen in ages |
Emerging stuff into a clean stage3 to test ebuilds for broken dependencies sounds quite reasonable to me.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Tue Apr 22, 2025 6:29 pm Post subject: |
|
|
| szatox wrote: | | Emerging stuff into a clean stage3 to test ebuilds for broken dependencies sounds quite reasonable to me. |
Sound and reason - completely different matters. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
