| View previous topic :: View next topic |
| Author |
Message |
skiwarz
Apprentice
Joined: 23 Feb 2014
Posts: 279
|
 Posted: Mon Feb 12, 2024 8:25 am Post subject: Change EMERGE_DEFAULT_OPTS per-package |
 |
|
I recently bought a new CPU and sextupled my core count. To take advantage of it, I'm trying to do the following:
In make.conf:
| Code: | MAKEOPTS="-j8"
EMERGE_DEFAULT_OPTS="--jobs=3" |
And then for a few specific packages (firefox, qtwebengine, etc - the BIG ones), I want to do the following, in package.env:
| Code: | MAKEOPTS="-j24"
EMERGE_DEFAULT_OPTS="--jobs=1" |
The settings in my make.conf work beautifully. When I emerge firefox by itself (emerge -qv firefox), for example, it correctly uses my package.env settings and compiles at a blistering pace.
However, if I have a large list of merges, portage ignores my package.env for those specific large packages, and compiles them with the settings specified in make.conf, along with the rest of the list. [EDIT: Portage DOES use my MAKEOPTS="-j24", but as I stated, it still tries to do "--jobs=3"]
Does portage have the ability to do this? (change --jobs or --makeopts in the middle of running)
I suppose the alternative is I just need to write a script to scan through my proposed merge list, merge the "big packages" first, then continue with my emerge @world. |
|
| Back to top |
|
 |
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Mon Feb 12, 2024 8:57 am Post subject: |
|
|
I think there was similar discussion and one of the developers said that currently it's not supported, but better try to find that thread, I don't think it was that long ago.
Best Regards,
Georgi |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9634
Location: beyond the rim
|
| Posted: Mon Feb 12, 2024 1:13 pm Post subject: |
|
|
EMERGE_DEFAULT_OPTS per package make no sense at all, the --jobs parameter even less given that it defines how many packages can be built/merged in parallel.
Just use an alias instead and maybe make use of the --load option. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55256
Location: 56N 3W
|
| Posted: Mon Feb 12, 2024 1:22 pm Post subject: |
|
|
skiwarz,
emerge --jobs tells portage to schedule --jobs in parallel at dependency resolution time.
It's a one time thing for the entire emerge invocation.
MAKEOPTS= can be set an a per package basis.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2986
|
| Posted: Mon Feb 12, 2024 2:23 pm Post subject: |
|
|
| Genone wrote: | | EMERGE_DEFAULT_OPTS per package make no sense at all |
True. Perhaps OP wants to limit parallel emerge jobs whenever certain packages are being compiled. Good luck with getting that implemented.
Best Regards,
Georgi
p.s. now I recall the other discussion I thought was about EMERGE_DEFAULT_OPS was for something else, more like per package FEATURES. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9634
Location: beyond the rim
|
| Posted: Tue Feb 13, 2024 2:41 pm Post subject: |
|
|
| logrusx wrote: | | Genone wrote: | | EMERGE_DEFAULT_OPTS per package make no sense at all |
True. Perhaps OP wants to limit parallel emerge jobs whenever certain packages are being compiled. Good luck with getting that implemented. |
Yeah, it is obvious what OP wants, but just as obvious EMERGE_DEFAULT_OPTS is the wrong tool for that. What he wants is a user-controllable RESTRICT=parallel-merge (there is no such thing). |
|
| Back to top |
|
|
skiwarz
Apprentice
Joined: 23 Feb 2014
Posts: 279
|
| Posted: Wed Feb 14, 2024 5:24 am Post subject: |
|
|
Thanks for all the responses.
I didn't know EMERGE_DEFAULT_OPTS is only read once, though it makes sense now I think about it. Good to know.
"--load-average=24.0 --jobs=6" seems to give me some semblance of what I'm looking for. When I start emerging the "big packages", my makeopts will go up to -j24 for those, and the load average will limit further jobs until later.
Fun story: The first time I tested with this, I saw the number of emerge jobs jump to about 52 concurrently, immediately after executing the command, and it made me think portage was ignoring the --jobs option. Turns out there's a counter displayed before the load average data that I'd never noticed before. Once that hit my limit, portage stopped spawning new processes. |
|
| Back to top |
|
|
c2
n00b
Joined: 14 Mar 2025
Posts: 37
|
| Posted: Thu Apr 24, 2025 12:23 am Post subject: |
|
|
| skiwarz wrote: | Thanks for all the responses.
I didn't know EMERGE_DEFAULT_OPTS is only read once, though it makes sense now I think about it. Good to know.
"--load-average=24.0 --jobs=6" seems to give me some semblance of what I'm looking for. When I start emerging the "big packages", my makeopts will go up to -j24 for those, and the load average will limit further jobs until later.
Fun story: The first time I tested with this, I saw the number of emerge jobs jump to about 52 concurrently, immediately after executing the command, and it made me think portage was ignoring the --jobs option. Turns out there's a counter displayed before the load average data that I'd never noticed before. Once that hit my limit, portage stopped spawning new processes. |
hello i just wanted to know if you ever went anywhere as what you mentioned in the first message in this thread is very intresting to me,
a way i came up with off the top of my head to get around EMERGE_DEFAULT_OPTS checking at runtime would be via a scripte that checks what packages are out of date (pretty much the view you see of all the packages when you run emerge -AuDN @world) then runs all the small ones in a emerge command then affter that runs the larger ones with some kind of way to deal with small packages having the larger package as a dependaicy.
also it might not be too hard to have the script automaticaly figure out what packages are "large" as tools like genlop can guess at how long a package will take to compile and i assume the portage logs most likely give time to complie data. tho this brings up a issue as if a package is flaged as small it will be given less cores so will natuarly take longer which could lead to a minor issue where packages flip flop between the 2 states. a way to deal with this would be by avreaging out the total restults and saving it to a file.
i mainly want to do this as when installing a large amount of small packages i am sitting at 2% cpu useage 0% drive usage and ~70 gb of ram free so i want more packages to compile then i have threads for all the small packages tho this obviously creates issues for large packages. i recently got a new cpu so i want to recomplie everything for it
can someone tell me if this is a stupid idea and if so why i should not do it. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55256
Location: 56N 3W
|
| Posted: Thu Apr 24, 2025 9:14 am Post subject: |
|
|
c2,
It's a bad idea for the same reason that planning to cross a chasm in two smaller jumps is a bad idea. :)
ebuilds express dependencies. When you attempt to do a subset of the required updates, proper dependency resolution may not be possible, so you have self induced conflicts.
Further, @system set packages are assumed to always be installed. Their ebuilds do not express all the dependencies. It's a way to avoid lots of circular dependencies. e.g. gcc is not usually listed as a depency anywhere, unless a minimal version is required.
This leads to @system packages being built sequentially.
In short, your idea may work some of the time and not others.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
c2
n00b
Joined: 14 Mar 2025
Posts: 37
|
| Posted: Thu Apr 24, 2025 10:20 am Post subject: |
|
|
| NeddySeagoon wrote: | c2,
It's a bad idea for the same reason that planning to cross a chasm in two smaller jumps is a bad idea. 
ebuilds express dependencies. When you attempt to do a subset of the required updates, proper dependency resolution may not be possible, so you have self induced conflicts.
Further, @system set packages are assumed to always be installed. Their ebuilds do not express all the dependencies. It's a way to avoid lots of circular dependencies. e.g. gcc is not usually listed as a depency anywhere, unless a minimal version is required.
This leads to @system packages being built sequentially.
In short, your idea may work some of the time and not others. |
yeah thanks, yeah it only working some of the time is fine, i already knew doing it in 2 batches was bound to create issues with depdancies so i would add some minor ways to deal with it but if it didn't work i would prolly just have it update all the packages it can with out issues then right to a file the ones it couldn't have it notify me somehow, i mean i'll awalys be able to to just run emerge -a affterwords. also i am just experimenting with my make.conf with some slightly unstable options using mold for example so i know it's going to come across errors. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55256
Location: 56N 3W
|
| Posted: Thu Apr 24, 2025 2:02 pm Post subject: |
|
|
c2,
It may be easier to | Code: | | emerge -uUDav @world --exclude="<list_of_big_packages_to_avoid>" |
if that mostly works, you can add --exclude="<list_of_big_packages_to_avoid"> do EMERGE_DEFAULT_OPTIONS.
It saves writing a script,
When you want to update "<list_of_big_packages_to_avoid>" you can | Code: | | emerge -1aqv "<list_of_big_packages_to_avoid>" --jobs=1 |
in that latter case a set called @list_of_big_packages_to_avoid will work but I've never tried --exclude=@set
Anyway, when it breaks you can keep all the pieces. :)
That's a feature of having your Gentoo your way,
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
c2
n00b
Joined: 14 Mar 2025
Posts: 37
|
| Posted: Thu Apr 24, 2025 3:21 pm Post subject: |
|
|
| NeddySeagoon wrote: | c2,
It may be easier to | Code: | | emerge -uUDav @world --exclude="<list_of_big_packages_to_avoid>" |
if that mostly works, you can add --exclude="<list_of_big_packages_to_avoid"> do EMERGE_DEFAULT_OPTIONS.
It saves writing a script,
When you want to update "<list_of_big_packages_to_avoid>" you can | Code: | | emerge -1aqv "<list_of_big_packages_to_avoid>" --jobs=1 |
in that latter case a set called @list_of_big_packages_to_avoid will work but I've never tried --exclude=@set
Anyway, when it breaks you can keep all the pieces.
That's a feature of having your Gentoo your way, |
Yeah thanks for the advice, but I want something more automatic, I started playing games like Factorio and oxygen not included when I was 7 so I have a innate desire to automate everything |
|
| Back to top |
|
|
|
Portage & Programming
