Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] MAKEOPTS and EMERGE_DEFAULT_OPTS
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
johngalt
Apprentice
Apprentice


Joined: 09 Sep 2004
Posts: 257
Location: 3rd Rock

PostPosted: Thu Jul 05, 2018 3:55 pm    Post subject: [SOLVED] MAKEOPTS and EMERGE_DEFAULT_OPTS Reply with quote

Hi all,

Reading a bunch of articles regarding the above 2 items. System is a Core i7 965 EE

So far, I've read:

  • https://wiki.gentoo.org/wiki//etc/portage/make.conf#MAKEOPTS
  • https://wiki.gentoo.org/wiki/MAKEOPTS
  • https://wiki.gentoo.org/wiki/EMERGE_DEFAULT_OPTS
  • https://forums.gentoo.org/viewtopic-t-847546-highlight-makeopts.html
  • https://wiki.gentoo.org/wiki/Talk:MAKEOPTS (just to see the commentary
  • https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
  • https://www.preney.ca/paul/archives/341
  • https://lists.gt.net/gentoo/user/269846


And other links. And the general consensus seems to be the `MAKEOPTS='j{N+1}' applies only if you have slow hardware. And the load is recommended to be N, but I also saw documentation on where load-average may be better off at 90% (to allow continued responsiveness). However, I also saw where ppl ignore that, and only run make routines when they don't need any other responsiveness.

And then there are the `EMERGE_DEFAULT_OPTS' of '--jobs' and '--load-average'.

Can anyone make heads or tails of this? Most of the discussions I found are dated rather old, as far back as 10 years ago, including this one https://forums.gentoo.org/viewtopic-t-715522-postdays-0-postorder-asc-start-25.html where the user Jan R used `MAKEOPTS='-J12' (but I believe that was on the original dual XEON build). I'm still reading through this thread, but I am seeing more references to -j9 now.

My hardware is not by any means considered 'fast' in this day and age, but I am using an SSD for full system disk, I have 12 GB RAM, and my CPU, while 1st generation, is well nigh fast enough, with 8 cores total listed (quad core CPU with HT enabled). In reading various pages, it seems that, primarily because of the SSD and RAM, I really don't need the N+1, and can go with N.

Am I off in making this assumption?

So, in /etc/portage/make.conf, I have:

Code:

MAKEOPTS="-j9 -l8"

EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --with-bdeps=y"


What say you?
_________________
desultory wrote:
If you want to retain credibility as a functional adult; when you are told that you are acting boorishly, the correct response is to consider that possibility and act accordingly to correct that behavior.


Amen.


Last edited by johngalt on Sat Jul 07, 2018 1:03 am; edited 1 time in total
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6968

PostPosted: Thu Jul 05, 2018 4:26 pm    Post subject: Reply with quote

the -j is a command for make
the --jobs is a command for emerge

-jX allow X make to spawn in //
--jobs=X allow emerge to emerge X package in //

this mean this:
emerge --jobs=4 world -> allow portage to build upto 4 packages at the same time
emerge --jobs=4 --nodeps gcc vlc -> allow upto 4 packages, but your list of packages is made of 2 (vlc and gcc), so 2 only

assuming gcc is using at max "10 make" jobs and vlc max "7 make jobs".
MAKE_OPTS="-j8" emerge --jobs=4 --nodeps vlc gcc -> allow 8 make per package(!), and you are also emerging two packages, now do the math :)
make children for gcc: 8 (gcc could spawn upto 10, but -j8 limit it to 8)
make children for vlc: 7
total make children 8 + 7 = 15 threads.
That's something to consider because your cpu will have to handle those 15 threads now.
Back to top
View user's profile Send private message
johngalt
Apprentice
Apprentice


Joined: 09 Sep 2004
Posts: 257
Location: 3rd Rock

PostPosted: Thu Jul 05, 2018 5:03 pm    Post subject: Reply with quote

So, it seems as if the better option is to keep MAKEOPTS and not worry about EMERGE... --jobs at all, right?
_________________
desultory wrote:
If you want to retain credibility as a functional adult; when you are told that you are acting boorishly, the correct response is to consider that possibility and act accordingly to correct that behavior.


Amen.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6968

PostPosted: Thu Jul 05, 2018 9:34 pm    Post subject: Reply with quote

you need to tune --jobs= with --load-average
because even if you set -j1000 if only one make job could be use, one only will spawn and your 3 other cores will do nothing, so they could had work instead, hence why --jobs=2 could had help making them busier
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13505

PostPosted: Fri Jul 06, 2018 1:05 am    Post subject: Reply with quote

It depends on what packages you build. If the package has good native support for parallel builds, then a high MAKEOPTS and a low emerge --jobs is likely better, since it will run few packages in parallel, but each package will run many jobs. If the package has poor or no support for parallel builds, then high MAKEOPTS and low emerge --jobs will make poor use of your CPU time. In that case, low MAKEOPTS and high emerge --jobs can provide your CPU with something to do even during the single-threaded portions of the individual packages.

Generally, parallel emerge jobs are a workaround to let you waste less CPU time when dealing with packages that do not parallelize well. You should prefer parallelism within packages when it is available. If you have any doubt, or if you decide to use high values on both, use the load average governors to avoid overloading the system. Starting number-of-CPUs parallel packages, each with number-of-CPUs parallel builds within the package can, absent any other limiters, quickly create excessive jobs. Excessive jobs leads to excessive memory use, which leads to the out-of-memory killer.
Back to top
View user's profile Send private message
johngalt
Apprentice
Apprentice


Joined: 09 Sep 2004
Posts: 257
Location: 3rd Rock

PostPosted: Sat Jul 07, 2018 1:02 am    Post subject: Reply with quote

So it comes down to knowing what the packages are capable of - or, using better terminology, knowing which packages are restricted to single core affinity when building versus those which allow for multiple core affinity when building.

I think that has gotten me moved toward the right track. I think that the default for me is to go with high MAPEOPTS and low emerge --jobs, and alter --jobs on a per-package bass when it needs it with packages that don't make use of multiple cores when building, for example.

Now to research which packages fall in to which of those two categories....

Thanks, Hu and krinn.
_________________
desultory wrote:
If you want to retain credibility as a functional adult; when you are told that you are acting boorishly, the correct response is to consider that possibility and act accordingly to correct that behavior.


Amen.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum