Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
revisiting make.conf MAKEOPTS
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Sun Mar 06, 2011 11:24 am    Post subject: revisiting make.conf MAKEOPTS Reply with quote

Ola,

I have taken time this weekend to reread gentoo installation handbook,
in order to learn more. It did a hell of a lot more sence now than almost
a year ago when I started with Gentoo.
I made only one change, but I found something, that I would like to ask first,
before I do such thing.

From handbook:
Quote:
MAKEOPTS

With MAKEOPTS you define how many parallel compilations should occur when you install a package. A good choice is the number of CPUs (or CPU cores) in your system plus one, but this guideline isn't always perfect.

Code Listing 4.3: MAKEOPTS for a regular, 1-CPU system

MAKEOPTS="-j2"


This will speed up compilation time:
-just speed it up, or
-do 2 or 3 simultaneous compilations?

Now, I understand this like so, that this wont hurt if I put it in my make.conf file?

I have 2 laptops, one is single cpu, so I can add that line safely,
and the other laptop is dual-core, 64 bit, and I could add in make.conf file line like so:
MAKEOPTS="-j3" or MAKEOPTS="-j4" (since it is 'dual')?
(I am newbie, so sorry if this is more of funny question)

here is my make.conf from dual-core laptop:
Code:
CFLAGS="-march=core2 -O2 -pipe"
CXXFLAGS="${CFLAGS}"
CHOST="x86_64-pc-linux-gnu"
USE="mmx sse sse2 gtk -gnome bzip2 gzip zip rar php -samba slp hpcups ppds new-hpcups scanner pam branding mplayer mp3 gimp usb X dbus -hal jpeg png gif tiff startup-notification thunar consolekit cairo lock session automount opengl avahi wmf xml apache2 mysql mdnsresponder-compat dvd alsa cdr -arts -kde -minimal -qt3 -qt4 -hpijs -static-ppds device-mapper udev -xfce"
GENTOO_MIRRORS="http://mirror.bih.net.ba/gentoo/"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
INPUT_DEVICES="evdev synaptic keyboard mouse"
VIDEO_CARDS="radeon"


Thank you.
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Sun Mar 06, 2011 11:37 am    Post subject: Re: revisiting make.conf MAKEOPTS Reply with quote

while true wrote:
I am newbie, so sorry if this is more of funny question

It is not a funny question indeed.
However, I suggest you use the search feature of this forum with MAKEOPTS as argument for topic and posters such as
krinn , Hu, NeddySeagoon, d2_racing... :wink:... The latter only when quoting Zac... :wink: :wink:

You should be able to find tons of threads dealing with this story.
Some of them are more than very interesting, including posts from truly experienced gurus.

You can for example start with this one : https://forums.gentoo.org/viewtopic-t-847546-highlight-makeopts.html
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Sun Mar 06, 2011 11:40 am    Post subject: Reply with quote

You can't really break anything with this option.

Here is the man page to make: http://unixhelp.ed.ac.uk/CGI/man-cgi?make
Quote:
-j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run simultaneously. If
there is more than one -j option, the last one is effective. If
the -j option is given without an argument, make will not limit
the number of jobs that can run simultaneously.

Developers can specify in their Makefiles, if a job can be done parallel during compile time.
Additionally some ebuilds reduce the number of jobs, if there were problems.

Just keep in mind:
not enough jobs = CPU not fully utilized and time/per job is too high
perfect number of jobs = CPU fully utilized and optimal time/job
too many jobs = CPU too much utilized and the overall time for the jobs is higher than it would be with less.
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Sun Mar 06, 2011 3:01 pm    Post subject: Reply with quote

Hello guys,

Thanks for fast reply.

Aha, so there is a slight chance of "over-doing" it?
Quote:
If the -j option is given without an argument, make will not limit
the number of jobs that can run simultaneously.

So MAKEOPTS="-j" would be the right syntax, to let it decide by itself?
or to avoid MAKEOPTS altogether?


off topic:
I have not said this enough: This forum's support is superb!
..but... I hate search option!!! It is not that I am not using it,
but I can not choose to search just topics,
or word and author in one search,
and results are sorted by time, not by some relevance...

but I guess there is a price for every beautifull thing, kheh
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Sun Mar 06, 2011 3:07 pm    Post subject: Reply with quote

If you use too many, the CPU will be more busy reading writing to disk (probably even swap memory) than translating the code :D

I haven't tested a blank "-j" option yet.

//edit: OK, you can break/slow things :D
MAKEOPTS="-j" ate up all my memory in a blink of an eye and made the system very very unresponsive... during compile
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Sun Mar 06, 2011 3:41 pm    Post subject: Reply with quote

aha, kheh, sorry disi, but that is how I understood, that is just '-j'...

Well, as newbie, I will wait for another year to feel more comfortable to hack this kind of stuff.

I just thought that since it is in handbook, I should give it a try, to speed up things a bit,
since some times, the update or emerging stuff is taking realy long time...

offtopic:
At times like updateing or emerging stuff,
I force myself to remember the time, when I had cassettes,
and 90 minutes one were real luxury (I was born in communist country),
but to record anything on them I had to be there
every minute... (And I did so many compilations for birthday gifts)
.. oh what do you know, it compiled already!

back on top:
So I guess i will not put that in my make.conf file,
thank you both for your time and input

(disi, sorry again for overloading your machine...)
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Sun Mar 06, 2011 3:56 pm    Post subject: Reply with quote

That was an ctrl+alt+SysRq [s|u|b]
My desktop is used to that... :)

You can also play around with PORTAGE_NICENESS to increase or decrease the nicelevel of emerge...

//edit: I am just testing with "-j" on console (no X) and it runs ~30 jobs(if it can)... awesome fast on my Phenom II ;)
//edit: just saw one package with ~80 times libtool running O.o
//edit: "ps aux | grep /usr/lib/gcc | wc -l" gives between 1 and 74 results for xulrunner
//edit: no good, it pops up at some point and uses ~7GB of memory (I have only 8GB), then stalls the compile (wants probably to swap but there is none)
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 9342

PostPosted: Sun Mar 06, 2011 5:41 pm    Post subject: Reply with quote

Specifying a bare -j with no associated number directs Make to launch as many jobs in parallel as it can, subject to dependency limitations. For any well-written large project, the number of jobs launched in this way will exceed what your system can comfortably handle. Running several dozen instances of gcc or g++ on a typical desktop system will exceed memory and/or CPU resources. This will manifest as poor interactivity, high swapping, or both.

The recommendation for the "ideal" number of jobs is based on the idea that you want to keep the CPUs as busy as possible, but do not want to have excess jobs waiting (and consuming memory) that cannot be run because there is no CPU available to run them. Reasonable people disagree on how to compute this ideal number, but the most commonly cited values are N, N+1, and (2N)+1, where N is the number of CPUs available. This ideal is probably influenced in part by available RAM. If your system is low on RAM, you should err on the side of too few jobs, since each job requires both a CPU to run it and RAM in which to store it. If your system has plenty of excess RAM, you can err on the side of a few too many jobs, since you will not miss the extra memory.
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Sun Mar 06, 2011 5:52 pm    Post subject: Reply with quote

Sure, I just wanted to see what's the limit and with -j20 it turns out to use a maximum of 2.5GB RAM. IMHO the physical limit is the memory in the machine...
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Mon Mar 07, 2011 5:54 am    Post subject: Reply with quote

Thanks guys!

So I guess I should try this option, kheh, but first I have questions:

If I start with N+1,
where and how should I monitor behaviour of cpu & ram?
Will top (or htop) do fine?
(monitoring cpu & ram)
Also, I guess I should emerge several stuff (not just one) to see activities in top?

And if cpu & ram is 100%, I should back up a bit?
And if cpu & ram is 40% I should step up?

Or, if I understood disi-s last post, ram is the limit?
(in which case I can do a bit overloading of cpu,
to fill most of the ram?)

And how will this be outputed in shell?
Right now I see emerging scrolling one after another,
(one finishes another starts),
or am I now just asking for it?

Thank you
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 2607
Location: Dallas area

PostPosted: Mon Mar 07, 2011 11:54 am    Post subject: Reply with quote

I like to keep my jobs moderate (-j6) to leave some good system interactivity (but that's just me)

Latest with this setting
Code:
     Sun Feb 27 09:31:36 2011 >>> sys-devel/gcc-4.4.5
       merge time: 9 minutes and 46 seconds.

     Sat Mar  5 05:00:01 2011 >>> sys-libs/glibc-2.11.3
       merge time: 9 minutes and 41 seconds.

     Mon Mar  7 04:22:04 2011 >>> mail-client/thunderbird-3.1.9
       merge time: 5 minutes and 54 seconds.


This with 8 gig ram, X6 1055 @ 3.5 ghz
I could push the jobs higher, but I'm happy with it the way it is.
_________________
Asus m5a99fx, FX 8320 - amd64-multilib, 3.15.9-zen, glibc-2.19, gcc-4.9.2, eudev
xorg-server-1.16, openbox w/lxpanel, nouveau, oss4(2011)
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Mon Mar 07, 2011 1:09 pm    Post subject: Reply with quote

kheh, this would be a nice statistic to have,
cpu/ram/swap vs "-jx" in terms of
50% ("to leave some good system interactivity")
and 100% usage.

Right now, from cli (no wm or de)
with no MAKEOPTS in make.conf's use,
top shows:
load: 1.47 1.30 1.27
Cpu: 75%us (avarage), 25%sy 1.4ghz
Mem: 508876k total, 370000k used
Swap: 1060280k total 29600k used


Since I do nothing on computer during the update,
my farmers mind tells me that I could add -j1 safely
to use whole ram and a nice bit of swap,
but to slightly overload cpu,
(with older laptop disks r/w might cause swap delayed response,
which might benefit time to overloaded cpu)

but enything more than -j1 would overfload swap as well,
resulting in poor performance.


Am I getting this equation right?

@ Anon-E-moose, minutes?
Minutes?!?
and not just any doble digit minutes,
but single digit minutes!!!

Thank you
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
d2_racing
Moderator
Moderator


Joined: 25 Apr 2005
Posts: 13046
Location: Ste-Foy,Canada

PostPosted: Mon Mar 07, 2011 1:38 pm    Post subject: Reply with quote

Hi, about the MAKEOPTS feature, I use this when someone ask me which value to use :

Code:

$ cat /proc/cpuinfo  | grep "model name" | wc -l


And I add 1 to the result.

That's for the MAKEOPTS option.

That's one half of the optimization that you can do.

The other part is the EMERGE_DEFAULT_OPTS with the jobs and load-average parm.

But on that part, I don't have any good advice, I used the Zac Medico setting.
_________________
Sysadmin of GentooQuébec.org
Wiki
Signature
IRC on Freenode : #gentoo-quebec
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Mon Mar 07, 2011 2:20 pm    Post subject: Reply with quote

disi
Quote:
You can also play around with PORTAGE_NICENESS to increase or decrease the nicelevel of emerge...


d2_racing
Quote:
The other part is the EMERGE_DEFAULT_OPTS with the jobs and load-average parm.



slow down guys, with this tempo I will beat Anon-E-moose's time eazyly, kheh

But on serious note, those two options are supposed to reside in /etc/make.config?
Where can I read more about this? At least where can I find examples of PORTAGE_NICENESS and EMERGE_DEFAULT_OPTS?

Thank you
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Mon Mar 07, 2011 2:44 pm    Post subject: Reply with quote

while true wrote:
disi
Quote:
You can also play around with PORTAGE_NICENESS to increase or decrease the nicelevel of emerge...


d2_racing
Quote:
The other part is the EMERGE_DEFAULT_OPTS with the jobs and load-average parm.



slow down guys, with this tempo I will beat Anon-E-moose's time eazyly, kheh

But on serious note, those two options are supposed to reside in /etc/make.config?
Where can I read more about this? At least where can I find examples of PORTAGE_NICENESS and EMERGE_DEFAULT_OPTS?

Thank you


PORTAGE_NICENESS is easy. It can go from -19 to 19 (-19 is OS killer, 0 is normal, 19 is lowest priority)
http://linux.die.net/man/2/nice

//edit: EMERGE_DEFAULT_OPTS can hold anything you find in "man emerge" as default options when you run emerge <whatever>
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
skellr
l33t
l33t


Joined: 18 Jun 2005
Posts: 706
Location: The Village

PostPosted: Mon Mar 07, 2011 3:51 pm    Post subject: Reply with quote

You can also limit the amount of jobs by load. "-j -l1"

edit: gtk+ has a good build system to test with. :)
Back to top
View user's profile Send private message
d2_racing
Moderator
Moderator


Joined: 25 Apr 2005
Posts: 13046
Location: Ste-Foy,Canada

PostPosted: Mon Mar 07, 2011 6:25 pm    Post subject: Reply with quote

I found this : http://blogs.gentoo.org/zmedico/2010/03/03/portage_parallel_builds/
_________________
Sysadmin of GentooQuébec.org
Wiki
Signature
IRC on Freenode : #gentoo-quebec
Back to top
View user's profile Send private message
while true
Guru
Guru


Joined: 07 Apr 2010
Posts: 363
Location: Ljubljana, Slovenia

PostPosted: Mon Mar 07, 2011 8:37 pm    Post subject: Reply with quote

wow!
Code:
grom@C6820s ~ $ cat /etc/make.conf
...
...
MAKEOPTS="-j3"
PORTAGE_NICENESS="-4"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=2.0"


This, I believe, did speed up things on my
hp laptop 2.2ghz dual-core 2gb ram 1gb swap.
(monitoring in htop in ratpoison wm:)
cpu was on average 87-100%
ram 90% (all together with used/buffers/cache)
swap 0MB

The only thing that I do not quite understand is --load-average.
Quote:
--load-average LOAD
Specifies that no new builds should be started if there are
other builds running and the load average is at least LOAD (a
floating-point number). This option is recommended for use in
combination with --jobs in order to avoid excess load. See
make(1) for information about analogous options that should
be configured via MAKEOPTS in make.conf(5).

Can I ask you guys to explain this to me?

Thank you
_________________
Kind regards, Goran Mitic

alive
while true
kick ass
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum