View previous topic :: View next topic |
Author |
Message |
d2_racing Bodhisattva
Joined: 25 Apr 2005 Posts: 13047 Location: Ste-Foy,Canada
|
Posted: Fri Mar 05, 2010 2:48 am Post subject: |
|
|
In fact, that's a great feature. |
|
Back to top |
|
|
nikaya Veteran
Joined: 13 May 2006 Posts: 1471 Location: Germany
|
Posted: Fri Mar 05, 2010 10:30 am Post subject: |
|
|
Ahhh. "l" as in --load. But I didn't know that it can be set in MAKEOPTS. Thought it must be specified seperately.
If I set --load-average to MAKEOPTS and to emerge isn't there a risk of overloading by setting it twice? _________________ Notes on Dhamma
How to waste your time: look for an explanation of consciousness, ask to know what feeling is. (Nanavira Thera) |
|
Back to top |
|
|
sera Retired Dev
Joined: 29 Feb 2008 Posts: 1017 Location: CET
|
Posted: Fri Mar 05, 2010 12:13 pm Post subject: |
|
|
d2_racing wrote: | In fact, that's a great feature. |
And how much faster is an emerge -e world now? Is it really that great? Numbers please |
|
Back to top |
|
|
d2_racing Bodhisattva
Joined: 25 Apr 2005 Posts: 13047 Location: Ste-Foy,Canada
|
Posted: Fri Mar 05, 2010 2:03 pm Post subject: |
|
|
I save maybe 5-10 minutes during the process. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9507 Location: beyond the rim
|
Posted: Fri Mar 05, 2010 2:22 pm Post subject: |
|
|
nikaya wrote: | Ahhh. "l" as in --load. But I didn't know that it can be set in MAKEOPTS. Thought it must be specified seperately.
If I set --load-average to MAKEOPTS and to emerge isn't there a risk of overloading by setting it twice? |
As I said before, the --jobs and --load-average options for emerge and make are completely independant of each other. So for example
Code: | MAKEOPTS="-j4 -l3.0" emerge --jobs=2 --load-average=2 | means that emerge will start a maximum of two parallel builds if the load average is below 2.0, but within each build process make can start up to four parallel commands as long as the load avgerage is below 3.0. So effectively emerge would only start a second build if the first one isn't cpu-limited at the moment, e.g. because it forces -j1 or is IO-limited. (All assuming the system is completely idle otherwise).
Last edited by Genone on Fri Mar 05, 2010 2:25 pm; edited 1 time in total |
|
Back to top |
|
|
costel78 Guru
Joined: 20 Apr 2007 Posts: 402
|
Posted: Fri Mar 05, 2010 2:23 pm Post subject: |
|
|
Ok, I tested paludis.
It is great when you compile a single package because prepare, install qmerge stage of ebuild are a faster than portage. Also memory needed is lower than portage.
Obviously, there are almost no change during compile phase - the hardest part of building and install of a package.
But when I compiled a "set" (system, world or installed) using portage that was faster than paludis in order of magnitude, simple because paludis lack the parallel build feature.
Due to dependencies, of course there are room for better. That's why in the begin of thread asked about implement parallel build in paludis.
O, and using --load-average both in emerge command (or in EMERGE_DEFAULT_OPTS as suggested previous) and in MAKEOPTS do no harm.
emerge does not start to build a new PACKAGE if system load average is greater than "n" and make does not start (or start only one, not "m" from jobs) to compile a NEW FILE from sources if system load if greater than "n".
For me, EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=7.0 --keep-going --with-bdeps y" and MAKEOPTS="-j8 --load-average=11" seems to be the better
Thank you everyone for your support and interest.
Hope we resurrect this thread soon when paludis will implement a better parallel scheduler than portage (wish list)
For now, I stick with portage. _________________ Sorry for my English. I'm still learning this language. |
|
Back to top |
|
|
seren Guru
Joined: 27 Aug 2005 Posts: 448 Location: Wisconsin
|
Posted: Sun Mar 07, 2010 9:24 pm Post subject: |
|
|
Yamakuzure wrote: | d2_racing wrote: | So with a CoreI7, you need to use --load-average=8.0, because I assume that it handle the hyperthreading. | AFAIK Core-X CPUs don't use Hyperthreading any more, because they have multiple cores. /proc/cpuinfo on my x86-Xeon-1 (with Hyperthreading) says I have two CPUs, and the Core-2-Duo on my laptop says the same. Do i7 have HT? What says cpuinfo? I'd really liked to know. |
Core i7 ,i5 and i3 DO use HT. So it would be best to use -j9 with a quad i7/i5 and -j5 on a dual i5/i3. Hyperthreading may be only a small gain I think estimates are in the range of 10-15% but if you don't saturate the all threads it will not allow the cpu to reach 100% load. You are right about Core-2 and all Conroe / Merom / Penryn / Wolfdale / Woodcrest / Yorkfield / Kentsfield and the Xeon Dunnington etc etc have no HT. Here is Wikipedias low down on Nehalem http://en.wikipedia.org/wiki/Intel_Nehalem Quoting that article Quote: | Simultaneous multithreading (SMT) by multiple cores which enables two threads per core. Intel calls this hyper-threading. Simultaneous multithreading has not been present on a consumer desktop Intel processor since 2006 with the Pentium 4 and Pentium XE. Intel reintroduced SMT with their Atom Architecture. | and more on SMT http://en.wikipedia.org/wiki/Simultaneous_multithreading . If anyone has different results I would be interested.
Also with my experience with using --jobs is that sometimes portage simply can not run parallel builds and this leaves you either not using all cores because you set MAKEOPTS according to combine --jobs, eg on a quad core no HT; set MAKEOPTS=-j2 and --jobs=2 so if running full load you end up with a total of 4 jobs, but when portage cannot run parallel builds you end up with only 2 threads, this goes the other way when you set MAKEOPTS=-j4 and --jobs=2 because when running parallel you have a total of 8 threads which taxes everything to a point where jobs are idle. This is the paradox I keep running in to and because of this on my quad 9550 I use MAKEOPTS=-j5 and let portage do its work without --jobs unless I do builds with LOTS of packages and then I manually set MAKEOPTS=-j3 emerge -j3 (I have an alias for this) just to be sure I use my cpu to its fullest at the cost of an idle make process now and then if anyone has a better solution I'm all ears.
Quote: | For me, EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=7.0 --keep-going --with-bdeps y" and MAKEOPTS="-j8 --load-average=11" seems to be the better
Thank you everyone for your support and interest. |
I don't think this is a good way for one you are using redundant load-averages and I doubt --load-average=11 is being used. You should allow portage to control the load average because it will manage load between all phases and not just the make. In my opinion you should use a combination of MAKEOPTS and --jobs equal +1 to the amount of processes your cpu can handle, in your case this is 9 total threads. With the configuration you are using you are allowing a total of 48!! threads at times and with your load average limit I imagine you don't run more than 3 jobs with a total of 24 threads at once (with many jobs running idle). I think a better solution would be to use EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=7.0 --keep-going --with-bdeps y" and MAKEOPTS="-j4" while this won't utilize your cpu when compiling one packgage you can use set MAKEOPTS="-j9" emerge foo via the command line and it will override make.conf. Running 8 jobs with 6 concurrent builds is way way overkill. |
|
Back to top |
|
|
costel78 Guru
Joined: 20 Apr 2007 Posts: 402
|
Posted: Mon Mar 08, 2010 1:05 pm Post subject: |
|
|
Fortunately, both load-average options prevent the things going to far avoid that overkill situation you describe to happen.
Using this settings I got the shortest time to compile entire world set.
And the max system load was 13.6 (kdelibs and mozilla-thunderbird compiled at the same time)
On the hand I need 8 in MAKEOPTS for packages which benefit by parallel make, on the other hand jobs=6 or 8 in emerge for small packages like perl-*, modular xorg or event kde.
I don't pretend that these settings are good for everyone. I said that seems to be the best for me. It depends by packages in world, use flags etc. _________________ Sorry for my English. I'm still learning this language. |
|
Back to top |
|
|
seren Guru
Joined: 27 Aug 2005 Posts: 448 Location: Wisconsin
|
Posted: Tue Mar 09, 2010 8:24 am Post subject: |
|
|
It really makes no sense to push something beyond what it can even do, and then use a governor to limit your excess jobs. Please because I'm 99% sure you are suffering placebo, do this Code: | time emerge `ls -d /var/db/pkg/dev-perl/* | sed -e "s/\/var\/db\/pkg\//\=/g"` -av1 | then change your settings to what I recommend and tell me which is faster.
You will be amazed that ricing wont get you anywhere. I have been doing this for 5 years and tried more than I can remember just to speed things up and eventually realized (as will you) that there is a point where less can be more.
On my machine with a Quad Q9550 (4.25ghz overclocked) I ran that (here I have 21 packages)
Code: |
[ebuild R ] dev-perl/HTML-Tagset-3.20 0 kB
[ebuild R ] dev-perl/Class-MethodMaker-2.15 0 kB
[ebuild R ] dev-perl/Net-SSLeay-1.36 0 kB
[ebuild R ] dev-perl/XML-Parser-2.36-r1 0 kB
[ebuild R ] dev-perl/YAML-Tiny-1.41 0 kB
[ebuild R ] dev-perl/TermReadKey-2.30 0 kB
[ebuild R ] dev-perl/HTML-Parser-3.64 USE="-test" 0 kB
[ebuild R ] dev-perl/IO-Socket-SSL-1.31 USE="-idn" 0 kB
[ebuild R ] dev-perl/Digest-SHA1-2.12 0 kB
[ebuild R ] dev-perl/Shell-EnvImporter-1.07 0 kB
[ebuild R ] dev-perl/URI-1.52 0 kB
[ebuild R ] dev-perl/Digest-HMAC-1.02 0 kB
[ebuild R ] dev-perl/HTML-Tree-3.23 0 kB
[ebuild R ] dev-perl/Net-SMTP-SSL-1.01 0 kB
[ebuild R ] dev-perl/PortageXS-0.02.10-r1 USE="-minimal" 0 kB
[ebuild R ] dev-perl/Authen-SASL-2.13 USE="-kerberos" 0 kB
[ebuild R ] dev-perl/DateManip-5.54 0 kB
[ebuild R ] dev-perl/Error-0.17.016 USE="-test" 0 kB
[ebuild R ] dev-perl/Crypt-SSLeay-0.57 0 kB
[ebuild R ] dev-perl/libwww-perl-5.834-r1 USE="ssl" 0 kB
[ebuild R ] dev-perl/Locale-gettext-1.05-r1 0 kB
|
with the settings of:
Code: |
MAKEOPTS="-j8"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --complete-graph -j6 --load-average=11" |
and got this result
Code: | 75.62s user 31.18s system 194% cpu 54.936 total |
I then ran it again with these settings
Code: | MAKEOPTS="-j5"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --complete-graph -j3 --load-average=11" |
This was the result
Code: | 74.04s user 30.87s system 177% cpu 59.265 total |
You can see the difference is negligible I have an extra second to spare so Its not really any improvement I would hope for at least 10-25% to even consider it.
Please post your results because I'm curious.
Last edited by seren on Tue Mar 09, 2010 7:36 pm; edited 1 time in total |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2280 Location: Adendorf, Germany
|
Posted: Tue Mar 09, 2010 12:32 pm Post subject: |
|
|
@seren: thanks alot for the information!
Just a quick note: Portage will only do parallel builds when it has to build packages that do _not_ depend upon each other. _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
seren Guru
Joined: 27 Aug 2005 Posts: 448 Location: Wisconsin
|
Posted: Tue Mar 09, 2010 7:35 pm Post subject: |
|
|
Yamakuzure wrote: | @seren: thanks alot for the information!
Just a quick note: Portage will only do parallel builds when it has to build packages that do _not_ depend upon each other. |
Yes sir that is probably the one thing that gets in the way of parallel compiles. |
|
Back to top |
|
|
costel78 Guru
Joined: 20 Apr 2007 Posts: 402
|
Posted: Wed Mar 10, 2010 7:30 pm Post subject: |
|
|
@seren: I put the info requested, also here
time emerge `ls -d /var/db/pkg/dev-perl/* | sed -e "s/\/var\/db\/pkg\//\=/g"` -v1
Code: | These are the packages that would be merged, in order:
Calculating dependencies ... done!
[ebuild R ] dev-perl/HTML-Tagset-3.20 0 kB
[ebuild R ] dev-perl/XML-NamespaceSupport-1.10 0 kB
[ebuild R ] dev-perl/IO-stringy-2.110 0 kB
[ebuild R ] dev-perl/Net-Daemon-0.43 0 kB
[ebuild R ] dev-perl/Net-SSLeay-1.36 0 kB
[ebuild R ] dev-perl/TimeDate-1.20 0 kB
[ebuild R ] dev-perl/XML-Parser-2.36-r1 0 kB
[ebuild R ] dev-perl/Apache-Reload-0.10 0 kB
[ebuild R ] dev-perl/Apache-Test-1.31 0 kB
[ebuild R ] dev-perl/Authen-PAM-0.16 0 kB
[ebuild R ] dev-perl/Crypt-PasswdMD5-1.3 0 kB
[ebuild R ] dev-perl/Geo-IP-1.38 0 kB
[ebuild R ] dev-perl/Locale-gettext-1.05-r1 0 kB
[ebuild R ] dev-perl/perl-tk-804.028-r3 0 kB
[ebuild R ] dev-perl/TermReadKey-2.30 0 kB
[ebuild R ] dev-perl/XML-Generator-1.03 0 kB
[ebuild R ] dev-perl/YAML-Tiny-1.41 0 kB
[ebuild R ] dev-perl/HTML-Parser-3.64 USE="-test" 0 kB
[ebuild R ] dev-perl/URI-1.52 0 kB
[ebuild R ] dev-perl/XML-SAX-0.96 0 kB
[ebuild R ] dev-perl/Digest-SHA1-2.12 0 kB
[ebuild R ] dev-perl/IO-Socket-SSL-1.32 USE="-idn" 0 kB
[ebuild R ] dev-perl/MailTools-2.06 USE="-test" 0 kB
[ebuild R ] dev-perl/PlRPC-0.2020-r1 0 kB
[ebuild R ] dev-perl/Archive-Zip-1.30 0 kB
[ebuild R ] dev-perl/DateManip-5.54 0 kB
[ebuild R ] dev-perl/Error-0.17.016 USE="-test" 0 kB
[ebuild R ] dev-perl/DBI-1.609 0 kB
[ebuild R ] dev-perl/Digest-HMAC-1.02 0 kB
[ebuild R ] dev-perl/HTML-Tree-3.23 0 kB
[ebuild R ] dev-perl/XML-LibXML-1.70 0 kB
[ebuild R ] dev-perl/Net-SMTP-SSL-1.01 0 kB
[ebuild R ] dev-perl/Authen-SASL-2.13 USE="-kerberos" 0 kB
[ebuild R ] dev-perl/DBD-mysql-4.01.3 0 kB
[ebuild R ] dev-perl/DBD-Pg-2.16.0-r1 0 kB
[ebuild R ] dev-perl/XML-Simple-2.18 0 kB
[ebuild R ] dev-perl/Crypt-SSLeay-0.57 0 kB
[ebuild R ] dev-perl/libwww-perl-5.834-r1 USE="ssl" 0 kB
[ebuild R ] dev-perl/MIME-tools-5.427 0 kB
Total: 39 packages (39 reinstalls), Size of downloads: 0 kB
>>> Verifying ebuild manifests
>>> Emerging (1 of 39) dev-perl/HTML-Tagset-3.20
>>> Emerging (2 of 39) dev-perl/XML-NamespaceSupport-1.10
>>> Emerging (3 of 39) dev-perl/IO-stringy-2.110
>>> Installing (1 of 39) dev-perl/HTML-Tagset-3.20
>>> Installing (2 of 39) dev-perl/XML-NamespaceSupport-1.10
>>> Installing (3 of 39) dev-perl/IO-stringy-2.110
>>> Emerging (4 of 39) dev-perl/Net-Daemon-0.43
>>> Emerging (5 of 39) dev-perl/Net-SSLeay-1.36
>>> Installing (4 of 39) dev-perl/Net-Daemon-0.43
>>> Emerging (6 of 39) dev-perl/TimeDate-1.20
>>> Emerging (7 of 39) dev-perl/XML-Parser-2.36-r1
>>> Installing (6 of 39) dev-perl/TimeDate-1.20
>>> Emerging (8 of 39) dev-perl/Apache-Reload-0.10
>>> Emerging (9 of 39) dev-perl/Apache-Test-1.31
>>> Installing (7 of 39) dev-perl/XML-Parser-2.36-r1
>>> Emerging (10 of 39) dev-perl/Authen-PAM-0.16
>>> Installing (5 of 39) dev-perl/Net-SSLeay-1.36
>>> Installing (8 of 39) dev-perl/Apache-Reload-0.10
>>> Emerging (11 of 39) dev-perl/Crypt-PasswdMD5-1.3
>>> Installing (9 of 39) dev-perl/Apache-Test-1.31
>>> Emerging (12 of 39) dev-perl/Geo-IP-1.38
>>> Installing (10 of 39) dev-perl/Authen-PAM-0.16
>>> Emerging (13 of 39) dev-perl/Locale-gettext-1.05-r1
>>> Installing (11 of 39) dev-perl/Crypt-PasswdMD5-1.3
>>> Emerging (14 of 39) dev-perl/perl-tk-804.028-r3
>>> Installing (12 of 39) dev-perl/Geo-IP-1.38
>>> Installing (13 of 39) dev-perl/Locale-gettext-1.05-r1
>>> Emerging (15 of 39) dev-perl/TermReadKey-2.30
>>> Emerging (16 of 39) dev-perl/XML-Generator-1.03
>>> Failed to emerge dev-perl/perl-tk-804.028-r3, Log file:
>>> '/var/tmp/portage/dev-perl/perl-tk-804.028-r3/temp/build.log'
>>> Installing (16 of 39) dev-perl/XML-Generator-1.03
>>> Installing (15 of 39) dev-perl/TermReadKey-2.30
>>> Jobs: 15 of 39 complete, 1 failed Load avg: 1.17, 0.54, 0.24
*** Resuming merge...
These are the packages that would be merged, in order:
Calcu ... done!
[ebuild R ] dev-perl/YAML-Tiny-1.41 0 kB
[ebuild R ] dev-perl/HTML-Parser-3.64 USE="-test" 0 kB
[ebuild R ] dev-perl/URI-1.52 0 kB
[ebuild R ] dev-perl/Digest-SHA1-2.12 0 kB
[ebuild R ] dev-perl/IO-Socket-SSL-1.32 USE="-idn" 0 kB
[ebuild R ] dev-perl/MailTools-2.06 USE="-test" 0 kB
[ebuild R ] dev-perl/PlRPC-0.2020-r1 0 kB
[ebuild R ] dev-perl/Digest-HMAC-1.02 0 kB
[ebuild R ] dev-perl/HTML-Tree-3.23 0 kB
[ebuild R ] dev-perl/Archive-Zip-1.30 0 kB
[ebuild R ] dev-perl/Net-SMTP-SSL-1.01 0 kB
[ebuild R ] dev-perl/DBI-1.609 0 kB
[ebuild R ] dev-perl/Authen-SASL-2.13 USE="-kerberos" 0 kB
[ebuild R ] dev-perl/DateManip-5.54 0 kB
[ebuild R ] dev-perl/Error-0.17.016 USE="-test" 0 kB
[ebuild R ] dev-perl/Crypt-SSLeay-0.57 0 kB
[ebuild R ] dev-perl/libwww-perl-5.834-r1 USE="ssl" 0 kB
[ebuild R ] dev-perl/MIME-tools-5.427 0 kB
[ebuild R ] dev-perl/DBD-mysql-4.01.3 0 kB
[ebuild R ] dev-perl/XML-SAX-0.96 0 kB
[ebuild R ] dev-perl/XML-LibXML-1.70 0 kB
[ebuild R ] dev-perl/XML-Simple-2.18 0 kB
[ebuild R ] dev-perl/DBD-Pg-2.16.0-r1 0 kB
Total: 23 packages (23 reinstalls), Size of downloads: 0 kB
>>> Emerging (1 of 23) dev-perl/YAML-Tiny-1.41
>>> Emerging (2 of 23) dev-perl/HTML-Parser-3.64
>>> Installing (1 of 23) dev-perl/YAML-Tiny-1.41
>>> Installing (2 of 23) dev-perl/HTML-Parser-3.64
>>> Emerging (3 of 23) dev-perl/URI-1.52
>>> Emerging (4 of 23) dev-perl/Digest-SHA1-2.12
>>> Emerging (5 of 23) dev-perl/IO-Socket-SSL-1.32
>>> Installing (3 of 23) dev-perl/URI-1.52
>>> Installing (4 of 23) dev-perl/Digest-SHA1-2.12
>>> Installing (5 of 23) dev-perl/IO-Socket-SSL-1.32
>>> Emerging (6 of 23) dev-perl/MailTools-2.06
>>> Emerging (7 of 23) dev-perl/PlRPC-0.2020-r1
>>> Installing (6 of 23) dev-perl/MailTools-2.06
>>> Installing (7 of 23) dev-perl/PlRPC-0.2020-r1
>>> Emerging (8 of 23) dev-perl/Digest-HMAC-1.02
>>> Emerging (9 of 23) dev-perl/HTML-Tree-3.23
>>> Emerging (10 of 23) dev-perl/Archive-Zip-1.30
>>> Installing (8 of 23) dev-perl/Digest-HMAC-1.02
>>> Installing (9 of 23) dev-perl/HTML-Tree-3.23
>>> Emerging (11 of 23) dev-perl/Net-SMTP-SSL-1.01
>>> Installing (10 of 23) dev-perl/Archive-Zip-1.30
>>> Installing (11 of 23) dev-perl/Net-SMTP-SSL-1.01
>>> Emerging (12 of 23) dev-perl/DBI-1.609
>>> Emerging (13 of 23) dev-perl/Authen-SASL-2.13
>>> Emerging (14 of 23) dev-perl/DateManip-5.54
>>> Installing (13 of 23) dev-perl/Authen-SASL-2.13
>>> Emerging (15 of 23) dev-perl/Error-0.17.016
>>> Installing (14 of 23) dev-perl/DateManip-5.54
>>> Installing (15 of 23) dev-perl/Error-0.17.016
>>> Emerging (16 of 23) dev-perl/Crypt-SSLeay-0.57
>>> Installing (12 of 23) dev-perl/DBI-1.609
>>> Installing (16 of 23) dev-perl/Crypt-SSLeay-0.57
>>> Emerging (17 of 23) dev-perl/XML-SAX-0.96
>>> Emerging (18 of 23) dev-perl/libwww-perl-5.834-r1
>>> Emerging (19 of 23) dev-perl/DBD-mysql-4.01.3
>>> Installing (17 of 23) dev-perl/XML-SAX-0.96
>>> Emerging (20 of 23) dev-perl/XML-LibXML-1.70
>>> Installing (18 of 23) dev-perl/libwww-perl-5.834-r1
>>> Emerging (21 of 23) dev-perl/MIME-tools-5.427
>>> Installing (21 of 23) dev-perl/MIME-tools-5.427
>>> Installing (19 of 23) dev-perl/DBD-mysql-4.01.3
>>> Emerging (22 of 23) dev-perl/DBD-Pg-2.16.0-r1
>>> Installing (20 of 23) dev-perl/XML-LibXML-1.70
>>> Installing (22 of 23) dev-perl/DBD-Pg-2.16.0-r1
>>> Emerging (23 of 23) dev-perl/XML-Simple-2.18
>>> Installing (23 of 23) dev-perl/XML-Simple-2.18
* The following package has failed to build or install:
*
* ('ebuild', '/', 'dev-perl/perl-tk-804.028-r3', 'merge'), Log file:
* '/var/tmp/portage/dev-perl/perl-tk-804.028-r3/temp/build.log'
*
* GNU info directory index is up-to-date.
real 4m29.104s
user 6m4.136s
sys 0m48.007s
|
_________________ Sorry for my English. I'm still learning this language. |
|
Back to top |
|
|
|