Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Paludis --multitask support for compiling
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Fri Feb 26, 2010 12:58 pm    Post subject: Paludis --multitask support for compiling Reply with quote

Hello,

First, I hope that it's the right place for my question as paludis is in portage.

I read that paludis is faster than portage per package build.
The problem, as far I was able to find and read on Net, paludis lack of --jobs --load-average=n support (portage equivalent)
All I found was --multitask option, but only for parallel sync.

Is the assumption that portage --jobs=10 --load-average=8 --keep-going -e world will be faster than paludis equivalent ?
On http://paludis.pioto.org I was unable to find regarding this or future plans.
Hardware configuration include an Intel Core i7 CPU, with 4GB RAM and /var/tmp/portage mounted as tmpfs in RAM.

Thank you!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Fri Feb 26, 2010 1:56 pm    Post subject: Reply with quote

Quote:
I read that paludis is faster than portage per package build.

Even if the case this is no reason to change the package manager.

Quote:
paludis lack of --jobs --load-average=n support (portage equivalent)

Portage just pass down unknown arguments, paludis does not. Use paludis' bashrc to accomplish your goal. In this case it's as simple as setting MAKE_OPTS.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Fri Feb 26, 2010 2:57 pm    Post subject: Reply with quote

My final goal is to shorten compile time. That's why I begun to read about paludis and put questions about it.
If it won't prove to be better, I won't switch.
sera wrote:

Portage just pass down unknown arguments, paludis does not. Use paludis' bashrc to accomplish your goal. In this case it's as simple as setting MAKE_OPTS.

I'm not sure if I understand you.
When I said "paludis lack of --jobs --load-average=n support" I referred to the fact that while paludis compile in order mc, p7zip and bzip2 for example one by one, by build them in parallel portage seems to be faster. That's why I asked about --multitask improvements in paludis.

Sorry for my English, I'm not a native English speaker.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Fri Feb 26, 2010 3:55 pm    Post subject: Reply with quote

Compile time has virtually nothing to do with the package manager but with the tools used under the hood. The two options -j and -l are for the tool make. Paludis requires you to configure make directly by setting the environment variable MAKE_OPTS in /etc/paludis/bashrc. The same goes for every other tool.

If you want to significantly reduce compile time reduce the optimization level, use ccache or distcc.

Quote:
If it won't prove to be better, I won't switch.

Uh, there are many reason why paludis could be a better fit for your needs, which all have nothing to do with speed.

Regarding speed paludis and portage are slow in different places. ;)
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Fri Feb 26, 2010 4:21 pm    Post subject: Reply with quote

I generally agree with you.
But by reducing compile time I should said reduce "TOTAL" compile time with better parallelism when compile entire world set for example.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Fri Feb 26, 2010 4:28 pm    Post subject: Reply with quote

sera wrote:
The two options -j and -l are for the tool make.

The --jobs and --load-averge options of emerge have nothing to do with make (for make options you use the MAKEOPTS variable in make.conf).
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Fri Feb 26, 2010 6:25 pm    Post subject: Reply with quote

Genone wrote:
sera wrote:
The two options -j and -l are for the tool make.

While make supports the same options they are not directly related.


It can't mean how may ebuild phases to schedule at once. Unfortunately man 5 portage does not contain those options. Could you please elaborate a little more.

@costel78: I've never done the test however from my personal experience I don't think you will see a significant difference between emerge -e world and it's equivalent. If you do decide to do some tests, please let us know the results.

Speed primarily matters for interactive usage. For my usage pattern paludis is in front.
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1931

PostPosted: Fri Feb 26, 2010 6:36 pm    Post subject: Reply with quote

man emerge wrote:
-j [JOBS], --jobs[=JOBS]
    Specifies the number of packages to build simultaneously. If this option is given without an argument, emerge will not limit the number of jobs that can run simultaneously. Also see the related --load-average option. Note that interactive packages currently force a setting of --jobs=1. This issue can be temporarily avoided by specifying --accept-properties=-interactive.

--load-average=LOAD
    Specifies that no new builds should be started if there are other builds running and the load average is at least LOAD (a floating-point number). This option is recommended for use in combination with --jobs in order to avoid excess load. See make(1) for information about analogous options that should be configured via MAKEOPTS in make.conf(5).
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Fri Feb 26, 2010 7:45 pm    Post subject: Reply with quote

This must be a portage 2.2 feature then. Thanks AM088 for looking it up for me. Maybe it's about time too look at :2.2 in more depth. The part about make was a misunderstanding.

Current paludis does not have a similar feature and as far as I know there is nothing like this planned in the near future.
Back to top
View user's profile Send private message
d2_racing
Bodhisattva
Bodhisattva


Joined: 25 Apr 2005
Posts: 13047
Location: Ste-Foy,Canada

PostPosted: Fri Feb 26, 2010 8:36 pm    Post subject: Reply with quote

Genone wrote:
The --jobs and --load-averge options of emerge have nothing to do with make (for make options you use the MAKEOPTS variable in make.conf).


In fact and the jobs and load-average are only use with Portage.

I can confirm what Genone says, because when Portage 2.2 hit the tree back in 2008-2009, I asked Zmedico about those params too.


Last edited by d2_racing on Sat Feb 27, 2010 4:07 am; edited 1 time in total
Back to top
View user's profile Send private message
yngwin
Retired Dev
Retired Dev


Joined: 19 Dec 2002
Posts: 4572
Location: Suzhou, China

PostPosted: Fri Feb 26, 2010 10:03 pm    Post subject: Reply with quote

Another thing to consider is that Gentoo officially only supports portage.
_________________
"Those who deny freedom to others deserve it not for themselves." - Abraham Lincoln
Free Culture | Defective by Design | EFF
Back to top
View user's profile Send private message
d2_racing
Bodhisattva
Bodhisattva


Joined: 25 Apr 2005
Posts: 13047
Location: Ste-Foy,Canada

PostPosted: Sat Feb 27, 2010 4:08 am    Post subject: Reply with quote

In fact, but Paludis is a popular alternative.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sat Feb 27, 2010 11:09 am    Post subject: Reply with quote

To clarify things my settings are
in /etc/make.conf:
Code:

MAKEOPTS="-j8 --load-average=11"

and to command to rebuild entire world set:
Code:

emerge --jobs=10 --load-average=8 --keep-going -e @installed


Due to dependencies for almost 50-55% of packages my CPU stand at 30-40% usage. That bother me.
And bottom line, for now paludis doesn't seem to solve now or in near future that problem.
Problem solved (somehow) for me.

Thank you everyone!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
d2_racing
Bodhisattva
Bodhisattva


Joined: 25 Apr 2005
Posts: 13047
Location: Ste-Foy,Canada

PostPosted: Sat Feb 27, 2010 4:12 pm    Post subject: Reply with quote

Maybe you could play with this too inside your /etc/make.conf :

Code:

EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8.0 --with-bdeps y"
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sat Feb 27, 2010 4:23 pm    Post subject: Reply with quote

d2_racing wrote:
Maybe you could play with this too inside your /etc/make.conf :

Code:

EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8.0 --with-bdeps y"


Interesting idea. This could improve portage package scheduler to be able to compile more packages at same time.
I'll try it tomorrow.
Thank you very much! I really appreciate.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
d2_racing
Bodhisattva
Bodhisattva


Joined: 25 Apr 2005
Posts: 13047
Location: Ste-Foy,Canada

PostPosted: Sat Feb 27, 2010 9:00 pm    Post subject: Reply with quote

No problem, that tweak is compatible with Portage 2.1 and 2.2 I think.
Back to top
View user's profile Send private message
nikaya
Veteran
Veteran


Joined: 13 May 2006
Posts: 1471
Location: Germany

PostPosted: Sun Feb 28, 2010 1:12 pm    Post subject: Reply with quote

I did not really understand the option --load-average. What does this closely mean? And what digit is the best for this option in conjunction with --jobs?
_________________
Notes on Dhamma
How to waste your time: look for an explanation of consciousness, ask to know what feeling is. (Nanavira Thera)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54028
Location: 56N 3W

PostPosted: Sun Feb 28, 2010 2:28 pm    Post subject: Reply with quote

Moved from Portage & Programming to Unsupported Software.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Wed Mar 03, 2010 11:03 am    Post subject: Reply with quote

http://en.wikipedia.org/wiki/Load_average wrote:
Systems calculate the load average as the exponentially damped/weighted moving average of the load number. The three values of load average refer to the past one, five, and fifteen minutes of system operation.

For single-CPU systems that are CPU-bound, one can think of load average as a percentage of system utilization during the respective time period. For systems with multiple CPUs, one must divide the number by the number of processors in order to get a comparable percentage.

For example, one can interpret a load average of "1.73 0.50 7.98" on a single-CPU system as:
  • during the last minute, the CPU was overloaded by 73% (1 CPU with 1.73 runnable processes, so that 0.73 processes had to wait for a turn)
  • during the last 5 minutes, the CPU was underloaded 50% (no processes had to wait for a turn)
  • during the last 15 minutes, the CPU was overloaded 698% (1 CPU with 7.98 runnable processes, so that 6.98 processes had to wait for a turn)
This means that this CPU could have handled all of the work scheduled for the last minute if it were 1.73 times as fast, or if there were two (1.73 rounded up) times as many CPUs, but that over the last five minutes it was twice as fast as necessary to prevent runnable processes from waiting their turn.
So equaling --load-average with the number of your CPU cores means you are allowing 100% cpu utilization, or, that portage won't start another parallel build when 100% is reached.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
d2_racing
Bodhisattva
Bodhisattva


Joined: 25 Apr 2005
Posts: 13047
Location: Ste-Foy,Canada

PostPosted: Wed Mar 03, 2010 12:51 pm    Post subject: Reply with quote

So with a CoreI7, you need to use --load-average=8.0, because I assume that it handle the hyperthreading.
Back to top
View user's profile Send private message
nikaya
Veteran
Veteran


Joined: 13 May 2006
Posts: 1471
Location: Germany

PostPosted: Thu Mar 04, 2010 9:35 am    Post subject: Reply with quote

Thanks, that cleared something. :)
_________________
Notes on Dhamma
How to waste your time: look for an explanation of consciousness, ask to know what feeling is. (Nanavira Thera)
Back to top
View user's profile Send private message
Yamakuzure
Advocate
Advocate


Joined: 21 Jun 2006
Posts: 2280
Location: Adendorf, Germany

PostPosted: Thu Mar 04, 2010 11:31 am    Post subject: Reply with quote

d2_racing wrote:
So with a CoreI7, you need to use --load-average=8.0, because I assume that it handle the hyperthreading.
AFAIK Core-X CPUs don't use Hyperthreading any more, because they have multiple cores. /proc/cpuinfo on my x86-Xeon-1 (with Hyperthreading) says I have two CPUs, and the Core-2-Duo on my laptop says the same. Do i7 have HT? What says cpuinfo? I'd really liked to know.
_________________
Important German:
  1. "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
  2. "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Thu Mar 04, 2010 3:51 pm    Post subject: Reply with quote

Hyperthreading only helps when the threads are accessing the same data. Something like this would probably work better:
Code:
MAKEOPTS="-j8 -l8" emerge --load-average 4 --jobs 4
Back to top
View user's profile Send private message
nikaya
Veteran
Veteran


Joined: 13 May 2006
Posts: 1471
Location: Germany

PostPosted: Fri Mar 05, 2010 1:42 am    Post subject: Reply with quote

Ant_P wrote:

Code:
MAKEOPTS="-j8 -l8" ...

What is this "-l8"? Is it the letter "ell"? Never heard about it before.
_________________
Notes on Dhamma
How to waste your time: look for an explanation of consciousness, ask to know what feeling is. (Nanavira Thera)
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Fri Mar 05, 2010 2:36 am    Post subject: Reply with quote

From `man make`:
Code:
       -l [load], --load-average[=load]
            Specifies  that  no  new  jobs (commands) should be started if there are others jobs running and the load
            average is at least load (a floating-point number).  With no argument, removes a previous load limit.


So you'll get up to 4 emerge jobs started at once, but if some of those finish early the longer running ones get more CPU time.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum