Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage Features Questions
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 18, 2013 6:06 pm    Post subject: Portage Features Questions Reply with quote

Portage used to have a Paralel build option but I don't find it anylonger. Is it still supported in make.conf?

I've got a Quad Core and it took forever to install things with make_opts="-j5" Does it get bette with higher options or are their other methods to speed things up?

My system is an AMD x4-640 with 16GB of memory and /dev/sda is a 7200rpm Seagate.
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2244

PostPosted: Sat May 18, 2013 6:19 pm    Post subject: Reply with quote

FastTurtle ...

besdies the '-j{N}' in MAKEOPTS, which is for "parallel make", there is the '-j' '--jobs' which "specifies the number of packages to build simultaneously". This can be added to make.conf, eg:

Code:
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} -j8 --load-average=8.00"

The above will allow 8 package builds with maximim cpu load for 8 CPU's ... so, adjust as needed.

best ... khay
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 18, 2013 11:57 pm    Post subject: Reply with quote

Is that -j{N} the same as the make_opts -j{N}?

If so, means I can combine everything into a single line.
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1394
Location: Bardowick, Germany

PostPosted: Sun May 19, 2013 8:12 pm    Post subject: Reply with quote

No.
Code:
MAKEOPTS="-j9 -l4"
This will add those options to calls to 'make', so 'make' will compile up to 9 files parallel unless the system load is 4.0 or higher.
Code:
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --jobs=3 --load-average=2.0"
This will have portage to start up to 3 parallel emerges unless the system load is 2.0 or highter.

The problem is, that the system load "builds up". So starting another emerge will not raise the system load immediately. This is even more imminent if portage has to download files first.

If you chose too high settings for EMERGE_DEFAULT_OPTS, you might end up with an insanely high load, because the compiling will be done with at least one process and not hold back this least one.
_________________
Boycott systemd!
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 25, 2013 5:21 pm    Post subject: Reply with quote

Finally got around to testing your settings on an -e system rebuild and seeing it. Looks like it will make a difference. May need to tweak the load average higher since I have plenty of headroom (sysgaurd is showing around 45% and I want around 75-80% loading. High enough for the build while leaving enough headroom to not overheat and kick in the thermal throttling.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32027
Location: 56N 3W

PostPosted: Sat May 25, 2013 5:24 pm    Post subject: Reply with quote

FastTurtle,

It sounds as if your colling system may be inadequate, ineffective, or both.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 25, 2013 5:55 pm    Post subject: Reply with quote

Neddy: I'll have to check it later as the CPU Fan hasn't spun up as I've heard when I managed to push the CPU to 80% loading. Right now I'm only seeing 40% avg with portage reporting less the 2.00 avg.

I've got a full 16GB of memory and yet with everything, I'm only using 1/4 of it (4GiB) so I'm not pushing the system load anywhere's near what it'll handle. Looks like more adjustments ahead.

First thing I'll change is the make_opts and push it to 7. Worked pretty decent on loading the cpu when I tested it or do I want to push the jobs higher? I've got enough ram and loading that makes more sense as it'll push the cpu loading as needed.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32027
Location: 56N 3W

PostPosted: Sat May 25, 2013 6:51 pm    Post subject: Reply with quote

FastTurtle,

Too many parallel builds cause the CPU cache to be flushed as task switching takes place to give another ready to run job some CPU time.
The CPU cache is the fastest RAM you have, it really slows things down to keep reloading it from main memory.
Thats what sets a limit on build speed.

With a tiny cache, MAKEOPS="-j2" and --jobs=1 may be the best you can get. With lots of cores and a huge cache, you can do a lot better.
A good place to start is --jobs=2 and MAKEOPS="-jX" where X is your cores+1.

It also depends on what you are building. The kernel and glibc contain a lot of very small files, so you can build more of them at once and still have all the work fit into your CPU cache.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 25, 2013 7:11 pm    Post subject: Reply with quote

Looking at cat /proc/cpuinfo I'm seeing 2 cores at 800, 1 core at 1800 and the last hitting 3000 so I it's looking like either it's overheating or I've got the wrong CPU settings in the kernel (conservative instead of on demand).

Needy: good explanation and reminder about cache swapping. Could be part of the problem but after seeing those two cores at such low clocks, I'm beginning to think you're right on the HSF not being up to the task or something not seated right. Next is to drop the jobs down to 4 and see if that combined with -j5 in make opts gets me the best speeds.

So I'll shutdown and pull the HSF to check, then reboot, change jobs to 4 while keeping the -j5 and see if that helps.
_________________
AsRock Q87M with E3-1230 Xeon (Haswell)
16GB RAM - Geforce GT640 2GB
Crucial M500 120GB SSD (Windows Boot)
Seagate 250GB Barracuda (linux disk)- Samsung 1TB Spinpoint - WD 2TB Green - Seagate 3TB
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32027
Location: 56N 3W

PostPosted: Sat May 25, 2013 7:29 pm    Post subject: Reply with quote

FastTurtle,

Some build jobs are I/O limited, not CPU limited, so the two idle cores may be waiting for I/O to complete.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sat May 25, 2013 8:49 pm    Post subject: Reply with quote

Don't think the issue was I/O lag though it's possible. Changing the jobs to 4 seems to helped a lot - Testing with an -e system and it's already hit half of what I'd been at the last test in 10 mins. Much better and what I'm looking for.

I think the key was your comment about cache but I'll know when it hits glibc again. That'll answer things for sure as glibc really is a big package.

Just looked at RAM use and it's a quarter of what I'd been using before - 1G.

It's looking like your comment about cache really was the key since the start loaded something like 20 jobs and sped through them then it got into the larger builds.

Have to say it's been a learning experience today.

Question I've now got is how many jobs does portage normally run?
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2801
Location: Oceanside, Ca

PostPosted: Sat May 25, 2013 10:09 pm    Post subject: Reply with quote

FastTurtle I've played with MAKEOPTS="-jX" a lot. A good place to try and see what is a good X is in your kernel
Code:
make clean ; time make -jX

You'll probably find "-j6 or 7" best for building your kernel. For glibc and gcc, I never remember which but, I normally run -j8 on a amd 1055 6 core, one prefers a j5 the other j8.

If you watch your load , I use htop in a second term. your actual load will climb to your "j" number and maybe +1 with builds like ,libreoffice gcc glibc boost qtcore.

I once accidentally ran make -j120 while testing build times in my kernel. It built but boy did it load up. Oh yes if you want to lock your system up run "-j" with no number.

To fine tune I use "time emerge system" which which builds your basic toolchain and few friends. On my system thats 42 packages or a stage1 basically.
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Sun May 26, 2013 3:41 pm; edited 1 time in total
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sun May 26, 2013 1:06 pm    Post subject: Reply with quote

hielvc: Thanks

I've used the -j7 for make_opts and it certainly does load the cpu. Seems to work great while doing a clean install and that's what I'm trying to shorten the time of. Even with this quad-core AMD 640 it takes 24 hours to do a clean install if nothing breaks during it, which is no better them the old dual core 240. Normal pkg installs and the occaisional rebuild are handled fine by konsole.

Just finished an -e world rebuild an am not impressed as it took over 9 hours and wasn't finished (libreoffice) due to all the interactive packages (gcc/glibc others) that I estimate added at least 3 hours to the build time. Right now I'm looking at some kind of drop but it doesn't look to be more then half and I was hoping for it being down to 6. Might need to adjust the make_opts back to -j7.

All:

One thing I've noticed is that portage does not seem to use my system memory like it should. I've got 16GB and never saw it use more then 4. Maybe a problem as it should use at least half of it. I know that the kernel reserves quite a bit plus the tmpfs and such
here's what top shows with kde and all my apps running including the libreoffice build.
  • KiB Mem: 16442068 total 3749520 used

The kernel is showing 1/4 of available system ram being used and that includes what's cached. Why isn't portage taking full advantage of this much memory or at least using half of it?

Neddy: why are there some interactive packages - gcc/glibc that force the system to build just them?
Is there any way to force them into paralel build mode to help reduce the build times? It's annoying as gcc alone takes over an hour and I don't have any clue why or where to start looking.

Code:
CFLAGS="-march=native -Os -pipe"


Last edited by FastTurtle on Sun May 26, 2013 1:43 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32027
Location: 56N 3W

PostPosted: Sun May 26, 2013 1:22 pm    Post subject: Reply with quote

FastTurtle,

What do you mean by "interactive packages - gcc/glibc" ? interdependant maybe?

gcc and glibc are interdependent for a number of reasons. you need >gcc-X to build glibc-Y and you must have >glibc-A to build gcc-B, so sometines but not always, the order is important. Indeed, if you leave updating gcc/glibc too long, you get a circular dependency, which you have to break by building intermediate versions of both packages.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Mon May 27, 2013 6:00 pm    Post subject: Reply with quote

The emerge, make.conf, portage manual all mention them and there's the package_properties="-interactive" option to temporarily mask them.

On the gcc/glibc issue, that's been true as long as I've used Gentoo but the option -interactive should at least prevent them from rebuilding.

As to others? I'm not positive but a console test rebuild took over 12 hours when I expected about 6 due to the 4 jobs paralel builds. So I'm investigating what other apps force the single job mode? Looks like I'll have to use the -interactive option and see what doesn't get rebuilt
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4152

PostPosted: Mon May 27, 2013 7:49 pm    Post subject: Reply with quote

It seems you have misunderstood the meaning of "interactive": eix -c --properties-interactive will show you which packages use this property: These are essentially games packages which e.g. require you to insert the original games DVD or something similar: As the name says, some interactive action (of the user) is required for succesfull emerge. This has nothing to do with gcc or glibc.
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Fri May 31, 2013 2:40 pm    Post subject: Reply with quote

mv: Tell me why the man pages don't at least include (games) with the description since all they indicate is -interactive with absolutely no clarification or explanation. To me, if you mention a feature that covers something as you've indicated, then explain it instead of just throwing it out there. I know it sounds like a bit of a rant but when I get dinged for it in various classes and work rejected needing rewrite/clarification, I learn very quickly to expand anything included (even if only to indicate - games) so folks get the idea.

All:

I've completed my build tweak tests and have to say it's been an interesting experience.

To perform the tests I took advantage of the new package sets feature as it allowed me to test with a known config - changes to one option at a time.

method used : emerge @fluxter (package.set in /etc/portage/sets) Note that I did not do a -e @fluxter test.

Personally, I see no reason to rebuild glibc/gcc everytime we do an -e world rebuild unless there's been a changed flag, chost or an update to either glibc or gcc. Otherwise quit wasting cycles doing something unneeded and shorten the install time for Gentoo. Even on my quad-core backed by 16GB of ram, it takes over 3 hours for just glibc/gcc to rebuild. For a system rebuild w/o the toolchain, it would take about an hour during installation. Not long at all and certainly worth while then but it's the reason my testing did not use the -e @fluxter as it would have taken another 3 - 5 hours with rebuilding the toolchain and everything else.

Code:
 Minimal Fluxbox Build
#

# System
net-misc/dhcpcd
sys-apps/mlocate
sys-boot/lilo
sys-kernel/gentoo-sources
app-portage/gentoolkit
media-sound/alsa-utils

# GUI
x11-base/xorg-server
x11-base/xorg-drivers
x11-drivers/radeon-ucode (yes I have a radeon)
x11-wm/fluxbox
x11-misc/fbpanel

# Apps
app-editors/qwriter
app-misc/mc
mail-client/sylpheed
mail-filter/bogofilter
media-sound/qmmp
media-video/vlc
www-client/firefox
www-plugins/adobe-flash
x11-misc/rednotebook
x11-misc/spacefm
x11-misc/tinymount
x11-terms/xterm


This was what I used for the testing so discussion about why/what I chose wont be answered.

Results were very interesting
Baseline test was done with make_opts="-j7" jobs=3 - time was 2 hours

Test 2 increased make_opts=-j8 and took 5 hours
Test 3 make_opts=-j9 - Time was 4 hours

The baseline test 5 changed to make_opts=-j5 jobs=2 took 5hours 20 minutes

Did not perform test 4 as I didn't see any major reduction in time for test 3 - if it had been down to 3 hours, I might have done it.

Discontinued testing after test 5 for the same reason I didn't perform test 4. If it had been less then 4 hours, I may have continued since it would have gave me hope of a reduction.

I may perform additional tests in the future based but right now I've at least got a baseline figure for my installation expectations.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4152

PostPosted: Fri May 31, 2013 5:47 pm    Post subject: Reply with quote

FastTurtle wrote:
mv: Tell me why the man pages don't at least include (games) with the description since all they indicate is -interactive with absolutely no clarification or explanation.

The feature is not limited to games. I would consider "interactive" rather self-explaining, but you can of course open a bug and suggest a better explanation.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2374
Location: UK

PostPosted: Sat Jun 01, 2013 4:45 pm    Post subject: Reply with quote

FastTurtle wrote:
mv: Tell me why the man pages don't at least include (games) with the description since all they indicate is -interactive with absolutely no clarification or explanation.

Code:
       ACCEPT_PROPERTIES = [space delimited list of properties]
              This variable is used to mask packages based on PROPERTIES restrictions.  In addition  to  property
              names, the * and -* wildcard tokens are also supported. This variable can be temporarily overridden
              using the --accept-properties option of emerge(1).  See ebuild(5) for more information about  PROP‐
              ERTIES.
              Defaults to the value of *.
              Examples:
              # Accept any properties
              ACCEPT_PROPERTIES="*"
              # Accept any properties except the "interactive" property
              ACCEPT_PROPERTIES="* -interactive"

Code:
       PROPERTIES = [interactive]
              A space delimited list of properties, with conditional syntax support.
              interactive
                     One or more ebuild phases will produce a prompt that requires user interaction.

What part of that man page description don't you understand?
Quote:
Otherwise quit wasting cycles doing something unneeded

Like running "emerge -e world" for specious reasons?
Back to top
View user's profile Send private message
FastTurtle
Guru
Guru


Joined: 03 Sep 2002
Posts: 409
Location: Shake & Bake

PostPosted: Sun Jun 02, 2013 3:09 pm    Post subject: Reply with quote

My comment on the -interactive was in reply to a suggestion that it only impacted pkgs such as games - which may have been valid but my reading of the appropriate man pages (portage, emerge,make.conf) all indicate that it's supposed to prevent the building of pckgs that force portage out of parallel mode to single mode eg: the toolchain and I was testing this to see if it actually worked - no it doesn't at this time - as my need is to reduce the build time from 24hours to something far lesss (goal was 6 hours).

The build time reduction was just one of the reasons I started this thread with such a wide ranging title. Heck I'm still learning what changes have been made to portage and gentoo in the last 5 years - yes it's been that long since I've really used Gentoo and there have been quite a few changes in that time. Things like the Systemd argument/decission so this discussion could change direction and probably will while staying related to portage. As to dropping into debates about things like http://forums.gentoo.org/viewtopic-t-960898-highlight-.html I opened here, that's not pertinent to this discussion.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum