Portage Features Questions

FastTurtle · Posted: Sat May 18, 2013 6:06 pm Post subject: Portage Features Questions

Portage used to have a Paralel build option but I don't find it anylonger. Is it still supported in make.conf?

I've got a Quad Core and it took forever to install things with make_opts="-j5" Does it get bette with higher options or are their other methods to speed things up?

My system is an AMD x4-640 with 16GB of memory and /dev/sda is a 7200rpm Seagate.

khayyam · Posted: Sat May 18, 2013 6:19 pm Post subject:

FastTurtle ...

besdies the '-j{N}' in MAKEOPTS, which is for "parallel make", there is the '-j' '--jobs' which "specifies the number of packages to build simultaneously". This can be added to make.conf, eg:

FastTurtle · Posted: Sat May 18, 2013 11:57 pm Post subject:

Is that -j{N} the same as the make_opts -j{N}?

If so, means I can combine everything into a single line.

Yamakuzure · Posted: Sun May 19, 2013 8:12 pm Post subject:

FastTurtle · Posted: Sat May 25, 2013 5:21 pm Post subject:

Finally got around to testing your settings on an -e system rebuild and seeing it. Looks like it will make a difference. May need to tweak the load average higher since I have plenty of headroom (sysgaurd is showing around 45% and I want around 75-80% loading. High enough for the build while leaving enough headroom to not overheat and kick in the thermal throttling.

NeddySeagoon · Posted: Sat May 25, 2013 5:24 pm Post subject:

FastTurtle,

It sounds as if your colling system may be inadequate, ineffective, or both.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FastTurtle · Posted: Sat May 25, 2013 5:55 pm Post subject:

Neddy: I'll have to check it later as the CPU Fan hasn't spun up as I've heard when I managed to push the CPU to 80% loading. Right now I'm only seeing 40% avg with portage reporting less the 2.00 avg.

I've got a full 16GB of memory and yet with everything, I'm only using 1/4 of it (4GiB) so I'm not pushing the system load anywhere's near what it'll handle. Looks like more adjustments ahead.

First thing I'll change is the make_opts and push it to 7. Worked pretty decent on loading the cpu when I tested it or do I want to push the jobs higher? I've got enough ram and loading that makes more sense as it'll push the cpu loading as needed.

NeddySeagoon · Posted: Sat May 25, 2013 6:51 pm Post subject:

FastTurtle,

Too many parallel builds cause the CPU cache to be flushed as task switching takes place to give another ready to run job some CPU time.
The CPU cache is the fastest RAM you have, it really slows things down to keep reloading it from main memory.
Thats what sets a limit on build speed.

With a tiny cache, MAKEOPS="-j2" and --jobs=1 may be the best you can get. With lots of cores and a huge cache, you can do a lot better.
A good place to start is --jobs=2 and MAKEOPS="-jX" where X is your cores+1.

It also depends on what you are building. The kernel and glibc contain a lot of very small files, so you can build more of them at once and still have all the work fit into your CPU cache.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FastTurtle · Posted: Sat May 25, 2013 7:11 pm Post subject:

Looking at cat /proc/cpuinfo I'm seeing 2 cores at 800, 1 core at 1800 and the last hitting 3000 so I it's looking like either it's overheating or I've got the wrong CPU settings in the kernel (conservative instead of on demand).

Needy: good explanation and reminder about cache swapping. Could be part of the problem but after seeing those two cores at such low clocks, I'm beginning to think you're right on the HSF not being up to the task or something not seated right. Next is to drop the jobs down to 4 and see if that combined with -j5 in make opts gets me the best speeds.

So I'll shutdown and pull the HSF to check, then reboot, change jobs to 4 while keeping the -j5 and see if that helps.
_________________
AsRock B550 Phantom Gaming 4
128GB 3200 Mhz memory
4x 4TB Sata - 2x 2TB Sata SSD - 4x 450GB SaS - 3x 900GB SaS - 72GB SaS for Boot
LSI 9211-8i in HBA mode for all of the SaS drives
Radeon 6800 (Non XT) for GPU

NeddySeagoon · Posted: Sat May 25, 2013 7:29 pm Post subject:

FastTurtle,

Some build jobs are I/O limited, not CPU limited, so the two idle cores may be waiting for I/O to complete.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FastTurtle · Posted: Sat May 25, 2013 8:49 pm Post subject:

Don't think the issue was I/O lag though it's possible. Changing the jobs to 4 seems to helped a lot - Testing with an -e system and it's already hit half of what I'd been at the last test in 10 mins. Much better and what I'm looking for.

I think the key was your comment about cache but I'll know when it hits glibc again. That'll answer things for sure as glibc really is a big package.

Just looked at RAM use and it's a quarter of what I'd been using before - 1G.

It's looking like your comment about cache really was the key since the start loaded something like 20 jobs and sped through them then it got into the larger builds.

Have to say it's been a learning experience today.

Question I've now got is how many jobs does portage normally run?

hielvc · Posted: Sat May 25, 2013 10:09 pm Post subject:

FastTurtle I've played with MAKEOPTS="-jX" a lot. A good place to try and see what is a good X is in your kernel

FastTurtle · Posted: Sun May 26, 2013 1:06 pm Post subject:

hielvc: Thanks

I've used the -j7 for make_opts and it certainly does load the cpu. Seems to work great while doing a clean install and that's what I'm trying to shorten the time of. Even with this quad-core AMD 640 it takes 24 hours to do a clean install if nothing breaks during it, which is no better them the old dual core 240. Normal pkg installs and the occaisional rebuild are handled fine by konsole.

Just finished an -e world rebuild an am not impressed as it took over 9 hours and wasn't finished (libreoffice) due to all the interactive packages (gcc/glibc others) that I estimate added at least 3 hours to the build time. Right now I'm looking at some kind of drop but it doesn't look to be more then half and I was hoping for it being down to 6. Might need to adjust the make_opts back to -j7.

All:

One thing I've noticed is that portage does not seem to use my system memory like it should. I've got 16GB and never saw it use more then 4. Maybe a problem as it should use at least half of it. I know that the kernel reserves quite a bit plus the tmpfs and such
here's what top shows with kde and all my apps running including the libreoffice build.

KiB Mem: 16442068 total 3749520 used

The kernel is showing 1/4 of available system ram being used and that includes what's cached. Why isn't portage taking full advantage of this much memory or at least using half of it?

Neddy: why are there some interactive packages - gcc/glibc that force the system to build just them?
Is there any way to force them into paralel build mode to help reduce the build times? It's annoying as gcc alone takes over an hour and I don't have any clue why or where to start looking.

NeddySeagoon · Posted: Sun May 26, 2013 1:22 pm Post subject:

FastTurtle,

What do you mean by "interactive packages - gcc/glibc" ? interdependant maybe?

gcc and glibc are interdependent for a number of reasons. you need >gcc-X to build glibc-Y and you must have >glibc-A to build gcc-B, so sometines but not always, the order is important. Indeed, if you leave updating gcc/glibc too long, you get a circular dependency, which you have to break by building intermediate versions of both packages.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

FastTurtle · Posted: Mon May 27, 2013 6:00 pm Post subject:

The emerge, make.conf, portage manual all mention them and there's the package_properties="-interactive" option to temporarily mask them.

On the gcc/glibc issue, that's been true as long as I've used Gentoo but the option -interactive should at least prevent them from rebuilding.

As to others? I'm not positive but a console test rebuild took over 12 hours when I expected about 6 due to the 4 jobs paralel builds. So I'm investigating what other apps force the single job mode? Looks like I'll have to use the -interactive option and see what doesn't get rebuilt

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

It seems you have misunderstood the meaning of "interactive": eix -c --properties-interactive will show you which packages use this property: These are essentially games packages which e.g. require you to insert the original games DVD or something similar: As the name says, some interactive action (of the user) is required for succesfull emerge. This has nothing to do with gcc or glibc.

FastTurtle · Posted: Fri May 31, 2013 2:40 pm Post subject:

mv: Tell me why the man pages don't at least include (games) with the description since all they indicate is -interactive with absolutely no clarification or explanation. To me, if you mention a feature that covers something as you've indicated, then explain it instead of just throwing it out there. I know it sounds like a bit of a rant but when I get dinged for it in various classes and work rejected needing rewrite/clarification, I learn very quickly to expand anything included (even if only to indicate - games) so folks get the idea.

All:

I've completed my build tweak tests and have to say it's been an interesting experience.

To perform the tests I took advantage of the new package sets feature as it allowed me to test with a known config - changes to one option at a time.

method used : emerge @fluxter (package.set in /etc/portage/sets) Note that I did not do a -e @fluxter test.

Personally, I see no reason to rebuild glibc/gcc everytime we do an -e world rebuild unless there's been a changed flag, chost or an update to either glibc or gcc. Otherwise quit wasting cycles doing something unneeded and shorten the install time for Gentoo. Even on my quad-core backed by 16GB of ram, it takes over 3 hours for just glibc/gcc to rebuild. For a system rebuild w/o the toolchain, it would take about an hour during installation. Not long at all and certainly worth while then but it's the reason my testing did not use the -e @fluxter as it would have taken another 3 - 5 hours with rebuilding the toolchain and everything else.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

FastTurtle · Posted: Sun Jun 02, 2013 3:09 pm Post subject:

My comment on the -interactive was in reply to a suggestion that it only impacted pkgs such as games - which may have been valid but my reading of the appropriate man pages (portage, emerge,make.conf) all indicate that it's supposed to prevent the building of pckgs that force portage out of parallel mode to single mode eg: the toolchain and I was testing this to see if it actually worked - no it doesn't at this time - as my need is to reduce the build time from 24hours to something far lesss (goal was 6 hours).

The build time reduction was just one of the reasons I started this thread with such a wide ranging title. Heck I'm still learning what changes have been made to portage and gentoo in the last 5 years - yes it's been that long since I've really used Gentoo and there have been quite a few changes in that time. Things like the Systemd argument/decission so this discussion could change direction and probably will while staying related to portage. As to dropping into debates about things like https://forums.gentoo.org/viewtopic-t-960898-highlight-.html I opened here, that's not pertinent to this discussion.