Parallel portage - How to reduce build times (download)

devsk · Last edited by devsk on Wed Jul 16, 2008 7:08 pm; edited 21 times in total

THIS FEATURE IS IN SVN NOW AND WILL BE RELEASED IN PORTAGE 2.2. Thanks all for the support and the pushing...

-devsk 07-16-2008.

https://bugs.gentoo.org/show_bug.cgi?id=147516

YOU SHOULD USE THE VERSION ATTACHED TO THE BUG.
01-28-2008: Updated to 2.1.4.
06-06-2007 : A fix for circular dependency related issues is added to the patch. The patch also includes a fix for world file related bug which was fixed on Apr 2 but only made it to bug attachment and NOT on my website. So, if you downloaded the patch from my website (comcast.net), please download again from BGO. Please download and apply.

Apply instructions:

***NOTE THAT VERSION NUMBER AND ATTACHMENT NUMBER WILL CHANGE in the following code piece from time to time. I will try to keep them updated but it doesn't hurt to verify against the bugdb. JUST ADD CC YOURSELF ON THE BUG AND YOU WILL GET AN EMAIL WHEN I PROVIDE A NEW FIX ***

Download the patch from the bugdb at https://bugs.gentoo.org/show_bug.cgi?id=147516 , emerge original and apply the patch on top.

Master Shake · Posted: Sun Jul 30, 2006 12:05 am Post subject:

For some of the packages -j1 is all that you can do. Lets take some program foo and break it up into three basic parts: foo.h foo2.h and main.cc. If foo2.h depends on foo.h (which can happen) then all you can do is compile each individually and link them one at a time. What I commly do to parallelize portage is to have two installs running at the same time (i don't recommend this I have run into problems before doing this). But yes besides that, that is an interesting way of handling things. However I don't see that ever happening because of the little minor bugs that can pop up from doing that.
_________________
System Specs:
64-bit gentoo linux
Q6600 @ 3.2Ghz
P35 Chipset
4 Gigs 800mhz 4-4-4-12
Nvidia GeForce 8800 GTX @ 630mhz

devsk · Posted: Sun Jul 30, 2006 12:16 am Post subject:

Philantrop · Posted: Sun Jul 30, 2006 12:22 am Post subject:

gdoubleu · Tux's lil' helper Joined: 07 May 2003 Posts: 80

That's funny, I was thinking about this exact same thing earlier this morning. Last night a started an emerge kde-meta before heading off for bed, hoping to wake up to some graphical goodnees. Instead, I wake up to find out that the emege didn't get very far, hitting this hangup. So, I work around that problem and then start my emerge again while I go off to run errands. Again, though, I return to find that portage hit this circular dependancy and failed once again (still extremely far from the end of my kde-meta emerge). So now, almost 24 hours afterwards, I'm still at 20 of 315 of my "third run" of kde-meta :cry:

.

A parallel emerge would be great in this case. Yes there were errors for a couple packages, but don't halt the entire emerge because of it. There are still plenty of other packages that could have been safely emerged. Maybe then I would be at 250 of 315 instead :wink:

.

Of course, there has been talk of making the compiling and fetching of packages parallel ever since I started using Gentoo (back when there were only 40,000 packages in the tree). So at that rate, we probably shouldn't be expecting to see this feature anytime before 2010. I certainly hope I am wrong though.

But in all seriousness I think this, and parallel compile/fetch, could be presented much better in a portage GUI.

Philantrop · Posted: Sun Jul 30, 2006 1:29 am Post subject:

Portage 2.1 does have parallel fetching/compiling.
_________________
If you feel the issues discussed in this thread have been resolved, please add a "[Solved]" to the subject of your original posting.

devsk · Posted: Sun Jul 30, 2006 1:43 am Post subject:

gdoubleu · Tux's lil' helper Joined: 07 May 2003 Posts: 80

cyrillic

devsk

Master Shake · Posted: Tue Aug 01, 2006 6:45 pm Post subject:

Well if your putting it out there, I'll help test it.
_________________
System Specs:
64-bit gentoo linux
Q6600 @ 3.2Ghz
P35 Chipset
4 Gigs 800mhz 4-4-4-12
Nvidia GeForce 8800 GTX @ 630mhz

devsk · Posted: Mon Sep 04, 2006 7:49 pm Post subject:

ok, finally got some time and I took a shot at it. I am at a point where in portage, I am ready to do parallel merges (patch only needs to change /usr/lib/portage/bin/emerge so far). But suddenly, I realised that I don't know what to do with the output of two emerge sessions? In serial case, it is just output on the terminal (and optionally goes to /var/log/portage). In parallel case, it will be interspersed from both merges, which is not only ugly but very confusing because setup/postinstall messages will be all mixed up.

One option (clean option) is to just output brief message like 'emerging blah to /', setup messages like OO might not compile if aggressive flags, post install messages like you need to manually run vmware-config.pl before running vmware etc. bracketed with package name to clearly identify where the message came from. Just eat up or dump (if LOG enabled) the rest of the build output in /var/log/portage. This will also reduce the verbosity of portage (is that a good thing? I think so because I am interested in verbosity only if an error occurs, and for that I can enable logging: also a standard technique that every other piece of software follows). But this seems like a lot of work which I didn't anticipate in the original scope of parallelism work.

Does anybody have any better ideas on how to go about handling output in case an emerge session decides to merge two packages at the same time?

Phenax · l33t Joined: 10 Mar 2006 Posts: 972

Just output warnings/errors and the rest can go to (a) log file(s) IMO..

devsk · Posted: Tue Sep 05, 2006 4:43 am Post subject:

without any mods to the output so far, the results are very decent on a dual-core cpu:

devsk · Posted: Tue Sep 05, 2006 4:53 am Post subject:

ok, I just tested the pipeline version and we have some more in our very small limited subset:

mudrii · l33t Joined: 26 Jun 2003 Posts: 789 Location: Singapore

Nice FEATURES I hope it will go into final product
_________________
www.gentoo.ro

cyrillic · Posted: Tue Sep 05, 2006 4:39 pm Post subject:

devsk · Posted: Tue Sep 05, 2006 5:42 pm Post subject:

I use a python variable which I can set based on MAKEOPTS e.g. -j3 becomes 2 (=numcpus). but the problem is if I start 2 (for -j3) emerges and both of them use -j3, we will see 6 gcc processes thrashing the system at certain times. This may not be ok, but it is something which we can't handle easily. Of course, the random mix of packages will average out the effect, and always keep the cpu busy.

If you wanna use distcc, I think then I can't base it on MAKEOPTS because with -j6, 5 forks will really thrash the local system. I want to limit it to local numcpus.

Is there a python way of finding the numcpus on the system?

cyrillic · Posted: Tue Sep 05, 2006 11:52 pm Post subject:

I'm not sure if this makes it harder for you to implement, but how about using a separate variable, instead of basing it on MAKEOPTS ?

devsk · Posted: Wed Sep 06, 2006 1:39 am Post subject:

I actually decided to do this simple thing:

razze · Posted: Wed Sep 06, 2006 9:17 am Post subject:

Hello!

The parallell emerging feature seems nice! I also hop to see it available some day.

As a little somewhat off-topic comment I got the rule for MAKEOPTS -j wrong and set it to 5 on my dual core AMD 64. I still have yet to see any donwside of this, everything works fine, and during emerges it keeps both cores busy while still leaving cpu cycles for other stuff. In fact I don't even notice if I have an emerge going on in the background other than by wathcing the CPU usage krells in gkrellm showing a lot of activity.
_________________
--------------------------
Linux User #393524

AMD Athlon 64 X2 3800+ @2,3 GHz, Asus A8V, VIA K8T800PRO, Nvidia GeForce 7600GS, 2 Gb Dual Channel DDR RAM, 2x Samsung SP120 200Gb, 2x WD 320Gb, Plextor PX-716A, linux-2.6.31-gentoo-r10

Zentoo · Posted: Wed Sep 06, 2006 1:00 pm Post subject: Parallel emerge !

That's so great to see it !

I'm ready to test the parallel emerging feature on my dual core 2.5Ghz on my whole world.
So yet a patch is ready, post it please as we could test it!
_________________
Kernel 5.14.15-zen | Gcc 11.2 | Glibc 2.34
Core i7 6700K @ 4.6GHz | 32Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"

devsk · Posted: Wed Sep 06, 2006 5:38 pm Post subject:

I hope to finish it by this weekend, and open a bug and put my patch there for eventual inclusion in the official portage.

PS: I have finished making changes for handling the output problem. With my patch in, you will see output like:

Zentoo · Posted: Wed Sep 06, 2006 6:35 pm Post subject: Order of emerge's output

Ok i think i should love this stuff since that could be a way to forget compilation stuff on console.
And all the compilation stuff could slown down a bad terminal and most especially virtual console in frame buffer.
I have yet make some gentoo installs on framebuffer console and some emerge were 2 to 5 times slower !
It's why I use screen then detach and reattach it to continue and don't lost time to print the compilation stuff on my virtual console.

So that's a point.

On your point, i don't hate it so !

but i wonder if the remaining output from emerge should be mixed by the parallel emerge.
I suppose that should be mixed. So may be you should write a line before and after each message to know which pkgs write out the message.
Just light me on this point please.

Another solution could be to wait a pkg have finished to be emerged to flush messages on the terminal. May be simple for users but more complex to implement I presume .

Cheers in advance for your work on this stuff ! People with dual core processor should really like it ! :wink:

_________________
Kernel 5.14.15-zen | Gcc 11.2 | Glibc 2.34
Core i7 6700K @ 4.6GHz | 32Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"

devsk · Posted: Wed Sep 06, 2006 6:54 pm Post subject:

rest of the stuff goes to respective log file for the package in /var/log/portage if you have PORT_LOGDIR set. If its not set, the configure/compile/install messages are gone. This is the standard for any piece of software. You enable logging and you get a log. You don't enable logging, you don't get anything, except for messages when it fails.

--debug argument to emerge will still throw everything at the terminal and also in the log file if PORT_LOGDIR enabled.
--debug will also disable the parallel feature.

PS: Restricting output of the emerge also speeds up emerge. Typically, scrolling text will waste cpu cycles in X as well in the specific terminal's code. Even if you minimize the window, the terminal still needs to track its scroll buffer and hence wastes cpu. I have seen gains with the minimal output.