Parallel portage - How to reduce build times (download)

[devsk]

THIS FEATURE IS IN SVN NOW AND WILL BE RELEASED IN PORTAGE 2.2. Thanks all for the support and the pushing... -devsk 07-16-2008.

https://bugs.gentoo.org/show_bug.cgi?id=147516

YOU SHOULD USE THE VERSION ATTACHED TO THE BUG.
01-28-2008: Updated to 2.1.4.
06-06-2007 : A fix for circular dependency related issues is added to the patch. The patch also includes a fix for world file related bug which was fixed on Apr 2 but only made it to bug attachment and NOT on my website. So, if you downloaded the patch from my website (comcast.net), please download again from BGO. Please download and apply.

Apply instructions:

***NOTE THAT VERSION NUMBER AND ATTACHMENT NUMBER WILL CHANGE in the following code piece from time to time. I will try to keep them updated but it doesn't hurt to verify against the bugdb. JUST ADD CC YOURSELF ON THE BUG AND YOU WILL GET AN EMAIL WHEN I PROVIDE A NEW FIX ***

Download the patch from the bugdb at https://bugs.gentoo.org/show_bug.cgi?id=147516 , emerge original and apply the patch on top.

Code: Select all

cd /tmp/
wget --output-document=portage.patch http://bugs.gentoo.org/attachment.cgi?id=142841
echo ">sys-apps/portage-2.1.4.1" >> /etc/portage/package.mask
emerge -1 "=sys-apps/portage-2.1.4.1"
echo "sys-apps/portage-2.1.4.1" >> /etc/portage/profile/package.provided
cd /usr/lib/portage
patch -p1 < /tmp/portage.patch

*************** OLDER INFO, read but don't follow the install instructions **********************

03-25-2007 : There is an updated patch for portage-2.1.2.2 at https://bugs.gentoo.org/show_bug.cgi?id=147516 . Fixes two bugs reported in this thread.
02-23-2007 : There is an updated patch for portage-2.1.2-r10 at https://bugs.gentoo.org/show_bug.cgi?id=147516 . If you are testing, its easier to download the ebuild overlay from my page.
02-18-2007 : Patch and ebuild tar updated to 2.1.2-r9 (current stable portage version). Please test, but at your own risk. Also, read this page thoroughly before beginning.

EDIT: A new version is available for download and testing at https://bugs.gentoo.org/show_bug.cgi?id=147516 :
Can we get some testing please?
-devsk 09-13-2006

Consider some facts (motivation for doing parallel merges) before we start:

1. Configuration, link and merge phases of packages are essentially single threaded. So, a multi core or multi cpu system is using at the most 50% during these phases of the package install. These phases are also I/O intensive so cpu usage is even lower most of the time.

2. Many packages do MAKEOPTS=-j1 which makes them not use the cpu upto 100%.

3. Not all packages in the dependency list need to be merged in sequentially one after the other.

Given these facts, consider this simple example:

I am trying to install gentoolkit. Some of the dependencies are:

gentoolkit->portage->pycrypto->gmp
gentoolkit->perl

perl and gmp are not related to each other and can be emerged at the same time.

In general, the dep graph will always have nodes which are not related to each other. This is probably always true if you are doing an update on world where there is a variety of potentially unrelated packages being emerged. If we have multiple leaf nodes, and portage finds them, portage can start a separate build for each leaf node for upto a maximum, like 2 single and 3 for dual cpu, at a time. Once leaves are done it can proceed to the next level of unrelated packages.

So, I went ahead and put together a patch for portage (its on portage-2.1.1).

Please follow these simple steps to test it out: NOTE THAT VERSION NUMBERS WILL CHANGE. Modify appropriately.

Download the patch from the bugdb at https://bugs.gentoo.org/show_bug.cgi?id=147516 .
Put ">sys-apps/portage-2.1.2.7" into your /etc/portage/package.mask
emerge -1 "=sys-apps/portage-2.1.2.7"
cd /usr/lib/portage
patch -p1 < [patch location]

****** THE FOLLOWING INSTRUCTIONS ARE OLD*****************
1. Backup your system/install. I mean it! I haven't seen anything bad happen to my system from it, but s**t happens, so you are warned... Do it in a chroot install if you don't want to backup.

2. Goto: https://bugs.gentoo.org/show_bug.cgi?id=147516 and download the patch.

3. quickpkg portage to save the old install. (make sure you are at least 2.1 to begin with if using tbz2 and 2.1.1 if using the patch).

4. cp /usr/portage/sys-apps/portage into /usr/local/portage/sys-apps with:

Code: Select all

cd /usr/local/portage
\rm -rf /usr/local/portage/sys-apps/portage
cp -a /usr/portage/sys-apps/portage /usr/local/portage/sys-apps/

make sure overlay is set to /usr/local/portage. emerge portage. This will give you a clean portage 2.1.1 with parallel patch in. [This may not work if you synced and got a newer than 2.1.1 version in tree. Put the newer version in package.provided as in step 6]

Or apply the patch against portage-2.1.1 by going to /usr/lib and patch < [localtion of downloaded patch]

5. test it out with some emerges (at least 2-3 on the cmdline) using --oneshot. FEATURES="parallel" enables the parallel feature and FEATURES="-parallel" disables it.

6. Make sure you put sys-apps/portage-2.1.1 (or the latest version in portag tree) in /etc/portage/profile/package.provided if you manually patched the portage or have synced the tree and have an upgrade available. This is to avoid getting your patched install being overwritten by the one in portage tree when you do an 'emerge -e system' for example.

For timing your results and reporting: It is best to use 'time FEATURES="parallel -ccache -distcc" emerge --oneshot <pkg1> <pkg2> <pkg3>' and then, 'time FEATURES="-parallel -ccache -distcc" emerge --oneshot <pkg1> <pkg2> <pkg3>' . Note the times. Repeat at least three times to rule out caching. genlop timings will not be useful because the packages individually may take slightly longer to emerge now.

The changes that you will see and and hopefully like are:

1. Parallel install of certain packages keeping your cpu pegged. note that it may not always be possible e.g. you want to emerge -O kdelibs kdebase...

2. The output is very concise if PORT_LOGDIR is set. Only important messages (like setup and post install messages which are currently lost in the garbage that flies by) are shown in the terminal, rest go to log files in /var/log/portage. I highly recommend setting it. If you don't, your configure/compile/install garbage will be thrown on the terminal as before. Less output on the terminal speeds up emerges a lot.

3. If a package fails, emerge doesn't stop immdiately. It will keep emerging as long as there are packages in the current slot to merge. It will stop after the current slot. A slot is like a level in the depgraph. All leaf nodes (packages which don't depend on anything) are at slot 0 (or 1). Next level packages are at slot 1, and so on. All packages at the same slot are not related to each other and can be emerged at the same time. In nutshell, portage will stop only when its absolutely impossible to proceed with any emerge.

Also, note that you will see some spurious looking merge slot related messages before the emerge starts. Don't worry they are telling you about which packages will be emerged at which slot. The emerging starts at slot 0 (if present, otherwise 1). I would like this output when you submit a problem. These will be removed when the patch becomes stable.

For the code reviewers: python is not my native language (I talk C). So, please make as many suggestions to improve the code as you can. I put this together real quick and I might have overlooked many things. So, please review and make suggestions.

Once I have ironed out basic problems, I will submit it for inclusion in official portage.

[Master Shake]

For some of the packages -j1 is all that you can do. Lets take some program foo and break it up into three basic parts: foo.h foo2.h and main.cc. If foo2.h depends on foo.h (which can happen) then all you can do is compile each individually and link them one at a time. What I commly do to parallelize portage is to have two installs running at the same time (i don't recommend this I have run into problems before doing this). But yes besides that, that is an interesting way of handling things. However I don't see that ever happening because of the little minor bugs that can pop up from doing that.

[devsk]

But yes besides that, that is an interesting way of handling things. However I don't see that ever happening because of the little minor bugs that can pop up from doing that.

bugs will pop only if I or you do 'emerge foo' and 'emerge bar' in two different windows. When portage does it, it will do it based on the dependency graph. The graph will clearly tell portage if bar should be emerged before/after foo or can be merged in parallel with it.

Its actually a very simple idea with lots of benefit. And the good thing is that the portage already has the data (namely the dep graph) that it needs to start doing this.

Only other way the bugs will pop is when the dep graph itself is messed. But if that is the case, that means that we have a broken portage tree, and you will probably soon see a bug about that in bugzilla.

[Philantrop]

For some of the packages -j1 is all that you can do.

-j1 is not really important for the proposal.

I like the idea of clean parallel emerges very much. And as devsk pointed out correctly this would mean implementing something unclean most of us do from time to time in a cleaner way.

[gdoubleu]

That's funny, I was thinking about this exact same thing earlier this morning. Last night a started an emerge kde-meta before heading off for bed, hoping to wake up to some graphical goodnees. Instead, I wake up to find out that the emege didn't get very far, hitting this hangup. So, I work around that problem and then start my emerge again while I go off to run errands. Again, though, I return to find that portage hit this circular dependancy and failed once again (still extremely far from the end of my kde-meta emerge). So now, almost 24 hours afterwards, I'm still at 20 of 315 of my "third run" of kde-meta .

A parallel emerge would be great in this case. Yes there were errors for a couple packages, but don't halt the entire emerge because of it. There are still plenty of other packages that could have been safely emerged. Maybe then I would be at 250 of 315 instead .

Of course, there has been talk of making the compiling and fetching of packages parallel ever since I started using Gentoo (back when there were only 40,000 packages in the tree). So at that rate, we probably shouldn't be expecting to see this feature anytime before 2010. I certainly hope I am wrong though.

But in all seriousness I think this, and parallel compile/fetch, could be presented much better in a portage GUI.

[Philantrop]

Portage 2.1 does have parallel fetching/compiling.

[devsk]

Portage 2.1 does have parallel fetching/compiling.

to me parallel-fetch is a useless feature and waste of programming resources. I mean there is no problem or error or wrong behaviour if I fire 'emerge -f kde > /dev/null 2>&1 &' and fire 'emerge kde' immd. (typically I put a sleep for few seconds in between to let the dep graph build) after that. Portage locking will make sure the package that 'emerge kde' wants to emerge, will wait for the fetch to complete which was started by 'emerge -f kde'. Classic waste of programming resources.

[gdoubleu]

Portage 2.1 does have parallel fetching/compiling.

That's some great news. Seems like that feature was left out of the newsletter. For all those not in the know, in make.conf you should add

Code: Select all

FEATURES="parallel-fetch"

[cyrillic]

What do you guys think about this proposal?

Sounds great !

When do we get to test it ?

[devsk]

What do you guys think about this proposal?

Sounds great !

When do we get to test it ?

in a couple of weeks, I guess? maybe a little more. Its been a while since I touched portage code and it has changed a LOT since 2.0.51.

[Master Shake]

Well if your putting it out there, I'll help test it.

[devsk]

ok, finally got some time and I took a shot at it. I am at a point where in portage, I am ready to do parallel merges (patch only needs to change /usr/lib/portage/bin/emerge so far). But suddenly, I realised that I don't know what to do with the output of two emerge sessions? In serial case, it is just output on the terminal (and optionally goes to /var/log/portage). In parallel case, it will be interspersed from both merges, which is not only ugly but very confusing because setup/postinstall messages will be all mixed up.

One option (clean option) is to just output brief message like 'emerging blah to /', setup messages like OO might not compile if aggressive flags, post install messages like you need to manually run vmware-config.pl before running vmware etc. bracketed with package name to clearly identify where the message came from. Just eat up or dump (if LOG enabled) the rest of the build output in /var/log/portage. This will also reduce the verbosity of portage (is that a good thing? I think so because I am interested in verbosity only if an error occurs, and for that I can enable logging: also a standard technique that every other piece of software follows). But this seems like a lot of work which I didn't anticipate in the original scope of parallelism work.

Does anybody have any better ideas on how to go about handling output in case an emerge session decides to merge two packages at the same time?

[Phenax]

Just output warnings/errors and the rest can go to (a) log file(s) IMO..

[devsk]

without any mods to the output so far, the results are very decent on a dual-core cpu:

Code: Select all

time FEATURES="-parallel -ccache" emerge --oneshot tcl tk gnome-common eog
real    2m2.693s
user    2m0.516s
sys     0m42.371s

time FEATURES="parallel -ccache" emerge --oneshot tcl tk gnome-common eog
real    1m44.824s
user    2m1.284s
sys     0m43.067s

That's a 15% gain in performance... not bad for such a small set of packages. And this is with 2-at-a-time approach (where I spawn two and wait for them to finish and start next two) and not with a pipeline approach (where in I can start 3rd when 1st finishes, still keeping active spawned to 2).

Not so interesting when run in a VM where only one core is available. The gain was a measely 4 seconds on 3 minutes. This was expected.

I need to clean up the patch (lot of debug and hacks in there), fix a dep bug, optimize in a pipeline fashion instead of blind "2 at a time", do something about the interspersed output, and I will be ready for a test release. Keep your fingers crossed.

[devsk]

ok, I just tested the pipeline version and we have some more in our very small limited subset:

Code: Select all

real    1m42.491s
user    2m0.776s
sys     0m42.583s

that makes it 17%... I think this will make a lot of difference in bigger package sets like emerge -e system or emerge -e world.

[mudrii]

Nice FEATURES I hope it will go into final product

[cyrillic]

And this is with 2-at-a-time approach (where I spawn two and wait for them to finish and start next two) and not with a pipeline approach (where in I can start 3rd when 1st finishes, still keeping active spawned to 2).

Please don't hard code it for 2 at a time, because I would like to try it with distcc and 6 machines (or more).

[devsk]

I use a python variable which I can set based on MAKEOPTS e.g. -j3 becomes 2 (=numcpus). but the problem is if I start 2 (for -j3) emerges and both of them use -j3, we will see 6 gcc processes thrashing the system at certain times. This may not be ok, but it is something which we can't handle easily. Of course, the random mix of packages will average out the effect, and always keep the cpu busy.

If you wanna use distcc, I think then I can't base it on MAKEOPTS because with -j6, 5 forks will really thrash the local system. I want to limit it to local numcpus.

Is there a python way of finding the numcpus on the system?

[cyrillic]

I'm not sure if this makes it harder for you to implement, but how about using a separate variable, instead of basing it on MAKEOPTS ?

FEATURES="distcc parallel"
MAKEOPTS="-j2"
FORKS="6"

This way the total number of compiler processes would be 12, but 10 of them would be distributed to other machines.

[devsk]

I actually decided to do this simple thing:

Code: Select all

                # dirty little trick to get number of cpus from the system
                fd_cpuinfo = os.popen("cat /proc/cpuinfo","r")
                cpu_count = 0
                for data_cpuinfo in fd_cpuinfo.readlines():
                        if string.find(data_cpuinfo,'cpu MHz') > -1 :
                                cpu_count += 1
                fd_cpuinfo.close()

                # if someone really screwed with /proc/cpuinfo output, we should not suffer
                if cpu_count == 0:
                        cpu_count = 1

                for x in self.merge_slot.keys():
                        # if previous slot failed, discontinue the emerge
                        if one_in_slot_failed:
                                print "An emerge in previous slot failed, bailing: slot=",x
                                break
                        if parallel:
                                num_at_atime = cpu_count + 1
                        else:
                                num_at_atime = 1

this is similar to guidelines for MAKEOPTS i.e. numcpus+1. The problem with a separate variable is portage folks are going to hate it...

Also, too many forks can undo all the gains we get from doing parallel emerges. So, less configurable is better in this case.

With this change and a slightly changed subset of packages (large enough to show the advantage of doing this but small enough to repeat many times), here are some results (after repeating 3 times to rule out gains from caching):

Code: Select all

$ time FEATURES="-parallel -ccache" emerge --oneshot tcl tk eog gnome-session

real    2m37.898s
user    2m31.493s
sys     0m56.996s

$ time FEATURES="parallel -ccache" emerge --oneshot tcl tk eog gnome-session

real    1m59.137s
user    2m31.117s
sys     0m57.584s

That's an impressive ~25% gain on a dual core cpu. I was surprised by the fact that it did 3 parallel emerges and had -j3 in MAKEOPTS, still the thrashing didn't lead to performance loss. I am probably underestimating the linux kernel in managing large number of cpu hungary concurrent processes....

[razze]

Hello!

The parallell emerging feature seems nice! I also hop to see it available some day.

As a little somewhat off-topic comment I got the rule for MAKEOPTS -j wrong and set it to 5 on my dual core AMD 64. I still have yet to see any donwside of this, everything works fine, and during emerges it keeps both cores busy while still leaving cpu cycles for other stuff. In fact I don't even notice if I have an emerge going on in the background other than by wathcing the CPU usage krells in gkrellm showing a lot of activity.

[Zentoo]

That's so great to see it !

I'm ready to test the parallel emerging feature on my dual core 2.5Ghz on my whole world.
So yet a patch is ready, post it please as we could test it!

[devsk]

I hope to finish it by this weekend, and open a bug and put my patch there for eventual inclusion in the official portage.

PS: I have finished making changes for handling the output problem. With my patch in, you will see output like:

Code: Select all

>>> Emerging (1 of 1) media-gfx/eog-2.14.3 to /
 * eog-2.14.3.tar.bz2 MD5 ;-) ...                                                                       [ ok ]
 * eog-2.14.3.tar.bz2 RMD160 ;-) ...                                                                 [ ok ]
 * eog-2.14.3.tar.bz2 SHA1 ;-) ...                                                                     [ ok ]
 * eog-2.14.3.tar.bz2 SHA256 ;-) ...                                                                 [ ok ]
 * eog-2.14.3.tar.bz2 size ;-) ...                                                                       [ ok ]
 * checking ebuild checksums ;-) ...                                                                  [ ok ]
 * checking auxfile checksums ;-) ...                                                                 [ ok ]
 * checking miscfile checksums ;-) ...                                                                [ ok ]
 * checking eog-2.14.3.tar.bz2 ;-) ...                                                                [ ok ]
 *
 * Please note that this EOG is very dangerous ebuild
 * You should stay away from -ftracer
 *
>>> Merging media-gfx/eog-2.14.3 to /
>>> Safely unmerging already-installed instance...
>>> Regenerating /etc/ld.so.cache...
>>> Original instance of package unmerged safely.
 *
 * Now that you have emerged the most dangerous package on the planet
 * run eog-setup to complete the install
 * Also, add yourself to eog-group...:)
 *
>>> Regenerating /etc/ld.so.cache...
>>> media-gfx/eog-2.14.3 merged.

>>> No packages selected for removal by clean.

>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.


 * GNU info directory index is up-to-date.
 * IMPORTANT: 1 config files in /etc need updating.
 * Type emerge --help config to learn how to update config files.

Note how setup and postinst messages (which are the most important part of the messages flying by with current portage) are shown but the rest of the garbage is gone to the logfiles. I just inserted pkg_setup and pkg_postinst with few dummy ewarn's in the eog ebuild.

The scheme is:

Emerging (x of y) blah...
setup messages...
Merging blah to blah
Unmerging installed blah
postinst messages
few more cleanup lines
etc-update message

Does anybody particularly hate this scheme? any ideas to improve it?

PS: some formatting may have been lost during copy-paste.

[Zentoo]

Ok i think i should love this stuff since that could be a way to forget compilation stuff on console.
And all the compilation stuff could slown down a bad terminal and most especially virtual console in frame buffer.
I have yet make some gentoo installs on framebuffer console and some emerge were 2 to 5 times slower !
It's why I use screen then detach and reattach it to continue and don't lost time to print the compilation stuff on my virtual console.

So that's a point.

On your point, i don't hate it so ! but i wonder if the remaining output from emerge should be mixed by the parallel emerge.
I suppose that should be mixed. So may be you should write a line before and after each message to know which pkgs write out the message.
Just light me on this point please.

Another solution could be to wait a pkg have finished to be emerged to flush messages on the terminal. May be simple for users but more complex to implement I presume .


Cheers in advance for your work on this stuff ! People with dual core processor should really like it !

[devsk]

rest of the stuff goes to respective log file for the package in /var/log/portage if you have PORT_LOGDIR set. If its not set, the configure/compile/install messages are gone. This is the standard for any piece of software. You enable logging and you get a log. You don't enable logging, you don't get anything, except for messages when it fails.

--debug argument to emerge will still throw everything at the terminal and also in the log file if PORT_LOGDIR enabled.
--debug will also disable the parallel feature.

PS: Restricting output of the emerge also speeds up emerge. Typically, scrolling text will waste cpu cycles in X as well in the specific terminal's code. Even if you minimize the window, the terminal still needs to track its scroll buffer and hence wastes cpu. I have seen gains with the minimal output.