Why isn't Portage multithreaded?

sparc · Posted: Mon Feb 28, 2011 10:47 pm Post subject: Why isn't Portage multithreaded?

Hi,

I have a very simple question to ask. Why aren't emerge, equery and the rest of the tools multithreaded?

I've searched the man pages and I haven't found an option to enable such a function. But let me elaborate more. I'm of course not referring to the --jobs functionality; performing a thorough emerge world with all the bells and wistles (-D --with-bdeps=y etc) has an initial overhead that accounts to multiple minutes. This becomes quite frustrating when one does not know what to expect and needs to verify the list of packages before. When this preview results in changes (mask, unmask and use flags) and the procedure has to be repeated, the whole task is just tedious. I have an i7, and I feel being significantly let down when gkrellm reports just one core at 99% while the rest of the system is idling.

As I can understand the process of dependency structuring, it has a big potential of parallelization; but to be honest I haven't bothered to go through the source code... yet! So here it is again. Why isn't this potential being exploited? Why isn't emerge and the rest of the tools multithreaded?

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

On my 4 year old Core2 duo:

sparc · Posted: Tue Mar 01, 2011 12:13 am Post subject:

You should read more carefully... I am just saying that the following:

Naib · Posted: Tue Mar 01, 2011 12:25 am Post subject:

such queries are more i/o bound then cpu bound. making it threaded wouldn't really provide any speed benefit if the same data has to be fetched from a slow turning hunk of plated glass
_________________

sparc · Posted: Tue Mar 01, 2011 12:54 am Post subject:

Thanks for the input Naib. I thought the same at first also. However, even an idling firefox produces more file I/O than this query on average (according to iotop). Of course there are a few spikes than climb up to the MB per sec layer, but they last less than a dozen seconds all together On the contrary, I get one hardware thread at constant 99%. That alone is clue that the process is not constraint by the hdd; if it was it would have to wait for the data, thus lowering the CPU utilization.

In any case, I'm not here to complain. I want to know if there is a specific design decision behind this lack of functionality. If not however, I would really love to help add it.

dol-sen · Posted: Tue Mar 01, 2011 1:23 am Post subject:

No portage is not multi threaded. As naib has said, much of the data loaded by emerge is disk/file based. So will not see much benefit from multi threading. There is also a timing issue for portage that needs data loaded in a certain order for some things in order for the profile and USE flag stacking to work as expected. Another is that one of the portage backends, namely the sqlite one cannot be run from a thread. In porthole we run portage in threads for many things. and it works, but porthole users cannot use the portage sqlite backend.

One of the big problems with emerge/portage in general is the vast amount of code that has to load and initialize, as well as the data. It has been improving these last years, but it is a very large app. Porthole loads portage once, so searches and things tend to be much faster than from the command line. What you need is an interactive session where portage & data stays loaded and ready. I have been working on gentoolkit code for awhile now. I got involved in order to make usable api's for many of its modules so that I can make a plug-in for it in porthole. Since equery loads and gathers it's data from portage, it too gets the slowdown by initializing and gathering data for each run of the command line. When I get the porthole plug-in done. It won't have that same slowdown since it will only load and initialize once at the start.

So for now, to get and see dependency graphs for pkgs, I would recommend you try porthole. It has a fully dynamic dependency view that is capable of following unsatisfied dependencies to the ultimate end. And it is quick too. I am currently working on layman code now, since I've taken over maintainership/development. I am adding a new api so other apps can use and run it. I have also started to create a plug-in for porthole. Once that is done it will be easier to create a gentoolkit plug-in with equery searches as well.

Oh and I think you'll find porthole can generate a complete upgrade list of every package on your system, not just pvnuD list in less than 20 seconds. Even when I haven't updated in a long while, it never takes more than 15 sec. typically about 3-5 sec. The main reason is that it doesn't build dependency graphs, it has a list of all installed pkgs, then checks the entire list for available upgrades. Then you can decide what you want to upgrade, what deps are needed for them, etc. Emerge will look after needed deps at emerge time. When I get more time, I'll add more portage api to calculate dependency graphs from portage. But that is a different project.

Another option is to try pkgcore. It was designed from the start for good code structure and speed and to be a next gen. portage. It does not have as many features as portage, but will run circles around it speed wise. A dependency graph like the one you posted, the times would be less than a minute, probably less than 30sec from my experience with using it.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

keywords, use and even overlays shouldnt make that big of an impact, what do you get with:

dE_logics · Advocate Joined: 02 Jan 2009 Posts: 2253 Location: $TERM

That's why multi core processors always suck. The coding is difficult, and many times impossible.

I think it all happened cause of Microsoft's rubbish CPU scheduler (worst in the world); to compensate for it the manufactures (like always) changed their design.

BTW when it comes to disk I/O intensive operations, multi threading is bound to give issues unless the task is waaaay too much CPU extensive, the HDD head will rocket here and there further reducing the data transfer.
_________________
My blog

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

As far as portage is concerned it is probably mainly because threads is an optional use flag in python, so the devs would have to put in additional code if python isn't threaded.

Make and alot of other programs seem to do very well in multiple core processors.

What really sucks is when you have a 24 core server and an app that only uses one core.

The Doctor · Moderator Joined: 27 Jul 2010 Posts: 2678

Yamakuzure · Posted: Tue Mar 01, 2011 9:04 am Post subject:

Maybe you might want to switch your portage to use sqlite instead of flat files?

aCOSwt · Posted: Tue Mar 01, 2011 9:28 am Post subject:

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

Keeping your portage tree in squashfs+aufs gives an enormous boost, since most of the IO-bound vanishes. I think it should decrease the 5 minutes to 30 seconds.
The 1000 lines config-files shouldn't be an issue - I have these also. (Yes, my cpu is much slower than yours).
Another hint to avoid frequent calling of emerge in your case is to use the new --autounmask option.

sparc · Posted: Tue Mar 01, 2011 9:54 pm Post subject:

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

Is this a better comparison?

sparc · Posted: Wed Mar 02, 2011 1:22 am Post subject:

Here is one test you asked previously:

dol-sen · Posted: Wed Mar 02, 2011 2:50 am Post subject:

Ok, to clarify, my experience a little more with threading portage. Most of it is thread safe, with one notable exception the sqlite backend which must run from the main thread only. In porthole the imported portage code is used for data gathering only, so there is no concern for configs and things getting out of sync between threads. That could very well be a different story for a fully utilized emerge process.

For getting more advice as to where to start, the perils and things to watch out for in portage code. Get on irc and join #gentoo-portage on the freenode network. You'll want to talk to zmedico
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

I don't approach any problem with one solution in mind, IO was just my first guess as I have seen people unknowingly using a generic hd driver, disable swap, mmap, or other kernel options.

Your problem could be caused by many different reasons, inconsistent symbols caused by changed USE flags, emerging glibc or kernel headers and not running emerge -e world, running ricersfs, or CFLAGS like -O3 --omg-optimized I was just trying to help you look outside of the box.

The amount of packages you have isnt as drastic a number as you think. I have had systems with >2000 packages on about half the hardware using much slower versions of portage and even then it didn't take 5 minutes to resolve dependencies.

I don't have a problem with threading portage, but the fact is that portage in it's current state should not be anywhere near that slow. I see engineers and programmers all the time try to compensate for misconfigurations or crap code by throwing more ram and processor cores at it instead of fixing_the_problem.

If that is the road you want to walk down be my guest.

Yamakuzure · Posted: Wed Mar 02, 2011 11:21 am Post subject:

Okay, I give it a try:

Dependencies in 5 minutes:

There seems to be something wrong with your machine or your setup. Even with the number of your packages:

sera · Retired Dev Joined: 29 Feb 2008 Posts: 1017 Location: CET

For me the issue sounds more like portage aborting at the first issue than an issue with multi threading. Imagine portage would tell you all changes needed in a single pass.
Sure there are potential problems but no more than with autoumask.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

sparc · Posted: Wed Mar 02, 2011 7:32 pm Post subject:

On my faulty machine I get the following:

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

boerKrelis · Posted: Wed Mar 02, 2011 11:22 pm Post subject: