Why isn't Portage multithreaded?

dE_logics · Advocate Joined: 02 Jan 2009 Posts: 2253 Location: $TERM

This discussion has grown big.

I gave another thought to this the next day when I was emerging VirtualBox.

After a -pv, most of the info that portage works on is cached to ram. Thus, in this case (i.e second or third emerge with -pv) is bound to give advantage.

+ Multi threading is possible. You can calculate the dependency of a package in separate threads; that wont change the outcome.
_________________
My blog

Genone · Posted: Thu Mar 03, 2011 5:31 pm Post subject:

Multi threading won't give much of a benefit anyway as dependency calculation can't be split up as much as you think, because decisions in one branch may influence decisions in other branches (e.g. any-of deps). Getting that right in a concurrent way would add so much overhead (and memory usage) to eat up any improvements, and would make debugging and maintenance much harder in the long run (debugging multithreaded apps is always a pain).
Also I'm not sure if this still applies, but python used to have some issues with multithreaded apps due to the global interpreter lock, in fact Guide recommended to use multiprocessing instead (which wouldn't be a good idea for dep resolution due to startup times and rather short-living processes).
And as has been mentioned, IO is always an issue with portage as it has to access (not necessarily read) up to many thousand files for a system-wide dep resolution.

While portage has had a lot of performance improvements in the last years, a good part is eaten up again by additional complexity in the dep resolved needed for new requirements like use-/slot-deps, improved package selection and EAPI support in general.

Yamakuzure · Posted: Fri Mar 04, 2011 11:45 am Post subject:

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

I got upto 1m6sec with this redicilious command.

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

Ohh snap,

I forgot to add --metadata to that string ;- )

jamapii · l33t Joined: 16 Sep 2004 Posts: 637

I use emerge -j4 with portage 2.2

This isn't perfect, but should be better than nothing.

However, sometimes builds fail, and then succeed when building serially.

_______0 · Guru Joined: 15 Oct 2012 Posts: 521

mm..

The title of the thread instantly caught my attention as I also find some things portage does utterly WTFs. The parallelization being talk here is good but not useful.

I think compiling world in gentoo would be cut by as much as 3/4 just by optimizing different stages of the emerging process. As cores keep increasing there are certain stages of the emerge process that look retarded and slow down the overall updating. Indeed packages do emerge faster but tha part that'd I've noticed that slows down installing stuff is simple things like "checking".

To illustrate my point let's examine the gcc ebuild. Emerge spends much of the initial time and other parts just doing dumb tasks like the followings:

Scimmia22 · n00b Joined: 24 Sep 2012 Posts: 10

________________________0, you're completely missing the fact that the "Checking" isn't a function of the ebuild or portage, but of the configure script included as part of the source code to setup the makefiles.

boerKrelis · Posted: Tue Oct 16, 2012 9:43 am Post subject:

________________________0, you are unclear on the concepts. Portage != automake/autoconf .

Also, you are talking about two things: 1) portage doing stuff that you think is unnecessary 2) portage not doing various stuff in parallel.

Also, many things you mention either cannot be parallelized (e.g., applying patches should be done in-order) or are I/O-bound.

Also, O("for thing in array_of_things; do $thing") == O(thing1 && thing2 && thing2 && thingN) == O(n) (if thingN == O(1)).
You ran

pigeon768 · l33t Joined: 02 Jan 2006 Posts: 683

_______0 · Guru Joined: 15 Oct 2012 Posts: 521

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Akkara · Posted: Wed Dec 19, 2012 2:15 am Post subject:

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

Yamakuzure · Posted: Wed Dec 19, 2012 9:41 am Post subject:

Well, he has one valid point with the "checking" of portage, though. Portage does check all files listed in the manifest and needed by the ebuild against various hashes. AFAIK this is done in a serial way, which takes a lot of time when ebuilds need only portions of a large bunch of large archives. (qt4 comes to my mind.)
_________________
Important German:

"Aha" - German reaction to pretend that you are really interested while giving no f*ck.
"Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

_______0 · Guru Joined: 15 Oct 2012 Posts: 521

wow I am retarded,