Slowing Portage

[cwr]

There's been some recent discussion on the gentoo-devel mailing list of Portage's increasing
slowness, and someone mentioned cProfile, which I'd forgotten about. Below is a run of cProfile
on emerge, first with a single package (vim) and then with dependencies (xorg-x11). (Both
packages had already been built on the systems concerned.) The exact commands used were:

[krinn]

- EAPI handling, the higher the EAPI version support by portage, the higher the complexity of EAPI, the slower it will be.
- portage features: some features are default enable, the older the portage version, the lighter or the less default features it would have, making it faster.
- vim, xorg... different version of programs can lead to high difference in ebuild or dependencies, maybe not for vim, but obviously, nobody could say an 1.4 xorg would be as light as an 1.0 xorg.

in order to see portage slowness, use same ebuild version and run it against two portage version that handle that EAPI version, using a make.conf set explicity with all features... enable and disable.
You would get results that show difference in portage, handling the same EAPI and complexity, on the same ebuild.

Because test like that is like if you want to tell an user is faster or slower at reading words by asking them to read a text.
The problem is that user1 read a 1000 words text, and user2 is reading a 3000 words text.
Even user2 is faster at reading a word, result could only show user1 is faster.

I would agree i feel portage is slower than before, but at least give it a real test scenario. This looks more like a "Who wants to drown his dog accuses him of rabies" solution (french expression translate, while it's typical french expression, i think anyone get it even in english)

[rogerx]

Or better yet, have a 32 bit platform (AKA Intel Pentium, i686, i586, ...) box alongside a 64 bit box with similar installed packages and compare the profiler output.

Examining the statistics will exponentially give you a good idea where the slow downs are occurring. Matter of fact, performing your tests entirely on those older processor platforms will show some exponential time lapses within the profiler output.

But I can tell you from my experience, Python over all is extremely cumbersome for any 32 bit i686 processors! Along with the non-standardized Python or non-backwards compatible Python functions, I tend to stick with Bourne shell or Bash scripting, as they run much faster than Python. With my i7-3770K with 32 GB RAM, I have no problems, but I still rely on Bash so others who maybe on older platforms are not hindered by my scripting/code.
_________________
Roger
http://rogerx.freeshell.org/

[Ant P.]

90% of Portage's slowness comes from the fact it has to parse 100k Bash scripts using the Bash shell. There was a project to write a libbash for this, which would've made that 90% go away, but like many Gentoo things it never got adopted.

[rogerx]

Question, libbash being written in C/C++ or Python?

Sounds like more redundant code, requiring more human resources dedicated to maintenance of the new code.

If one simply compares 32 bit performance to 64 bit performance, Python overall simply requires a lot of resources to begin with. Probably best to get to the root of the problem, versus dancing around in my opinion. I'm guessing this is why the idea of libbash was dropped. But with all that said, I also realize the important portability features of Python, as well as the fact Python simplifies tasks; probably out-weighed the emerge command possibly being written in Bourne Shell or BASH. And Python being easier to write than C or C++, hence bugs are likely fixed much faster.

I should also note within krinn's excellent above explanation of slowing, I'm pretty sure I'm seeing a lot of EBuild author's simply citing and implying "EAPI=5" within all their EBuild's instead of noticing that they do not use or do not need EAPI=5. A good example is my recent rewrite of the SiteCopy ebuild (ie. Gentoo Bug #500070) for which somebody recently questioned not using the later version. Later version incorrectly citing EAPI=5 is better than EAPI=2, when in reality the higher number only includes more features, hence significantly increasing overhead for smaller or slower platforms!
_________________
Roger
http://rogerx.freeshell.org/

[Ant P.]

Furthermore, the majority of code in /usr/lib/portage/python2.7/ is also Bash scripts - ebuilds are executed by using the 25KB ebuild.sh file as a wrapper, which in turn loads four more Bash scripts bringing the amount to 100KB of Bash code loaded. This is, of course, the overhead Bash imposes before portage can even do a single thing with said ebuild.

[rogerx]

Ant, I don't understand. It's as if you're saying Python is faster than Bash?

From my benchmarking here, Python is much slower and requires more resources than Bourne Shell or Bash. An example of code faster than Bourne Shell or Bash, is C; or C++ if you must. Code faster than C or C++ is Assembly Language.

Of course there maybe a point where Python may run faster on more faster or resourceful systems with larger amounts of data, but on x86 32 bit my benchmarks show Bourne Shell or Bash to be faster by far, likely due to limited CPU speed.
_________________
Roger
http://rogerx.freeshell.org/

[Ant P.]

My real-world experience doesn't match that benchmark: Paludis (C++11) is by far the slowest package manager I've ever used by an order of magnitude, using exactly the same data as Portage.

I'd like to see how you're measuring these.

[rogerx]

I've also researched Paludis within the past month or so, and upon grepping the configure.in found the following:

/var/tmp/portage/sys-apps/paludis-2.0.0/work/paludis-2.0.0/configure: as_fn_error $? "Bad ricer. No bagel. Try again with non-broken compiler flags." "$LINENO" 5
/var/tmp/portage/sys-apps/paludis-2.0.0/work/paludis-2.0.0/configure.ac: AC_MSG_ERROR([Bad ricer. No bagel. Try again with non-broken compiler flags.])

Real world experience is like comparing apples to oranges, when we're talking facts. ;-)
_________________
Roger
http://rogerx.freeshell.org/

[cwr]

The aim of the original post was nol to compare Bash with Python, but to compare the standard ebuild
usage with the standard ebuild usage. If you think the tests aren't relevant, that's fair enough, but
you then need to produce more and better tests to show that your views are correct. Just saying
you don't like the answers doesn't cut it.

Will

[nitm]

Portage is not just getting slower. It is getting unbareably slow.
And the limitting factor is not disk, but cpu:

[rogerx]

Python is just inherently slow. Compare Python to Bash, SED and AWK, or C just using something like printf.

Atop of this, the more features added without thoroughly thinking through "Programming from the Ground Up" techniques, will tend to also cause slowing.

About the only thing I see Python good for, are Python's cross platform capabilities. Other than this; Bash, SED or AWK, and C can easily outperform Python.
_________________
Roger
http://rogerx.freeshell.org/

[bstaletic]

[steveL]

[Ottre]

[The_Great_Sephiroth]

I am just curious as to why this isn't in C or C++. I would think that such a core functionality would be best served by actual binaries.
_________________
Ever picture systemd as what runs "The Borg"?

[Ottre]

[rogerx]

Ottre: Some interesting tidbits there on why Portage was written using Python instead of Bash or C.

Ditto concerning Bash sub-shells, as Bash sub-shells inherently loose script global variable definitions as well. (Just tracked a long unknown bug within abcde CD utility, likely using sub-shells without knowing this effect.)

But given enough time; I think Bash, Sed and/or AWK could be utilized to rewrite Portage with far better results. Personally, I still see Python as somewhat of a trend or "what's hip" versus what is best, and gambling faster processing units will offset the slowness. (I looked at Paludis C++ version of Portage awhile ago, and was somewhat dismayed at it's direction, leadership or sense of responsibility.)

However, everybody (whom doesn't like Python) is probably stuck with this issue concerning Portage slowness due to Python now. Nor do I want to open a can of worms before I get hungry enough to eat the whole can!
_________________
Roger
http://rogerx.freeshell.org/

[nitm]

[haarp]

[hasufell]

It's the input.

[John R. Graham]

[rogerx]

Geez. I knew it wouldn't be long before the complaining would start. Shrugs.

Anyways, I didn't get the comment concerning SystemD. (I'm not a fan of SystemD, due to it's complications.)

Updated: I missed a comment whom misconceived a previous comment. No I'm not suggesting Bash/SED/AWK being faster than C/C++. From my benchmarking, fastest to slowest language; Assembly, C/C++, Bash/SED/AWK, Python. Python tends to require large amounts of overhead (or system resources) in order to be robust.
_________________
Roger
http://rogerx.freeshell.org/

[John R. Graham]

My intuition is that you're incorrect about Python being significantly slower than Awk and for that matter Awk being comparable to Bash as Bash has the reputation of being relatively slow. Sed doesn't even count since it's not anywhere near a complete programming language. I'd be interested in what benchmarks you used to come to this conclusion. (I'd also claim that such a benchmark for Sed doesn't even exist.)

Speaking of facts, here are a few trivial "empty loop" benchmarks for you:

[Akkara]