Why not migrate Portage to C language?

[jhon987]

Hi,
Just to clarify things out, I'm asking this from a user perspective, i.e. presuming it's technically feasible... but first:

Why?

As a user, I find many apps, plugins, etc... written in Python are considerably slower than those written in many other languages such as javascript or better yet C.
Besides that, when Portage is written in Python it's dependent on a specific Python version, for instance 2.7, because Python versions aren't (for the most part) backward compatible.
This, in turn, may require users to have more than one Python version installed on their machines at all times, since other apps might require different Python version.

To sum up, the main reasons IMHO Portage would gain by being rewritten in other programing language (namely C) are:

• Python is slow, rewriting Portage in other language might make it faster
• Python isn't backwards compatible, thus rewriting Portage would rid the user of Python specific version dependency...

So far I've found 2 threads concerning that issue yet, both are fairly old and aren't directly asking the herein question - http://forums.gentoo.org/viewtopic-t-54 ... rt-50.html
- http://forums.gentoo.org/viewtopic-t-31 ... 378cdbfa61

This is my opinion, does anyone agree?

[John R. Graham]

Why don't you try out sys-apps/paludis? It's a Portage tree compatible package manager written in C++.

- John

[szatox]

"i don't care" option is missing. I use it every few days and let it run while I do other things. E.g. go sleep. Or go for a walk. Or play games. Does it matter if I do 1 or 2 rounds in that time? No, I will play 3 or 4 (or 20 ), so I wouldn't even notice the difference anyway
Do you sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark?

[Naib]

• Python is slow, rewriting Portage in other language might make it faster

Bad python is slow like bad C++ is slow. While python is a scripted language and thus is slower than a compiled, it isn't painfully slow. Equally python is a very good boilerplate langage if the actual speed concerns are correctly quantified and then that code dropped to C

[*]Python isn't backwards compatible, thus rewriting Portage would rid the user of Python specific version dependency...[/list]

Python is very backwards compatible with the py2 and py3 being the obvious exception as the python dev's broke compatibility to fix some initial architectual issues. a py3.3 application will work perfectly fine on py3.4
If you use py3.4 specific extensions then yes you can only use py3.4 but portage as a whole doesn't use anything that exotic

[John R. Graham]

"i don't care" option is missing.

@jhon987, I can add an "I don't care" option to the poll if you like.

- John

[jhon987]

"i don't care" option is missing.

@jhon987, I can add an "I don't care" option to the poll if you like.

- John

sure, why not...
And thanks for the tip (Paludis, the Other Package Mangler)

[jhon987]

"i don't care" option is missing. I use it every few days and let it run while I do other things. E.g. go sleep. Or go for a walk. Or play games. Does it matter if I do 1 or 2 rounds in that time? No, I will play 3 or 4 (or 20 ), so I wouldn't even notice the difference anyway
Do you sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark?

I have intel i7 8-core, so usually I do "sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark?"
(except when I emerge a web-browser, libre-office or other really huge package which takes approx 40 mins to compile)

[ct85711]

I wouldn't mind if portage was rewritten in C IF it would improve the performance. However, it's been said in numerous threads, the part in portage that takes the most time, is the dependency tree. That isn't going to be any faster in which ever language it is written in. Beyond that, I see no benefits where portage would be noticeably faster.

Like Naib said, portage is hardly fixed on one version of Python. Since I've used Gentoo, I've updated python numerous times, and never had an issue recompiling portage to use the new python (it always supported the new versions of python almost immediately after it is released)

If anything, I think you should be complaining why so many packages still only support python 2.7, and never updated to support 3 (so that we won't have multiple copies of python).

[The Doctor]

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while C, C#, C++, lisp, haskell, etc. would make it more difficult (or, in the case of haskell, more lazy.) Actually, haskell might not be a bad language to use, except for the rewriting required. That would be a side effect, which violates too many rules to consider.

[The_Great_Sephiroth]

I would like to see it in C++ personally, but C is good too. My reasons are simply, I am damn tired of scripting languages being used for everything, bloating my systems with interpreters. It has always been my firm belief that an OS and its tools should be done in a good, binary-compiled language. I could then cut crap like Python out of my OS.

With that said, I do not see Portage and its tools as "slow". The compilation process takes time and the system does not affect that. Hardware affects that! I am happy with Gentoo and its tools, and if they stay the way they are, I won't mind a bit. Now if they switch things around so the entire OS is Python and all of the software is suddenly Python, I am outta' here! :p

[ulenrich]

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while ...

+1
And I think this poll is dumb and blind in an obvious way:
The "Gentoo Comunity" is of maintainers and developers. Gentoo forum users do not decide anything. It is docracy! Thus you do not need a poll but a coder!

If the solving part of portage takes the most of the time, you could think of a more narrow solution of your performance issue: Make the solver part of portage pluggable like Debian does to provide alternatives. There was a french akademic project, financed by European Union science funds, that provided some more advanced algos. I think also openSUSE took some of that for their super fast zypper tool.

[Yamakuzure]

If you use py3.4 specific extensions then yes you can only use py3.4 but portage as a whole doesn't use anything that exotic

And if you rely on features introduced with -std=gnu11, it won't compile using -std=c99.

[WWWW]

There's already a protage in C:

PALUDIS

[WWWW]

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while ...

+1
And I think this poll is dumb and blind in an obvious way:
The "Gentoo Comunity" is of maintainers and developers. Gentoo forum users do not decide anything. It is docracy! Thus you do not need a poll but a coder!

If the solving part of portage takes the most of the time, you could think of a more narrow solution of your performance issue: Make the solver part of portage pluggable like Debian does to provide alternatives. There was a french akademic project, financed by European Union science funds, that provided some more advanced algos. I think also openSUSE took some of that for their super fast zypper tool.

Nothing beats eix. Try find, filter updates etc, in debian.

[Fitzcarraldo]

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable.

+2.

It's not that I would mind if Portage were written in C, so I voted "Don't care", but rewriting something as complex as Portage in any language would likely introduce new bugs, which would take time to shake out. No doubt Portage has bugs currently, but just think what re-implementing the whole thing in a different language could do in the short term.

[Ant P.]

I couldn't care less about what Portage is written in. But I sure would like reading ebuild metadata and arbitrary bash script execution to be separate operations.

[hasufell]

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while C, C#, C++, lisp, haskell, etc. would make it more difficult (or, in the case of haskell, more lazy.) Actually, haskell might not be a bad language to use, except for the rewriting required. That would be a side effect, which violates too many rules to consider.

You wouldn't "rewrite" any imperative program in haskell. You re-design it and then implement it. At which point there's not much similarity left, even in the algorithms (which should be reverse-engineered and optimized for FP).
Also, whether your functions/algorithms are actually lazy depends on a lot of things and there are lots of ways to break laziness without knowing. Haskell on it's own doesn't make all your functions lazy.

For the topic... it implies that there is anything particularly useful/reusable/understandable in the portage code. Since I don't find any indication that this is the case, I don't see a reason to reimplement a broken piece of code in a random language.

[eccerr0r]

I always thought, though portage takes a while to compute, emerging packages will equally be slow on that same machine. So it doesn't really matter that much in the whole scheme of things unless the typical usage model is to emerge packages one at a time.

As far as I can tell, it's the disk database that's the slowest thing. Having to read in the disk database and then traversing it is not going to be fast no matter what language - at least when I compare portage on NFS, portage on mechanical disk, and portage on SSD. Having said that, I'm not sure if a SQL/SQLite database holding the portage tree is the right solution either, easily peeking into the ebuild is nice too.

I guess it is somewhat consistent at least, speed does not seem to be a concern to most Gentoo users... or at least the solution is "solve it with hardware." Yes, agreed, portage is virtually a no-op on my SSD i7 too.

But my poor, poor 486 it takes a two minutes or so to run "emerge" to get the other white meat ...

[mistbow]

I'd rather rewrite Portage on Golang.

[Muso]

I'd rather rewrite Portage on Golang.

Go for it.

[axl]

C wouldn't be a good programming language to write portage in.

portage makes heavy use of strings and regexp expressions. there's no way to get out of the interpreter box.

python|perl|php

these are the choices. well, the old tested choices. using C here is just weird.

what you CAN do is optimize the portage tree, the /var/db/pkg files, maybe the distfiles. these things contribute to the overall speed of how portage behaves.

[axl]

If I would vote for portage to change, I would vote for multi-cpu feature. smp. multi-core. whatever you wanna call it.

As it stands, we're 20 years into the future, and portage still uses one core (FOR ITSELF... I'm not suggesting it compiles with one core. it functions with one core). Check signatures. equery belongs. emerge. think of any command... it runs in a single core. which in 2020 soon... it's kinda... uhm, unperfect. unideal. unoptimized.

I remember the 90's when I bought my first smp computer. one motherboard. 2 cpu's. a smp kernel. back then we didn't have multicore. which came after hyperthreading. but they did come, one after another and now ... applications which run in a single thread are ... very old-fashioned.

that aside, there's another issues with portage. the stupid memory usage. sorry for the offtopic, but it's kinda ontopic.

i'm talking about portage on raspberry pi.

you run emerge... emerge runs some stuff, makes a list of packages to ebuild, and when you actually get to start building anything, the emerge command/python command already has 200-300 megs of ram consumated. not doing anything. once you have a list of packages, you can just ebuild {suite} (whatever suite is.. configure, compile, package or merge or both or whatever). which is another thing that you might want to do. maybe you want to build 2-3 packages at the time with 4-8 threads for each package. portage can't do that. an admin can, but portage can't.


I'm pretty sure it would be a monumental impossible task, but on the other hand, it's 2020. and emerge is single core. can you ... ? I can't. I'd like that to be different. and it can be done. python has pthread. perl has. hell, even php can mime it with a little bit of help from some statically build executables/scripts.


But let's start with the easy stuff. let go of the memory. (ON PI) when you figure out what needs to be upgraded & stuff, just run a script. why is emerge 200-300Mb of memory. just emerge itself. the python behind emerge. can't we free() all that memory and just run the list?

[axl]

I'll put it another way. I feel like emerge should just "forget" all the dependency stuff that it has calculated, once it starts building.

I know it doesn't do that. Because you can just look at top. That's what would be a good first step toward a better portage.

And a multi-core portage. multi-cpu. multi-threading. I don't care how you call it as long as it hauls ass. My old computer has 8 cores. My Pi has 4. I am ashamed to say out loud how many cores just stand and do nothing.

And lastly, use tmpfs/ramfs and often backups ... but that's up to the admins.

[pa4wdh]

In my general opinion a package manager should have the least number of dependencies possible so that if something on your system breaks you have the best chance your package manager still works to fix it.
As a result of that I'd like to see portage in C (or any other compiled language) with the option to build it with all it's dependencies statically linked in. this will eliminate huge dependencies such as the python language.

I still remember an update many years ago which broke wget, which caused the download for the fix to fail ...

[Goverp]

From my experience, rewriting in a new language rarely makes much performance improvement, and of course it introduces bugs - especially in a language lacking many, many features of the one currently in use, so the rewrite is non-trivial. When there's a performance issue, (a) IMHO you want at least 3* better speed to be worthwhile (this is for applications; infrastructure gains of 10% are good 'cos they're cumulative) (b) it's invariably about the algorithms, and the starting point is instrumentation to detect the hotspots.

Starting with the assumption that it will be faster in C is doomed; find out what's actually slow, and see if there's a better way of doing it.

<edit>
Oh, I find myself contradicted, in an interesting way:
https://www.phoronix.com/scan.php?page= ... -To-Python
</edit>