Why not migrate Portage to C language?

jhon987 · Apprentice Joined: 18 Nov 2013 Posts: 297

Hi,
Just to clarify things out, I'm asking this from a user perspective, i.e. presuming it's technically feasible... but first:

Why?

As a user, I find many apps, plugins, etc... written in Python are considerably slower than those written in many other languages such as javascript or better yet C.
Besides that, when Portage is written in Python it's dependent on a specific Python version, for instance 2.7, because Python versions aren't (for the most part) backward compatible.
This, in turn, may require users to have more than one Python version installed on their machines at all times, since other apps might require different Python version.

To sum up, the main reasons IMHO Portage would gain by being rewritten in other programing language (namely C) are:

Python is slow, rewriting Portage in other language might make it faster
Python isn't backwards compatible, thus rewriting Portage would rid the user of Python specific version dependency...

So far I've found 2 threads concerning that issue yet, both are fairly old and aren't directly asking the herein question - https://forums.gentoo.org/viewtopic-t-54862-postdays-0-postorder-asc-start-50.html
- https://forums.gentoo.org/viewtopic-t-311291-postdays-0-postorder-asc-highlight-portagec-start-50.html?sid=3462e3cdb24530af6c93c8378cdbfa61

This is my opinion, does anyone agree?

John R. Graham · Posted: Wed Dec 10, 2014 7:15 pm Post subject:

Why don't you try out sys-apps/paludis? It's a Portage tree compatible package manager written in C++.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3131

"i don't care" option is missing. I use it every few days and let it run while I do other things. E.g. go sleep. Or go for a walk. Or play games. Does it matter if I do 1 or 2 rounds in that time? No, I will play 3 or 4 (or 20

), so I wouldn't even notice the difference anyway :lol:

Do you sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark?

Naib · Last edited by Naib on Wed Dec 10, 2014 9:36 pm; edited 1 time in total

John R. Graham · Posted: Wed Dec 10, 2014 9:01 pm Post subject:

jhon987 · Apprentice Joined: 18 Nov 2013 Posts: 297

jhon987 · Apprentice Joined: 18 Nov 2013 Posts: 297

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

I wouldn't mind if portage was rewritten in C IF it would improve the performance. However, it's been said in numerous threads, the part in portage that takes the most time, is the dependency tree. That isn't going to be any faster in which ever language it is written in. Beyond that, I see no benefits where portage would be noticeably faster.

Like Naib said, portage is hardly fixed on one version of Python. Since I've used Gentoo, I've updated python numerous times, and never had an issue recompiling portage to use the new python (it always supported the new versions of python almost immediately after it is released)

If anything, I think you should be complaining why so many packages still only support python 2.7, and never updated to support 3 (so that we won't have multiple copies of python).

The Doctor · Moderator Joined: 27 Jul 2010 Posts: 2678

Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while C, C#, C++, lisp, haskell, etc. would make it more difficult (or, in the case of haskell, more lazy.) Actually, haskell might not be a bad language to use, except for the rewriting required. That would be a side effect, which violates too many rules to consider.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.

The_Great_Sephiroth · Posted: Wed Dec 10, 2014 10:16 pm Post subject:

I would like to see it in C++ personally, but C is good too. My reasons are simply, I am damn tired of scripting languages being used for everything, bloating my systems with interpreters. It has always been my firm belief that an OS and its tools should be done in a good, binary-compiled language. I could then cut crap like Python out of my OS.

With that said, I do not see Portage and its tools as "slow". The compilation process takes time and the system does not affect that. Hardware affects that! I am happy with Gentoo and its tools, and if they stay the way they are, I won't mind a bit. Now if they switch things around so the entire OS is Python and all of the software is suddenly Python, I am outta' here! :p
_________________
Ever picture systemd as what runs "The Borg"?

ulenrich · Veteran Joined: 10 Oct 2010 Posts: 1480

Yamakuzure

WWWW · Tux's lil' helper Joined: 30 Nov 2014 Posts: 143

There's already a protage in C:

PALUDIS

WWWW · Tux's lil' helper Joined: 30 Nov 2014 Posts: 143

Fitzcarraldo · Posted: Thu Dec 11, 2014 8:05 pm Post subject:

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

I couldn't care less about what Portage is written in. But I sure would like reading ebuild metadata and arbitrary bash script execution to be separate operations.

hasufell · Retired Dev Joined: 29 Oct 2011 Posts: 429

eccerr0r · Posted: Wed Dec 17, 2014 11:30 pm Post subject:

I always thought, though portage takes a while to compute, emerging packages will equally be slow on that same machine. So it doesn't really matter that much in the whole scheme of things unless the typical usage model is to emerge packages one at a time.

As far as I can tell, it's the disk database that's the slowest thing. Having to read in the disk database and then traversing it is not going to be fast no matter what language - at least when I compare portage on NFS, portage on mechanical disk, and portage on SSD. Having said that, I'm not sure if a SQL/SQLite database holding the portage tree is the right solution either, easily peeking into the ebuild is nice too.

I guess it is somewhat consistent at least, speed does not seem to be a concern to most Gentoo users... or at least the solution is "solve it with hardware." Yes, agreed, portage is virtually a no-op on my SSD i7 too.

But my poor, poor 486 it takes a two minutes or so to run "emerge" to get the other white meat ...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

mistbow · n00b Joined: 13 Apr 2019 Posts: 4

I'd rather rewrite Portage on Golang.

Muso · Posted: Fri Oct 18, 2019 7:11 pm Post subject:

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

C wouldn't be a good programming language to write portage in.

portage makes heavy use of strings and regexp expressions. there's no way to get out of the interpreter box.

python|perl|php

these are the choices. well, the old tested choices. using C here is just weird.

what you CAN do is optimize the portage tree, the /var/db/pkg files, maybe the distfiles. these things contribute to the overall speed of how portage behaves.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

If I would vote for portage to change, I would vote for multi-cpu feature. smp. multi-core. whatever you wanna call it.

As it stands, we're 20 years into the future, and portage still uses one core (FOR ITSELF... I'm not suggesting it compiles with one core. it functions with one core). Check signatures. equery belongs. emerge. think of any command... it runs in a single core. which in 2020 soon... it's kinda... uhm, unperfect. unideal. unoptimized.

I remember the 90's when I bought my first smp computer. one motherboard. 2 cpu's. a smp kernel. back then we didn't have multicore. which came after hyperthreading. but they did come, one after another and now ... applications which run in a single thread are ... very old-fashioned.

that aside, there's another issues with portage. the stupid memory usage. sorry for the offtopic, but it's kinda ontopic.

i'm talking about portage on raspberry pi.

you run emerge... emerge runs some stuff, makes a list of packages to ebuild, and when you actually get to start building anything, the emerge command/python command already has 200-300 megs of ram consumated. not doing anything. once you have a list of packages, you can just ebuild {suite} (whatever suite is.. configure, compile, package or merge or both or whatever). which is another thing that you might want to do. maybe you want to build 2-3 packages at the time with 4-8 threads for each package. portage can't do that. an admin can, but portage can't.

I'm pretty sure it would be a monumental impossible task, but on the other hand, it's 2020. and emerge is single core. can you ... ? I can't. I'd like that to be different. and it can be done. python has pthread. perl has. hell, even php can mime it with a little bit of help from some statically build executables/scripts.

But let's start with the easy stuff. let go of the memory. (ON PI) when you figure out what needs to be upgraded & stuff, just run a script. why is emerge 200-300Mb of memory. just emerge itself. the python behind emerge. can't we free() all that memory and just run the list?

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

I'll put it another way. I feel like emerge should just "forget" all the dependency stuff that it has calculated, once it starts building.

I know it doesn't do that. Because you can just look at top. That's what would be a good first step toward a better portage.

And a multi-core portage. multi-cpu. multi-threading. I don't care how you call it as long as it hauls ass. My old computer has 8 cores. My Pi has 4. I am ashamed to say out loud how many cores just stand and do nothing.

And lastly, use tmpfs/ramfs and often backups ... but that's up to the admins.

pa4wdh · l33t Joined: 16 Dec 2005 Posts: 811

In my general opinion a package manager should have the least number of dependencies possible so that if something on your system breaks you have the best chance your package manager still works to fix it.
As a result of that I'd like to see portage in C (or any other compiled language) with the option to build it with all it's dependencies statically linked in. this will eliminate huge dependencies such as the python language.

I still remember an update many years ago which broke wget, which caused the download for the fix to fail ... :roll:

_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com

Goverp · Veteran Joined: 07 Mar 2007 Posts: 1997

From my experience, rewriting in a new language rarely makes much performance improvement, and of course it introduces bugs - especially in a language lacking many, many features of the one currently in use, so the rewrite is non-trivial. When there's a performance issue, (a) IMHO you want at least 3* better speed to be worthwhile (this is for applications; infrastructure gains of 10% are good 'cos they're cumulative) (b) it's invariably about the algorithms, and the starting point is instrumentation to detect the hotspots.

Starting with the assumption that it will be faster in C is doomed; find out what's actually slow, and see if there's a better way of doing it.

<edit>
Oh, I find myself contradicted, in an interesting way:
https://www.phoronix.com/scan.php?page=news_item&px=Solaris-Old-C-Code-To-Python
</edit>
_________________
Greybeard