Portage rewrite in C?

charlieg · Posted: Mon May 19, 2003 9:33 am Post subject: Portage rewrite in C?

I just read this post containing the following statement:

Lovechild · Posted: Mon May 19, 2003 10:08 am Post subject:

There was a project going at one time to rewrite portage in c++ using some of the database features in QT (something called tinyQT was being developed for this I believe).

But the notion that Portage was to be written in something other than Python has been talked about before but there was to my knowledge been made no formal statement from the Portage developers. However developers of some of the portage GUIs have made statements in that regards - which they claim is based on developer information.

So it might be true, but I will remain doubtful untill I hear Carpaski make an official statement with reasons for doing this.

idl · Posted: Mon May 19, 2003 10:08 am Post subject:

C does have one major advantage though... Speed. Portage uses scripts a lot , and they can be called many times during a merge of a large package. The KDE ebuild maintainer said the reason they don't offer KDE in individual packages is because of the amount of time it would take to run all the portage scripts over and over.

I personaly would like portage to be writen in C and run as a daemon.
_________________
a.k.a port001
Found a bug? Please report it: Gentoo Bugzilla

Scandium · Posted: Mon May 19, 2003 10:26 am Post subject:

According to carpaski (just talked to him) Portage will remain python, although there may very well be a C API.

Lovechild · Posted: Mon May 19, 2003 10:38 am Post subject:

Ari Rahikkala · Guru Joined: 02 Oct 2002 Posts: 370 Location: Finland

Somebody will have to explain me why Portage should be a daemon... the only difference that I can see it would make is that it would use a lot of memory all the time instead of only when you're emerging something...

charlieg · Posted: Mon May 19, 2003 1:37 pm Post subject:

panserg · Apprentice Joined: 16 Apr 2003 Posts: 188

Portage has a higher level of fine-grained control than other package management systems. I think that require a much highel level of programming language than C. Also Portage is very intensively developed. Therefore the scripting language is very essential.

So, re-writing Portage to the compilable language of lower level then Python will kill the project or it will end up as Mozilla: overbloated custom-build problem-oriented interpreter of XUL. Remember Greenspun's Tenth Rule of Programming?

charlieg · Posted: Mon May 19, 2003 1:58 pm Post subject:

Lovechild · Posted: Mon May 19, 2003 3:39 pm Post subject:

Lin-matt and "some guy I can't remember" is writting an SQL version of portage so to speed up database lookups - and having tried out an early test release I must say it's BLAZING fast for lookups, but again that requires one to have a real SQL server running which isn't suitable for everyone - and it's not intended for "Joe Average" as far as I understand, but it opens op a whole heap of possibilities.

sundiver2k · Posted: Mon May 19, 2003 5:00 pm Post subject:

I don't get it.

What would rewriting Portage in C allow us to do that we can't do now? I don't think that Portage is slow. It's the configure and compiles that are slow. Portage just fires them off, so rewriting it in C won't give you much of a speedup.

As far as storing stuff in a database, I vote against that. Ever had a corrupted RPM database? It absolutely sucks. At least we can edit the individual files in Portage with a text editor and fix what might be broken.

If someone wants to do this for fun, I'm all for that, but I don't believe there is a need to convert Portage.

shm · Posted: Tue May 20, 2003 10:29 pm Post subject:

BradB · Posted: Wed May 21, 2003 12:11 am Post subject:

Optimise the thing that takes up most of your time first. If emerging a package foo takes 10 minutes, and only 30 seconds of that (5%) is portage doing stuff, then nothing you do in portage will give any noticable speedup, as I see it for actually emerging (compiling) stuff portage needs nothing done to it because it is not the slow part.
However, for lookups & searches which is all Portage, then we can actually make some improvement. I don't know how it works now, but I would think the search could be helped with the aid of a cache or database system. The database wouldn't need to be very fancy and you know exactly the data you will have (ebuilds) so for searching you could implement the database in Portage and not rely on other languages.
My Rules of Optimising
1) Optimise the bigest % of time first
2) Optimise the algorithm
3) Use a faster language (this is like the final resort, unless the language is actually the largest % of time)

Brad

panserg · Apprentice Joined: 16 Apr 2003 Posts: 188

ebrostig · Posted: Wed May 21, 2003 2:04 am Post subject:

Sometimes it is interesting to see all the suggestions about changes to Portage. Most of them is about rewriting Portage in a different language and doing so is not adding anything to Portage that is not there today or that can not be done in Portage.

I have to agree with all of you who points out that the work actually done by Portage is minimal compared to the rest of the emerge process (download, configure, compile and install).

Why does everything has to be re-written into C or C++? My basic take on program development has always been: Use the language best suited to the task you need to perform.

I'm not a Python guru, nor a perl guru, but I know my way around many languages and so far I haven't seen any indication that Python is not well-suited to the task.

I'm sure the development time is much less in Python than in C given the more "mature" syntax in many ways. Now, I would never suggest rewriting the Linux kernel nor the GCC system in Python either, it just doesn't make sense

Until someone can demonstrate features that is necessary in Portage that can not be implemented in Python, I suggest we leave Portage as it is and let the developers concentrate on making Gentoo even better.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

masseya · Posted: Wed May 21, 2003 4:18 pm Post subject:

asimon · Posted: Wed May 21, 2003 5:06 pm Post subject:

Before one makes any thoughts about how to best rewrite and when in what language Portage, I think it would make sense to profile Portage first. Only if you know where the actuall bottlenecks are can you dispose them. Otherwise you do the same errors, just in an other language. Doesn't help.

telex4 · l33t Joined: 21 Sep 2002 Posts: 704 Location: Reading, UK

Personally I don't care a jot about Portage taking a few seconds to get started, or even taking 30 seconds to scan a database of text files, rather than 5 seconds as an optimised C app accessing a proper database. It's not exactly much time, especially when you consider how long compiling takes, and all the advantages of having text-editable Python files to my mind outweigh the negligible speed "problems" (if they can be called that).

APIs in other languages would be good, but I'd rather not see Portage be rewritten in C just for the sake of it. Actually, I've not looked at the Portage code much, so this might already be in place, but it'd be cool just to have an extendable Portage Python module so we could write Portage-based apps without having to pipe portage and work with its output

ebrostig · Posted: Wed May 21, 2003 6:32 pm Post subject:

telex4, you are right on the money! Well, said!

masseya, you agree witrh me?????????? OMG! (pops open a bottle of Champagne)

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

beowulf · Apprentice Joined: 07 Apr 2003 Posts: 225

Forgive me if i'm wrong, but isn't the bottleneck of portage searching and calculating dependencies the filesystem? A re-write in C would be nice, and you may save a millesecond or two... but wouldn't a change in the method that portage stores the info help more? I've been watching PortageSQL, a project i seen on these forums... although i wouldn't need something so robust, i can imagine the speed benefit would be exponentially higher than a re-write in C.

My 2 cents

_________________
I have nothing witty to say here... ever :-(

sundiver2k · Posted: Wed May 21, 2003 11:27 pm Post subject:

C'mon folks. We're talking about Portage here. If you're using Portage all the time, you aren't using your computer.

Portage is supposed to make it easy to get software on your computer. It does that very well. If all you ever do is install software, then feel free to optimize the hell out of Portage. Use all the databases you can think of. Write the interface in C and hand-tune assembly language. Profile it, optimize and tune, tune tune away. If that's what you want to do, that's fine with me.

I, on the other hand, will be using my computer(s) for more things than just installing software packages. I don't want to spend all of my time installing software just to see how fast it happens. I just want to use the software on my computer not install it all the time.

liquidx · Posted: Thu May 22, 2003 3:47 am Post subject:

asimon · Posted: Thu May 22, 2003 9:12 am Post subject:

So some people want a faster Portage. And who has the needed time and competence to do this? I think this discussion is totaly pointless. As always with such things, talking brings nothing (are the actuall Portage developers even included here? No? Better file a wish into bugzilla, oh ... tuning Portage is already there.), coding on the other hand can. IMHO there need first some code or interesting profiling data before such a discussion makes sense. People want a rewrite in Language X and don't even have proof that it brings something. They don't even have the time to do it, well at least I see no code here. So what's the point? Telling the actuall Protage developers how they can do their work better? Don't you think they know the bottlenecks better than someone who did'nt even ever read and understood the Portage source code?

charlieg · Posted: Thu May 22, 2003 10:15 am Post subject: