Emerge - the future?

Message

Genone · Post by **Genone** » Wed Aug 20, 2003 1:00 am

For faster searching the esearch and fastsearch scripts (see Pythonheads portage utilities thread )are already available and reduce search times close to zero (using a static index).

vikwiz · Post by **vikwiz** » Wed Aug 20, 2003 1:02 am

() wrote:Is there some lightweight (possibly object oriented) database system that could be worth looking at apart from metakit?

ZODB (the database of Zope) would be a good choice.
It can run standalone, too, and provides a great webinterface if used in Zope.
It's fully object database, transactional, undoable, versionable, etc.

Forgot to mention: it's written in python, maybe some C routines.

vikwiz · Post by **vikwiz** » Wed Aug 20, 2003 1:37 am

Hoeken wrote:
-Instead of the regular 50 lines/second build dump, display good overall progress indicators
And how would the program know how far the build process has come? Make does not provide such information.

All these things have been discussed many times in other threads.

I would store the count of log lines in the database, so you can follow the progress.

RedBeard0531 · Post by **RedBeard0531** » Wed Aug 20, 2003 3:25 am

I actual wrote a script that creates the description file
http://forums.gentoo.org/viewtopic.php?t=57291
scroll down, its in there.

Tazz_ZA · Post by **Tazz_ZA** » Wed Aug 20, 2003 3:52 am

Tronic wrote: -Some intelligent system for pretty much automatically sharing the downloaded packages (distfiles) in a LAN (especially useful in campuses or home networks with many Gentoo boxes behind a single network link). Should probably be P2P, and not require setting up a real server and configuring all boxes to use it (and this is big enough to be a separate software project).

<comment>
NFS is what I use at work and accross a couple of WAN links - works for me
</comment>

raptor · Post by **raptor** » Wed Aug 20, 2003 7:10 am

ok here goes one :")

grep DESCRIPTION -r * > /tmp/desc

on my comp ~470Kb, not that big.

after that u just can use grep to search... one crontab that does this everyday and it ok...
so the only thing left is the format of the this file to be specified... after that -s and -S will be easy as cake :")), even depencities can be solved..
I'm proposing this :
ebuild-path/name|description|url|size|depend|rdepend

even better :

/usr/portage/ebuildDB.format
contains :

ebuildDB=/var/tmp/ebuild.db
fieldSeparator=|
name=$currentEbuild
description=/DESCRIPTION="(.*?)"/
url=/URL="(.*?)"/
#no extension just the filename
size=qx{du -h /usr/portage/distfiles/$justFilename}

u get the idea... there are couple of vars that are available on the behalf of the script i.e. the full path, fielname, fielname.ext etc..
if u dont know qx// is execute this..

andvin · Post by **andvin** » Wed Aug 20, 2003 8:27 am

Senso wrote:
MrPyro wrote: Also, one thing I find annoying about emerge at the moment is that some packages have warning or informational messages that come up during installations: not compiler warnings, things like the message that tells you an easy way to configure Apache to use mod_php during the mod_php build. I tend to start my "emerge -pU --deep world" process running then go out somewhere, rather than sit and stare at my monitor watching compiler output, so I miss these messages. Some system where these kinds of messages are logged, so that they can be read later, would be helpful.
Just a quick idea, but you could log normal output (1) to a file, cat it and remove all lines starting with "gcc" with sed. If you add more similar rules, you would maybe still have a lot of crap but it would be easier to browse the log and find useful messages.
An eventual command-line option doing this automatically is a good idea.

How about just adding "-s" to the MAKEOPTS variable? It should make the build process a lot more silent. Haven't tried it yet myself though...

Senso · Post by **Senso** » Wed Aug 20, 2003 11:07 am

andvin wrote:
Senso wrote: Just a quick idea, but you could log normal output (1) to a file, cat it and remove all lines starting with "gcc" with sed. If you add more similar rules, you would maybe still have a lot of crap but it would be easier to browse the log and find useful messages.
An eventual command-line option doing this automatically is a good idea.
How about just adding "-s" to the MAKEOPTS variable? It should make the build process a lot more silent. Haven't tried it yet myself though...

I just tried it and yes, it does remove GCC barf. You still get to see the errors (STD_ERR) and bash output. That's all I need to see, thanks for the tip.

pandora · Post by **pandora** » Wed Aug 20, 2003 11:43 am

The performance of portage is good enough as it is now. What is far more important is to improve the features: make it easier to remove packages that are not in the world file and have no dependancies, try to combat the explosion of USE flags.

One item which might be of significent value would be an rsync replacement that tells emerge clients what has changed since a date rather than the scanning everything across the Net.

I'd also like to see an option to check for just security releases. I currently emerge the production servers every day and even that is less often than I would like but I don't want to hammer the mirrors. On the other hand, I don't want to get hacked by someone downloading and running some sample exploit code in the 24hrs between my updates.

As for re-writing portage in C++: get a life! There are few languages less well suited to portage (or, indeed, any other task) than C++.

The idea of a custom database does at least make some sense: a database written for your one, single application will always be: a) easier to write than a general purpose DB by at least an order of magnitude, and b) out perform a general purpose DB by at least two orders of magnitude. Having said that, I don't think we actually need one.

RedBeard0531 · Post by **RedBeard0531** » Wed Aug 20, 2003 2:45 pm

raptor wrote:ok here goes one :")

grep DESCRIPTION -r * > /tmp/desc

on my comp ~470Kb, not that big.

after that u just can use grep to search... one crontab that does this everyday and it ok...
so the only thing left is the format of the this file to be specified... after that -s and -S will be easy as cake :")), even depencities can be solved..
I'm proposing this :
ebuild-path/name|description|url|size|depend|rdepend

even better :

/usr/portage/ebuildDB.format
contains :

ebuildDB=/var/tmp/ebuild.db
fieldSeparator=|
name=$currentEbuild
description=/DESCRIPTION="(.*?)"/
url=/URL="(.*?)"/
#no extension just the filename
size=qx{du -h /usr/portage/distfiles/$justFilename}

u get the idea... there are couple of vars that are available on the behalf of the script i.e. the full path, fielname, fielname.ext etc..
if u dont know qx// is execute this..

Take a look at /var/cache/edb/dep/

it doesnt help much. As to the DB, python has support for dictionaries (IIRC they're called assosiative arrays in other langauges), and that provides almost all the db features portage needs. Python also allows you to save any variable/object to disk. I have some patches that do this. The first onme is most likely to make it into portage.

http://bugs.gentoo.org/show_bug.cgi?id=26964

http://bugs.gentoo.org/show_bug.cgi?id=26447

rauar · Post by **rauar** » Thu Aug 21, 2003 2:06 pm

Check out this thread as I think this is one of the major problems with portage.

http://forums.gentoo.org/viewtopic.php?t=75565

Note that a more efficient database (faster read/write access) would result at least in this case in a much improved time performance.

NB: This IS definitly a problem !

Cheerz Al[/url]

ebrostig · Post by **ebrostig** » Thu Aug 21, 2003 6:18 pm

rauar wrote:Check out this thread as I think this is one of the major problems with portage.

http://forums.gentoo.org/viewtopic.php?t=75565

Note that a more efficient database (faster read/write access) would result at least in this case in a much improved time performance.

NB: This IS definitly a problem !

Cheerz Al[/url]

No, it's not a problem. See my response in the thread.

Why is it a Portage problem that the guy does not have enough disk space?

Erik

rauar · Post by **rauar** » Thu Aug 21, 2003 6:39 pm

ebrostig wrote: No, it's not a problem. See my response in the thread.

Why is it a Portage problem that the guy does not have enough disk space?

Erik

Really nice that portage is that much bloated. Gentoo as fully flexible and configurable linux distribution where one can tweak to death with a package management which takes up more than 50% of a minimum installation is a joke. And portage doesn't really need to hold 300mb of data. There's not THAT much information in the tree.

I got my stuff compiled by deleting some directories in the tree. Ok, nice workaround. And yes, I CAN buy a new harddrive. But the other threads intention was to show that portage is (noticable) bigger and slower than it needs to be.

As I said. I can stick with it. But I bet the gentoo distribution wouldn't need that many mirrors to hold the daily loads of rsyncs if someone would think about this a little more closer.

Have fun...

carambola5 · Post by **carambola5** » Thu Aug 21, 2003 8:36 pm

I know it wouldn't seem like much at first, but what if there was a single flat file that held all of the descriptions of the packages? This way, the DESCRIPTION line (and quite possible HOMEPAGE, if one were so inclined) could be omitted from all of the ebuilds. Updates (rare as they would be) to the flat file could be handled with diff files.

Like I said, it wouldn't be much savings for a single gentoo user, but the results could be quite favorable for the rsync server. And we all love our rsync servers, right?

mrchaotica · Post by **mrchaotica** » Thu Aug 21, 2003 11:21 pm

fdavid wrote:Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps. This is quite annoying to download and compile additional 40-60 MB for an app with 3-4MB size, even if it's not really needed. Not speaking of the unneeded network and server load. Besides, if you emerge these apps with the -O option, you will always have their dependencies in your update world list, which is also disturbing, because I can think that I really need the ebuild, because I can't remember that it was just an unneeded dependency.
It would be nice to make difference between "must have" and "nice to have" dependencies.

+1 Insightful! : )

It might be nice not to have portage trying to emerge X just because some program has an optional GUI!

far · Post by **far** » Thu Aug 21, 2003 11:36 pm

mrchaotica wrote:It might be nice not to have portage trying to emerge X just because some program has an optional GUI!

This is what USE flags are for. Just add "-X" to USE in /etc/make.conf

ebrostig · Post by **ebrostig** » Fri Aug 22, 2003 1:09 am

rauar wrote:
ebrostig wrote: No, it's not a problem. See my response in the thread.

Why is it a Portage problem that the guy does not have enough disk space?

Erik
Really nice that portage is that much bloated. Gentoo as fully flexible and configurable linux distribution where one can tweak to death with a package management which takes up more than 50% of a minimum installation is a joke. And portage doesn't really need to hold 300mb of data. There's not THAT much information in the tree.

Maybe you would care to point out what is bloat among the information in the Portage tree? As far as I can see, there isn't much that could be removed.

The only thing I see is the metadata directory and all the patches etc that is now in the Portage tree whereas earlier the patches were downloaded when you started the emerge of a package.

rauar wrote: I got my stuff compiled by deleting some directories in the tree. Ok, nice workaround. And yes, I CAN buy a new harddrive. But the other threads intention was to show that portage is (noticable) bigger and slower than it needs to be.

As I said. I can stick with it. But I bet the gentoo distribution wouldn't need that many mirrors to hold the daily loads of rsyncs if someone would think about this a little more closer.

Have fun...

You are more than welcome to work out a different approach to the problem and submit patches/suggestions etc. Just complaining and not presenting a solution can be fun, but not very productive.

My take on Portage is as follows:
1. Yes, it may be possible to use a database, but the size will not be smaller, but larger if that gets implemented.

2. Better to concentrate on changing the behaviour of portage when it comes to searching than to completly rewrite it.

3. Total time spent inside the Portage code during an install of upgrade is very low compared to the time it takes to download and compile the packages, hence it is not much to gain from messing up Portage totally.

Erik

Post by **KingTaco** » Fri Aug 22, 2003 1:13 am

-Threaded - after figuring out the dependancies, it could probably begin downloading all the packages (there could be a limit for simultaneus downloads, of course) and after some downloads are finished, the building of those packages can begin (assuming of course that the deps required for building are met). This gives really much improved performance, because network I/O and CPU-intensive compiling running side-by-side don't give any performance hit to each-other.

This will also cause a performance hit if more than 1-3 makes are going on at the same time. Why not set up multiple processes working on a central queue i.e. 2 procs downloading from 2 mirrors and 3 makes?

Genone · Post by **Genone** » Fri Aug 22, 2003 1:39 am

ebrostig wrote:Maybe you would care to point out what is bloat among the information in the Portage tree? As far as I can see, there isn't much that could be removed.

The only thing I see is the metadata directory and all the patches etc that is now in the Portage tree whereas earlier the patches were downloaded when you started the emerge of a package.

...

My take on Portage is as follows:
1. Yes, it may be possible to use a database, but the size will not be smaller, but larger if that gets implemented.

While I generally agree with you I think you missed the point here: The portage tree in /usr/portage takes up >200 MB of disc space, but if you only take the content from the files in there you end up with <50 MB, so there are >150 MB space wasted for the filesystem. You can verify that with

Code: Select all

find /usr/portage -type f | xargs -i cat {} >> /tmp/portage-tree
ls -l /tmp/portage-tree

(after moving distfiles and packages out of /usr/portage of course).

I don't think a real database would produce >75% overhead as it is now.

Yossarian · Post by **Yossarian** » Fri Aug 22, 2003 3:19 am

I think it would be useful if portage could somehow make a guestimate (sp?) of compile time based on the specs of your system, before you emerge the package. So you'd know whether you should compile it while you sleep, or take a backpacking trip through the andes etc.

fdavid · Post by **fdavid** » Fri Aug 22, 2003 9:18 am

far wrote:
mrchaotica wrote:It might be nice not to have portage trying to emerge X just because some program has an optional GUI!
This is what USE flags are for. Just add "-X" to USE in /etc/make.conf

Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies.
And, as it was discussed before, system wide USE flags don't always give the required result.

far · Post by **far** » Fri Aug 22, 2003 9:27 am

fdavid wrote:Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies.

Just do "etcat -u your_package". Or do you want a "USE=-everything" flag?

fdavid wrote:And, as it was discussed before, system wide USE flags don't always give the required result.

Then that is the fault of the ebuild writer, not portage itself.

fdavid · Post by **fdavid** » Fri Aug 22, 2003 10:05 am

far wrote:
fdavid wrote:Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies.
Just do "etcat -u your_package". Or do you want a "USE=-everything" flag?

It's just not what i meant. This whole thing is not abaout USE flags. Please read my original post:

fdavid wrote:Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps. This is quite annoying to download and compile additional 40-60 MB for an app with 3-4MB size, even if it's not really needed. Not speaking of the unneeded network and server load. Besides, if you emerge these apps with the -O option, you will always have their dependencies in your update world list, which is also disturbing, because I can think that I really need the ebuild, because I can't remember that it was just an unneeded dependency.

It would be nice to make difference between "must have" and "nice to have" dependencies.

So USE flags are not everything and cannot be used everywhere efficiently. So your answer to mrchaotica was a bit offtopic, because we were not discussing about USE flags. Yes, in case of X, a USE flag can be a solution of the problem, but generally not, and i think that the emphasis was on the "optional" and not on "GUI".

far wrote:
fdavid wrote:And, as it was discussed before, system wide USE flags don't always give the required result.
Then that is the fault of the ebuild writer, not portage itself.

It's also not what i meant. It's not about weather a USE flag in a specific ebuild really works or not. It's just about that general USE flags are not always a good idea. There was a discussion in this topic of specializing USE flags for ebuilds. I don't know if it's the right solution, but i do agree that general USE flags are not always optimal.

far · Post by **far** » Fri Aug 22, 2003 10:58 am

fdavid wrote:
fdavid wrote:Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps.
...
It would be nice to make difference between "must have" and "nice to have" dependencies.
So USE flags are not everything and cannot be used everywhere efficiently.

What use is a "nice to have" dependency? Either you need something or you don't. A "nice to have" dependency would only serve as a comment and would not be useful when installing something automatically.

fdavid wrote:There was a discussion in this topic of specializing USE flags for ebuilds.

Yeah, well it sounds like a good idea.

fdavid · Post by **fdavid** » Fri Aug 22, 2003 11:27 am

far wrote:What use is a "nice to have" dependency? Either you need something or you don't. A "nice to have" dependency would only serve as a comment and would not be useful when installing something automatically.

Maybe you are right, but sometimes it could be handy. If we stay at my previous example kdeveleop: It's nice to have i.g. doxygen together with kdevelop, and it is well integrated in it, once you have doxygen installed. But you don't need doxygen to run kdevelop. Maybe some people would install and use it, some other not. At the moment kdevelop does depend on doxygen in portage. I'm not totally against it, because we would lose some information otherwise. But this dependency should be marked soemhow "nice to have" and should also behave a bit different from the "must have" dependencies.

But i absolutely agree with you, that it's better not to have a "nice to have" dependency as dependency, than to have it as "must have".

Emerge - the future?

ZODB

Re: Emerge - the future?

Re: Emerge - the future?

one easy,fasr and aproximatly good solution

A dissenting voice

Re: one easy,fasr and aproximatly good solution