| View previous topic :: View next topic |
| Author |
Message |
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9527
Location: beyond the rim
|
 Posted: Wed Aug 20, 2003 1:00 am Post subject: |
 |
|
| For faster searching the esearch and fastsearch scripts (see Pythonheads portage utilities thread )are already available and reduce search times close to zero (using a static index). |
|
| Back to top |
|
 |
vikwiz
n00b
Joined: 01 Mar 2003
Posts: 50
Location: Budapest
|
| Posted: Wed Aug 20, 2003 1:02 am Post subject: ZODB |
|
|
| () wrote: | | Is there some lightweight (possibly object oriented) database system that could be worth looking at apart from metakit? |
ZODB (the database of Zope) would be a good choice.
It can run standalone, too, and provides a great webinterface if used in Zope.
It's fully object database, transactional, undoable, versionable, etc.
Forgot to mention: it's written in python, maybe some C routines. |
|
| Back to top |
|
|
vikwiz
n00b
Joined: 01 Mar 2003
Posts: 50
Location: Budapest
|
| Posted: Wed Aug 20, 2003 1:37 am Post subject: Re: Emerge - the future? |
|
|
| Hoeken wrote: |
| Quote: | | -Instead of the regular 50 lines/second build dump, display good overall progress indicators |
And how would the program know how far the build process has come? Make does not provide such information.
All these things have been discussed many times in other threads. |
I would store the count of log lines in the database, so you can follow the progress. |
|
| Back to top |
|
|
RedBeard0531
Guru
Joined: 21 Sep 2002
Posts: 415
Location: maryland
|
| Posted: Wed Aug 20, 2003 3:25 am Post subject: |
|
|
I actual wrote a script that creates the description file
https://forums.gentoo.org/viewtopic.php?t=57291
scroll down, its in there.
_________________
OH MY GOD! Kenny just killed Kenny!
That Basterd! |
|
| Back to top |
|
|
Tazz_ZA
n00b
Joined: 02 Jul 2003
Posts: 60
Location: Durban, South Africa
|
| Posted: Wed Aug 20, 2003 3:52 am Post subject: Re: Emerge - the future? |
|
|
| Tronic wrote: |
-Some intelligent system for pretty much automatically sharing the downloaded packages (distfiles) in a LAN (especially useful in campuses or home networks with many Gentoo boxes behind a single network link). Should probably be P2P, and not require setting up a real server and configuring all boxes to use it (and this is big enough to be a separate software project).
<comment>
NFS is what I use at work and accross a couple of WAN links - works for me
</comment>
|
|
|
| Back to top |
|
|
raptor
Apprentice
Joined: 20 Sep 2002
Posts: 171
|
| Posted: Wed Aug 20, 2003 7:10 am Post subject: one easy,fasr and aproximatly good solution |
|
|
ok here goes one :")
grep DESCRIPTION -r * > /tmp/desc
on my comp ~470Kb, not that big.
after that u just can use grep to search... one crontab that does this everyday and it ok...
so the only thing left is the format of the this file to be specified... after that -s and -S will be easy as cake :")), even depencities can be solved..
I'm proposing this :
ebuild-path/name|description|url|size|depend|rdepend
even better :
/usr/portage/ebuildDB.format
contains :
ebuildDB=/var/tmp/ebuild.db
fieldSeparator=|
name=$currentEbuild
description=/DESCRIPTION="(.*?)"/
url=/URL="(.*?)"/
#no extension just the filename
size=qx{du -h /usr/portage/distfiles/$justFilename}
u get the idea... there are couple of vars that are available on the behalf of the script i.e. the full path, fielname, fielname.ext etc..
if u dont know qx// is execute this.. |
|
| Back to top |
|
|
andvin
n00b
Joined: 27 May 2003
Posts: 23
Location: Linköping, Sweden
|
| Posted: Wed Aug 20, 2003 8:27 am Post subject: |
|
|
| Senso wrote: | | MrPyro wrote: |
Also, one thing I find annoying about emerge at the moment is that some packages have warning or informational messages that come up during installations: not compiler warnings, things like the message that tells you an easy way to configure Apache to use mod_php during the mod_php build. I tend to start my "emerge -pU --deep world" process running then go out somewhere, rather than sit and stare at my monitor watching compiler output, so I miss these messages. Some system where these kinds of messages are logged, so that they can be read later, would be helpful. |
Just a quick idea, but you could log normal output (1) to a file, cat it and remove all lines starting with "gcc" with sed. If you add more similar rules, you would maybe still have a lot of crap but it would be easier to browse the log and find useful messages.
An eventual command-line option doing this automatically is a good idea. |
How about just adding "-s" to the MAKEOPTS variable? It should make the build process a lot more silent. Haven't tried it yet myself though... |
|
| Back to top |
|
|
Senso
Apprentice
Joined: 17 Jun 2003
Posts: 250
Location: Montreal, Quebec
|
| Posted: Wed Aug 20, 2003 11:07 am Post subject: |
|
|
| andvin wrote: | | Senso wrote: |
Just a quick idea, but you could log normal output (1) to a file, cat it and remove all lines starting with "gcc" with sed. If you add more similar rules, you would maybe still have a lot of crap but it would be easier to browse the log and find useful messages.
An eventual command-line option doing this automatically is a good idea. |
How about just adding "-s" to the MAKEOPTS variable? It should make the build process a lot more silent. Haven't tried it yet myself though... |
I just tried it and yes, it does remove GCC barf. You still get to see the errors (STD_ERR) and bash output. That's all I need to see, thanks for the tip.  |
|
| Back to top |
|
|
pandora
Tux's lil' helper
Joined: 25 Sep 2002
Posts: 93
Location: UK
|
| Posted: Wed Aug 20, 2003 11:43 am Post subject: A dissenting voice |
|
|
The performance of portage is good enough as it is now. What is far more important is to improve the features: make it easier to remove packages that are not in the world file and have no dependancies, try to combat the explosion of USE flags.
One item which might be of significent value would be an rsync replacement that tells emerge clients what has changed since a date rather than the scanning everything across the Net.
I'd also like to see an option to check for just security releases. I currently emerge the production servers every day and even that is less often than I would like but I don't want to hammer the mirrors. On the other hand, I don't want to get hacked by someone downloading and running some sample exploit code in the 24hrs between my updates.
As for re-writing portage in C++: get a life! There are few languages less well suited to portage (or, indeed, any other task) than C++.
The idea of a custom database does at least make some sense: a database written for your one, single application will always be: a) easier to write than a general purpose DB by at least an order of magnitude, and b) out perform a general purpose DB by at least two orders of magnitude. Having said that, I don't think we actually need one.
Last edited by pandora on Wed Aug 20, 2003 3:00 pm; edited 1 time in total |
|
| Back to top |
|
|
RedBeard0531
Guru
Joined: 21 Sep 2002
Posts: 415
Location: maryland
|
| Posted: Wed Aug 20, 2003 2:45 pm Post subject: Re: one easy,fasr and aproximatly good solution |
|
|
| raptor wrote: | ok here goes one :")
grep DESCRIPTION -r * > /tmp/desc
on my comp ~470Kb, not that big.
after that u just can use grep to search... one crontab that does this everyday and it ok...
so the only thing left is the format of the this file to be specified... after that -s and -S will be easy as cake :")), even depencities can be solved..
I'm proposing this :
ebuild-path/name|description|url|size|depend|rdepend
even better :
/usr/portage/ebuildDB.format
contains :
ebuildDB=/var/tmp/ebuild.db
fieldSeparator=|
name=$currentEbuild
description=/DESCRIPTION="(.*?)"/
url=/URL="(.*?)"/
#no extension just the filename
size=qx{du -h /usr/portage/distfiles/$justFilename}
u get the idea... there are couple of vars that are available on the behalf of the script i.e. the full path, fielname, fielname.ext etc..
if u dont know qx// is execute this.. |
Take a look at /var/cache/edb/dep/
it doesnt help much. As to the DB, python has support for dictionaries (IIRC they're called assosiative arrays in other langauges), and that provides almost all the db features portage needs. Python also allows you to save any variable/object to disk. I have some patches that do this. The first onme is most likely to make it into portage.
https://bugs.gentoo.org/show_bug.cgi?id=26964
https://bugs.gentoo.org/show_bug.cgi?id=26447
_________________
OH MY GOD! Kenny just killed Kenny!
That Basterd! |
|
| Back to top |
|
|
rauar
n00b
Joined: 15 Mar 2003
Posts: 37
|
| Posted: Thu Aug 21, 2003 2:06 pm Post subject: |
|
|
Check out this thread as I think this is one of the major problems with portage.
https://forums.gentoo.org/viewtopic.php?t=75565
Note that a more efficient database (faster read/write access) would result at least in this case in a much improved time performance.
NB: This IS definitly a problem !
Cheerz Al[/url] |
|
| Back to top |
|
|
ebrostig
Bodhisattva
Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl
|
| Posted: Thu Aug 21, 2003 6:18 pm Post subject: |
|
|
| rauar wrote: | Check out this thread as I think this is one of the major problems with portage.
https://forums.gentoo.org/viewtopic.php?t=75565
Note that a more efficient database (faster read/write access) would result at least in this case in a much improved time performance.
NB: This IS definitly a problem !
Cheerz Al[/url] |
No, it's not a problem. See my response in the thread.
Why is it a Portage problem that the guy does not have enough disk space?
Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.' |
|
| Back to top |
|
|
rauar
n00b
Joined: 15 Mar 2003
Posts: 37
|
| Posted: Thu Aug 21, 2003 6:39 pm Post subject: |
|
|
| ebrostig wrote: |
No, it's not a problem. See my response in the thread.
Why is it a Portage problem that the guy does not have enough disk space?
Erik |
Really nice that portage is that much bloated. Gentoo as fully flexible and configurable linux distribution where one can tweak to death with a package management which takes up more than 50% of a minimum installation is a joke. And portage doesn't really need to hold 300mb of data. There's not THAT much information in the tree.
I got my stuff compiled by deleting some directories in the tree. Ok, nice workaround. And yes, I CAN buy a new harddrive. But the other threads intention was to show that portage is (noticable) bigger and slower than it needs to be.
As I said. I can stick with it. But I bet the gentoo distribution wouldn't need that many mirrors to hold the daily loads of rsyncs if someone would think about this a little more closer.
Have fun... |
|
| Back to top |
|
|
carambola5
Apprentice
Joined: 10 Jul 2002
Posts: 214
|
| Posted: Thu Aug 21, 2003 8:36 pm Post subject: |
|
|
I know it wouldn't seem like much at first, but what if there was a single flat file that held all of the descriptions of the packages? This way, the DESCRIPTION line (and quite possible HOMEPAGE, if one were so inclined) could be omitted from all of the ebuilds. Updates (rare as they would be) to the flat file could be handled with diff files.
Like I said, it wouldn't be much savings for a single gentoo user, but the results could be quite favorable for the rsync server. And we all love our rsync servers, right? |
|
| Back to top |
|
|
mrchaotica
n00b
Joined: 14 Dec 2002
Posts: 45
|
| Posted: Thu Aug 21, 2003 11:21 pm Post subject: |
|
|
| fdavid wrote: | Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps. This is quite annoying to download and compile additional 40-60 MB for an app with 3-4MB size, even if it's not really needed. Not speaking of the unneeded network and server load. Besides, if you emerge these apps with the -O option, you will always have their dependencies in your update world list, which is also disturbing, because I can think that I really need the ebuild, because I can't remember that it was just an unneeded dependency.
It would be nice to make difference between "must have" and "nice to have" dependencies. |
+1 Insightful! : )
It might be nice not to have portage trying to emerge X just because some program has an optional GUI!
_________________
forty-two is the Answer |
|
| Back to top |
|
|
far
Guru
Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden
|
| Posted: Thu Aug 21, 2003 11:36 pm Post subject: |
|
|
| mrchaotica wrote: | | It might be nice not to have portage trying to emerge X just because some program has an optional GUI! |
This is what USE flags are for. Just add "-X" to USE in /etc/make.conf
_________________
The Porthole Portage Frontend |
|
| Back to top |
|
|
ebrostig
Bodhisattva
Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl
|
| Posted: Fri Aug 22, 2003 1:09 am Post subject: |
|
|
| rauar wrote: | | ebrostig wrote: |
No, it's not a problem. See my response in the thread.
Why is it a Portage problem that the guy does not have enough disk space?
Erik |
Really nice that portage is that much bloated. Gentoo as fully flexible and configurable linux distribution where one can tweak to death with a package management which takes up more than 50% of a minimum installation is a joke. And portage doesn't really need to hold 300mb of data. There's not THAT much information in the tree.
|
Maybe you would care to point out what is bloat among the information in the Portage tree? As far as I can see, there isn't much that could be removed.
The only thing I see is the metadata directory and all the patches etc that is now in the Portage tree whereas earlier the patches were downloaded when you started the emerge of a package.
| rauar wrote: |
I got my stuff compiled by deleting some directories in the tree. Ok, nice workaround. And yes, I CAN buy a new harddrive. But the other threads intention was to show that portage is (noticable) bigger and slower than it needs to be.
As I said. I can stick with it. But I bet the gentoo distribution wouldn't need that many mirrors to hold the daily loads of rsyncs if someone would think about this a little more closer.
Have fun... |
You are more than welcome to work out a different approach to the problem and submit patches/suggestions etc. Just complaining and not presenting a solution can be fun, but not very productive.
My take on Portage is as follows:
1. Yes, it may be possible to use a database, but the size will not be smaller, but larger if that gets implemented.
2. Better to concentrate on changing the behaviour of portage when it comes to searching than to completly rewrite it.
3. Total time spent inside the Portage code during an install of upgrade is very low compared to the time it takes to download and compile the packages, hence it is not much to gain from messing up Portage totally.
Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.' |
|
| Back to top |
|
|
KingTaco
Developer
Joined: 22 Aug 2003
Posts: 207
Location: Bay Area, CA
|
| Posted: Fri Aug 22, 2003 1:13 am Post subject: |
|
|
| Quote: | | -Threaded - after figuring out the dependancies, it could probably begin downloading all the packages (there could be a limit for simultaneus downloads, of course) and after some downloads are finished, the building of those packages can begin (assuming of course that the deps required for building are met). This gives really much improved performance, because network I/O and CPU-intensive compiling running side-by-side don't give any performance hit to each-other. |
This will also cause a performance hit if more than 1-3 makes are going on at the same time. Why not set up multiple processes working on a central queue i.e. 2 procs downloading from 2 mirrors and 3 makes?
_________________
Explaining the obvious to the oblivious.
Adopt an unanswered post today -- https://forums.gentoo.org/search.php?search_id=unanswered |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9527
Location: beyond the rim
|
| Posted: Fri Aug 22, 2003 1:39 am Post subject: |
|
|
| ebrostig wrote: | Maybe you would care to point out what is bloat among the information in the Portage tree? As far as I can see, there isn't much that could be removed.
The only thing I see is the metadata directory and all the patches etc that is now in the Portage tree whereas earlier the patches were downloaded when you started the emerge of a package.
...
My take on Portage is as follows:
1. Yes, it may be possible to use a database, but the size will not be smaller, but larger if that gets implemented. |
While I generally agree with you I think you missed the point here: The portage tree in /usr/portage takes up >200 MB of disc space, but if you only take the content from the files in there you end up with <50 MB, so there are >150 MB space wasted for the filesystem. You can verify that with
| Code: | find /usr/portage -type f | xargs -i cat {} >> /tmp/portage-tree
ls -l /tmp/portage-tree |
(after moving distfiles and packages out of /usr/portage of course).
I don't think a real database would produce >75% overhead as it is now. |
|
| Back to top |
|
|
Yossarian
Tux's lil' helper
Joined: 15 Jul 2003
Posts: 117
Location: Austin, Tx.
|
| Posted: Fri Aug 22, 2003 3:19 am Post subject: |
|
|
| I think it would be useful if portage could somehow make a guestimate (sp?) of compile time based on the specs of your system, before you emerge the package. So you'd know whether you should compile it while you sleep, or take a backpacking trip through the andes etc. |
|
| Back to top |
|
|
fdavid
Tux's lil' helper
Joined: 27 Apr 2003
Posts: 82
Location: Graz, Austria, EU
|
| Posted: Fri Aug 22, 2003 9:18 am Post subject: |
|
|
| far wrote: | | mrchaotica wrote: | | It might be nice not to have portage trying to emerge X just because some program has an optional GUI! |
This is what USE flags are for. Just add "-X" to USE in /etc/make.conf |
Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies.
And, as it was discussed before, system wide USE flags don't always give the required result. |
|
| Back to top |
|
|
far
Guru
Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden
|
| Posted: Fri Aug 22, 2003 9:27 am Post subject: |
|
|
| fdavid wrote: | | Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies. |
Just do "etcat -u your_package". Or do you want a "USE=-everything" flag?
| fdavid wrote: | | And, as it was discussed before, system wide USE flags don't always give the required result. |
Then that is the fault of the ebuild writer, not portage itself.
_________________
The Porthole Portage Frontend |
|
| Back to top |
|
|
fdavid
Tux's lil' helper
Joined: 27 Apr 2003
Posts: 82
Location: Graz, Austria, EU
|
| Posted: Fri Aug 22, 2003 10:05 am Post subject: |
|
|
| far wrote: | | fdavid wrote: | | Yes, this one exists, but X is just one thing. So USE falgs hardly can be used for distinguishing between different types of dependencies. |
Just do "etcat -u your_package". Or do you want a "USE=-everything" flag? |
It's just not what i meant. This whole thing is not abaout USE flags. Please read my original post:
| fdavid wrote: | Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps. This is quite annoying to download and compile additional 40-60 MB for an app with 3-4MB size, even if it's not really needed. Not speaking of the unneeded network and server load. Besides, if you emerge these apps with the -O option, you will always have their dependencies in your update world list, which is also disturbing, because I can think that I really need the ebuild, because I can't remember that it was just an unneeded dependency.
It would be nice to make difference between "must have" and "nice to have" dependencies. |
So USE flags are not everything and cannot be used everywhere efficiently. So your answer to mrchaotica was a bit offtopic, because we were not discussing about USE flags. Yes, in case of X, a USE flag can be a solution of the problem, but generally not, and i think that the emphasis was on the "optional" and not on "GUI".
| far wrote: | | fdavid wrote: | | And, as it was discussed before, system wide USE flags don't always give the required result. |
Then that is the fault of the ebuild writer, not portage itself. |
It's also not what i meant. It's not about weather a USE flag in a specific ebuild really works or not. It's just about that general USE flags are not always a good idea. There was a discussion in this topic of specializing USE flags for ebuilds. I don't know if it's the right solution, but i do agree that general USE flags are not always optimal. |
|
| Back to top |
|
|
far
Guru
Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden
|
| Posted: Fri Aug 22, 2003 10:58 am Post subject: |
|
|
| fdavid wrote: | | fdavid wrote: | Another idea:
The importance of dependendcies can be different. For example, there is kdevelop or k3b with a plenty of depending ebuilds, but just a few (or none) of them are really needed to run these apps.
...
It would be nice to make difference between "must have" and "nice to have" dependencies. |
So USE flags are not everything and cannot be used everywhere efficiently. |
What use is a "nice to have" dependency? Either you need something or you don't. A "nice to have" dependency would only serve as a comment and would not be useful when installing something automatically.
| fdavid wrote: | | There was a discussion in this topic of specializing USE flags for ebuilds. |
Yeah, well it sounds like a good idea.
_________________
The Porthole Portage Frontend |
|
| Back to top |
|
|
fdavid
Tux's lil' helper
Joined: 27 Apr 2003
Posts: 82
Location: Graz, Austria, EU
|
| Posted: Fri Aug 22, 2003 11:27 am Post subject: |
|
|
| far wrote: | | What use is a "nice to have" dependency? Either you need something or you don't. A "nice to have" dependency would only serve as a comment and would not be useful when installing something automatically. |
Maybe you are right, but sometimes it could be handy. If we stay at my previous example kdeveleop: It's nice to have i.g. doxygen together with kdevelop, and it is well integrated in it, once you have doxygen installed. But you don't need doxygen to run kdevelop. Maybe some people would install and use it, some other not. At the moment kdevelop does depend on doxygen in portage. I'm not totally against it, because we would lose some information otherwise. But this dependency should be marked soemhow "nice to have" and should also behave a bit different from the "must have" dependencies.
But i absolutely agree with you, that it's better not to have a "nice to have" dependency as dependency, than to have it as "must have". |
|
| Back to top |
|
|
|
Gentoo Chat
