Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
epkg, ept-get, genorphan announcement
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
norayr
n00b
n00b


Joined: 25 Oct 2005
Posts: 20
Location: Yerevan, Armenia

PostPosted: Mon Jul 24, 2006 6:52 am    Post subject: epkg, ept-get, genorphan announcement Reply with quote

Hello.
I wrote some new tools, and three of them are already available at sourceforge.

http://epkg.sf.net
http://ept-get.sf.net
http://genorphan.sf.net

epkg is something like replacement for debian's dpkg (like epm which accepts rpm syntax)
Firstly I was used epm for my ept-get, byt after realize, that written in an interpreted language epm is too slow, and it doesn't satisfy my needs.
Now, I am using "epkg -l" to get packages currently installed on my system, and it is faster at least for 10 times.
epkg also has another feature which I was need for ept-get.
Firstly I have used "equery d packagename" but it was too slow again, so epkg has s feature which doesn't have dpkg so
I can call epkg with "epkg query packagename" and it works again much more faster then equery (because it isn't written in an interpreted language"

So, ept-get depends on epkg
I wrote ept-get because there was no tool to provide recoursive remove for gentoo
Now I can do
# ept-get remove packagename
and ept-get will calculate dependencies like apt-get and remove them all.
Of caurse there are still things to do, for instance I want ept-get report disk size which be freed after removal, like apt

genorphan is a tool like debian deborphan

I want to ask here to prepare ebuilds and try to commit them to the portage tree.
Because I am too busy at work now and hardly finding a time to write software, and when I got a time I want to spend it writing open source software rather then ebuilds.
So, I am asking here if someone has a time and like my tools, please, write ebuilds and try to commit them

I know, there are build dependency from fpc (FreePascalCompiler) but if someone do not want to have fpc installed to compile the software, then it is possible to install statically compiled binaries, which included in distribution
BTW, As a rule, fpc compiled programs does not depend on libc at all.
So, binaries included in distribution will work on any platform without change


Thanks

Good Luck

Norayr


Last edited by norayr on Mon Jul 24, 2006 12:32 pm; edited 2 times in total
Back to top
View user's profile Send private message
norayr
n00b
n00b


Joined: 25 Oct 2005
Posts: 20
Location: Yerevan, Armenia

PostPosted: Mon Jul 24, 2006 12:28 pm    Post subject: Reply with quote

I have to tell, that utility genfoster is almost finished and I'll upload it to sf.net soon.

Utility works similiar to debian debfoster, checks hich packages bring this or that package and could remove them all
Back to top
View user's profile Send private message
Sohail
Tux's lil' helper
Tux's lil' helper


Joined: 14 May 2005
Posts: 118
Location: Pakistan.

PostPosted: Mon Jul 24, 2006 6:40 pm    Post subject: Reply with quote

Hey there, I'm not an ebuild expert and I've never before compiled a pascal source but tomorrow in the evening I'm free so I'll do the ebuilds then. I have just fetched the fpc sources.

A question, should I use the provided compile.sh script or write a Makefile.

Regards.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9236
Location: beyond the rim

PostPosted: Mon Jul 24, 2006 7:38 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

norayr wrote:
it works again much more faster then equery (because it isn't written in an interpreted language)


Nope. It's faster because it cheats by bypassing the portage API and accessing the storage layer directly leading to inconsistent behavior. Also from a quick look it has quite a bunch of other problems, like the depend parser only using DEPEND (and ignoring [RP]DEPEND) or ignoring conditionals and ||-dependencies. Of course you're faster if you only implement 10% of the required featureset. Oh, and of course using the storage layer directly you're prone to breakage whenever we change the storage layer in any way.

Sorry if this looks like bashing you, just getting annoyed by people claiming that portage and it's tools are slow because they are written in a scripting language, that's simply bullshit.
Back to top
View user's profile Send private message
Sohail
Tux's lil' helper
Tux's lil' helper


Joined: 14 May 2005
Posts: 118
Location: Pakistan.

PostPosted: Tue Jul 25, 2006 5:45 am    Post subject: Reply with quote

Your ebuilds....

http://www.stupidcomputing.com/downloads/ebuilds.tar.bz2
Back to top
View user's profile Send private message
norayr
n00b
n00b


Joined: 25 Oct 2005
Posts: 20
Location: Yerevan, Armenia

PostPosted: Tue Jul 25, 2006 6:09 am    Post subject: Reply with quote

Sohail wrote:
Your ebuilds....

http://www.stupidcomputing.com/downloads/ebuilds.tar.bz2



Thank You very very very much for ebuilds.
Yes, writing Makefile is a preferred way.

Thanks again

Norayr
Back to top
View user's profile Send private message
norayr
n00b
n00b


Joined: 25 Oct 2005
Posts: 20
Location: Yerevan, Armenia

PostPosted: Tue Jul 25, 2006 6:46 am    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

Genone wrote:
norayr wrote:
it works again much more faster then equery (because it isn't written in an interpreted language)


Nope. It's faster because it cheats by bypassing the portage API and accessing the storage layer directly leading to inconsistent behavior.


Firstly I wrote ept-get and genorphan they depend on "epm" and "equery" and there was no epkg at that time.
But please, count how much time epm spends to just show installed packages.
And how much time equery spends to show who depend on some package?
Do you think it is an acceptable speed?
May be that means something wrong with portage API, not with my software?
I had to waint 30 minutes to genorphan just show unused libraries when it used equery.
But now it's do the same job immeditially using epkg.

Why should I having modern computer with 3.2ghz cpu wait more than I am waiting using debian's dpkg or apt-get or deborphan on a pentium 1 ???
I think that it is definitely big problem described by N. Wirth at 1995 in his article "A Plea For Lean Software".
http://cr.yp.to/bib/1995/wirth.pdf
Niklaus Wirth wrote:

With a touch of humor, the following two laws reflect the state of the art admirably well:
Software expands to fill the available memory (Parkinson)
Software is getting slower more rapidly than hardware becomes faster (Reiser)



Genone wrote:

Oh, and of course using the storage layer directly you're prone to breakage whenever we change the storage layer in any way.

If accessing portage API I have to wait such a long time, I prefer to not access it.
If you'll change storage layer I should change my software, that's true. But I have no other choice.
No, in fact I have, I am thinking about re-implementing portage myself, but I know, it is a really huge work.
Now I have a very limited time to spend on my own projects.

Genone wrote:

Sorry if this looks like bashing you, just getting annoyed by people claiming that portage and it's tools are slow because they are written in a scripting language, that's simply bullshit.

That's OK. Thank you very much for your critics.
Here I should make a quote from an author of Shed Skin http://mark.dufour.googlepages.com/home, Python to C++ translator
That means portage is slow because it is not written well (what is not true IMO) or it is slow because it is written in an interpreted language( which is more likely true). Accessing portage API with another scripts does not speedup process.
I know that scripts has many advantages, I really understand it. But interpreted code is never faster than compiled.

Mark Dufour, author of the Shed Skin wrote:

For a set of 16 non-trivial test programs, measurements show a typical speedup of 2-40 over Psyco, about 12 on average, and 2-220 over CPython, about 45 on average



Genone wrote:

Also from a quick look it has quite a bunch of other problems, like the depend parser only using DEPEND (and ignoring [RP]DEPEND) or ignoring conditionals and ||-dependencies.


No, its' only using RDEPEND for now, because it is aimed to deal with runtime dependencies.
I have just implemented functionality I wanted to implement.
I'll be really thankful if you'll write down here all other problems you noticed.
And I'll really be happy to implement other features I(want or people ask me to implement

Genone wrote:

Of course you're faster if you only implement 10% of the required featureset.


I'll be faster even if I'll implement 200% of the features, I am sure.

And I want to mention again.
I have been forced to reimplement some epm and equery functionality !
Just because it was too slow.
I do not want to write that was already written.
I do not want to invent a vehicle.
What I wanted is to have recoursive remove, like using "apt-get remove" which is yet not implemented in a portage.
Also I wanted to have some tool to show unused libraries and binaries, which could be safely removed.
And I wanted a tool which will show me which packages brought that package to clean that all together. (genfoster)

That's why I am writing.
I do not want to offend somebody, or tell them write their software in another language or using another compiler.
I just wanted to implement aome functionality which didn't implemented yet.
And at that time I was forced to re-implement some features.

OK, Anyway thanks for responces.

Good Luck.

Norayr
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Wed Sep 27, 2006 8:01 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

Wow that's you told ;)
Genone wrote:
Sorry if this looks like bashing you, just getting annoyed by people claiming that portage and it's tools are slow because they are written in a scripting language, that's simply bullshit.

So why are they slow then?
Back to top
View user's profile Send private message
ferringb
Retired Dev
Retired Dev


Joined: 03 Apr 2003
Posts: 355
Location: USA

PostPosted: Thu Sep 28, 2006 1:11 am    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

steveL wrote:
Wow that's you told ;)
Genone wrote:
Sorry if this looks like bashing you, just getting annoyed by people claiming that portage and it's tools are slow because they are written in a scripting language, that's simply bullshit.

So why are they slow then?

Craptastic algorithms, bad api forcing shitty calling patterns; strace portage/equery some time and take a *real* look at what it's doing.

http://gentooexperimental.org/~ferringb/run.log

Might be of interest for those claiming that interpretted is slow; pkgcore is python, with a few select hot spots in c. Drop the c components, adds a bit of proc time but not a *huge* amount; the difference is partly due to the fact the code is written expecting the c extensions to be in use also due to it being faster to do approach X when the extension is active. This is also without deviating *at all* from standard portage data structures, and passing through abstraction layers (iow, it's not hardcoded lookups into dirs).

What you'll notice in those stats is that the real slow down is IO, with pquery being faster due to the fact it's api is designed to minimize IO access; take a look at cached, you'll see the actual comparison of python vs c++ vs python/c that the gap *really* is not that huge.

Portage is slow due to the fact it's implementation goes to disk a helluva lot more then it should, beyond that it's litered with O(N^2) ops. The internals plain suck; blaming the language because someone wrote crap code in it doesn't make sense, interpretted can be pretty fricking fast as long as you know what you're doing (same with any language I might add).
_________________
I don't want to be buried in a pet cemetery. ~Ramones
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Sep 28, 2006 2:21 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

ferringb wrote:
Craptastic algorithms, bad api forcing shitty calling patterns; strace portage/equery some time and take a *real* look at what it's doing.
..
Portage is slow due to the fact it's implementation goes to disk a helluva lot more then it should, beyond that it's litered with O(N^2) ops. The internals plain suck; blaming the language because someone wrote crap code in it doesn't make sense, interpretted can be pretty fricking fast as long as you know what you're doing (same with any language I might add).

I agree with that; no language will protect you from stupid algorithms.
So pkgcore is the other package manager, which you develop? Can it maintain a system (is it reliable etc)?
Back to top
View user's profile Send private message
ferringb
Retired Dev
Retired Dev


Joined: 03 Apr 2003
Posts: 355
Location: USA

PostPosted: Fri Sep 29, 2006 6:22 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

steveL wrote:
ferringb wrote:
Craptastic algorithms, bad api forcing shitty calling patterns; strace portage/equery some time and take a *real* look at what it's doing.
..
Portage is slow due to the fact it's implementation goes to disk a helluva lot more then it should, beyond that it's litered with O(N^2) ops. The internals plain suck; blaming the language because someone wrote crap code in it doesn't make sense, interpretted can be pretty fricking fast as long as you know what you're doing (same with any language I might add).

I agree with that; no language will protect you from stupid algorithms.
So pkgcore is the other package manager, which you develop? Can it maintain a system (is it reliable etc)?

Well, release will be 0.1; for immutable (data api, queries, etc), it's fine, for mutable (merge/unmerge) it works, but will be require --force for all ops for 0.1.

Namely, first release, should work, but want only folks who are *aware* they're venturing off the beaten path to play with that functionality for the first release. If not adventurous, would wait for 0.2 personally.

Standpoint of the api, folks have been using it for a long while (namely for pcheck, repoman replacement); that's immutable access vs mutable though.
_________________
I don't want to be buried in a pet cemetery. ~Ramones
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Fri Sep 29, 2006 6:39 pm    Post subject: Reply with quote

Now that you have our interest, show us the goods :)
Where could we get it from (cvs, svn etc anonymous access)?
[EDIT]I've found it :) [/EDIT]
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Fri Sep 29, 2006 9:27 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

ferringb wrote:
http://gentooexperimental.org/~ferringb/run.log


those results you get with paludis are *really* wierds, and really far from the reality.., this is for example what I have
Code:
uncached: paludis -q bsdiff xorg-server
real    0m2.434s
user    0m0.519s
sys     0m0.225s

cached: paludis -q bsdiff xorg-server
real    0m0.581s
user    0m0.488s
sys     0m0.089s

_________________
The End of the Internet!
Back to top
View user's profile Send private message
ferringb
Retired Dev
Retired Dev


Joined: 03 Apr 2003
Posts: 355
Location: USA

PostPosted: Fri Sep 29, 2006 10:03 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

truc wrote:
ferringb wrote:
http://gentooexperimental.org/~ferringb/run.log


those results you get with paludis are *really* wierds, and really far from the reality.., this is for example what I have
Code:
uncached: paludis -q bsdiff xorg-server
real    0m2.434s
user    0m0.519s
sys     0m0.225s

cached: paludis -q bsdiff xorg-server
real    0m0.581s
user    0m0.488s
sys     0m0.089s

Those stats are with caching totally flushed (sync ; echo 3 > /proc/sys/vm/drop_caches); the 2.4s you're getting there indicates either
  • you ain't actually testing uncached (partial cached) thus your results aren't all that valid for uncached
  • alternatively, you're not using a pure portage vdb; paludis initially forced an incompatible collapsed old style virtuals vdb; the testing is for a *normal* vdb, one that portage is capable of reading/writing to properly; iow, the format that's been in place for the last few years.
  • you've got an insanely fast disk for /var/db/pkg (which case you would see matching speedups for the other 2).
  • you have absolutely *nothing* in your vdb; cached stats should be far lower if that were the case (again, would see matching decrease in runtime for the other 2).


So... clarify; the stats posted have been duplicated on multiple machines also; fs obviously has an effect (a fresh vdb shaves about 10s off of uncached full vdb walks for all manages), but the base search speeds will stack up in the same order.
_________________
I don't want to be buried in a pet cemetery. ~Ramones
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Fri Sep 29, 2006 10:32 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

ferringb wrote:
Those stats are with caching totally flushed
I had just rebooted, so it's possible it wasn't perfect.

then I've tested as I should:
Quote:
(sync ; echo 3 > /proc/sys/vm/drop_caches); the 2.4s you're getting there indicates either
  • you ain't actually testing uncached (partial cached) thus your results aren't all that valid for uncached


results are a bit differents
Code:
uncached: paludis -q bsdiff xorg-server
real    0m5.521s
user    0m0.544s
sys     0m0.303s

cached: paludis -q bsdiff xorg-server
real    0m0.575s
user    0m0.486s
sys     0m0.080s

Quote:
  • alternatively, you're not using a pure portage vdb; paludis initially forced an incompatible collapsed old style virtuals vdb; the testing is for a *normal* vdb, one that portage is capable of reading/writing to properly; iow, the format that's been in place for the last few years.

  • You're right too, I'm using paludis format
    Code:
    VDB_FORMAT
    paludis-2

    But I don't really see why it would impact theses results...

    Quote:
  • you've got an insanely fast disk for /var/db/pkg (which case you would see matching speedups for the other 2).

  • I'd really like, but no, i don't .. the only thing particular is that /var is on reseirfs .
    Quote:
  • you have absolutely *nothing* in your vdb; cached stats should be far lower if that were the case (again, would see matching decrease in runtime for the other 2).

  • Code:
    paludis --list-packages --repository installed | egrep '^\*' | wc -l
    485
    so vsb is not that empty

    Quote:
    So... clarify; the stats posted have been duplicated on multiple machines also; fs obviously has an effect (a fresh vdb shaves about 10s off of uncached full vdb walks for all manages), but the base search speeds will stack up in the same order.


    From what you said this results would be due to the paludis vdb format. Even if I don't see why it would be the case, why not showing theses results in the run.log, so that one can see if they want to use paludis vdb format with paludis or not?
    _________________
    The End of the Internet!
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Fri Sep 29, 2006 11:04 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    truc wrote:
    ferringb wrote:
    Those stats are with caching totally flushed
    I had just rebooted, so it's possible it wasn't perfect.

    then I've tested as I should:
    ferringb wrote:
    (sync ; echo 3 > /proc/sys/vm/drop_caches); the 2.4s you're getting there indicates either
    [list][*] you ain't actually testing uncached (partial cached) thus your results aren't all that valid for uncached


    results are a bit differents
    Code:
    uncached: paludis -q bsdiff xorg-server
    real    0m5.521s
    user    0m0.544s
    sys     0m0.303s

    cached: paludis -q bsdiff xorg-server
    real    0m0.575s
    user    0m0.486s
    sys     0m0.080s

    Quote:
    alternatively, you're not using a pure portage vdb; paludis initially forced an incompatible collapsed old style virtuals vdb; the testing is for a *normal* vdb, one that portage is capable of reading/writing to properly; iow, the format that's been in place for the last few years.

    You're right too, I'm using paludis format
    Code:
    VDB_FORMAT
    paludis-2

    But I don't really see why it would impact theses results...

    Standard vdb format forces a full VDB walk to collect virtuals; that's why you see the horrid 30s spike for all managers in the stats for uncached (pkgcore is exempted in select cases due to being designed to JIT just about everything, including the old style virtuals pull).

    truc wrote:
    From what you said this results would be due to the paludis vdb format. Even if I don't see why it would be the case, why not showing theses results in the run.log, so that one can see if they want to use paludis vdb format with paludis or not?

    The test is a comparison of using existant portage data; for paludis-2, have to convert the vdb (or more likely build the system from scratch with paludis), reverting from paludis-2 to standard portage compatible also requires a rather nasty wiping of virtual (it works, but sucks as a migration path). Finally... said format isn't safely usable with portage (portage will not update the collapsed caching).

    Stats were done from a normal build of a system- having to rebuild the system isn't much of an option for existing user base, further, requiring everyone to change over their vdb kind of sucks; since portage cannot safely update it, you're stuck with paludis at that point.

    Finally, there aren't any special tricks enabled for any of the three; can speed up portage/pkgcore via using faster cache backends for example, nothing of the sort was enabled- intention was a vanilla testing of the 3.
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ciaranm
    Retired Dev
    Retired Dev


    Joined: 19 Jul 2003
    Posts: 1719
    Location: In Hiding

    PostPosted: Sat Sep 30, 2006 3:58 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ferringb wrote:

    Standard vdb format forces a full VDB walk to collect virtuals; that's why you see the horrid 30s spike for all managers in the stats for uncached

    Indeed. And ebuilds rely upon standard VDB format. Highly annoying.

    Quote:
    (pkgcore is exempted in select cases due to being designed to JIT just about everything, including the old style virtuals pull).

    Are you assuming that a) all old style virtuals are virtual/* and b) that there are no PROVIDEs that aren't listed in a profile virtuals file? Without that I don't see how you can assume that querying for foo can avoid a VDB scan.

    Quote:
    truc wrote:
    From what you said this results would be due to the paludis vdb format. Even if I don't see why it would be the case, why not showing theses results in the run.log, so that one can see if they want to use paludis vdb format with paludis or not?

    The test is a comparison of using existant portage data; for paludis-2, have to convert the vdb (or more likely build the system from scratch with paludis), reverting from paludis-2 to standard portage compatible also requires a rather nasty wiping of virtual (it works, but sucks as a migration path). Finally... said format isn't safely usable with portage (portage will not update the collapsed caching).

    Hm? paludis-2 doesn't write old style virtuals to vdb any more. And migrating to portage should just be a case of rm -fr vdb/.cache (one of the Portage utilities doesn't ignore .dirs in vdb).
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Sat Sep 30, 2006 4:26 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ciaranm wrote:
    ferringb wrote:

    Standard vdb format forces a full VDB walk to collect virtuals; that's why you see the horrid 30s spike for all managers in the stats for uncached

    Indeed. And ebuilds rely upon standard VDB format. Highly annoying.

    Ongoing cleaning it up; oddly, that's not the real issue imo; I can monkey patch on the fly hardcoded paths out for certain abuses to work around that within the ebuild env. Don't claim it's clean, but it's one route for the remaining slackers. Main issue is the direct invocation of portageq externally, makes it far harder to wrap all access to the vdb, thus folks wind up trying a sort of compatible vdb format that lives at /var/db/pkg.

    ciaranm wrote:
    ferringb wrote:
    (pkgcore is exempted in select cases due to being designed to JIT just about everything, including the old style virtuals pull).

    Are you assuming that a) all old style virtuals are virtual/*

    Portage looks for and specially treats virtual/*; iow, all old style *are* virtual categrory. This is exactly why the glep37 implementation went to hell, folks used it before it had time to cook and spread to all users systems, thus the special cased handling blew on up --sync.
    ciaranm wrote:
    and b) that there are no PROVIDEs that aren't listed in a profile virtuals file? Without that I don't see how you can assume that querying for foo can avoid a VDB scan.

    Profile virtuals are pkgs that dep on other nodes; nice name for effective aliasing (although since they can specify range limiters, bit more then aliasing); point is, vdb isn't involved there beyond being a potential satisfier of the virtual pkgs deps (same as portdir/binpkgs).

    ciaranm wrote:
    ferringb wrote:
    ... said format isn't safely usable with portage (portage will not update the collapsed caching).

    Hm? paludis-2 doesn't write old style virtuals to vdb any more.

    Writes it to where then? .cache? Point above is more that whatever form y'all use to collapse the vdb walk, portage does _not_ know how to update it, so unless y'all are relying on mtime verification of directories to try and track any changes portage may have done, portage having write access to the vdb is still an issue.

    ciaranm wrote:
    And migrating to portage should just be a case of rm -fr vdb/.cache (one of the Portage utilities doesn't ignore .dirs in vdb).

    Utilities name would be useful since y'all have world stored in the base also...

    Also, y'all don't handle versioned provides in the VDB all that well; it's rarely used (more of a holdover from before glep37), but vdb provide keys *can* be versioned, and paludis spits chunks when it encounters it. I've disabled writiing that form out for pkgcore for the sake of avoiding bitching, but would be wise to clean up the relevant code.
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Sat Sep 30, 2006 4:35 pm    Post subject: Reply with quote

    reminds me...

    VDB_FORMAT, while useful as a way to mark the entries format, doesn't give any indication of who wrote out the entry; would suggest tagging in a PKGMANAGER key to vdb nodes so that regardless of the format (since I due intend to add paludis-2 read/write format at some point), can tell who actually wrote the entry rather then assuming that if it's VDB_FORMAT blah, it must have come from foo.
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ciaranm
    Retired Dev
    Retired Dev


    Joined: 19 Jul 2003
    Posts: 1719
    Location: In Hiding

    PostPosted: Sat Sep 30, 2006 4:36 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ferringb wrote:
    ciaranm wrote:
    and b) that there are no PROVIDEs that aren't listed in a profile virtuals file? Without that I don't see how you can assume that querying for foo can avoid a VDB scan.

    Profile virtuals are pkgs that dep on other nodes; nice name for effective aliasing (although since they can specify range limiters, bit more then aliasing); point is, vdb isn't involved there beyond being a potential satisfier of the virtual pkgs deps (same as portdir/binpkgs).

    The question is... If no profile lists virtual/foo, and an installed package PROVIDEs virtual/foo, and there is a package named app-misc/foo, and you query foo (no category), does pkgcore correctly identify that there are two foos? Handling this correctly is why paludis always does the vdb scan.

    ferringb wrote:
    ciaranm wrote:
    ferringb wrote:
    ... said format isn't safely usable with portage (portage will not update the collapsed caching).

    Hm? paludis-2 doesn't write old style virtuals to vdb any more.

    Writes it to where then? .cache? Point above is more that whatever form y'all use to collapse the vdb walk, portage does _not_ know how to update it, so unless y'all are relying on mtime verification of directories to try and track any changes portage may have done, portage having write access to the vdb is still an issue.

    We don't write it anywhere. It's generated totally on the fly, which is why there's that nasty thirty second VDB load.

    I *want* to .cache it, and tell users that they have to nuke .cache whenever switching between Portage and Paludis.

    Quote:
    ciaranm wrote:
    And migrating to portage should just be a case of rm -fr vdb/.cache (one of the Portage utilities doesn't ignore .dirs in vdb).

    Utilities name would be useful since y'all have world stored in the base also...

    Mike's script just symlinks that one.

    Quote:
    Also, y'all don't handle versioned provides in the VDB all that well; it's rarely used (more of a holdover from before glep37), but vdb provide keys *can* be versioned, and paludis spits chunks when it encounters it. I've disabled writiing that form out for pkgcore for the sake of avoiding bitching, but would be wise to clean up the relevant code.

    That one's intentional.
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Sat Sep 30, 2006 4:53 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ciaranm wrote:
    ferringb wrote:
    ciaranm wrote:
    and b) that there are no PROVIDEs that aren't listed in a profile virtuals file? Without that I don't see how you can assume that querying for foo can avoid a VDB scan.

    Profile virtuals are pkgs that dep on other nodes; nice name for effective aliasing (although since they can specify range limiters, bit more then aliasing); point is, vdb isn't involved there beyond being a potential satisfier of the virtual pkgs deps (same as portdir/binpkgs).

    The question is... If no profile lists virtual/foo, and an installed package PROVIDEs virtual/foo, and there is a package named app-misc/foo, and you query foo (no category), does pkgcore correctly identify that there are two foos? Handling this correctly is why paludis always does the vdb scan.

    Yep, handles it fine (although the virtual output annoys the piss out of me, rarely what I'm looking for).

    ciaranm wrote:
    We don't write it anywhere. It's generated totally on the fly, which is why there's that nasty thirty second VDB load.

    Then I'm extremely curious how the dude above is getting 5s scans, unless he started with older paludis which was writing the collapsed entries to disk...

    ciaranm wrote:
    I *want* to .cache it, and tell users that they have to nuke .cache whenever switching between Portage and Paludis.

    Yeah well, welcome to the club. Been wanting to restructure the vdb for a couple of years now, but getting all the ducks lined up and shot is a pita; tree is reasonably clean, but still need to have the external interaction with the manager passed through indirection (eselect pkg-manager, sans the hardcoded prefer- ask kugelfang for details as to why that blows).

    ciaranm wrote:
    ferringb wrote:
    ciaranm wrote:
    one of the Portage utilities doesn't ignore .dirs in vdb.

    Utilities name would be useful since y'all have world stored in the base also...

    Mike's script just symlinks that one.

    Was looking for something that wasn't a dirty hack... If his shit gets broke, well, known up front it's totally wrong/icky way of doing it, horking it is a given.

    ciaranm wrote:
    ferringb wrote:
    Also, y'all don't handle versioned provides in the VDB all that well; it's rarely used (more of a holdover from before glep37), but vdb provide keys *can* be versioned, and paludis spits chunks when it encounters it. I've disabled writiing that form out for pkgcore for the sake of avoiding bitching, but would be wise to clean up the relevant code.

    That one's intentional.


    Well I could intentionally re-enable writing out a finalized PROVIDE, thus causing paludis to bitch and moan; looking for a better reason then "cause I say so", since it's long time supported; intentionally changing the support requires kind of a good reason since the capability isn't a bug of portage (it's the basis of pkg.provided after all).
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Sat Sep 30, 2006 4:59 pm    Post subject: Reply with quote

    Could at least give me credit for pointing out rev1537 :P
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ciaranm
    Retired Dev
    Retired Dev


    Joined: 19 Jul 2003
    Posts: 1719
    Location: In Hiding

    PostPosted: Sat Sep 30, 2006 5:04 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    Quote:
    ciaranm wrote:
    We don't write it anywhere. It's generated totally on the fly, which is why there's that nasty thirty second VDB load.

    Then I'm extremely curious how the dude above is getting 5s scans, unless he started with older paludis which was writing the collapsed entries to disk...

    I get a full VDB load off a crappy laptop drive in under five seconds because I don't have KDE or Gnome installed and because I use ext3 with dir_index turned on. But even with a huge VDB, if it's in cache then pulling it in is RAM I/O bound rather than programming language performance bound for us.

    Quote:
    Well I could intentionally re-enable writing out a finalized PROVIDE, thus causing paludis to bitch and moan; looking for a better reason then "cause I say so", since it's long time supported; intentionally changing the support requires kind of a good reason since the capability isn't a bug of portage (it's the basis of pkg.provided after all).

    Automatically atomising it sounds risky, especially if updates/ ever supports version renaming...

    ferringb wrote:
    Could at least give me credit for pointing out rev1537 :P

    Hah. If everyone else is using that dirty hack to rig benchmark scores in carefully selected situations then we might as well too.
    Back to top
    View user's profile Send private message
    ferringb
    Retired Dev
    Retired Dev


    Joined: 03 Apr 2003
    Posts: 355
    Location: USA

    PostPosted: Sat Sep 30, 2006 5:14 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ciaranm wrote:
    Quote:
    ciaranm wrote:
    We don't write it anywhere. It's generated totally on the fly, which is why there's that nasty thirty second VDB load.

    Then I'm extremely curious how the dude above is getting 5s scans, unless he started with older paludis which was writing the collapsed entries to disk...

    I get a full VDB load off a crappy laptop drive in under five seconds because I don't have KDE or Gnome installed and because I use ext3 with dir_index turned on. But even with a huge VDB, if it's in cache then pulling it in is RAM I/O bound rather than programming language performance bound for us.

    Ironically, I'm running ext3 with dir_index on also; have a gb of ram, but the vdb/tree isn't always in cache, thus it *does* matter; hence explicitly flushing kernel caches for testing.

    Kde/Gnome installed isn't the main issue (the stats are from a system with neither installed), # of vdb entries is.

    ciaranm wrote:
    ferringb wrote:
    Well I could intentionally re-enable writing out a finalized PROVIDE, thus causing paludis to bitch and moan; looking for a better reason then "cause I say so", since it's long time supported; intentionally changing the support requires kind of a good reason since the capability isn't a bug of portage (it's the basis of pkg.provided after all).

    Automatically atomising it sounds risky, especially if updates/ ever supports version renaming...

    Versioning it, not atomising it (it's not an atom oddly enough); further, if updates/ ever supports version renaming, it needs a base version to rename from; if after just a flat out "write this provides into the vdb entry", versioned vs versionless matters not at all.

    ciaranm wrote:
    ferringb wrote:
    Could at least give me credit for pointing out rev1537 :P

    Hah. If everyone else is using that dirty hack to rig benchmark scores in carefully selected situations then we might as well too.

    Gracious per the norm. Reduction of runtime from 30s for uncached (common enough for users) to 6s is "rigging the benchmark", whatever.

    Seriously, don't be a jack ass (this thread so far was nice); it's an improvement, one portage picked up from early pkgcore designs, now paludis did the same. Can call it rigging (and be a child about it), but users will benefit either way.
    _________________
    I don't want to be buried in a pet cemetery. ~Ramones
    Back to top
    View user's profile Send private message
    ciaranm
    Retired Dev
    Retired Dev


    Joined: 19 Jul 2003
    Posts: 1719
    Location: In Hiding

    PostPosted: Sat Sep 30, 2006 5:25 pm    Post subject: Re: epkg, ept-get, genorphan announcement Reply with quote

    ferringb wrote:
    Ironically, I'm running ext3 with dir_index on also; have a gb of ram, but the vdb/tree isn't always in cache, thus it *does* matter; hence explicitly flushing kernel caches for testing.

    The VDB load is going to kick in for pretty much everything anyway... The only way around the load times is to change the format, which we've discussed already.

    Quote:
    Kde/Gnome installed isn't the main issue (the stats are from a system with neither installed), # of vdb entries is.

    KDE and Gnome are how you get lots of VDB entries.

    ferringb wrote:
    Versioning it, not atomising it (it's not an atom oddly enough); further, if updates/ ever supports version renaming, it needs a base version to rename from; if after just a flat out "write this provides into the vdb entry", versioned vs versionless matters not at all.

    Ya, but if updates says "move foo version 20060930 to version 0.1_alpha20060930", versioning PROVIDE in VDB would mean you'd have to reversion that as well, rather than just the package entry -- except how would the package manager know whether the version in PROVIDE came from the ebuild or whether it was added in by the package manager?

    ferringb wrote:
    but users will benefit either way.

    Except that very few users run -q cat/pkg off a cold cache. That's the only thing it affects. The -q pkg case still needs to do the VDB load in case there're any bar PROVIDEs.
    Back to top
    View user's profile Send private message
    Display posts from previous:   
    Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
    Goto page 1, 2  Next
    Page 1 of 2

     
    Jump to:  
    You cannot post new topics in this forum
    You cannot reply to topics in this forum
    You cannot edit your posts in this forum
    You cannot delete your posts in this forum
    You cannot vote in polls in this forum