Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Fast portage search script
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Sat Jul 12, 2003 12:25 am    Post subject: Fast portage search script Reply with quote

Ok, that might be interesting for other people as well, so I'll post it: Because it takes ages to do emerge -s or especially emerge -S on my server (that serves the tree to the rest of my boxes) I've written a small script to create a search index and use that for searching. And because I was bored I integrated it into emerge 8). The script can be found at http://gentoo.devel-net.org/download/fastsearch, the patch (against portage-2.0.48-r1) at http://gentoo.devel-net.org/download/emerge.patch (both can also be found in this thread in the german forum, but I don't recommend copy+paste from there because of tab/space conversions)

A few hints for installation:
  • the script can be used standalone without the patch
  • don't try the patch if you don't know how to patch
  • the patch can't be used without the script
  • the patch needs fastsearch in /usr/lib/portage/bin/
  • the patch enables the following functionality in emerge:
    • if you set "fastsearch" in FEATURES the search file will be recreated on every emerge sync, mostly useful for cronjobs
    • you can now use "emerge --fastsearch -s" and "emerge --fastsearch -S" to use the script instead of emerge's normal search method (long options work too)

  • the default behavior of emerge is NOT changed
  • the output format is completely different from normal emerge search output
  • it might find fewer packages because only the latest ebuild of a package is indexed

The script is very fast as it uses grep on a single text file, it does not even take a second on my server to get a result for a descripitionsearch :), but beware that this is a private script that needs testing.

If I get some positive feedback I'll improve it and put it in bugzilla.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sat Jul 12, 2003 1:15 am    Post subject: Reply with quote

great, gonna check this out first thing tomorrow :) (need some sleep first ;) )
Back to top
View user's profile Send private message
Pythonhead
Developer
Developer


Joined: 16 Dec 2002
Posts: 1801
Location: Redondo Beach, Republic of Calif.

PostPosted: Sat Jul 12, 2003 1:49 am    Post subject: Reply with quote

Very cool.

A couple minor quirks. The -u option in the standalone version only finds things in PORTDIR_OVERLAY unless I cd to PORTDIR. Also, the help shows an -a option, but I think its actually -s?
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Sat Jul 12, 2003 1:05 pm    Post subject: Reply with quote

Ok, seems that I forgot to rename the -s to -a (--all seems more logical to me). The problem with $PORTDIR is, that I don't have a default value for it if you don't set it in make.conf, I've fixed this now (same for $PORTDIR_OVERLAY). The new script is online under the same name as the old one.

Thanks for testing.
Back to top
View user's profile Send private message
sessionID
Apprentice
Apprentice


Joined: 11 Nov 2002
Posts: 266
Location: hungary

PostPosted: Sat Jul 12, 2003 2:34 pm    Post subject: Re: Fast portage search script Reply with quote

Genone wrote:
... The script is very fast ...


WOW That's speed :D
Really nice, thanks!
_________________
((( WeFunk )))
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Sat Jul 19, 2003 10:35 pm    Post subject: Reply with quote

Okay, I'm taking the discussion here:

Pythonhead wrote:

I have been using both fastsearch and esearch extensively while coding ebuilds and they both work great and fill a slightly different need. Variety is good.

One feature I'd like to see for developers is a switch to allow just PORTDIR_OVERLAY get searched. I guess this would take two separate databases; I haven't looked at the code for either. I only emerge sync once a week or so but I'm constantly doing work in PORTDIR_OVERLAY. Not a big deal, but it would be nice if it wouldn't take a lot of effort to add it.

It would also be very sweet to have a switch to show what etcat -v shows. Of course this would get out of date very quickly as you emerge packages, so could be misleading to rely on. Maybe not so good an idea, unless you emerge sync every day.

Another feature would be to show the time you last updated the database and the last time you did emerge sync every time it displays the search results.


I've added a new --info option to fastsearch (shows dates and db size), it is exclusive to update or search modes, but an alias should get you the behavior you wanted. Version information is not possible with fastsearch as it knows nothing about versions.
For the overlay separation: it would be simple to integrate it into the update/search procedure, but my option parsing is too stupid at the moment to recognize more than one option before the pattern. I might implement it at a later time.
Back to top
View user's profile Send private message
Petyr
Guru
Guru


Joined: 08 Jan 2003
Posts: 471
Location: San Diego, CA, USA

PostPosted: Sun Jul 20, 2003 1:40 am    Post subject: Reply with quote

Please get this incorperated into the distributed version of portage ^^
I have to do emerge -s a LOT because I'm stupid and can't remember stuff for more than like ohh.... 30 seconds at best ^_~
Anyhow, can you submit this one to bugtraq please

Petyr
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Sun Jul 20, 2003 2:38 am    Post subject: Reply with quote

Already done, bug 24756. But as carpaski has about 400 open bugs I'm not very optimistic that it will be added soon.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum