Fast portage search script

[Genone]

Ok, that might be interesting for other people as well, so I'll post it: Because it takes ages to do emerge -s or especially emerge -S on my server (that serves the tree to the rest of my boxes) I've written a small script to create a search index and use that for searching. And because I was bored I integrated it into emerge . The script can be found at http://gentoo.devel-net.org/download/fastsearch, the patch (against portage-2.0.48-r1) at http://gentoo.devel-net.org/download/emerge.patch (both can also be found in this thread in the german forum, but I don't recommend copy+paste from there because of tab/space conversions)

A few hints for installation:

• the script can be used standalone without the patch
• don't try the patch if you don't know how to patch
• the patch can't be used without the script
• the patch needs fastsearch in /usr/lib/portage/bin/
• the patch enables the following functionality in emerge:
  • if you set "fastsearch" in FEATURES the search file will be recreated on every emerge sync, mostly useful for cronjobs
  • you can now use "emerge --fastsearch -s" and "emerge --fastsearch -S" to use the script instead of emerge's normal search method (long options work too)
• the default behavior of emerge is NOT changed
• the output format is completely different from normal emerge search output
• it might find fewer packages because only the latest ebuild of a package is indexed

The script is very fast as it uses grep on a single text file, it does not even take a second on my server to get a result for a descripitionsearch , but beware that this is a private script that needs testing.

If I get some positive feedback I'll improve it and put it in bugzilla.

[neuron]

great, gonna check this out first thing tomorrow (need some sleep first )

[Pythonhead]

Very cool.

A couple minor quirks. The -u option in the standalone version only finds things in PORTDIR_OVERLAY unless I cd to PORTDIR. Also, the help shows an -a option, but I think its actually -s?

[Genone]

Ok, seems that I forgot to rename the -s to -a (--all seems more logical to me). The problem with $PORTDIR is, that I don't have a default value for it if you don't set it in make.conf, I've fixed this now (same for $PORTDIR_OVERLAY). The new script is online under the same name as the old one.

Thanks for testing.

[sessionID]

... The script is very fast ...

WOW That's speed
Really nice, thanks!

[Genone]

Okay, I'm taking the discussion here:

I have been using both fastsearch and esearch extensively while coding ebuilds and they both work great and fill a slightly different need. Variety is good.

One feature I'd like to see for developers is a switch to allow just PORTDIR_OVERLAY get searched. I guess this would take two separate databases; I haven't looked at the code for either. I only emerge sync once a week or so but I'm constantly doing work in PORTDIR_OVERLAY. Not a big deal, but it would be nice if it wouldn't take a lot of effort to add it.

It would also be very sweet to have a switch to show what etcat -v shows. Of course this would get out of date very quickly as you emerge packages, so could be misleading to rely on. Maybe not so good an idea, unless you emerge sync every day.

Another feature would be to show the time you last updated the database and the last time you did emerge sync every time it displays the search results.

I've added a new --info option to fastsearch (shows dates and db size), it is exclusive to update or search modes, but an alias should get you the behavior you wanted. Version information is not possible with fastsearch as it knows nothing about versions.
For the overlay separation: it would be simple to integrate it into the update/search procedure, but my option parsing is too stupid at the moment to recognize more than one option before the pattern. I might implement it at a later time.

[Petyr]

Please get this incorperated into the distributed version of portage ^^
I have to do emerge -s a LOT because I'm stupid and can't remember stuff for more than like ohh.... 30 seconds at best ^_~
Anyhow, can you submit this one to bugtraq please

Petyr

[Genone]

Already done, bug 24756. But as carpaski has about 400 open bugs I'm not very optimistic that it will be added soon.