portagedb-0.2.1_alpha1 - with overlay support

[xororand]

portagedb is now eix
the new discussion can be found here: http://forums.gentoo.org/viewtopic.php?t=278819

Portagedb is a small program for searching the portage tree. Similar to app-portage/esearch it uses an index file (/var/cache/portagedb) which makes it very fast.

Usage:

Code: Select all

xoa@pluto ~ $ portagedb
Usage: portagedb [options] command <search string>
Commands: u(pdate)           Updates the database -> /var/cache/portagedb
          s(earch)           Search for ebuilds (posix regex)
          S(earchdesc)       Search for ebuilds + descriptions (posix regex)
Options:  -n,--nocolor       Do not use ANSI color codes
          -c,--compact       Compact search results
          -v,--verbose       Verbose search results

Example output:

Code: Select all

pluto ~ # pdb s bogo
* app-text/bogosort
     Available versions:  ~0.4.2
     Installed:           no
     Homepage:            http://www.lysator.liu.se/~qha/bogosort/
     Description:         A file sorting program which uses the bogosort algorithm

* mail-filter/bogofilter
     Available versions:  0.92.8  [M]~0.93.2
     Installed:           no
     Homepage:            http://bogofilter.sourceforge.net/
     Description:         Bayesian spam filter designed with fast algorithms, and tuned for speed.

Screenshot: https://sourceforge.net/project/screens ... _id=117574

Project page: http://portdb.sourceforge.net
Installation: emerge portagedb
(Latest version with overlay support: http://prdownloads.sourceforge.net/port ... 2?download)

A little benchmark:
#!/bin/bash
MEMSIZE=512
flushcache() { dd if=/dev/hda of=/dev/null ibs=1048576 count=$[ MEMSIZE*2 ] 2>/dev/null; }
gettime_real( ) {
echo -n "flushcache; "
flushcache
echo -e -n $1 "\r\t\t\t\t\t"
(time $1 >/dev/null) 2>&1 |egrep real |egrep "[[:digit:]]*m.*s" -o
}
gettime_real "portagedb u"
gettime_real "eupdatedb"
gettime_real "portagedb S rare"
gettime_real "esearch -S rare"
gettime_real "emerge --searchdesc rare"

Updates:

Code: Select all

-------------------------------------------------------------------------------
Changelog for portagedb 
-------------------------------------------------------------------------------
Project page: http://portdb.sf.net
Discussion: http://forums.gentoo.org/viewtopic.php?t=214547&highlight=portagedb
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Ideas for future versions:
-------------------------------------------------------------------------------

- Support slotted versions, bug #1025517
  https://sourceforge.net/tracker/index.php?func=detail&aid=1025517&group_id=117574&atid=678491
- Provide a way to automatically "portagedb u" after emerge
- Tests on other platforms than x86
- Option for arbitrary formatting
- Replicate esearch's esync.py
- Option '-i' for restricting search results to installed packages
- Improve error handling

-------------------------------------------------------------------------------
v0.2.1_alpha1:
-------------------------------------------------------------------------------
* Execution speed of "portagedb s" is back to normal again, 
  i.e. no unnecessary ~1 sec delay when the program is started.

* Portage overlays now are supported!

-------------------------------------------------------------------------------
v0.2.0:
-------------------------------------------------------------------------------
- nothing changed. alpha4 appears to be stable on x86 and amd64.
  we're in the portage tree now :)

-------------------------------------------------------------------------------
v0.2.0_alpha4:
-------------------------------------------------------------------------------
- xororand * Fixed memory management issues which lead to a crash on amd64
  (Discovered valgrind and removed all delete/delete[]/free() mismatches)

-------------------------------------------------------------------------------
v0.2.0_alpha3:
-------------------------------------------------------------------------------
- LinuxCommando * Fixed bug: "portagedb s" now throws an error message

- LinuxCommando * Added: "portagedb u" checks for root rights

- LinuxCommando * Added: 
  "portagedb s abc xyz" searches for all packages with either abc or xyz in their name
  equivalent to: portagedb s "\(abc\)\|\(xyz\)"

- xororand * Fixed: /etc/portage/package.unmask is now used

-------------------------------------------------------------------------------
v0.2.0_alpha2:
-------------------------------------------------------------------------------

* Fixed bug #1025515: Version numbers now are sorted properly
  https://sourceforge.net/tracker/index.php?func=detail&aid=1025515&group_id=117574&atid=678491
  Reference: http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap2

* Fixed bug #1025519: Ebuild is available
  https://sourceforge.net/tracker/index.php?func=detail&aid=1025519&group_id=117574&atid=678491

* Stability and masking of packages are distinguished:
  Masked:		Red / [M]
  Stable:		Green / No prefix
  Unstable:	Brown / ~

* Different levels of verbosity:
  -c	Compact: One line per result
  -v	Verbose: Print licenses

* Color of packages changed from light green to normal green 
  -> looks better in black-on-white terminals

* The arch is currently determined by a call to "portageq envvar ARCH" which is kinda
  slow, taking about 1 second on my system. If someone knows a faster, _reliable_ way,
  please tell me.

------------------------------------------------------------------------------
v0.1.3-r1:
------------------------------------------------------------------------------
* Fixed bug #1048527: 'portagedb s' searches in package names AND descriptions
  https://sourceforge.net/tracker/index.php?func=detail&aid=1048527&group_id=117574&atid=678491

-------------------------------------------------------------------------------
v0.1.3:
-------------------------------------------------------------------------------

* PORTDIR in /etc/make.conf is checked

* Colored text only uses foreground color so it looks nice on
  Black-On-White terminals

* Added new option "--nocolor"

-------------------------------------------------------------------------------
v0.1.2:
-------------------------------------------------------------------------------

* fixed a stupid bug with directory handling. thank you wiebel for debugging.

[dannycool]

Code: Select all

df@kahlann ~ $ psearch() { locate $* | grep "^/usr/portage/[^/\-]*-.*/" | cut -d/ -f4,5 | sort | uniq; }
df@kahlann ~ $ psearch bonobo
app-doc/ebook-bonobo
dev-cpp/libbonobomm
dev-cpp/libbonobouimm
dev-python/bonobo-python
gnome-base/bonobo
gnome-base/bonobo-activation
gnome-base/libbonobo
gnome-base/libbonoboui
        (...)

*mumble* UNIX *mumble* destined to reinvent *mumble*

SCNR

[xororand]

SCNR

np

[Pythonhead]

It fails if PORTDIR != /usr/portage so it should get that value from /etc/make.conf

[asph]

just instaled it, VERY nice!!
now i can search ebuilds fast

[xororand]

It fails if PORTDIR != /usr/portage so it should get that value from /etc/make.conf

PORTDIR is now checked in version 0.1.3. Thanks for pointing that out.

[Pythonhead]

It fails if PORTDIR != /usr/portage so it should get that value from /etc/make.conf

PORTDIR is now checked in version 0.1.3. Thanks for pointing that out.

Thanks. A lot of developers only use CVS, not 'emerge sync' so they don't use /usr/portage.

[devon]

I did not see an ebuild for this on Bugzilla, so I submitted one. Worked fine for me on my x86 system using /usr/local/portage/app-portage/ as the directory.

[asph]

i think it would be great to incorporate more functions to modify how the output is done, do you think it would be posible to incorporate a function like the "c" (short listing) option in esearch? it is very useful and fast to read, since using |more on long searches is a not very practical

keep up good work

*edit*
it seems to be a little bug, since even if you use only the "s" option, it also looks at the descriptions, as if you used "sS"
*/edit*

[bludger]

Is it planned to integrate such indexing functionality into portage?

[bludger]

Maybe my question wasn't clear enough. Doing an esync updates the database so that esearch lists the current state of all applications. If I then install and application, eg. foo, and then do an esearch foo, it is still shown as uninstalled. It stays this way until I do another esync.

It would be great if this database was integrated into portage, so that emerge foo updates the database.

[GrayFox]

Strange but a very cool thing here:

Code: Select all

sudo ./portagedb u  0.77s user 0.66s system 85% cpu 1.680 total

I don't know any other way to check how long it took and I was to lazy to copy your script. Updating in <2s ? nice

Code: Select all

./portagedb s notexistant  0.05s user 0.01s system 91% cpu 0.066 total

really nice! I bet you could somehow manage that the shell appends a portagedb u after every emerge command.

[LifesizeKenDoll]

I'd have to say this is pretty cool, although I had to do a little hacking to change the packagename text from white to default (cause I have a black-on-white xterm)

[vdboor]

Just wondering, the esearch package also has a tool called "esync". This tool compares the old cached portage tree with the new one. Do you think you can reproduce that functionality as well?

[nsahoo]

it apparently doesn't index the portage_overlay yet. It could be really useful if it can index them as well.

thanks.

[Goya13]

WOW, what an awesome app! I love it.

Just one request - licenses in the output???

thanks so much

[xororand]

Thanks for your encouragement! So, portagedb hasn't moved alot during the last 2 months. That will change I have implemented some of your suggestions and also some other new features.

I did not see an ebuild for this on Bugzilla, so I submitted one. Smile Worked fine for me on my x86 system using /usr/local/portage/app-portage/ as the directory.

An ebuild will come with 0.1.4.

it seems to be a little bug, since even if you use only the "s" option, it also looks at the descriptions, as if you used "sS"

fixed in 0.1.3-r1 -> added 5 bytes to the sourcecode

I'd have to say this is pretty cool, although I had to do a little hacking to change the packagename text from white to default (cause I have a black-on-white xterm)

I'll check that again in 0.1.4.

Just wondering, the esearch package also has a tool called "esync". This tool compares the old cached portage tree with the new one. Do you think you can reproduce that functionality as well?

Later, sorry.

it apparently doesn't index the portage_overlay yet. It could be really useful if it can index them as well.

That is planned for the version after 0.1.4.

Just one request - licenses in the output???

Version 0.1.4

Here's the current ChangeLog:

Code: Select all

-------------------------------------------------------------------------------
Changelog for portagedb 
-------------------------------------------------------------------------------
Project page: http://portdb.sf.net
Discussion: http://forums.gentoo.org/viewtopic.php?t=214547&highlight=portagedb
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Ideas for future versions:
-------------------------------------------------------------------------------

-	Support slotted versions, bug #1025517
	https://sourceforge.net/tracker/index.php?func=detail&aid=1025517&group_id=117574&atid=678491

- Support for portage overlays

- Provide a way to automatically update portagedb after emerges

- Test functionality on other platforms than x86

- Option for arbitrary formatting

- Replicate esearch's esync.py

-	Option '-i' for restricting search results to installed packages

- Bash completion for emerge/genlop etc.

- Test performance with Intel's C++ Compiler

-------------------------------------------------------------------------------
v0.1.4:
-------------------------------------------------------------------------------

- Fixed bug #1025515: Version numbers now are sorted properly
	https://sourceforge.net/tracker/index.php?func=detail&aid=1025515&group_id=117574&atid=678491

- Fixed bug #1025519: Ebuild is available
	https://sourceforge.net/tracker/index.php?func=detail&aid=1025519&group_id=117574&atid=678491

- Stability and masking of packages are distinguished:
	Stable-Unmasked:		Green / No prefix
	Unstable-Unmasked:	Brown / ~
	Masked:							Red / [M]

- Different levels of verbosity:
	-c	Compact: One line per result
	-v	Verbose: [insert desc]

- Improved error handling

50-70% of the planned code is done. I hope to be ready in about 1 or 2 weeks.

[SuperJudge]

I was frightened how quick that was. I had gotten used to starting emerge -S and not paying attention to it for a long time.

[xororand]

I added an ebuild file of version 0.1.3-r1 for easier installation.
Get it here.

[skally]

0.1.4!
0.1.4!
This is what we're waiting for!

[Pythonhead]

I had a couple problems with it. First it failed with libtool version mismatches:

*** libtool.m4 and ltmain.sh have a version mismatch! ***
*** (libtool.m4 = 1.5.10, ltmain.sh = 1.5a) ***

Changing src_compile() to this fixes it:

Code: Select all

src_compile() {
	aclocal
	libtoolize --force --copy
	autoconf
	econf || die "econf failed"
	emake || die "emake failed"
}

Second, it doesn't work unless you run

Code: Select all

/usr/bin/portagedb u

from your root directory

# portagedb u
Updating portage database
Using portage cache: usr/portage/metadata/cache/
Error: scandir64("usr/portage/metadata/cache/", ..) failed
Database update failed

Also, it'd be easier if you named your tarball files without the -r1. Its just a little confusing since you have S=${WORKDIR}/${P}-r1 and the ebuild named with -r1.

Runs amazingly fast, nice job. Number one feature request: overlay directories

[SuperJudge]

I had a couple problems with it. First it failed with libtool version mismatches:

*** libtool.m4 and ltmain.sh have a version mismatch! ***
*** (libtool.m4 = 1.5.10, ltmain.sh = 1.5a) ***

Changing src_compile() to this fixes it:

Code: Select all

src_compile() {
	aclocal
	libtoolize --force --copy
	autoconf
	econf || die "econf failed"
	emake || die "emake failed"
}

Second, it doesn't work unless you run

Code: Select all

/usr/bin/portagedb u

from your root directory

# portagedb u
Updating portage database
Using portage cache: usr/portage/metadata/cache/
Error: scandir64("usr/portage/metadata/cache/", ..) failed
Database update failed

Also, it'd be easier if you named your tarball files without the -r1. Its just a little confusing since you have S=${WORKDIR}/${P}-r1 and the ebuild named with -r1.

Runs amazingly fast, nice job. Number one feature request: overlay directories

Is there a way to fix libtool, every other damn ebuild needs to be modified because of libtool version errors. I used to have an older version of libtool installed, but I'm tired of switching back.

[Pythonhead]

Is there a way to fix libtool, every other damn ebuild needs to be modified because of libtool version errors. I used to have an older version of libtool installed, but I'm tired of switching back.

libtool isn't broken, ebuilds are:
http://article.gmane.org/gmane.linux.ge ... libtoolize

So its going to be a pain for a while but should fix some real errors in the long run.

[SuperJudge]

Is there a way to fix libtool, every other damn ebuild needs to be modified because of libtool version errors. I used to have an older version of libtool installed, but I'm tired of switching back.

libtool isn't broken, ebuilds are:
http://article.gmane.org/gmane.linux.ge ... libtoolize

So its going to be a pain for a while but should fix some real errors in the long run.

I figured it was the ebuild, but I couldn't search out a good fix, and it seemed to work fine when I used an older ebuild. Thanks for the info.

[xororand]

I had a couple problems with it. First it failed with libtool version mismatches. Changing src_compile() to this fixes it:

Thanks for the detailed notice. I'll fix it when 0.2.0 is final. The 0.1.3-r1 eBuild will be fixed as soon as possible.

Second, it doesn't work unless you run

Code: Select all

/usr/bin/portagedb u

from your root directory

Unfortunately, I can't reproduce this behaviour. Do you use a custom PORTDIR (make.conf)?

Also, it'd be easier if you named your tarball files without the -r1. Its just a little confusing since you have S=${WORKDIR}/${P}-r1 and the ebuild named with -r1.

Thanks again for the hint. Meanwhile I've studied the portage help document and learned about proper version numbering. The new version is 0.2.0 - patch releases will be named with a _p suffix from now on.

By the way, 0.2.0_alpha2 is available on Sourceforge I hope it works.