two new metadata query tools

Message

Genone · Post by **Genone** » Tue Jun 28, 2005 4:23 am

Guess it's about time I release my little metadata query tool to the public, I've kept it for myself long enough:
http://dev.gentoo.org/~genone/scripts/metascan
What does it do? It scans the portage tree (or your vardb) for packages/ebuilds matching certain metadata conditions, examples:

Code: Select all

# - find all packages in the tree supporting the berkdb use flag: metascan IUSE berkdb
# - find all packages in the tree providing an mta: metascan PROVIDE virtual/mta
# - find all packages in the tree providing an mta and not supporting the maildir use flag:
# 		metascan -a PROVIDE virtual/mta IUSE -maildir
# - find all installed packages containing net-nds/openldap in DEPEND or RDEPEND:
# 		metascan -i -n DEPEND net-nds/openldap RDEPEND net-nds/openldap

Please note that you should be familiar with the basic ebuild syntax to use this tool. If you can't follow the examples above, this tool is not for you.

It's intended as a low-level query tool for maintenance tasks to complement the higher-level functions of equery (in the long run it might also be integrated into equery). I also hope to post the counterpart script auxget (maybe with a different name) shortly (needs some more polishing first).

EDIT Updated to work tihe the new portage-2.1_pre versions (including several refactorings and other enhancements)

Genone · Post by **Genone** » Wed Jun 29, 2005 4:45 am

Ok, as said I also uploaded auxget now:
http://dev.gentoo.org/~genone/scripts/auxget

It's the complement to metascan as it allows you to read the individual variables for a given package, for example:

Code: Select all

auxget app-admin/sudo
      DEPEND: pam? ( || ( virtual/pam sys-libs/pam ) ) ldap? ( >=net-nds/openldap-2.1.30-r1 ) skey? ( >=app-admin/skey-1.1.5-r1 ) sys-devel/bison virtual/mta !bootstrap? ( sys-devel/patch userland_GNU? ( sys-apps/shadow ) )
     RDEPEND: pam? ( || ( virtual/pam sys-libs/pam ) ) ldap? ( >=net-nds/openldap-2.1.30-r1 ) skey? ( >=app-admin/skey-1.1.5-r1 ) sys-devel/bison virtual/mta ldap? ( dev-lang/perl )
        SLOT: 0
     SRC_URI: ftp://ftp.sudo.ws/pub/sudo/sudo-1.6.8p9.tar.gz
    RESTRICT: 
    HOMEPAGE: http://www.sudo.ws/
     LICENSE: Sudo
 DESCRIPTION: Allows certain users/groups to run commands as root
    KEYWORDS: alpha amd64 hppa ia64 ppc ppc64 sparc x86
   INHERITED: multilib pam eutils
        IUSE: pam skey offensive ldap
     CDEPEND: 
     PDEPEND: 
     PROVIDE:

Also has a few options to tune the output to your needs. Like metascan this is a low-level tool, so you should know a little bit about portage if you want to use it.

TrueDFX · Post by **TrueDFX** » Wed Jun 29, 2005 6:39 am

Very nice. Two questions though:

I noticed auxget's -r option doesn't respect package.use. Is that intentional? (Also, in combination with -i, -r uses the current USE flags, not those with which the package was installed, but that doesn't matter to me.)

The second question: would it be possible to add a similar option to metascan to search only through the USE-reduced *DEPENDs?

Genone · Post by **Genone** » Wed Jun 29, 2005 10:48 am

TrueDFX wrote:Very nice. Two questions though:

I noticed auxget's -r option doesn't respect package.use. Is that intentional?

Should work and works for me ...

(Also, in combination with -i, -r uses the current USE flags, not those with which the package was installed, but that doesn't matter to me.)

Yeah, support for vardb is limited because the interface is very different.

The second question: would it be possible to add a similar option to metascan to search only through the USE-reduced *DEPENDs?

Done (just added that code to auxget a few minutes before posting the link).

TrueDFX · Post by **TrueDFX** » Wed Jun 29, 2005 11:54 am

What am I missing, then? I have nocxx in my USE flags, overridden for gcc with package.use.

Code: Select all

$ emerge -pv sys-devel/gcc

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild   R   ] sys-devel/gcc-4.0.1_pre20050616  (-altivec) -bootstrap -build +fortran -gcj* +gtk -hardened -ip28 (-multilib) -multislot (-n32) (-n64) +nls -nocxx +objc -static 31,018 kB 
[ebuild  N    ] sys-libs/libstdc++-v3-3.3.4  (-multilib) +nls 22,784 kB 

Total size of downloads: 53,803 kB
$ python auxget sys-devel/gcc PDEPEND
     PDEPEND: sys-devel/gcc-config !nocxx? ( !n32? ( !n64? ( !elibc_uclibc? ( !build? ( sys-libs/libstdc++-v3 ) ) ) ) )
$ python auxget -r sys-devel/gcc PDEPEND
     PDEPEND: sys-devel/gcc-config

And thanks for the metascan improvement, but unfortunately it doesn't work with portage-2.0*: portage_dep is not imported, and if I add that to the import line, I get another error:

Code: Select all

$ python metascan.1 -i -c -p DEPEND dev-lang/python
Reading package list ... done
Scanning packages for ['DEPEND'] ... Traceback (most recent call last):
  File "metascan.1", line 194, in ?
    pvalues[i] = " ".join(portage.flatten(portage_dep.use_reduce(portage_dep.paren_reduce(pvalues[i]), settings["USE"])))
IndexError: list index out of range

(Edit: I read the comment in the script, tried adding -r, and success. So I guess importing portage_dep is all. Great job. thanks

)

Genone · Post by **Genone** » Wed Jun 29, 2005 4:45 pm

TrueDFX wrote:What am I missing, then? I have nocxx in my USE flags, overridden for gcc with package.use.

Weird. That exact example works here (same gcc version too).

TrueDFX · Post by **TrueDFX** » Wed Jun 29, 2005 5:49 pm

Another portage 2.1* change? I managed to get it working by changing

Code: Select all

portage_dep.use_reduce(..., settings["USE"])

to

Code: Select all

portage_dep.use_reduce(..., " ".split(settings["USE"]))

So actually, it looks like it had nothing to do with package.use at all, and would probably print wrong results even without package.use. Sorry for the confusion.

Genone · Post by **Genone** » Thu Jun 30, 2005 12:00 pm

Hmm, might have had a bug there, please try the version I just uploaded (featuring a new --show-cpv option also).

TrueDFX · Post by **TrueDFX** » Thu Jun 30, 2005 12:10 pm

No problem here with your new version.

Genone · Post by **Genone** » Wed Sep 21, 2005 7:35 am

Added some fixes and enhancements to metascan (most notably a --debug and --regular-expressions switch).
NOTE: the syntax for negated queries has changed as the old syntax didn't allow to query for stuff like KEYWORDS="-amd64".
Old syntax: metascan KEY -value
New syntax: metascan -KEY value

dpetka2001 · Post by **dpetka2001** » Sat Oct 22, 2005 10:24 am

just downloaded metascan and am not able to use it...it says

Code: Select all

-bash: /usr/bin/local/metascan: Permission denied

why is that?? I am logged in as root...and another question...would you suggest metascan to find reverse dependencies for a package compared to unclepine?? thanks...

Genone · Post by **Genone** » Sat Oct 22, 2005 10:39 am

Already asked in the other thread, but you gave it execute permissions, right?
As for the other question, I don't know unclepine, but metascan does things the right way (as it just queries portage instead of playing around with vdb/porttree directly)

dpetka2001 · Post by **dpetka2001** » Sat Oct 22, 2005 11:01 am

it was the execute permission that was the problem...fixed it and it works now...so in order to find reverse dependencies for a specific package I have to type

Code: Select all

metascan -n DEPEND category/application-version

?? what is it better to use?? the portage tree or the vardb as database??

Genone · Post by **Genone** » Sat Oct 22, 2005 11:18 am

Depends: you want the actual reverse deps for a installed package or the potential reverse deps for a non-installed package?
For the first you use vdb, for the second porttree.
Oh, and you don't specify a version when using -n (generally using versions with metascan is tricky).

dpetka2001 · Post by **dpetka2001** » Sat Oct 22, 2005 12:33 pm

so to find the reverse dependencies of an installed package on my system I would use

Code: Select all

metascan -i -n DEPEND category/package

right?? is any other Key needed?? what about RDEPEND that you use in your example??

Genone · Post by **Genone** » Sat Oct 22, 2005 1:42 pm

dpetka2001 wrote:so to find the reverse dependencies of an installed package on my system I would use
Code: Select all
metascan -i -n DEPEND category/package
right?

Yep

is any other Key needed?? what about RDEPEND that you use in your example??

Well, there are three different types of dependencies:
- DEPEND: build time dependencies
- RDEPEND: run time dependencies
- PDEPEND: run time dependencies that should be merged after the package depending on them
It depends on the specific situation which key(s) you should use, e.g. for unmerge safety checks you want RDEPEND and PDEPEND but not DEPEND while for test coverage checks you want all of them.

dberkholz · Post by **dberkholz** » Tue Dec 20, 2005 3:25 am

Downloaded both and neither run, with the portage-2.1:

Code: Select all

$ metascan --help
Traceback (most recent call last):
  File "/home/donnie/dev/public_html/scripts/metascan", line 32, in ?
    isvalidatom = portage_dep.isvalidatom
AttributeError: 'module' object has no attribute 'isvalidatom'

Ideas?

Genone · Post by **Genone** » Tue Jan 17, 2006 7:32 pm

Ups, sorry, forgot to upload the new versions after I put them in subversion. Current versions should work with both 2.0 and new 2.1.

Genone · Post by **Genone** » Mon Feb 13, 2006 5:16 am

As another addition auxget has now an alternate personality that is triggered when it is invoked as mass_auxget, this allows reading metadata keys for multiple packages at once (instead of calling auxget for each of them), example:

Code: Select all

$ mass_auxget IUSE LICENSE sys-apps/portage app-portage/gentoolkit app-portage/porthole
sys-apps/portage-2.1_pre4-r1: IUSE="build doc selinux" LICENSE="GPL-2" 
app-portage/gentoolkit-0.2.2_pre2: IUSE="" LICENSE="GPL-2" 
app-portage/porthole-0.4.1: IUSE="debug" LICENSE="GPL-2"

When no packages are specified all packages are looked up, same for keys (so without any arguments it will look up all keys for all packages).

eyebex · Post by **eyebex** » Mon Feb 26, 2007 5:21 pm

For those seeking metalib.py, like me, here's the link:
http://dev.gentoo.org/~genone/scripts/metalib.py