Unmerging and reverse dependencies-- need testers

Message

spb · Post by **spb** » Sat Aug 14, 2004 9:15 pm

What with all the people asking for 'emerge -C' to clear out packages that depend upon the one you're removing, I thought I'd try to implement it.

The revdep algorithm is based on ecatmur's dep script, but reimplemented from scratch, and probably slightly different.

Patch against emerge from 2.0.51_pre17 is here. I'm still working on this, and it's by no means finished, but I'd appreciate some feedback.

Known issues at the moment:
-- It breaks unmerge of specific versions. (ie 'emerge -C gtk+' works, but 'emerge -C =gtk+-1.2*' doesn't.) (This seems to be fixed now)
-- Virtual handling seems to work, but might well be flaky.
-- The way it interacts with the existing unmerge code is horrible. I'll fix that once I get the other issues out of the way.

[edit] See further down for newer versions.

gurke · Post by **gurke** » Sun Aug 15, 2004 10:56 am

great idea, and finally someone found the time to implement it. i hope i can try it later today.
a nice feature would be if the unmerging worked like uninstalling with "debfoster" (asking you with each package in dependancy, if it should be kept). its a reverse dependencies program for debian, which works really nice. take a look at it. it logs the dependencies installed, while installing a package and then uses them for uninstallment, iirc.

robmoss · Post by **robmoss** » Sun Aug 15, 2004 5:29 pm

gurke - that works fine with Debian, as once a binary is in there, the .deb remains identical until revision or version bumped. But Gentoo ebuilds don't necessarily stay the same, so that won't work on Gentoo. Things must be done on-the-fly.

Syntaxis · Post by **Syntaxis** » Mon Aug 16, 2004 2:28 am

If that's the problem, why not just mandate bumping the version or revision number whenever any change is made to an ebuild?

I see that the existing ebuild policy states that such bumping should be restricted to

when the ebuild has changed to the point where users would want to upgrade

but couldn't some mechanism instead be devised whereby the user could just snake down a copy of the Gentoo changelog for the ebuild in question, open it up in the editor of his choice and then judge for himself whether or not the change is worth an upgrade?

oberyno · Post by **oberyno** » Mon Aug 16, 2004 4:16 am

Syntaxis: You might want to check out this.

Nifty patch thebell. It worked with panotools and kdelibs, but failed on qt.

Code: Select all

root> emerge -Cp qt                                               /home/oberyno
['x11-libs/qt-3.3.2']                                                          \adding 'x11-libs/qt-3.3.2'; parent= None
[]
calculating revdeps for x11-libs/qt-3.3.2
checking app-cdr/k3b-0.11.12-r1
Found revdep app-cdr/k3b-0.11.12-r1 for x11-libs/qt-3.3.2
checking kde-base/kdelibs-3.2.3
Found revdep kde-base/kdelibs-3.2.3 for x11-libs/qt-3.3.2
checking media-video/avifile-0.7.38.20030710-r1
checking net-www/opera-7.54
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2781, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2023, in unmerge
    mydepgraph.create(mypkg)
  File "/usr/bin/emerge", line 1986, in create
    for mydep in self.direct_revdeps(cpv):
  File "/usr/bin/emerge", line 1948, in direct_revdeps
    if self.parse_dep(myatom, cpv, myinstcpvs):
  File "/usr/bin/emerge", line 1904, in parse_dep
    if not cpv_satisfies_dep(cpv, atom):
  File "/usr/bin/emerge", line 1803, in cpv_satisfies_dep
    return ( len(portage.match_from_list(atom, [cpv])) > 0 )
  File "/usr/lib/portage/pym/portage.py", line 4183, in match_from_list
    cat,pkg = catsplit(mycpv)
ValueError: unpack list of wrong size

Anyway, great work.

spb · Post by **spb** » Mon Aug 16, 2004 10:23 am

oberyno wrote:Nifty patch thebell. It worked with panotools and kdelibs, but failed on qt.

Code: Select all

root> emerge -Cp qt                                               /home/oberyno
['x11-libs/qt-3.3.2']                                                          \adding 'x11-libs/qt-3.3.2'; parent= None
[]
calculating revdeps for x11-libs/qt-3.3.2
checking app-cdr/k3b-0.11.12-r1
Found revdep app-cdr/k3b-0.11.12-r1 for x11-libs/qt-3.3.2
checking kde-base/kdelibs-3.2.3
Found revdep kde-base/kdelibs-3.2.3 for x11-libs/qt-3.3.2
checking media-video/avifile-0.7.38.20030710-r1
checking net-www/opera-7.54
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2781, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2023, in unmerge
    mydepgraph.create(mypkg)
  File "/usr/bin/emerge", line 1986, in create
    for mydep in self.direct_revdeps(cpv):
  File "/usr/bin/emerge", line 1948, in direct_revdeps
    if self.parse_dep(myatom, cpv, myinstcpvs):
  File "/usr/bin/emerge", line 1904, in parse_dep
    if not cpv_satisfies_dep(cpv, atom):
  File "/usr/bin/emerge", line 1803, in cpv_satisfies_dep
    return ( len(portage.match_from_list(atom, [cpv])) > 0 )
  File "/usr/lib/portage/pym/portage.py", line 4183, in match_from_list
    cat,pkg = catsplit(mycpv)
ValueError: unpack list of wrong size

Hmm...I thought I'd got rid of that one. Can you go to line 1903, uncomment the print statement, and try it again? That should give a bit more info on what's causing it to die. This code does seem to choke on seemingly invalid DEPEND statements that the rest of emerge accepts. I'm not sure why, since most of the depend atom handling is in existing code.

Anyway, great work.

Thanks.

gurke · Post by **gurke** » Mon Aug 16, 2004 10:43 am

robmoss, an option would be logging 1st tier depenencies of each package as emerged. this also avoids trouble when changing useflags between the emerge and the unmerge. at an unmerge you could check for all non "worldfiled" packages in the logged dependency tree of the unmerged package and mark em for removal. then you would need to check if these packages are in dep. of any other worldfile packages. this should be computable pretty fast since you dont have to double check packages and since the average youser shouldnt have more packages then 1k.

oberyno · Post by **oberyno** » Mon Aug 16, 2004 10:44 am

Sure.

Code: Select all

oberyno> emerge -Cp qt                                                                                          
['x11-libs/qt-3.3.2']                                                                                           \adding 'x11-libs/qt-3.3.2'; parent= None
[]
calculating revdeps for x11-libs/qt-3.3.2
checking media-video/avifile-0.7.38.20030710-r1
Simple atom >=media-libs/jpeg-6b
Simple atom >=media-libs/divx4linux-20030428
Simple atom >=media-libs/win32codecs-0.90
Simple atom >=media-video/ffmpeg-0.4
Simple atom >=media-libs/xvid-0.9.0
Simple atom >=media-sound/lame-3.90
Simple atom >=media-libs/audiofile-0.2.3
Simple atom >=sys-apps/sed-4
Simple atom >=media-sound/madplay-0.14.2b
Simple atom >=sys-devel/patch-2.5.9
Simple atom >=media-libs/freetype-2.1
Simple atom >=media-libs/libsdl-1.2.2
Simple atom >=media-libs/a52dec-0.7
Simple atom >=sys-libs/zlib-1.1.3
Simple atom >=media-libs/libvorbis-1.0
Simple atom >=media-libs/alsa-lib-0.9.0_rc2
checking net-www/opera-7.54
Simple atom >=media-libs/fontconfig-2.1.94-r1
Simple atom media-libs/libexif
Simple atom x11-libs/openmotif
Simple atom app-text/aspell
Simple atom !amd64
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2781, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2023, in unmerge
    mydepgraph.create(mypkg)
  File "/usr/bin/emerge", line 1986, in create
    for mydep in self.direct_revdeps(cpv):
  File "/usr/bin/emerge", line 1948, in direct_revdeps
    if self.parse_dep(myatom, cpv, myinstcpvs):
  File "/usr/bin/emerge", line 1904, in parse_dep
    if not cpv_satisfies_dep(cpv, atom):
  File "/usr/bin/emerge", line 1803, in cpv_satisfies_dep
    return ( len(portage.match_from_list(atom, [cpv])) > 0 )
  File "/usr/lib/portage/pym/portage.py", line 4183, in match_from_list
    cat,pkg = catsplit(mycpv)
ValueError: unpack list of wrong size

I'll check my overlay for ebuilds with !amd64.

spb · Post by **spb** » Mon Aug 16, 2004 11:06 am

OK, it doesn't like Opera's RDEPEND:

Code: Select all

RDEPEND="virtual/x11
        >=media-libs/fontconfig-2.1.94-r1
        media-libs/libexif
        x11-libs/openmotif
        spell? ( app-text/aspell )
        amd64? ( app-emulation/emul-linux-x86-xlibs )
        !amd64 ( !sparc? ( !static? ( =x11-libs/qt-3* ) ) )"

It's that last line that's causing the problem; unless I'm much mistaken, that '!amd64 ('ought to be '!amd64? ('. Seemingly the 'normal' emerge code is fine with this, but use_reduce doesn't handle it properly. I had this problem with mozilla too; editing /var/db/pkg/cat/package-ver/RDEPEND and putting the ? in fixes it. I really should come up with a proper workaround.

oberyno · Post by **oberyno** » Mon Aug 16, 2004 11:28 am

Yep, the ? fixes it. Ideally though, this should be an option to unmerge in my opinion.

spb · Post by **spb** » Mon Aug 16, 2004 11:42 am

OK, new version here that should accept those weird RDEPEND lines. It's been tested against mozilla so far; let me know if it fixes opera.

oberyno · Post by **oberyno** » Mon Aug 16, 2004 11:53 am

Works like a charm.

Code: Select all

checking net-www/opera-7.54
mydep= virtual/x11 >=media-libs/fontconfig-2.1.94-r1 media-libs/libexif x11-libs/openmotif spell? ( app-text/aspell ) amd64? ( app-emulation/emul-linux-x86-xlibs ) !amd64 ( !sparc? ( !static? ( =x11-libs/qt-3* ) ) )
workaround: myrawdep= virtual/x11 >=media-libs/fontconfig-2.1.94-r1 media-libs/libexif x11-libs/openmotif spell? ( app-text/aspell ) amd64? ( app-emulation/emul-linux-x86-xlibs ) !amd64 ( !sparc? ( !static? ( =x11-libs/qt-3* ) ) )
['virtual/x11', '>=media-libs/fontconfig-2.1.94-r1', 'media-libs/libexif', 'x11-libs/openmotif', 'spell?', '(', 'app-text/aspell', ')', 'amd64?', '(', 'app-emulation/emul-linux-x86-xlibs', ')', '!amd64?', '(', '!sparc?', '(', '!static?', '(', '=x11-libs/qt-3*', ')', ')', ')']
mydep= virtual/x11 >=media-libs/fontconfig-2.1.94-r1 media-libs/libexif x11-libs/openmotif spell? ( app-text/aspell ) amd64? ( app-emulation/emul-linux-x86-xlibs ) !amd64? ( !sparc? ( !static? ( =x11-libs/qt-3* ) ) )
Found revdep net-www/opera-7.54 for x11-libs/qt-3.3.2
checking sys-apps/dbus-0.22.20040725

spb · Post by **spb** » Mon Aug 16, 2004 11:58 am

Brilliant. I've reuploaded it with slightly fewer debug lines enabled. Now all I need to do is clean up the interaction with the existing unmerge code. Now this is going to be fun...

spb · Post by **spb** » Mon Aug 16, 2004 9:13 pm

OK, new version here, now with the revdep code integrated into unmerge(). It *feels* pretty much stable now, and I haven't found any outstanding issues with this version. Anyone feel like testing it?

oberyno · Post by **oberyno** » Mon Aug 16, 2004 10:14 pm

I'm getting an error:

Code: Select all

root> emerge -Cp xorg-x11                                                                            /home/oberyno

>>> These are the packages that I would unmerge:

Calculating reverse dependencies \Traceback (most recent call last):
  File "/usr/bin/emerge", line 2783, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2142, in unmerge
    myrdepgraph.create(y)
  File "/usr/bin/emerge", line 2005, in create
    for mydep in self.direct_revdeps(cpv):
  File "/usr/bin/emerge", line 1955, in direct_revdeps
    mydep = portage_dep.dep_opconvert(mydep)
AttributeError: 'module' object has no attribute 'dep_opconvert'

I changed line 1955 to look like this:

Code: Select all

	mydep = portage.dep_opconvert(mydep,myuse,myopts)

as dep_opconvert is in portage.py for me. I'm just learning python, but that works for me. Well, it works for xorg-x11 and qt. Glibc spits this out

Code: Select all

root> emerge -Cp glibc                                                                               /home/oberyno

>>> These are the packages that I would unmerge:

Calculating reverse dependencies -Traceback (most recent call last):
  File "/usr/bin/emerge", line 2783, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2142, in unmerge
    myrdepgraph.create(y)
  File "/usr/bin/emerge", line 2006, in create
    self.create( mydep, cpv, 1 )
  File "/usr/bin/emerge", line 2005, in create
    for mydep in self.direct_revdeps(cpv):
  File "/usr/bin/emerge", line 1963, in direct_revdeps
    if self.parse_dep(myatom, cpv, myinstcpvs):
  File "/usr/bin/emerge", line 1895, in parse_dep
    if mycp not in provide:
NameError: global name 'provide' is not defined

The --nodeps part works perfectly.

Edit:Perl and python give the same error. It might actually be a good thing to not allow users to unmerge these things as the system gets broken without them. Still, if there is going to be an error, it should be more graceful.

oberyno · Post by **oberyno** » Tue Aug 17, 2004 12:14 am

Ok, I upgraded to the latest portage (_pre18) and no longer have to change the dep_opconvert line. Your patch is against _pre17 though, so I patched that emerge.

neonik · Post by **neonik** » Tue Aug 17, 2004 7:56 am

_pre20 is in Portage now. Gonna try to patch and see how it works.

The idea is great.

neonik · Post by **neonik** » Tue Aug 17, 2004 8:19 am

One single reject. 10 seconds to fix.

So here thebell's patch a little bit adjusted to work with emerge from sys-apps/portage-2.0.51_pre20: emerge-unmerge-rdep-v4-2.0.51_pre20.diff

Edit: updated the link.

neonik · Post by **neonik** » Tue Aug 17, 2004 8:36 am

Tested and working:
x11-base/xorg-x11
x11-libs/qt
kde-base/*
net-www/apache
x11-libs/gtk+
sys-devel/gcc
sys-libs/glibc

What I don't understand, though: trying to unmerge arts, it wants to unmerge kdeartwork as well as kdebase and kdelibs, but trying to unmerge kdeartwork explicitly, there's nothing else it's going to unmerge but kdeartwork self.
What I understand from that is that arts is dependent on all of those packages above while kdeartwork isn't. Although, according to the ebuild kdeartwork depends on kdebase.

DEPEND="opengl? ( virtual/opengl )
~kde-base/kdebase-${PV}"

Is it supposed to be so or am I misunderstanding something?

By the way, great job, thebell.

neonik · Post by **neonik** » Tue Aug 17, 2004 9:03 am

Running

Code: Select all

packages=`qpkg -I -nc` ; ./emerge-rdep -pC $packages

I get the following:

>>> These are the packages that I would unmerge:

...done

...done

...done

...done

...done

...done

!!! Trying to unmerge package(s) in system profile. 'app-arch/bzip2'
!!! This could be damaging to your system.

Then it stops and emerge spits out this python error message in STDERR:

Traceback (most recent call last):
File "./emerge-rdep", line 2807, in ?
if 1==unmerge(myaction, myfiles):
File "./emerge-rdep", line 2166, in unmerge
myrdepgraph.create(y)
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2029, in create
for mydep in self.direct_revdeps(cpv):
File "./emerge-rdep", line 1987, in direct_revdeps
if self.parse_dep(myatom, cpv, myinstcpvs):
File "./emerge-rdep", line 1919, in parse_dep
if mycp not in provide:
NameError: global name 'provide' is not defined

I admit, it's pointless to try to unmerge the whole system according to revdeps. But should this feature replace the original way to unmerge, I guess it should also let the user unmerge the whole system, since the original unmerge function lets do this.

spb · Post by **spb** » Tue Aug 17, 2004 9:09 am

@oberyno: yes, sorry about that. I updated portage, and suddenly it didn't work, since paren_reduce got simplified (it used to do a dep_opconvert, and now doesn't). I guess I should have warned you.

neonik wrote:What I don't understand, though: trying to unmerge arts, it wants to unmerge kdeartwork as well as kdebase and kdelibs, but trying to unmerge kdeartwork explicitly, there's nothing else it's going to unmerge but kdeartwork self.
What I understand from that is that arts is dependent on all of those packages above while kdeartwork isn't. Although, according to the ebuild kdeartwork depends on kdebase.
DEPEND="opengl? ( virtual/opengl )
~kde-base/kdebase-${PV}"
Is it supposed to be so or am I misunderstanding something?

The idea is that when you try to unmerge arts, it removes everything that has a runtime dependency on arts, which in this case includes kdebase and kdelibs. kdebase then depends on kdebase, so it gets removed too. Unmerging unneeded forward dependencies is the next thing I'm going to try.

By the way, great job, thebell.

Thanks.

spb · Post by **spb** » Tue Aug 17, 2004 9:28 am

neonik wrote:
Code: Select all
packages=`qpkg -I -nc` ; ./emerge-rdep -pC $packages

Interesting...I tried that, and it's still calculating revdeps 10 minutes later. This thing ain't exactly fast.

And thinking about things, it's doing dependency calculation in a stupid way. I should improve that.

[edit]Hmm... it worked perfectly (seemingly; the list overflowed my terminal buffer), eventually. Have you got a package installed with weird depend lines that I haven't come across yet?

[edit2] New version here. I've modified the depgraph creation a bit; it should be a lot quicker for huge revdepgraphs now (especially the unmerge-complete-system case), and also rediffed against _pre20. I still haven't managed to recreate neonik's problem.

neonik · Post by **neonik** » Tue Aug 17, 2004 10:30 am

thebell wrote:Have you got a package installed with weird depend lines that I haven't come across yet?

That's very possible. I'll take an attempt to spot it/them.

spb · Post by **spb** » Tue Aug 17, 2004 10:34 am

neonik wrote:
thebell wrote:Have you got a package installed with weird depend lines that I haven't come across yet?
That's very possible. I'll take an attempt to spot it/them.

Uncommenting lines 1971 and 1985 in the latest version would probably help.

neonik · Post by **neonik** » Tue Aug 17, 2004 10:46 am

Packages causing the exception:
#1: sys-apps/baselayout
#2: sys-apps/coreutils

I guess there might be more.

Here the traceback from sys-apps/coreutils:

Traceback (most recent call last):
File "./emerge-rdep", line 2807, in ?
if 1==unmerge(myaction, myfiles):
File "./emerge-rdep", line 2166, in unmerge
myrdepgraph.create(y)
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2029, in create
for mydep in self.direct_revdeps(cpv):
File "./emerge-rdep", line 1987, in direct_revdeps
if self.parse_dep(myatom, cpv, myinstcpvs):
File "./emerge-rdep", line 1919, in parse_dep
if mycp not in provide:
NameError: global name 'provide' is not defined

Here from sys-apps/baselayout:

Traceback (most recent call last):
File "./emerge-rdep", line 2807, in ?
if 1==unmerge(myaction, myfiles):
File "./emerge-rdep", line 2166, in unmerge
myrdepgraph.create(y)
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2030, in create
self.create( mydep, cpv, 1 )
File "./emerge-rdep", line 2029, in create
for mydep in self.direct_revdeps(cpv):
File "./emerge-rdep", line 1987, in direct_revdeps
if self.parse_dep(myatom, cpv, myinstcpvs):
File "./emerge-rdep", line 1919, in parse_dep
if mycp not in provide:
NameError: global name 'provide' is not defined