Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Helper script: distfiles cleanup
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
me22
n00b
n00b


Joined: 30 Jun 2002
Posts: 45
Location: Canada

PostPosted: Fri Jan 01, 2010 8:13 pm    Post subject: Helper script: distfiles cleanup Reply with quote

After a number of years, my /usr/portage/distfiles has gotten up to about 18 GB of stuff, so I figured that for the new year I'd clean it out a bit. Since it's almost 10k files, though I knew I'd need a script if I didn't just want to wipe it. Here's what I came up with.

It attempts to guess outdated packages by regex and modification date, so without arguments it just shows the most recent file and the size of the older ones, ordered by the file groups taking the most space:
Code:

$ old_versions.pl
linux-*.tar.bz2: linux-2.6.30.tar.bz2
 50355835 linux-2.6.27.tar.bz2
 49441874 linux-2.6.26.tar.bz2
 48601689 linux-2.6.25.tar.bz2
 46737783 linux-2.6.24.tar.bz2
 45488158 linux-2.6.23.tar.bz2
 45119878 linux-2.6.22.tar.bz2
 43375937 linux-2.6.20.tar.bz2
 41272919 linux-2.6.17.tar.bz2
 42733268 linux-2.6.19.tar.bz2
 41863580 linux-2.6.18.tar.bz2
454990921 total
gcc-*.tar.bz2: gcc-4.4.2.tar.bz2
 59368714 gcc-4.3.4.tar.bz2
 62869928 gcc-4.4.1.tar.bz2
 58929447 gcc-4.3.2.tar.bz2
 59369954 gcc-4.3.3.tar.bz2
 58964610 gcc-4.3.1.tar.bz2
 39707720 gcc-4.1.2.tar.bz2
 28193401 gcc-3.4.6.tar.bz2
 23972413 gcc-3.3.6.tar.bz2
391376187 total


With a --rm argument, it will rm -i the ones it thinks are no longer used:
Code:

distfiles # old_versions.pl --rm
boost_*.tar.bz2: boost_1_41_0.tar.bz2
rm: remove regular file `boost_1_39_0.tar.bz2'? 
rm: remove regular file `boost_1_35_0.tar.bz2'?
rm: remove regular file `boost_1_34_1.tar.bz2'?
rm: remove regular file `boost_1_34_0.tar.bz2'?
rm: remove regular file `boost_1_34_pre20061214.tar.bz2'? y
rm: remove regular file `boost_1_33_1.tar.bz2'?
rm: remove regular file `boost_1_33_0.tar.bz2'?
fgfs-base-*.tar.bz2: Only have fgfs-base-0.9.10.tar.bz2
mplayer-*.tar.bz2: mplayer-1.0_rc4_p20090919.tar.bz2
rm: remove regular file `mplayer-1.0_rc2_p20090731.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p20090322.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p28450.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p28348.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p28288.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p28058.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p27725.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p27458.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p27120.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p26753.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p26454.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p26450.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p26300.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p26258.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p25993.tar.bz2'? y
rm: remove regular file `mplayer-1.0_rc2_p24929.tar.bz2'? y


Here's the perl code:
Code:

#!/usr/bin/perl

use strict;

my %seen = ();
my %keep = ();
my %gone = ();
my %size = ();

foreach my $file (split(/\n/, `ls -t /usr/portage/distfiles`)) {
    my $pkg = $file;
    $pkg =~ s{\d+([._-]\d+)*([a-z]\d?)?([_-]r\d+(.\d+)*)?([_-]rc\d+)?([_-]pre\d+)?(?=.)}{*}g;
    $pkg =~ tr/A-Z/a-z/;
    if ($pkg =~ m{^\*\.(zip|tar\.(gz|bz2))$}) { next; }
    $size{$pkg} += (-s "$file");
    if ($seen{$pkg}) {
        $gone{$pkg} .= "$file\n";
    } else {
        $seen{$pkg} = $file;
        $keep{$pkg} .= "$file\n";
    }
}

foreach my $pkg (reverse sort {$size{$a} <=> $size{$b}} keys %seen) {
    if ($gone{$pkg}) {
        print "$pkg: $keep{$pkg}";
        if ($ARGV[0] eq "--rm") {
            system("rm", "-i", split(/\n/, $gone{$pkg}));
        } else {
            system("wc", "-c", split(/\n/, $gone{$pkg}));
        }
    } else {
        print "$pkg: Only have $keep{$pkg}";
    }
}


Of course, since it's distfiles, killing extra stuff is safe, but I prefer not to do that.

I hope someone else finds this useful too.
_________________
~ Scott "me22" McMurray
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Fri Jan 01, 2010 9:01 pm    Post subject: Reply with quote

eclean, contained in app-portage/gentoolkit, could have been used instead.
Back to top
View user's profile Send private message
me22
n00b
n00b


Joined: 30 Jun 2002
Posts: 45
Location: Canada

PostPosted: Fri Jan 01, 2010 9:30 pm    Post subject: Reply with quote

sera wrote:
eclean, contained in app-portage/gentoolkit, could have been used instead.

Ah, thanks. Guess I didn't look hard enough for the "right" way.

That said, eclean -p distfiles seems to take a few minutes to run, so I'm still happy to have my quick approximation. (Especially since I get most of the advantage just from cleaning up NVidia drivers and a handful of other packages.)
_________________
~ Scott "me22" McMurray
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Sat Jan 02, 2010 8:58 am    Post subject: Reply with quote

There is no "right" way. The difference is eclean is well tested and maintained for you.
Yours is certainly easier to expand if the need should arise. For instance if you want to mv instead of rm.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum