Cleaning out stale distfiles

[alexbuell]

It's just amazing how almost on every forum i'm in, there's a While E. Coyote nick =)

Beep beep!

[ghwomb]

Just posted so I would get a mail if anything happened. Don't seem to be much activity here though. Perhaps the scripts is being tested the Debian way; lots of testing on uncommon systems and no releases.

[far]

Eh, you could just click "Watch this topic for replies" in the lower left corner of the page.

A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball.

[erik_swanson]

A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball.

Hence the reason I believe that it's better to only remove distfiles that are truly dead (no longer used by anything in portage).
This is what my one-liner does, which I'm reposting because I've improved it to auto-detect your portdir/distdir/overlays using 'emerge info.'

Code: Select all

bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|")

[patrickfo]

i have 3 adsl gateway ( ipsec ). behind each there is two or more pc runing gentoo.
i choose to keep the sources files for each domain on a samba directory which is mounted at boot for every station.
just for the fun:
on the samba server in smb.conf :

Code: Select all

...
[distfiles]
  comment = no comment ;)
  path = /my_sources_files_dir
  valid users = root
  read only = no
  create mask = 0700
  browseable = no

on each station:
a file ( smcred ) somewhere in etc:

Code: Select all

username = root
password = afunnypassword

Code: Select all

smbpassword -a root

you enter here your password (not necessary to be the same as root login!)
for security reasons:

Code: Select all

chmod 0700 smcred

next i put in /etc/fstab

Code: Select all

//servername/distfiles  /mnt/distfiles  smbfs credentials=/etc/smcred  0 0

then in /etc/make.conf on each station

Code: Select all

DISTDIR=/mnt/distfiles

needless to say it is better to cp your files from your original distfile dir on the samba server BEFORE running unused_sf.sh, because missing files will be downloaded!!!

and now the scripts:
unused_sf.sh : create a list of sources files needed by each station in a directory named .cleandf/stationname/ which i put on the samba server in the distdir. ( i have modified the script that it can handle a single station with no distdir defined and tesded it just only once so be carefull!!!)
this script show also the packages that are installed and no mode in the portage tree, and download also files that are needed by installed packages and are missing ( i have found some patches for xfree more recent than te ones used for the original build on one or two stations and some other rewrited ( i suppose ) between the time of my compil and the time i run the script exemple: foomatic 3.0 )

expunge_sf.sh : list all the files and show the ones that are no more needed by any station and, file by file you can move them in a dir that you can later delete.
i am not a bash specialist, so be carefull! and also these scripts are very intensive time and resources consuming... and now the scripts
----------------

Code: Select all

#!/bin/bash
# unusedsf: Find unused sources files in DISTDIR
	
### erreurs possibles
RPT_IS_NOT_A_DIR=65
UNKNOWN_OPTION=66

### mes variables

# dir in which we (euh... I ) put the files...
RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2 )

if [ ${#RPT} != 0 ]; then
	RPT="$RPT/.cleandf/$HOSTNAME"
else
	RPT="$HOME/.cleandf/$HOSTNAME"
fi

### gestion des options...

while getopts :d: OPTION ;
	do
		case $OPTION in
		d)
			RPT=$OPTARG
			if [[ -d $RPT ]]; then
				echo ">>> dir of files produced changed to $RPT!"
			fi
			;;
		*)
			echo -e "\n!!! option $OPTARG unknown!\n"
			echo ">>> usage: $0 [-d <dir>]"
			echo -e "     -d directory\n"
			exit $UNKNOWN_OPTION
			;;
		esac;
done

# creating DISTDIR/.cleandf/HOSTNAME if not exist
if [[ -e $RPT ]]; then
	if [[ -d $RPT ]]; then
		echo "$RPT found."
	else
		echo "$RPT must be a directory!"
		exit $RPT_IS_NOT_A_DIR
	fi
else
	mkdir -p $RPT
	echo "$RPT created."
fi

### do some cleaning...
for FICH in emerge_list qpkg qpkgV ;
	do
		if [[ -f $RPT/$FICH.txt ]]; then
			rm -f $RPT/$FICH.txt
		fi;
done

### création des fichiers de travail

echo -en "doing \"qpkg -nc -I\" ...\t\t"
qpkg -nc -I | sort > $RPT/qpkg.txt
echo "ok."
echo -en "doing \"qpkg -nc -I -v\" ...\t"
qpkg -nc -I -v | sort > $RPT/qpkgV.txt
echo "ok."
echo 

for PACKAGE in $( cat $RPT/qpkg.txt );
	do
	for VERSION in $( grep $PACKAGE- $RPT/qpkgV.txt | cut -d '/' -f2 );
		do
		echo -e ">>> $PACKAGE/$VERSION"
		emerge --nospinner -f $PACKAGE/$VERSION.ebuild > $RPT/.temp
		if [ $? != 0 ]; then
		#packages not in portage tree, out of sync ( fixpackages!! )
			echo ">>> $PACKAGE/$VERSION output:" >> $RPT/ERREURS.txt
			cat $RPT/.temp >> $RPT/ERREURS.txt
			echo "============" >> $RPT/ERREURS.txt
		else
		#package that have the .ebuild file modif ( xfree-4.3. !!... )
			grep -i download $RPT/.temp 2>&1 >/dev/null
			if [ $? = 0 ]; then
			   echo ">>> $PACKAGE/$VERSION should be rebuild:" >> $RPT/to_REBUILD.txt
			   grep -i download $RPT/.temp >> $RPT/to_REBUILD.txt
			   echo "============" >> $RPT/to_REBUILD.txt
			fi
		fi;
		cat $RPT/.temp >> $RPT/emerge_list.txt
	done ;
done


# reporting problems
clear
if [ -e $RPT/to_REBUILD.txt ]; then
	echo "Packages that should be rebuild:"
	cat $RPT/to_REBUILD.txt
	echo
else
	echo "Well, nothing to rebuild... ;-)"
fi

if [ -e $RPT/ERREURS.txt ]; then
	echo "Actions must be done on:"
	cat $RPT/ERREURS.txt
	echo
else
	echo "Very well, no errors found... ;-)"
fi

================

Code: Select all

#!/bin/bash
# expunge_sf.sh: remove unecessaries files from DISTDIR

NOTHING_TO_DO=65
SAUVE_NOT_A_DIR=66

RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2)
SAUVE="/sauve/old_distfiles"
if [ ${#RPT} != 0 ]; then
	DISTDIR=$RPT
	RPT="$RPT/.cleandf"
else
	RPT="$HOME/.cleandf"
	DISTDIR="/usr/portage/distfiles"
fi

### gestion des options...

while getopts :d:s: OPTION ;
	do
		case $OPTION in
		d)
			RPT=$OPTARG
			;;
		s)
			SAUVE=$OPTARG
			;;
		*)
			echo -e "\n!!! option $OPTARG unknown!\n"
			echo ">>> usage: $0 [-d <dir>] -s <dir>"
			echo -e "     -d directory to put temporary files\n"
			echo -e "     -s directory to store backups\n"
			exit $UNKNOWN_OPTION
			;;
		esac;
done

if [ -e $RPT ]; then
	echo "$RPT found."
else
	echo "You must run unused_sf BEFORE expunge_sf!!!"
	exit $NOTHING_TO_DO
fi
	
if [[ ( -e $SAUVE ) && ( -d $SAUVE ) ]]; then
	echo "$SAUVE found."
else	
	echo "You must supply a dir for backup files!!!"
	exit $SAUVE_NOT_A_DIR
fi

# some cleaning from previous use...

if [ -f $RPT/fichiers_inutiles.txt ]; then
	rm -rf $RPT/fichiers_inutiles.txt
fi

if [ -f $RPT/tous_fichiers.txt ]; then
	rm -rf $RPT/tous_fichiers.txt
fi;

# list of files in DISTDIR

station=$(ls $RPT)

for FICH in $(ls $DISTDIR --ignore="cvs-src") ;
	do 
	echo $FICH
	used=0
	for ordi in $station;
		do
		grep $FICH $RPT/$ordi/emerge_list.txt >/dev/null 2>&1
		result=$?
		if [ $result = 0 ]; then
			used=1
		fi;

	done
	if [ $used -eq 1 ]; then
		echo "     $FICH" >> $RPT/tous_fichiers.txt
	else
		echo $FICH >> $RPT/fichiers_inutiles.txt
		echo "dead $FICH" >> $RPT/tous_fichiers.txt
	fi;
done

if [[ !( -e $RPT/fichiers_inutiles.txt) ]]; then
	echo "no files to move :( "
	exit 0
fi

resultat="hit control-C to break"

for FICH in $(cat $RPT/fichiers_inutiles.txt);
	do
	clear
	echo -e "$resultat\n"
	grep -A 15 -B3 --colour $FICH $RPT/tous_fichiers.txt

	echo -e "\n$FICH"
	echo -en "move $FICH to $SAUVE/: ? (y/n): \t"
	read reponse
	if [[ $reponse = "y" || $reponse = "Y" ]]; then
		mv $DISTDIR/$FICH $SAUVE/
		resultat="$FICH moved"
	else
		resultat="$FICH remain"
	fi;
done

============
good luck with them
any comment to ameliorate will be welcome, and injury must be directed to /dev/null ))
Patrick
PS: the indent is gone away... try to recreat it or say me and i will send you this files...

[esprout]

Ok, I figure I'll throw my script into the foray.

This one interactively processes the PKGDIR directory for old versions. Just run it without any arguments, and it will prompt you before deleting each file.

It'd be nice to have one script to process both PKGDIR and DISTDIR. Or maybe one of them already do, I missed it? So many scripts now.

I stole a couple lines/ideas from previous scripts, if something looks familiar

Code: Select all

#!/usr/bin/perl

# Root check
if ($< != 0) {
        print "You must be root to run this script.\n";
        exit 0;
}

my $DISTDIR;
my $PKGDIR;

# Figure out Portage Directories
my @emergeinfo = `emerge info`;
foreach my $info (@emergeinfo) {
        if ($info =~ m/^DISTDIR=/) {
                ($DISTDIR=$info) =~ s/^DISTDIR="(.*)"\n$/$1/;
        } elsif ($info =~ m/^PKGDIR=/) {
                ($PKGDIR=$info) =~ s/^PKGDIR="(.*)"\n$/$1/;
        }
}

print "Processing files in $PKGDIR...\n";

# Process files in $PKGDIR
chdir("$PKGDIR");
my $old;
my $new;
my $candidate;

# Generate list of .tbz2 files to process
@FILES = `find . -name \*.tbz2 | sort`;

# Process list of files
foreach (@FILES) {
        # Only process symlinks to reduce user interaction
        next if ($_ =~ m|^./All/|);
        $_ =~ s|\n$||;
        $new = $_;
        $new =~ s|(\./.+/.+)-[\d]+.*|$1|;
        #print "new: $_\n";
        if ($old) {
                # new files might be same as old file
                if ("$old" eq "$new") {
                        my $input;
                        $candidate =~ s|.tbz2$||;
                        # strip .tbz2 extension and use perl sort
                        # to do a better guess of which files is older
                        $_ =~ s|.tbz2$||;
                        ($candidate, $_) = sort($candidate, $_);
                        # reappend .tbz2 extension
                        $candidate .= ".tbz2";
                        $_ .= ".tbz2";
                        #print "dup: $old, $new, $candidate\n";
                        # Figure out corresponding file in All directory
                        (my $allcandidate = $candidate) =~ s|^./.*/|./All/|;
                        # Ask user to confirm deletion
                        print "rm:   $allcandidate and\n      $candidate\nkeep: $_ [y(es)/n(o)/s(kip)]? : ";
                        ($input = (<STDIN>)) =~ s|\n$||;
                        # Delete away
                        if ("$input" eq "y") {
                                unlink($candidate, $allcandidate);
                                print "$candidate, $allcandidate deleted\n";
                                $old = $new;
                                $candidate = $_;
                        # Ask user for swapped deletion
                        } elsif ("$input" eq "n") {
                                (my $allcandidate = $_) =~ s|^./.*/|./All/|;
                                print "rm:   $allcandidate and\n      $_\nkeep: $candidate [y(es)/n(o)]? : ";
                                ($input = (<STDIN>)) =~ s|\n$||;
                                if ("$input" eq "y") {
                                        unlink($_, $allcandidate);
                                        print "$_, $allcandidate delete\n";
                                } else {
                                        $old = $new;
                                        $candidate = $_;
                                }
                        }
                } else {
                        #print "nodup: $old, $new, $candidate\n";
                        $old = $new;
                        $candidate = $_;
                }
        } else {
                #print "first round: $old, $new, $candidate\n";
                $old = $new;
                $candidate = $_;
        }
}

[far]

Please put your source code inside BBCode [code] blocks like so:

[code]10 PRINT "Hello, World!"[/code]

If you don't, you will lose the indentation and the code will be difficult to read.

[patrickfo]

Please put your source code inside BBCode

Code: Select all

 blocks like so:

[code]10 PRINT "Hello, World!"

If you don't, you will lose the indentation and the code will be difficult to read.

thank you
i have corrected my post

[raptor]

small chages to original to show the old files size and total size that will be saved after stuff are deleted ...
example......
Would delete: ttmkfdir-3.0.4 [19 kB] in favour of ttmkfdir-3.0.9
Would delete: usbutils-0.11 [216 kB] in favour of usbutils-0.9
Would delete: vte-0.10.25 [759 kB] in favour of vte-0.10.26
Would delete: xscreensaver-4.05 [3133 kB] in favour of xscreensaver-4.06
Would delete: yelp-1.0.6 [478 kB] in favour of yelp-2.2.0
Would delete: zapping-0.6.4 [1058 kB] in favour of zapping-0.6.5
Would delete: zvbi-0.2.1 [486 kB] in favour of zvbi-0.2.3
Total size that will be reclaimed : 450038 kB

wow 450 MB not bad :")
===================================
#!/usr/bin/perl
use strict;

my $lastname = 0;
my $lastversion;
my $lastext;
my $totalSize;
my $lvSize;
my @stalefiles;
my @files;
........
if ($lastname && $name eq $lastname) {
if ($version gt $lastversion) {
$lvSize = (stat ("/usr/portage/distfiles/$name$version$lastext"))[7];
push (@stalefiles, [$name, $version, $lastversion, $lastext,$lvSize]);
}
}
......................

..........................
else
{
# Safe mode (default)!
print "\7*Pretend* mode, will only pretend to delete files.\nTo actually delete the files, reinvoke with the --nopretend parameter.\n\n";
foreach (@stalefiles) {
my ($name, $version, $lastversion, $ext, $lvSize) = @{$_};
$totalSize += $lvSize;
$lvSize = sprintf "%.2u kB", $lvSize/1024;
print "Would delete: $name$lastversion [$lvSize] in favour of $name$version\n";
}

print "Total size that will be reclaimed : " . (sprintf "%.2u kB", $totalSize/1024) . "\n";
}
1;

[SavageMindz]

This is a "one-liner" I wrote to safely clean out obsolete distfiles that are no longer referenced by any ebuild in portage.

nice one liner, just cleared me out about 4gb or old distfiles, that was quite sorely needed . Modified your command so I could get a list of what it got rid of and it seems to have worked a charm.

cheers

Savage

[cbradney]

Got this one from Kerframil:

Code: Select all

#!/usr/bin/env python
## distclean.py version 0.1 (20 Aug 2003,
## This is the first version with a version number)
##
## Removes source files for Gentoo
## packages that are no longer installed
## Use with '-p' (pretend) flag to just get a list of files
## that would be removed
##
## Copyright (c) 2003, Fredrik Arnerup (e97_far@e.kth.se)
## All rights reserved.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted provided that the following conditions are met:
##
##   * Redistributions of source code must retain the above copyright notice,
##     this list of conditions and the following disclaimer.
##
##   * Redistributions in binary form must reproduce the above copyright
##     notice, this list of conditions and the following disclaimer in the
##     documentation and/or other materials provided with the distribution.
##
## THIS SOFTWARE IS PROVIDED BY FREDRIK ARNERUP "AS IS" AND ANY
## EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
## WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
## DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
## FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
## DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
## SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
## CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
## LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
## OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
## DAMAGE.

import sys, os, os.path, getopt, portage

opt_p = 0
try:
    if len(sys.argv) > 1:
        opt_p = getopt.getopt(sys.argv[1:], 'p')[0][0][0] == '-p'
except getopt.GetoptError:
    pass

distdir = portage.config().environ()['DISTDIR']
print 'DISTDIR =', distdir
    
vartree = portage.db['/']['vartree']
packages = []
for name in vartree.getallnodes():
    packages.extend(vartree.dep_match(name))

files = {}
for package in packages:
    try:
        package_files = portage.portdb.aux_get(package, ['SRC_URI'])[0].split()
        package_files = [(url.split('/')[-1]) for url in package_files]
        for filename in package_files:
            files[filename] = 1
    except:
        print 'Failed to get file list for', package

if not files:
    sys.exit("No package files found.  This can't be right.\n")

try:
    list = portage.listdir(distdir)
except os.OSError:
    sys.exit('Failed to read ' + distdir)

size = 0; count = 0
for file in list:
    abs_file = distdir + '/' + file
    if (os.path.isfile(abs_file) and (not os.path.islink(abs_file)) 
        and (not file in files)):
        size += os.stat(abs_file).st_size
        count += 1
        if opt_p:
            print 'Would remove', abs_file
        else:
            try:
                os.remove(abs_file)
                print 'Removed', abs_file
            except OSError:
                print 'Failed to remove', abs_file


size /= 1048576  ## MB
print '%i files, total size: %i MB' % (count, size)

[far]

Yes, that is the script from my webpage ...

[cbradney]

Works wonderfully! Thank you!

[raptor]

thanx for the scripts "robm" and others.. a small modification at the end of the script :
Now it will not delete the files but will move them in /usr/portage/olddist. This way we can restore if something goes wrong, and delete only when nececary....

here are the results :

Code: Select all

$du -shc /usr/portage/distfiles/
2.6G    /usr/portage/distfiles
2.6G    total

$ ./distbuster 
.....

$ du -shc /usr/portage/distfiles/
599M    /usr/portage/distfiles
599M    total
$ du -shc /usr/portage/olddist
2.0G    /usr/portage/olddist
2.0G    total

not bad...

Code: Select all

opendir(DIR,$makeconf{"DISTDIR"})|| die "can't open ".$makeconf{"DISTDIR"};
#new line
mkdir "/usr/portage/olddist";
while ($file = readdir DIR) {
  next unless -f $makeconf{"DISTDIR"}."/$file";
  if  (! $files{$file}) {
    if ($opt_p) {
#      print "Would erase ".$makeconf{"DISTDIR"}."/$file\n";
      print "Would move  $makeconf{DISTDIR}/$file -> /usr/portage/olddist \n";
    }
    else {
#      unlink $makeconf{"DISTDIR"}."/$file";
       rename "$makeconf{DISTDIR}/$file", "/usr/portage/olddist/$file";
    }
  }
}

[raptor]

As I used this script I found one big problem atleast for me...
I usualy use several gentoo machines and from time i'm rsync-ing their distfiles... Now the problem is that after I ran the modified-robm script all files that do not belong to packages i have installed are deleted/moved which is BAD.
The whole purpose for rsync-ing is that i do not need to dload files which are already dloaded on other machines...

Is there a way to preserve files that do not belong to already installed packges...!?!

i will apricate any help...

tia

[wolf31o2]

Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.

Now, if only it did PORT_OVERLAY also and was in python, it would be perfect. In fact, it would be good enough that I think a bug would definitely need to be opened for the portage team to include it in a future version of portage.

[far]

Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.

Now, if only it did PORT_OVERLAY also and was in python, it would be perfect.

We aim for perfection

Code: Select all

#!/usr/bin/env python

import os, os.path, portage

env = portage.config().environ()
distdir = env['DISTDIR']
portdir = env['PORTDIR']
overlay = env.get('PORTDIR_OVERLAY', '')
distfiles = portage.listdir(distdir)
keep = []

def visit(spam, dir, files):
    for file in files:
        if 'digest-' in file:
            path = os.path.join(dir, file)
            if not os.path.isfile(path): continue
            for line in open(path).readlines():
                try: keep.append(line.split()[2])
                except: pass

os.path.walk(portdir, visit, None)
if overlay: os.path.walk(overlay, visit, None)
remove = []
for file in distfiles:
    if file not in keep: remove.append(file)
for file in remove:
    file = os.path.join(distdir, file)
    if os.path.isfile(file):
        print file

Note: it only prints a list of files that you may remove. It is pretty slow.

[wolf31o2]

Sweet... what would I have to change on the "print file" line to make it actually remove them?

Sorry, I know nothing of python.

[far]

Sweet... what would I have to change on the "print file" line to make it actually remove them?

Change it to "os.remove(file)"

[TGL]

/me also likes this script, thanks. I took the permission to modify it a bit so that it uses the /var/cache/edb/dep files instead of the digests, which is faster. But it should give the same result in general, allthough there are some theoretical cases where it would not (for instance if you add an ebuild in your overlay, and then manually add its distfiles, and then don't use the ebuild, and then launch the cleanup script before your next sync, then your distfiles won't be referenced in the cache... well, who cares about such bugs?).

Code: Select all

EDIT: removed the old code, see below for an updated version.

Also, I think you had a bug in your version in case of multiple overlays (they are allowed, at least in the .50_pre serie), but this would be easy to fix (for ov in string.split(overlay): ...).

[TGL]

Added some colors to give it the Gentoo's touch. For cron jobs, this ugliness can be disable with the "-q" option. Also, if all you want is the raw list of files to delete (to pipe it to something else for instance), use "-p -q". Otherwise, a simple "-p" first to check that everything is ok, and then no option to actually clean the files.

Code: Select all

EDIT: added --ask (-a) for per-file confirmation. Default is still to blindly delete everything.
EDIT 2: removed old code. You can now find the script in bug #33877... no need to make this thread longer than it already is.

[gebner]

Code: Select all

bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|")

Mine does practically the same, but is 5 secs faster :

Code: Select all

(eval export `emerge info | egrep 'PORTDIR|DISTDIR'`; find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' | xargs -n10 cut -d' ' -f3 | perl -ne 'chomp;$f{$_}=1;END{$d=$ENV{DISTDIR};opendir$h,$d; while($f=readdir$h){print"$d/$f\n"if!$f{$f}&&!-d"$d/$f"}}')

This shows only the files that are no longer necessary, if you want to actually delete them you'd have to append a '| xargs rm'.

Or in a bit more readable fashion (as I have it in /usr/local/bin/clean-distfiles):

Code: Select all

#!/bin/sh
eval export `emerge info | egrep 'PORTDIR|DISTDIR'`

find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' \
	| xargs -n10 cut -d' ' -f3 \
	| perl -ne 'chomp;
		$f{$_} = 1;
		END{
			$d = $ENV{DISTDIR};
			opendir $h, $d;
			while($f = readdir $h){
				print "$d/$f\n" if not $f{$f} and not -d "$d/$f";
			}
		}' \
	| if [ "$1" = "-d" ]; then
		xargs rm -v
	else
		cat
	fi

---
ChangeLog:

• 1.1 added pretty-printed version and corrected old one not to delete directories
• 1.0 initial post

[TGL]

Not bad, but still 8 times slower than mine :p

[wolf31o2]

TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit?

[TGL]

TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit?

I first thought i should ask karltk if he would be interrested, and then forgot about it. Thanks for the reminder, goto #gentoo-dev.