View previous topic :: View next topic |
Author |
Message |
alexbuell Guru
Joined: 18 Jul 2002 Posts: 490 Location: "Hemp"shire, UK
|
Posted: Sun Jul 27, 2003 6:40 pm Post subject: |
|
|
RoadRunner wrote: |
It's just amazing how almost on every forum i'm in, there's a While E. Coyote nick =)
|
Beep beep! _________________ Cheers,
Alex.
Linux - the best text adventure game ever. |
|
Back to top |
|
|
ghwomb n00b
Joined: 04 Aug 2003 Posts: 1 Location: Kumla, Sweden
|
Posted: Mon Aug 04, 2003 4:33 pm Post subject: |
|
|
Just posted so I would get a mail if anything happened. Don't seem to be much activity here though. Perhaps the scripts is being tested the Debian way; lots of testing on uncommon systems and no releases. |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Mon Aug 04, 2003 4:43 pm Post subject: |
|
|
Eh, you could just click "Watch this topic for replies" in the lower left corner of the page.
A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball. _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
erik_swanson Retired Dev
Joined: 02 Feb 2003 Posts: 123 Location: Corvallis, OR USA
|
Posted: Mon Aug 04, 2003 6:38 pm Post subject: |
|
|
far wrote: | A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball. |
Hence the reason I believe that it's better to only remove distfiles that are truly dead (no longer used by anything in portage).
This is what my one-liner does, which I'm reposting because I've improved it to auto-detect your portdir/distdir/overlays using 'emerge info.'
Code: | bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|") |
|
|
Back to top |
|
|
patrickfo Tux's lil' helper
Joined: 30 Jun 2002 Posts: 79 Location: France
|
Posted: Mon Aug 04, 2003 11:18 pm Post subject: if this can help just two bash scripts that i use ... |
|
|
i have 3 adsl gateway ( ipsec ). behind each there is two or more pc runing gentoo.
i choose to keep the sources files for each domain on a samba directory which is mounted at boot for every station.
just for the fun:
on the samba server in smb.conf :
Code: | ...
[distfiles]
comment = no comment ;)
path = /my_sources_files_dir
valid users = root
read only = no
create mask = 0700
browseable = no
|
on each station:
a file ( smcred ) somewhere in etc:
Code: | username = root
password = afunnypassword |
Code: | smbpassword -a root |
you enter here your password (not necessary to be the same as root login!)
for security reasons:
next i put in /etc/fstab
Code: | //servername/distfiles /mnt/distfiles smbfs credentials=/etc/smcred 0 0 |
then in /etc/make.conf on each station
Code: | DISTDIR=/mnt/distfiles |
needless to say it is better to cp your files from your original distfile dir on the samba server BEFORE running unused_sf.sh, because missing files will be downloaded!!!
and now the scripts:
unused_sf.sh : create a list of sources files needed by each station in a directory named .cleandf/stationname/ which i put on the samba server in the distdir. ( i have modified the script that it can handle a single station with no distdir defined and tesded it just only once so be carefull!!!)
this script show also the packages that are installed and no mode in the portage tree, and download also files that are needed by installed packages and are missing ( i have found some patches for xfree more recent than te ones used for the original build on one or two stations and some other rewrited ( i suppose ) between the time of my compil and the time i run the script exemple: foomatic 3.0 )
expunge_sf.sh : list all the files and show the ones that are no more needed by any station and, file by file you can move them in a dir that you can later delete.
i am not a bash specialist, so be carefull! and also these scripts are very intensive time and resources consuming... and now the scripts
----------------
Code: | #!/bin/bash
# unusedsf: Find unused sources files in DISTDIR
### erreurs possibles
RPT_IS_NOT_A_DIR=65
UNKNOWN_OPTION=66
### mes variables
# dir in which we (euh... I ) put the files...
RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2 )
if [ ${#RPT} != 0 ]; then
RPT="$RPT/.cleandf/$HOSTNAME"
else
RPT="$HOME/.cleandf/$HOSTNAME"
fi
### gestion des options...
while getopts :d: OPTION ;
do
case $OPTION in
d)
RPT=$OPTARG
if [[ -d $RPT ]]; then
echo ">>> dir of files produced changed to $RPT!"
fi
;;
*)
echo -e "\n!!! option $OPTARG unknown!\n"
echo ">>> usage: $0 [-d <dir>]"
echo -e " -d directory\n"
exit $UNKNOWN_OPTION
;;
esac;
done
# creating DISTDIR/.cleandf/HOSTNAME if not exist
if [[ -e $RPT ]]; then
if [[ -d $RPT ]]; then
echo "$RPT found."
else
echo "$RPT must be a directory!"
exit $RPT_IS_NOT_A_DIR
fi
else
mkdir -p $RPT
echo "$RPT created."
fi
### do some cleaning...
for FICH in emerge_list qpkg qpkgV ;
do
if [[ -f $RPT/$FICH.txt ]]; then
rm -f $RPT/$FICH.txt
fi;
done
### création des fichiers de travail
echo -en "doing \"qpkg -nc -I\" ...\t\t"
qpkg -nc -I | sort > $RPT/qpkg.txt
echo "ok."
echo -en "doing \"qpkg -nc -I -v\" ...\t"
qpkg -nc -I -v | sort > $RPT/qpkgV.txt
echo "ok."
echo
for PACKAGE in $( cat $RPT/qpkg.txt );
do
for VERSION in $( grep $PACKAGE- $RPT/qpkgV.txt | cut -d '/' -f2 );
do
echo -e ">>> $PACKAGE/$VERSION"
emerge --nospinner -f $PACKAGE/$VERSION.ebuild > $RPT/.temp
if [ $? != 0 ]; then
#packages not in portage tree, out of sync ( fixpackages!! )
echo ">>> $PACKAGE/$VERSION output:" >> $RPT/ERREURS.txt
cat $RPT/.temp >> $RPT/ERREURS.txt
echo "============" >> $RPT/ERREURS.txt
else
#package that have the .ebuild file modif ( xfree-4.3. !!... )
grep -i download $RPT/.temp 2>&1 >/dev/null
if [ $? = 0 ]; then
echo ">>> $PACKAGE/$VERSION should be rebuild:" >> $RPT/to_REBUILD.txt
grep -i download $RPT/.temp >> $RPT/to_REBUILD.txt
echo "============" >> $RPT/to_REBUILD.txt
fi
fi;
cat $RPT/.temp >> $RPT/emerge_list.txt
done ;
done
# reporting problems
clear
if [ -e $RPT/to_REBUILD.txt ]; then
echo "Packages that should be rebuild:"
cat $RPT/to_REBUILD.txt
echo
else
echo "Well, nothing to rebuild... ;-)"
fi
if [ -e $RPT/ERREURS.txt ]; then
echo "Actions must be done on:"
cat $RPT/ERREURS.txt
echo
else
echo "Very well, no errors found... ;-)"
fi
| ================
Code: | #!/bin/bash
# expunge_sf.sh: remove unecessaries files from DISTDIR
NOTHING_TO_DO=65
SAUVE_NOT_A_DIR=66
RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2)
SAUVE="/sauve/old_distfiles"
if [ ${#RPT} != 0 ]; then
DISTDIR=$RPT
RPT="$RPT/.cleandf"
else
RPT="$HOME/.cleandf"
DISTDIR="/usr/portage/distfiles"
fi
### gestion des options...
while getopts :d:s: OPTION ;
do
case $OPTION in
d)
RPT=$OPTARG
;;
s)
SAUVE=$OPTARG
;;
*)
echo -e "\n!!! option $OPTARG unknown!\n"
echo ">>> usage: $0 [-d <dir>] -s <dir>"
echo -e " -d directory to put temporary files\n"
echo -e " -s directory to store backups\n"
exit $UNKNOWN_OPTION
;;
esac;
done
if [ -e $RPT ]; then
echo "$RPT found."
else
echo "You must run unused_sf BEFORE expunge_sf!!!"
exit $NOTHING_TO_DO
fi
if [[ ( -e $SAUVE ) && ( -d $SAUVE ) ]]; then
echo "$SAUVE found."
else
echo "You must supply a dir for backup files!!!"
exit $SAUVE_NOT_A_DIR
fi
# some cleaning from previous use...
if [ -f $RPT/fichiers_inutiles.txt ]; then
rm -rf $RPT/fichiers_inutiles.txt
fi
if [ -f $RPT/tous_fichiers.txt ]; then
rm -rf $RPT/tous_fichiers.txt
fi;
# list of files in DISTDIR
station=$(ls $RPT)
for FICH in $(ls $DISTDIR --ignore="cvs-src") ;
do
echo $FICH
used=0
for ordi in $station;
do
grep $FICH $RPT/$ordi/emerge_list.txt >/dev/null 2>&1
result=$?
if [ $result = 0 ]; then
used=1
fi;
done
if [ $used -eq 1 ]; then
echo " $FICH" >> $RPT/tous_fichiers.txt
else
echo $FICH >> $RPT/fichiers_inutiles.txt
echo "dead $FICH" >> $RPT/tous_fichiers.txt
fi;
done
if [[ !( -e $RPT/fichiers_inutiles.txt) ]]; then
echo "no files to move :( "
exit 0
fi
resultat="hit control-C to break"
for FICH in $(cat $RPT/fichiers_inutiles.txt);
do
clear
echo -e "$resultat\n"
grep -A 15 -B3 --colour $FICH $RPT/tous_fichiers.txt
echo -e "\n$FICH"
echo -en "move $FICH to $SAUVE/: ? (y/n): \t"
read reponse
if [[ $reponse = "y" || $reponse = "Y" ]]; then
mv $DISTDIR/$FICH $SAUVE/
resultat="$FICH moved"
else
resultat="$FICH remain"
fi;
done
| ============
good luck with them
any comment to ameliorate will be welcome, and injury must be directed to /dev/null ))
Patrick
PS: the indent is gone away... try to recreat it or say me and i will send you this files...
Last edited by patrickfo on Fri Aug 08, 2003 9:54 pm; edited 2 times in total |
|
Back to top |
|
|
esprout n00b
Joined: 09 Jan 2003 Posts: 22
|
Posted: Fri Aug 08, 2003 7:21 am Post subject: Script for interactively processing PKGDIR |
|
|
Ok, I figure I'll throw my script into the foray.
This one interactively processes the PKGDIR directory for old versions. Just run it without any arguments, and it will prompt you before deleting each file.
It'd be nice to have one script to process both PKGDIR and DISTDIR. Or maybe one of them already do, I missed it? So many scripts now.
I stole a couple lines/ideas from previous scripts, if something looks familiar
Code: |
#!/usr/bin/perl
# Root check
if ($< != 0) {
print "You must be root to run this script.\n";
exit 0;
}
my $DISTDIR;
my $PKGDIR;
# Figure out Portage Directories
my @emergeinfo = `emerge info`;
foreach my $info (@emergeinfo) {
if ($info =~ m/^DISTDIR=/) {
($DISTDIR=$info) =~ s/^DISTDIR="(.*)"\n$/$1/;
} elsif ($info =~ m/^PKGDIR=/) {
($PKGDIR=$info) =~ s/^PKGDIR="(.*)"\n$/$1/;
}
}
print "Processing files in $PKGDIR...\n";
# Process files in $PKGDIR
chdir("$PKGDIR");
my $old;
my $new;
my $candidate;
# Generate list of .tbz2 files to process
@FILES = `find . -name \*.tbz2 | sort`;
# Process list of files
foreach (@FILES) {
# Only process symlinks to reduce user interaction
next if ($_ =~ m|^./All/|);
$_ =~ s|\n$||;
$new = $_;
$new =~ s|(\./.+/.+)-[\d]+.*|$1|;
#print "new: $_\n";
if ($old) {
# new files might be same as old file
if ("$old" eq "$new") {
my $input;
$candidate =~ s|.tbz2$||;
# strip .tbz2 extension and use perl sort
# to do a better guess of which files is older
$_ =~ s|.tbz2$||;
($candidate, $_) = sort($candidate, $_);
# reappend .tbz2 extension
$candidate .= ".tbz2";
$_ .= ".tbz2";
#print "dup: $old, $new, $candidate\n";
# Figure out corresponding file in All directory
(my $allcandidate = $candidate) =~ s|^./.*/|./All/|;
# Ask user to confirm deletion
print "rm: $allcandidate and\n $candidate\nkeep: $_ [y(es)/n(o)/s(kip)]? : ";
($input = (<STDIN>)) =~ s|\n$||;
# Delete away
if ("$input" eq "y") {
unlink($candidate, $allcandidate);
print "$candidate, $allcandidate deleted\n";
$old = $new;
$candidate = $_;
# Ask user for swapped deletion
} elsif ("$input" eq "n") {
(my $allcandidate = $_) =~ s|^./.*/|./All/|;
print "rm: $allcandidate and\n $_\nkeep: $candidate [y(es)/n(o)]? : ";
($input = (<STDIN>)) =~ s|\n$||;
if ("$input" eq "y") {
unlink($_, $allcandidate);
print "$_, $allcandidate delete\n";
} else {
$old = $new;
$candidate = $_;
}
}
} else {
#print "nodup: $old, $new, $candidate\n";
$old = $new;
$candidate = $_;
}
} else {
#print "first round: $old, $new, $candidate\n";
$old = $new;
$candidate = $_;
}
}
|
Last edited by esprout on Wed Aug 13, 2003 4:54 am; edited 1 time in total |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Fri Aug 08, 2003 5:39 pm Post subject: |
|
|
Please put your source code inside BBCode [code] blocks like so:
[code]10 PRINT "Hello, World!"[/code]
If you don't, you will lose the indentation and the code will be difficult to read. _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
patrickfo Tux's lil' helper
Joined: 30 Jun 2002 Posts: 79 Location: France
|
Posted: Mon Aug 11, 2003 6:00 am Post subject: thank you |
|
|
far wrote: | Please put your source code inside BBCode [code] blocks like so:
[code]10 PRINT "Hello, World!"[/code]
If you don't, you will lose the indentation and the code will be difficult to read. |
thank you
i have corrected my post |
|
Back to top |
|
|
raptor Apprentice
Joined: 20 Sep 2002 Posts: 171
|
Posted: Wed Aug 20, 2003 1:43 pm Post subject: Size matters :") |
|
|
small chages to original to show the old files size and total size that will be saved after stuff are deleted ...
example......
Would delete: ttmkfdir-3.0.4 [19 kB] in favour of ttmkfdir-3.0.9
Would delete: usbutils-0.11 [216 kB] in favour of usbutils-0.9
Would delete: vte-0.10.25 [759 kB] in favour of vte-0.10.26
Would delete: xscreensaver-4.05 [3133 kB] in favour of xscreensaver-4.06
Would delete: yelp-1.0.6 [478 kB] in favour of yelp-2.2.0
Would delete: zapping-0.6.4 [1058 kB] in favour of zapping-0.6.5
Would delete: zvbi-0.2.1 [486 kB] in favour of zvbi-0.2.3
Total size that will be reclaimed : 450038 kB
wow 450 MB not bad :")
===================================
#!/usr/bin/perl
use strict;
my $lastname = 0;
my $lastversion;
my $lastext;
my $totalSize;
my $lvSize;
my @stalefiles;
my @files;
........
if ($lastname && $name eq $lastname) {
if ($version gt $lastversion) {
$lvSize = (stat ("/usr/portage/distfiles/$name$version$lastext"))[7];
push (@stalefiles, [$name, $version, $lastversion, $lastext,$lvSize]);
}
}
......................
..........................
else
{
# Safe mode (default)!
print "\7*Pretend* mode, will only pretend to delete files.\nTo actually delete the files, reinvoke with the --nopretend parameter.\n\n";
foreach (@stalefiles) {
my ($name, $version, $lastversion, $ext, $lvSize) = @{$_};
$totalSize += $lvSize;
$lvSize = sprintf "%.2u kB", $lvSize/1024;
print "Would delete: $name$lastversion [$lvSize] in favour of $name$version\n";
}
print "Total size that will be reclaimed : " . (sprintf "%.2u kB", $totalSize/1024) . "\n";
}
1; |
|
Back to top |
|
|
SavageMindz Tux's lil' helper
Joined: 28 Oct 2002 Posts: 87 Location: The bit of hell that has frozen over.
|
Posted: Sun Nov 23, 2003 7:01 am Post subject: |
|
|
erik.swanson wrote: | This is a "one-liner" I wrote to safely clean out obsolete distfiles that are no longer referenced by any ebuild in portage. |
nice one liner, just cleared me out about 4gb or old distfiles, that was quite sorely needed . Modified your command so I could get a list of what it got rid of and it seems to have worked a charm.
cheers
Savage |
|
Back to top |
|
|
cbradney Apprentice
Joined: 08 May 2003 Posts: 245 Location: Luxembourg
|
Posted: Tue Dec 16, 2003 6:09 pm Post subject: |
|
|
Got this one from Kerframil:
Code: |
#!/usr/bin/env python
## distclean.py version 0.1 (20 Aug 2003,
## This is the first version with a version number)
##
## Removes source files for Gentoo
## packages that are no longer installed
## Use with '-p' (pretend) flag to just get a list of files
## that would be removed
##
## Copyright (c) 2003, Fredrik Arnerup (e97_far@e.kth.se)
## All rights reserved.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted provided that the following conditions are met:
##
## * Redistributions of source code must retain the above copyright notice,
## this list of conditions and the following disclaimer.
##
## * Redistributions in binary form must reproduce the above copyright
## notice, this list of conditions and the following disclaimer in the
## documentation and/or other materials provided with the distribution.
##
## THIS SOFTWARE IS PROVIDED BY FREDRIK ARNERUP "AS IS" AND ANY
## EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
## WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
## DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
## FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
## DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
## SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
## CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
## LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
## OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
## DAMAGE.
import sys, os, os.path, getopt, portage
opt_p = 0
try:
if len(sys.argv) > 1:
opt_p = getopt.getopt(sys.argv[1:], 'p')[0][0][0] == '-p'
except getopt.GetoptError:
pass
distdir = portage.config().environ()['DISTDIR']
print 'DISTDIR =', distdir
vartree = portage.db['/']['vartree']
packages = []
for name in vartree.getallnodes():
packages.extend(vartree.dep_match(name))
files = {}
for package in packages:
try:
package_files = portage.portdb.aux_get(package, ['SRC_URI'])[0].split()
package_files = [(url.split('/')[-1]) for url in package_files]
for filename in package_files:
files[filename] = 1
except:
print 'Failed to get file list for', package
if not files:
sys.exit("No package files found. This can't be right.\n")
try:
list = portage.listdir(distdir)
except os.OSError:
sys.exit('Failed to read ' + distdir)
size = 0; count = 0
for file in list:
abs_file = distdir + '/' + file
if (os.path.isfile(abs_file) and (not os.path.islink(abs_file))
and (not file in files)):
size += os.stat(abs_file).st_size
count += 1
if opt_p:
print 'Would remove', abs_file
else:
try:
os.remove(abs_file)
print 'Removed', abs_file
except OSError:
print 'Failed to remove', abs_file
size /= 1048576 ## MB
print '%i files, total size: %i MB' % (count, size)
|
|
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
|
Back to top |
|
|
cbradney Apprentice
Joined: 08 May 2003 Posts: 245 Location: Luxembourg
|
Posted: Tue Dec 16, 2003 7:37 pm Post subject: |
|
|
Works wonderfully! Thank you! |
|
Back to top |
|
|
raptor Apprentice
Joined: 20 Sep 2002 Posts: 171
|
Posted: Sun Jan 11, 2004 1:36 pm Post subject: Re: mod for PORTAGE_OVERLAY, etc. |
|
|
thanx for the scripts "robm" and others.. a small modification at the end of the script :
Now it will not delete the files but will move them in /usr/portage/olddist. This way we can restore if something goes wrong, and delete only when nececary....
here are the results :
Code: |
$du -shc /usr/portage/distfiles/
2.6G /usr/portage/distfiles
2.6G total
$ ./distbuster
.....
$ du -shc /usr/portage/distfiles/
599M /usr/portage/distfiles
599M total
$ du -shc /usr/portage/olddist
2.0G /usr/portage/olddist
2.0G total
|
not bad...
Code: |
opendir(DIR,$makeconf{"DISTDIR"})|| die "can't open ".$makeconf{"DISTDIR"};
#new line
mkdir "/usr/portage/olddist";
while ($file = readdir DIR) {
next unless -f $makeconf{"DISTDIR"}."/$file";
if (! $files{$file}) {
if ($opt_p) {
# print "Would erase ".$makeconf{"DISTDIR"}."/$file\n";
print "Would move $makeconf{DISTDIR}/$file -> /usr/portage/olddist \n";
}
else {
# unlink $makeconf{"DISTDIR"}."/$file";
rename "$makeconf{DISTDIR}/$file", "/usr/portage/olddist/$file";
}
}
}
|
|
|
Back to top |
|
|
raptor Apprentice
Joined: 20 Sep 2002 Posts: 171
|
Posted: Sun Jan 11, 2004 1:47 pm Post subject: not so good |
|
|
As I used this script I found one big problem atleast for me...
I usualy use several gentoo machines and from time i'm rsync-ing their distfiles... Now the problem is that after I ran the modified-robm script all files that do not belong to packages i have installed are deleted/moved which is BAD.
The whole purpose for rsync-ing is that i do not need to dload files which are already dloaded on other machines...
Is there a way to preserve files that do not belong to already installed packges...!?!
i will apricate any help...
tia |
|
Back to top |
|
|
wolf31o2 Retired Dev
Joined: 31 Jan 2003 Posts: 628 Location: Mountain View, CA
|
Posted: Mon Jan 26, 2004 6:04 pm Post subject: |
|
|
Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.
Now, if only it did PORT_OVERLAY also and was in python, it would be perfect. In fact, it would be good enough that I think a bug would definitely need to be opened for the portage team to include it in a future version of portage. _________________ Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Mon Jan 26, 2004 8:20 pm Post subject: |
|
|
wolf31o2 wrote: | Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.
Now, if only it did PORT_OVERLAY also and was in python, it would be perfect. |
We aim for perfection
Code: | #!/usr/bin/env python
import os, os.path, portage
env = portage.config().environ()
distdir = env['DISTDIR']
portdir = env['PORTDIR']
overlay = env.get('PORTDIR_OVERLAY', '')
distfiles = portage.listdir(distdir)
keep = []
def visit(spam, dir, files):
for file in files:
if 'digest-' in file:
path = os.path.join(dir, file)
if not os.path.isfile(path): continue
for line in open(path).readlines():
try: keep.append(line.split()[2])
except: pass
os.path.walk(portdir, visit, None)
if overlay: os.path.walk(overlay, visit, None)
remove = []
for file in distfiles:
if file not in keep: remove.append(file)
for file in remove:
file = os.path.join(distdir, file)
if os.path.isfile(file):
print file |
Note: it only prints a list of files that you may remove. It is pretty slow. _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
wolf31o2 Retired Dev
Joined: 31 Jan 2003 Posts: 628 Location: Mountain View, CA
|
Posted: Mon Jan 26, 2004 9:25 pm Post subject: |
|
|
Sweet... what would I have to change on the "print file" line to make it actually remove them?
Sorry, I know nothing of python. _________________ Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Mon Jan 26, 2004 9:37 pm Post subject: |
|
|
wolf31o2 wrote: | Sweet... what would I have to change on the "print file" line to make it actually remove them? |
Change it to "os.remove(file)" _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
TGL Bodhisattva
Joined: 02 Jun 2002 Posts: 1978 Location: Rennes, France
|
Posted: Tue Jan 27, 2004 12:02 am Post subject: |
|
|
/me also likes this script, thanks. I took the permission to modify it a bit so that it uses the /var/cache/edb/dep files instead of the digests, which is faster. But it should give the same result in general, allthough there are some theoretical cases where it would not (for instance if you add an ebuild in your overlay, and then manually add its distfiles, and then don't use the ebuild, and then launch the cleanup script before your next sync, then your distfiles won't be referenced in the cache... well, who cares about such bugs?). Code: | EDIT: removed the old code, see below for an updated version. |
Also, I think you had a bug in your version in case of multiple overlays (they are allowed, at least in the .50_pre serie), but this would be easy to fix (for ov in string.split(overlay): ...).
Last edited by TGL on Wed Jan 28, 2004 9:00 pm; edited 1 time in total |
|
Back to top |
|
|
TGL Bodhisattva
Joined: 02 Jun 2002 Posts: 1978 Location: Rennes, France
|
Posted: Wed Jan 28, 2004 8:58 pm Post subject: |
|
|
Added some colors to give it the Gentoo's touch. For cron jobs, this ugliness can be disable with the "-q" option. Also, if all you want is the raw list of files to delete (to pipe it to something else for instance), use "-p -q". Otherwise, a simple "-p" first to check that everything is ok, and then no option to actually clean the files.
EDIT: added --ask (-a) for per-file confirmation. Default is still to blindly delete everything.
EDIT 2: removed old code. You can now find the script in bug #33877... no need to make this thread longer than it already is.
Last edited by TGL on Mon Feb 09, 2004 11:22 pm; edited 2 times in total |
|
Back to top |
|
|
gebner Tux's lil' helper
Joined: 01 Jan 2004 Posts: 100 Location: Wien, EU
|
Posted: Thu Feb 05, 2004 2:14 pm Post subject: |
|
|
erik.swanson wrote: | Code: | bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|") |
|
Mine does practically the same, but is 5 secs faster :
Code: | (eval export `emerge info | egrep 'PORTDIR|DISTDIR'`; find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' | xargs -n10 cut -d' ' -f3 | perl -ne 'chomp;$f{$_}=1;END{$d=$ENV{DISTDIR};opendir$h,$d; while($f=readdir$h){print"$d/$f\n"if!$f{$f}&&!-d"$d/$f"}}') |
This shows only the files that are no longer necessary, if you want to actually delete them you'd have to append a '| xargs rm'.
Or in a bit more readable fashion (as I have it in /usr/local/bin/clean-distfiles):
Code: | #!/bin/sh
eval export `emerge info | egrep 'PORTDIR|DISTDIR'`
find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' \
| xargs -n10 cut -d' ' -f3 \
| perl -ne 'chomp;
$f{$_} = 1;
END{
$d = $ENV{DISTDIR};
opendir $h, $d;
while($f = readdir $h){
print "$d/$f\n" if not $f{$f} and not -d "$d/$f";
}
}' \
| if [ "$1" = "-d" ]; then
xargs rm -v
else
cat
fi |
---
ChangeLog:
- 1.1 added pretty-printed version and corrected old one not to delete directories
- 1.0 initial post
_________________ clue = if 0 == 0 then clue else error "The poster didn't have any clue when he posted this." |
|
Back to top |
|
|
TGL Bodhisattva
Joined: 02 Jun 2002 Posts: 1978 Location: Rennes, France
|
Posted: Fri Feb 06, 2004 1:13 am Post subject: |
|
|
Not bad, but still 8 times slower than mine :p |
|
Back to top |
|
|
wolf31o2 Retired Dev
Joined: 31 Jan 2003 Posts: 628 Location: Mountain View, CA
|
Posted: Sun Feb 08, 2004 7:51 pm Post subject: |
|
|
TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit? _________________ Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org |
|
Back to top |
|
|
TGL Bodhisattva
Joined: 02 Jun 2002 Posts: 1978 Location: Rennes, France
|
Posted: Sun Feb 08, 2004 8:01 pm Post subject: |
|
|
wolf31o2 wrote: | TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit? |
I first thought i should ask karltk if he would be interrested, and then forgot about it. Thanks for the reminder, goto #gentoo-dev. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|