View previous topic :: View next topic |
Author |
Message |
eeknay Guru
Joined: 07 Jul 2003 Posts: 402 Location: EndOfTheRainbow
|
Posted: Sun Mar 21, 2004 9:42 pm Post subject: |
|
|
thanks for the help. _________________ Linda: "The holiday season is time of celebration for most but it is also the time to remember the tragic suffering of the less fortunate."
Morbo: "Earthlings do not yet know the meaning of suffering." |
|
Back to top |
|
|
Larcen Apprentice
Joined: 21 Mar 2004 Posts: 174
|
Posted: Sun Mar 21, 2004 11:55 pm Post subject: |
|
|
Hello, I was using distclean-0.1.py and I continuously got this error:
aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)
many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory? |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Mon Mar 22, 2004 12:02 am Post subject: |
|
|
Larcen wrote: | Hello, I was using distclean-0.1.py and I continuously got this error:
aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)
many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory? |
Probably means what it says, that there is something wrong with that ebuild.
It shouldn't have any impact on what distclean does.
The portage module is annoying in that way. Some errors it reports directly to stderr, instead of throwing exceptions.
The error message will probably go away after an emerge sync. _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
Larcen Apprentice
Joined: 21 Mar 2004 Posts: 174
|
Posted: Mon Mar 22, 2004 12:09 am Post subject: |
|
|
Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over? |
|
Back to top |
|
|
far Guru
Joined: 10 Mar 2003 Posts: 394 Location: Stockholm, Sweden
|
Posted: Mon Mar 22, 2004 12:18 am Post subject: |
|
|
Larcen wrote: | Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over? |
Wipe out distfiles? Yes, that's probably harmless, if you don't mind maybe having to download stuff again.
Does this error cause distclean to abort? _________________ The Porthole Portage Frontend |
|
Back to top |
|
|
Larcen Apprentice
Joined: 21 Mar 2004 Posts: 174
|
Posted: Mon Mar 22, 2004 12:35 am Post subject: |
|
|
Mind downloading what I emerge? No I don't. Does it cause it to abort? No, it continues right along. |
|
Back to top |
|
|
juam n00b
Joined: 01 Jan 2004 Posts: 3
|
Posted: Sat Apr 10, 2004 9:47 pm Post subject: |
|
|
Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/
For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).
New policy are welcome. |
|
Back to top |
|
|
Jowilly Guru
Joined: 22 Jun 2002 Posts: 319
|
Posted: Sat May 08, 2004 8:03 pm Post subject: |
|
|
juam wrote: | Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/
For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).
New policy are welcome. |
Excellent !
I've put it in /etc/conf.d/local.start |
|
Back to top |
|
|
kaidon n00b
Joined: 01 Nov 2003 Posts: 72
|
Posted: Sat Jun 05, 2004 8:58 pm Post subject: |
|
|
@juam
wonderfull
you just gave me back 1.6 gigs
thanx for that |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Tue Jul 06, 2004 11:10 pm Post subject: |
|
|
juam wrote: | Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/
For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).
New policy are welcome. | [3732]> Code: | distcleaner --pretend --action=not_installed
Traceback (most recent call last):
File "/opt/portage/bin/distcleaner", line 294, in ?
main()
File "/opt/portage/bin/distcleaner", line 289, in main
d.perform()
File "/opt/portage/bin/distcleaner", line 116, in perform
self.select_files_to_delete(hash)
File "/opt/portage/bin/distcleaner", line 192, in select_files_to_delete
hash.pop(file,0)
AttributeError: 'dict' object has no attribute 'pop' |
I don't know what happened there. gentoolkit-0.2.0_pre8 is installed |
|
Back to top |
|
|
carpman Advocate
Joined: 20 Jun 2002 Posts: 2202 Location: London - UK
|
Posted: Tue Aug 17, 2004 11:27 am Post subject: |
|
|
Hello, been using this script to keep distfile dir clean but have now moved over to http-replicator for network distfile download cache.
What this means is that the local distfiles dir contents can be cleaned out after emerging is done but on the http-replicator server all distfiles are moved to http-replicator dir, i tried editing the script so it pointed to /var/cache/http-replicator but the script did not work, there was no errors but no output even though i know there was files that could have been deleted.
How can i get this script to work with http-replicator?
cheers _________________ Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
Back to top |
|
|
gindaani n00b
Joined: 28 Apr 2003 Posts: 15
|
Posted: Fri Aug 27, 2004 8:32 pm Post subject: |
|
|
What about the files in /var/tmp/portage?
I have a directory in there for every ebuild that has ever been run. Some of the directories only contain one file, temp/successful.
Is it safe to wipe all of these out or do we need something to selectively delete them?
Thanks,
Dan |
|
Back to top |
|
|
wolf31o2 Retired Dev
Joined: 31 Jan 2003 Posts: 628 Location: Mountain View, CA
|
Posted: Fri Aug 27, 2004 9:00 pm Post subject: |
|
|
You can delete them safely. They are simply temporary files created by portage during an emerge. The reason they're around, is they also provide valuable information to portage during each stage of the install of a package. It won't give you much room, but it will free up a good number of inodes. _________________ Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org |
|
Back to top |
|
|
asph l33t
Joined: 25 Aug 2003 Posts: 741 Location: Barcelona, Spain
|
Posted: Sat Aug 28, 2004 6:07 pm Post subject: |
|
|
does this script (the one on the first post of this thread) also delete files not installed? for example, if i emerge -f xmms, but finally i don't install it, when i run the script will those files i downloaded get deleted? they should
also check this little curious 'bug'
Code: | Would delete: font-arial-iso-8859-1 in favour of font-arial-iso-8859-2 |
_________________ gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep |
|
Back to top |
|
|
Nightweaver n00b
Joined: 24 Jul 2002 Posts: 40
|
Posted: Mon Sep 27, 2004 11:48 pm Post subject: |
|
|
juam wrote: | Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/
For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).
New policy are welcome. |
I just upgraded to a newer version of gentoolkit (_pre9), and the 0.0.1 script no longer works
Any chance of what I hope would be a simple update? I run this script every time after I do an emerge -uD world, but now it reports every single file in /usr/portage/distfiles as being 'not installed'.
Thanks. |
|
Back to top |
|
|
houtworm Guru
Joined: 08 Mar 2003 Posts: 391 Location: Den Haag, Netherlands
|
Posted: Wed Oct 13, 2004 12:59 am Post subject: |
|
|
distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version? _________________ niemand is onbekwamer, dan een timmerman zonder hamer
Kees |
|
Back to top |
|
|
calle2003 n00b
Joined: 21 Oct 2004 Posts: 63
|
Posted: Thu Oct 21, 2004 8:36 pm Post subject: |
|
|
Hi all,
I found this thread because I'm not happy about old distfiles hanging around and wanted to delete them. I'm a bit concerned finding problems with the script with version numbers and so on, I'd rather have it delete exactly those files in DISTDIR that do not belong to an installed package. I think the approach should be to read the distfiles using the usual portage functionality and then delete all the rest. Unfortunately I couldn't find any possibilty to output the results of an ebuild's SRC_URI to the command line. Is there any?
Because of this, I did a small patch on portage.py (2.0.51);
Code: |
--- src-2.0.51/portage.py 2004-10-21 21:52:26.753228088 +0200
+++ portage.py 2004-10-21 19:46:07.496450256 +0200
@@ -2210,6 +2210,7 @@
validcommands = ["help","clean","prerm","postrm","preinst","postinst",
"config","setup","depend","fetch","digest",
+ "distfiles",
"unpack","compile","test","install","rpm","qmerge","merge",
"package","unmerge", "manifest"]
@@ -2494,6 +2495,10 @@
except:
pass
+ if mydo=="distfiles":
+ print string.join(checkme,"\n")
+ return 0
+
if not fetch(fetchme, mysettings, listonly=listonly, fetchonly=fetchonly):
return 1
|
This adds a "distfiles" command to ebuild so that "ebuild <file> distfiles" prints all the distfiles of that package. It works with the usual portage syntax.
Second, I created a script to show and delete the obsolete files (distclean):
Code: |
#!/bin/bash
# distclean utility
# Christian Schaefer
# License: public domain
# Use this script on your own risc. The author does
# not guarantue that doesn't destroy your system
# or is useful at all.
TMP1=tmp.distfiles.installed
TMP2=tmp.distfiles.obsolete
TMP3=tmp.distfiles.installed.tmp
DISTDIR=`portageq envvar DISTDIR`
function print_obsolete() {
$PAGER $TMP2;
}
function delete_obsolete() {
cat $TMP2 | xargs -r -l rm;
rm $TMP1 $TMP2;
}
function print_statistics() {
cat $TMP2 | xargs -r du --human-readable --total | tail -n 1;
}
function build_lists() {
# create list of distfiles for installed packages
# (distfiles patch to portage.py needed)
if [ ! -e "$TMP1" ]; then
rm $TMP2 >/dev/nul 2>&1;
qpkg -I -v --nocolor |
xargs -l printf "=%s\n" |
xargs -l equery which |
xargs -i ebuild {} distfiles |
sort | uniq > "$TMP3" ;
mv "$TMP3" "$TMP1";
fi;
# create list of distfiles present on disc
# but not belonging to installed packages
if [ ! -e "$TMP2" ]; then
find $DISTDIR -type f -printf "%f\n" |
sort | uniq |
comm -1 -3 "$TMP1" - |
xargs -r -l printf "$DISTDIR/%s\n" > $TMP2 ;
fi;
}
function print_usage() {
echo Utility to clean up the distfiles directory of portage
echo Files not belonging to installed packages are considered
echo obsolete and deleted.
echo
echo Usage: $0 option
echo
echo Options:
echo
echo "-p --pretend print a list of obsolete files"
echo "-s --size print size of obsolete files"
echo "--delete delete obsolete files"
}
case $1 in
( -p | --pretend )
build_lists;
print_obsolete;
;;
( --delete )
build_lists;
delete_obsolete;
;;
( -s | --size )
build_lists;
print_statistics;
;;
( * )
print_usage;
;;
esac
|
The script leaves two files in the current directory, if no delete is done:
tmp.distfiles.installed
tmp.distfiles.obsolete
They are reused on a second run (so you can delete after checking quite fast), but must be deleted manually when anything changes in the DISTDIR.
This is anything else but fast, in fact it takes several minutes to produce an output.
I would like to know your opinion on this approach. I have no knowledge of python and only very basic knowledge of shell scripts so I'm sure there are a lot of things that could be optimized. At least it works for me, an "emerge -ef world" after "distclean --delete" fetched nothing, but I've got 1.4 gigabytes of disk space back.
I do not recommend to use this script, it is just experimental!
Best regards,
Christian. |
|
Back to top |
|
|
abionescu n00b
Joined: 11 Nov 2004 Posts: 2
|
Posted: Thu Nov 11, 2004 4:33 pm Post subject: |
|
|
You can use the following script to delete source files not used in your system:
Quote: |
#!/bin/bash
echo "Building index file..."
rm index.file
echo "What software do you have installed?"
for installed in `find /var/db/pkg/ -name *.ebuild`
do
basename=`dirname $installed | xargs dirname | xargs basename`
filename=`dirname $installed | xargs basename`
for configs in `find "/usr/portage/"$basename -name "digest-"$filename -print`
do
echo "Adding..."$configs
cat $configs >> index.file 2>&1
done
done
echo "Comparing files..."
rm remove.file
for i in `ls /usr/portage/distfiles/*.*`
do
chksum_full=`md5sum $i`
chksum_new=${chksum_full:0:32}
no=`cat index.file | grep $chksum_new | wc -l`
if [[ $no == 0 ]]
then
echo "Remove"$'\t'$i
echo $i >> remove.file
else
echo "Keep"$'\t'$i
fi
done
echo "Cleaning up..."
rm index.file
|
which will create a list with files from /usr/portage/distfiles folder which has no reason to be there.
After you check the content of the file you can issue the command:
Quote: |
cat remove.file | xargs rm
|
With some tunning in "index" creation part you can remove only the obsolete files (not found into portage) |
|
Back to top |
|
|
languste n00b
Joined: 20 Dec 2004 Posts: 1
|
Posted: Mon Dec 20, 2004 1:05 am Post subject: |
|
|
use this script if you have problems with recent portage versions. it cleans your dist and package files.
Code: | #!/bin/bash
#
# filename: genclean
#
# Removes deprecated distfiles and packages of a gentoo system
#
# Tested with:
# - portage 2.0.51-r3
# - gentoolkit 0.2.0
#
# Shoud work with later versions as well
#
# v1.0, languste, Mon Dec 20 00:33:48 GMT 2004
# source needed vars
#
source_vars() {
eval export `emerge info | egrep 'PORTDIR|DISTDIR|PKGDIR'`
}
# clean distfiles
#
distfiles_clean() {
DIST_FILES=`find "$DISTDIR" -maxdepth 1 -type f -printf "%f\n" | sort`
DIGEST_FILES=`find "$PORTDIR" "$PORTDIR_OVERLAY" -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u`
OLD_DIST_FILES=`comm -23 <(echo "$DIST_FILES") <(echo "$DIGEST_FILES")`
remove_files "$DISTDIR" "$OLD_DIST_FILES"
}
# clean packages
#
packages_clean() {
QPKG_FILES=`qpkg -I -v -nc | cut -d "/" -f2 | sort | sed -e 's/$/\.tbz2/g' -e 's,^,All/,g'`
QPKG_LINKS=`qpkg -I -v -nc | sort | sed 's/$/\.tbz2/g'`
PKG_FILES=`cd "$PKGDIR" && find ./All/* | cut -d "/" -f2- | sort`
PKG_LINKS=`cd "$PKGDIR" && find . -type l | cut -d "/" -f2- | sort`
OLD_PKG_FILES=`comm -23 <(echo "$PKG_FILES") <(echo "$QPKG_FILES")`
OLD_PKG_LINKS=`comm -23 <(echo "$PKG_LINKS") <(echo "$QPKG_LINKS")`
remove_files "$PKGDIR" "$OLD_PKG_FILES $OLD_PKG_LINKS"
}
# remove files
#
remove_files() {
DIR="$1"
FILES="$2"
if [ "x${FILES}" = "x" -o "x${FILES}" = "x " ]; then
echo "no files to remove."
exit
fi
echo -e "These files (and links) in "$DIR" will be removed:\n"
echo "Size Filename"
echo "---- --------"
cd "${DIR}"
du -hc $FILES
echo -en "\nRemove these files? (Type YES to continue): "
read YES
[ "x$YES" = "xYES" ] || exit
echo "removing files..."
rm `echo "$FILES"`
}
# main
#
if [ "$1" = "distfiles" ]; then
source_vars
distfiles_clean
elif [ "$1" = "packages" ]; then
source_vars
packages_clean
else
echo -e "Usage: `basename $0` <distfiles|packages>\n"
echo "This little script will remove unused \"distfiles\" or \"packages\"."
echo "No files will be deleted whithout confirmation!"
exit
fi |
p.s. i'am a debian user playing with gentoo and i can't belive that there is no tool inside "gentoolkit" or wherever. handling this problem. in my 4 year old debian system there would be now terrabytes of old and deprecated stuff filling up my disk |
|
Back to top |
|
|
heikole Tux's lil' helper
Joined: 04 Oct 2004 Posts: 148 Location: Berlin, Germany
|
Posted: Thu Dec 23, 2004 2:04 pm Post subject: |
|
|
houtworm wrote: | distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version? |
Same problem for me. It seems that portage 2.0.51 is a little bit broken. distcleaner expects the call
Code: | src = pkg.get_env_var('SRC_URI') |
to return some value in src, but there isn't anything in it. pkg.get_env_var is defined in class Package in
/usr/lib/gentoolkit/pym/gentoolkit/gentoolkit.py
and called in distcleaner itself. Does somebody have any ideas how to fix this? |
|
Back to top |
|
|
skander n00b
Joined: 30 Dec 2004 Posts: 12
|
Posted: Tue Jan 04, 2005 8:03 pm Post subject: thanks robm! |
|
|
Thanks for the great script! |
|
Back to top |
|
|
fistfullast33l n00b
Joined: 26 May 2004 Posts: 28
|
Posted: Fri Jan 21, 2005 5:54 am Post subject: Delete Everything? |
|
|
Someone remind me why I can't just delete all the files in distfiles?
Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point. |
|
Back to top |
|
|
vonhelmet l33t
Joined: 06 Apr 2004 Posts: 770 Location: Somewhere in a school
|
Posted: Fri Jan 21, 2005 9:46 am Post subject: Re: Delete Everything? |
|
|
fistfullast33l wrote: | Someone remind me why I can't just delete all the files in distfiles?
Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point. |
Yes, but the gentoo mirrors might mind you using up all their bandwidth.
Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways. _________________ My blog
nvtuner software - enhance your AGP Geforce 6800 or 6200! |
|
Back to top |
|
|
fistfullast33l n00b
Joined: 26 May 2004 Posts: 28
|
Posted: Fri Jan 21, 2005 1:41 pm Post subject: Re: Delete Everything? |
|
|
vonhelmet wrote: |
Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways. |
That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed. |
|
Back to top |
|
|
skander n00b
Joined: 30 Dec 2004 Posts: 12
|
Posted: Fri Jan 21, 2005 1:46 pm Post subject: Re: Delete Everything? |
|
|
fistfullast33l wrote: | vonhelmet wrote: |
Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways. |
That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed. |
It may have been a little harsh, but what did you take the "Don't sync more than once a day" message on the gentoo mirrors to mean? |
|
Back to top |
|
|
|