Cleaning out stale distfiles

[eeknay]

thanks for the help.

[Larcen]

Hello, I was using distclean-0.1.py and I continuously got this error:

aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)

many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory?

[far]

Hello, I was using distclean-0.1.py and I continuously got this error:

aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)

many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory?

Probably means what it says, that there is something wrong with that ebuild.
It shouldn't have any impact on what distclean does.
The portage module is annoying in that way. Some errors it reports directly to stderr, instead of throwing exceptions.

The error message will probably go away after an emerge sync.

[Larcen]

Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over?

[far]

Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over?

Wipe out distfiles? Yes, that's probably harmless, if you don't mind maybe having to download stuff again.
Does this error cause distclean to abort?

[Larcen]

Mind downloading what I emerge? No I don't. Does it cause it to abort? No, it continues right along.

[juam]

Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.

[Jowilly]

Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.

Excellent !

I've put it in /etc/conf.d/local.start

[kaidon]

@juam

wonderfull

you just gave me back 1.6 gigs

thanx for that

[devsk]

Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.

[3732]>

Code: Select all

distcleaner --pretend --action=not_installed
Traceback (most recent call last):
  File "/opt/portage/bin/distcleaner", line 294, in ?
    main()
  File "/opt/portage/bin/distcleaner", line 289, in main
    d.perform()
  File "/opt/portage/bin/distcleaner", line 116, in perform
    self.select_files_to_delete(hash)
  File "/opt/portage/bin/distcleaner", line 192, in select_files_to_delete
    hash.pop(file,0)
AttributeError: 'dict' object has no attribute 'pop'

I don't know what happened there. gentoolkit-0.2.0_pre8 is installed

[carpman]

Hello, been using this script to keep distfile dir clean but have now moved over to http-replicator for network distfile download cache.

What this means is that the local distfiles dir contents can be cleaned out after emerging is done but on the http-replicator server all distfiles are moved to http-replicator dir, i tried editing the script so it pointed to /var/cache/http-replicator but the script did not work, there was no errors but no output even though i know there was files that could have been deleted.


How can i get this script to work with http-replicator?

cheers

[gindaani]

What about the files in /var/tmp/portage?

I have a directory in there for every ebuild that has ever been run. Some of the directories only contain one file, temp/successful.

Is it safe to wipe all of these out or do we need something to selectively delete them?

Thanks,

Dan

[wolf31o2]

You can delete them safely. They are simply temporary files created by portage during an emerge. The reason they're around, is they also provide valuable information to portage during each stage of the install of a package. It won't give you much room, but it will free up a good number of inodes.

[asph]

does this script (the one on the first post of this thread) also delete files not installed? for example, if i emerge -f xmms, but finally i don't install it, when i run the script will those files i downloaded get deleted? they should

also check this little curious 'bug'

Code: Select all

Would delete: font-arial-iso-8859-1 in favour of font-arial-iso-8859-2

[Nightweaver]

Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.

I just upgraded to a newer version of gentoolkit (_pre9), and the 0.0.1 script no longer works

Any chance of what I hope would be a simple update? I run this script every time after I do an emerge -uD world, but now it reports every single file in /usr/portage/distfiles as being 'not installed'.

Thanks.

[houtworm]

distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version?

[calle2003]

Hi all,

I found this thread because I'm not happy about old distfiles hanging around and wanted to delete them. I'm a bit concerned finding problems with the script with version numbers and so on, I'd rather have it delete exactly those files in DISTDIR that do not belong to an installed package. I think the approach should be to read the distfiles using the usual portage functionality and then delete all the rest. Unfortunately I couldn't find any possibilty to output the results of an ebuild's SRC_URI to the command line. Is there any?

Because of this, I did a small patch on portage.py (2.0.51);

Code: Select all

--- src-2.0.51/portage.py       2004-10-21 21:52:26.753228088 +0200
+++ portage.py  2004-10-21 19:46:07.496450256 +0200
@@ -2210,6 +2210,7 @@
 
        validcommands = ["help","clean","prerm","postrm","preinst","postinst",
                        "config","setup","depend","fetch","digest",
+                       "distfiles",
                        "unpack","compile","test","install","rpm","qmerge","merge",
                        "package","unmerge", "manifest"]
 
@@ -2494,6 +2495,10 @@
        except:
                pass
 
+       if mydo=="distfiles":
+               print string.join(checkme,"\n")
+               return 0
+
        if not fetch(fetchme, mysettings, listonly=listonly, fetchonly=fetchonly):
                return 1

This adds a "distfiles" command to ebuild so that "ebuild <file> distfiles" prints all the distfiles of that package. It works with the usual portage syntax.

Second, I created a script to show and delete the obsolete files (distclean):

Code: Select all

#!/bin/bash
# distclean utility
# Christian Schaefer
# License: public domain
# Use this script on your own risc. The author does
# not guarantue that doesn't destroy your system
# or is useful at all.

TMP1=tmp.distfiles.installed
TMP2=tmp.distfiles.obsolete
TMP3=tmp.distfiles.installed.tmp
DISTDIR=`portageq envvar DISTDIR`

function print_obsolete() {
        $PAGER $TMP2;
}

function delete_obsolete() {
        cat $TMP2 | xargs -r -l rm;
        rm $TMP1 $TMP2;
}

function print_statistics() {
        cat $TMP2 | xargs -r du --human-readable --total | tail -n 1;
}

function build_lists() {
# create list of distfiles for installed packages
# (distfiles patch to portage.py needed)
if [ ! -e "$TMP1" ]; then
        rm $TMP2 >/dev/nul 2>&1;
        qpkg -I -v --nocolor |
                xargs -l printf "=%s\n" | 
                xargs -l equery which |
                xargs -i ebuild {} distfiles | 
                sort | uniq > "$TMP3" ;
        mv "$TMP3" "$TMP1";
fi;

# create list of distfiles present on disc
# but not belonging to installed packages
if [ ! -e "$TMP2" ]; then
        find $DISTDIR -type f -printf "%f\n" | 
                sort | uniq | 
                comm -1 -3 "$TMP1" - |
                xargs -r -l printf "$DISTDIR/%s\n" > $TMP2 ;
fi;
}

function print_usage() {
        echo Utility to clean up the distfiles directory of portage
        echo Files not belonging to installed packages are considered
        echo obsolete and deleted.
        echo
        echo Usage: $0 option
        echo
        echo Options:
        echo
        echo "-p --pretend    print a list of obsolete files"
        echo "-s --size       print size of obsolete files"
        echo "--delete        delete obsolete files"
}

case $1 in 
        ( -p | --pretend )
                build_lists;
                print_obsolete;
        ;;
        ( --delete )
                build_lists;
                delete_obsolete;
        ;;
        ( -s | --size )
                build_lists;
                print_statistics;
        ;;
        ( * )
                print_usage;
        ;;
esac

The script leaves two files in the current directory, if no delete is done:

tmp.distfiles.installed
tmp.distfiles.obsolete

They are reused on a second run (so you can delete after checking quite fast), but must be deleted manually when anything changes in the DISTDIR.

This is anything else but fast, in fact it takes several minutes to produce an output.

I would like to know your opinion on this approach. I have no knowledge of python and only very basic knowledge of shell scripts so I'm sure there are a lot of things that could be optimized. At least it works for me, an "emerge -ef world" after "distclean --delete" fetched nothing, but I've got 1.4 gigabytes of disk space back.

I do not recommend to use this script, it is just experimental!

Best regards,
Christian.

[abionescu]

You can use the following script to delete source files not used in your system:

#!/bin/bash
echo "Building index file..."
rm index.file
echo "What software do you have installed?"
for installed in `find /var/db/pkg/ -name *.ebuild`
do
basename=`dirname $installed | xargs dirname | xargs basename`
filename=`dirname $installed | xargs basename`
for configs in `find "/usr/portage/"$basename -name "digest-"$filename -print`
do
echo "Adding..."$configs
cat $configs >> index.file 2>&1
done
done
echo "Comparing files..."
rm remove.file
for i in `ls /usr/portage/distfiles/*.*`
do
chksum_full=`md5sum $i`
chksum_new=${chksum_full:0:32}
no=`cat index.file | grep $chksum_new | wc -l`
if [[ $no == 0 ]]
then
echo "Remove"$'\t'$i
echo $i >> remove.file
else
echo "Keep"$'\t'$i
fi
done
echo "Cleaning up..."
rm index.file

which will create a list with files from /usr/portage/distfiles folder which has no reason to be there.

After you check the content of the file you can issue the command:

cat remove.file | xargs rm

With some tunning in "index" creation part you can remove only the obsolete files (not found into portage)

[languste]

use this script if you have problems with recent portage versions. it cleans your dist and package files.

Code: Select all

#!/bin/bash
#
# filename: genclean
#
# Removes deprecated distfiles and packages of a gentoo system
#
# Tested with:
#              - portage 2.0.51-r3
#              - gentoolkit 0.2.0
#
# Shoud work with later versions as well
#
# v1.0, languste, Mon Dec 20 00:33:48 GMT 2004

# source needed vars
#
source_vars() {
        eval export `emerge info | egrep 'PORTDIR|DISTDIR|PKGDIR'`
}

# clean distfiles
#
distfiles_clean() {
        DIST_FILES=`find "$DISTDIR" -maxdepth 1 -type f -printf "%f\n" | sort`
        DIGEST_FILES=`find "$PORTDIR" "$PORTDIR_OVERLAY"  -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u`
        OLD_DIST_FILES=`comm -23 <(echo "$DIST_FILES") <(echo "$DIGEST_FILES")`

        remove_files "$DISTDIR" "$OLD_DIST_FILES"
}

# clean packages
#
packages_clean() {
        QPKG_FILES=`qpkg -I -v -nc | cut -d "/" -f2 | sort | sed -e 's/$/\.tbz2/g' -e 's,^,All/,g'`
        QPKG_LINKS=`qpkg -I -v -nc | sort | sed 's/$/\.tbz2/g'`

        PKG_FILES=`cd "$PKGDIR" && find ./All/* | cut -d "/" -f2- | sort`
        PKG_LINKS=`cd "$PKGDIR" && find .  -type l | cut -d "/" -f2- | sort`
        
        OLD_PKG_FILES=`comm -23 <(echo "$PKG_FILES") <(echo "$QPKG_FILES")`
        OLD_PKG_LINKS=`comm -23 <(echo "$PKG_LINKS") <(echo "$QPKG_LINKS")`

        remove_files "$PKGDIR" "$OLD_PKG_FILES $OLD_PKG_LINKS"
}

# remove files
#
remove_files() {
        DIR="$1"
        FILES="$2"

        if [ "x${FILES}" = "x" -o "x${FILES}" = "x " ]; then
                echo "no files to remove."
                exit
        fi

        echo -e "These files (and links) in "$DIR" will be removed:\n"
        echo "Size    Filename"
        echo "----    --------"

        cd "${DIR}"
        du -hc $FILES

        echo -en "\nRemove these files? (Type YES to continue): "
        read YES

        [ "x$YES" = "xYES" ] || exit

        echo "removing files..."
        rm `echo "$FILES"`
}

# main
#
if [ "$1" = "distfiles" ]; then
        source_vars
        distfiles_clean
elif [ "$1" = "packages" ]; then
        source_vars
        packages_clean
else
        echo -e "Usage: `basename $0` <distfiles|packages>\n"
        echo "This little script will remove unused \"distfiles\" or \"packages\"."
        echo "No files will be deleted whithout confirmation!"
        exit
fi

p.s. i'am a debian user playing with gentoo and i can't belive that there is no tool inside "gentoolkit" or wherever. handling this problem. in my 4 year old debian system there would be now terrabytes of old and deprecated stuff filling up my disk

[heikole]

distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version?

Same problem for me. It seems that portage 2.0.51 is a little bit broken. distcleaner expects the call

Code: Select all

src = pkg.get_env_var('SRC_URI')

to return some value in src, but there isn't anything in it. pkg.get_env_var is defined in class Package in

• /usr/lib/gentoolkit/pym/gentoolkit/gentoolkit.py

and called in distcleaner itself. Does somebody have any ideas how to fix this?

[skander]

Thanks for the great script!

[fistfullast33l]

Someone remind me why I can't just delete all the files in distfiles?

Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point.

[vonhelmet]

Someone remind me why I can't just delete all the files in distfiles?

Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point.

Yes, but the gentoo mirrors might mind you using up all their bandwidth.

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.

[fistfullast33l]

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.

That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed.

[skander]

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.

That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed.

It may have been a little harsh, but what did you take the "Don't sync more than once a day" message on the gentoo mirrors to mean?