Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cleaning out stale distfiles
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
eeknay
Guru
Guru


Joined: 07 Jul 2003
Posts: 402
Location: EndOfTheRainbow

PostPosted: Sun Mar 21, 2004 9:42 pm    Post subject: Reply with quote

thanks for the help.
_________________
Linda: "The holiday season is time of celebration for most but it is also the time to remember the tragic suffering of the less fortunate."
Morbo: "Earthlings do not yet know the meaning of suffering."
Back to top
View user's profile Send private message
Larcen
Apprentice
Apprentice


Joined: 21 Mar 2004
Posts: 174

PostPosted: Sun Mar 21, 2004 11:55 pm    Post subject: Reply with quote

Hello, I was using distclean-0.1.py and I continuously got this error:

aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)

many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory?
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Mon Mar 22, 2004 12:02 am    Post subject: Reply with quote

Larcen wrote:
Hello, I was using distclean-0.1.py and I continuously got this error:

aux_get(): (1) Error in x11-misc/ttmkfdir-3.0.9-r1 ebuild.
Check for syntax error or corruption in the ebuild. (--debug)

many many times, then it removed a handfull of files.. Is this common and should be ignored, or what? Do I need to run the file from a specific directory?

Probably means what it says, that there is something wrong with that ebuild.
It shouldn't have any impact on what distclean does.
The portage module is annoying in that way. Some errors it reports directly to stderr, instead of throwing exceptions.

The error message will probably go away after an emerge sync.
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
Larcen
Apprentice
Apprentice


Joined: 21 Mar 2004
Posts: 174

PostPosted: Mon Mar 22, 2004 12:09 am    Post subject: Reply with quote

Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over?
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Mon Mar 22, 2004 12:18 am    Post subject: Reply with quote

Larcen wrote:
Wow, that was a fast reply. Thank you. So if after a few syncs, it doesn't go away, would it hurt to wipe out the directory and just start over?

Wipe out distfiles? Yes, that's probably harmless, if you don't mind maybe having to download stuff again.
Does this error cause distclean to abort?
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
Larcen
Apprentice
Apprentice


Joined: 21 Mar 2004
Posts: 174

PostPosted: Mon Mar 22, 2004 12:35 am    Post subject: Reply with quote

Mind downloading what I emerge? No I don't. Does it cause it to abort? No, it continues right along.
Back to top
View user's profile Send private message
juam
n00b
n00b


Joined: 01 Jan 2004
Posts: 3

PostPosted: Sat Apr 10, 2004 9:47 pm    Post subject: Reply with quote

Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.
Back to top
View user's profile Send private message
Jowilly
Guru
Guru


Joined: 22 Jun 2002
Posts: 319

PostPosted: Sat May 08, 2004 8:03 pm    Post subject: Reply with quote

juam wrote:
Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.


Excellent !

I've put it in /etc/conf.d/local.start
Back to top
View user's profile Send private message
kaidon
n00b
n00b


Joined: 01 Nov 2003
Posts: 72

PostPosted: Sat Jun 05, 2004 8:58 pm    Post subject: Reply with quote

@juam

wonderfull :D

you just gave me back 1.6 gigs

thanx for that
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2732
Location: Bay Area, CA

PostPosted: Tue Jul 06, 2004 11:10 pm    Post subject: Reply with quote

juam wrote:
Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.
[3732]>
Code:
distcleaner --pretend --action=not_installed
Traceback (most recent call last):
  File "/opt/portage/bin/distcleaner", line 294, in ?
    main()
  File "/opt/portage/bin/distcleaner", line 289, in main
    d.perform()
  File "/opt/portage/bin/distcleaner", line 116, in perform
    self.select_files_to_delete(hash)
  File "/opt/portage/bin/distcleaner", line 192, in select_files_to_delete
    hash.pop(file,0)
AttributeError: 'dict' object has no attribute 'pop'


I don't know what happened there. gentoolkit-0.2.0_pre8 is installed
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Tue Aug 17, 2004 11:27 am    Post subject: Reply with quote

Hello, been using this script to keep distfile dir clean but have now moved over to http-replicator for network distfile download cache.

What this means is that the local distfiles dir contents can be cleaned out after emerging is done but on the http-replicator server all distfiles are moved to http-replicator dir, i tried editing the script so it pointed to /var/cache/http-replicator but the script did not work, there was no errors but no output even though i know there was files that could have been deleted.


How can i get this script to work with http-replicator?

cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
gindaani
n00b
n00b


Joined: 28 Apr 2003
Posts: 15

PostPosted: Fri Aug 27, 2004 8:32 pm    Post subject: Reply with quote

What about the files in /var/tmp/portage?

I have a directory in there for every ebuild that has ever been run. Some of the directories only contain one file, temp/successful.

Is it safe to wipe all of these out or do we need something to selectively delete them?

Thanks,

Dan
Back to top
View user's profile Send private message
wolf31o2
Retired Dev
Retired Dev


Joined: 31 Jan 2003
Posts: 628
Location: Mountain View, CA

PostPosted: Fri Aug 27, 2004 9:00 pm    Post subject: Reply with quote

You can delete them safely. They are simply temporary files created by portage during an emerge. The reason they're around, is they also provide valuable information to portage during each stage of the install of a package. It won't give you much room, but it will free up a good number of inodes.
_________________
Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org
Back to top
View user's profile Send private message
asph
l33t
l33t


Joined: 25 Aug 2003
Posts: 737
Location: Barcelona, Spain

PostPosted: Sat Aug 28, 2004 6:07 pm    Post subject: Reply with quote

does this script (the one on the first post of this thread) also delete files not installed? for example, if i emerge -f xmms, but finally i don't install it, when i run the script will those files i downloaded get deleted? they should :)

also check this little curious 'bug' ;)
Code:
Would delete: font-arial-iso-8859-1 in favour of font-arial-iso-8859-2

_________________
gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep
Back to top
View user's profile Send private message
Nightweaver
n00b
n00b


Joined: 24 Jul 2002
Posts: 40

PostPosted: Mon Sep 27, 2004 11:48 pm    Post subject: Reply with quote

juam wrote:
Hi, i've implemented a program, that let you define policies about what you want to clean:
http://www.leak.com.ar/~juam/code/distcleaner/

For now there are two policies. One deletes all the source that aren't needed to rebuild the system, and another, deletes all the files smaller that 1024 KB (those can be downloaded relative in an small time, and are the most common).

New policy are welcome.


I just upgraded to a newer version of gentoolkit (_pre9), and the 0.0.1 script no longer works :-(

Any chance of what I hope would be a simple update? I run this script every time after I do an emerge -uD world, but now it reports every single file in /usr/portage/distfiles as being 'not installed'.

Thanks.
Back to top
View user's profile Send private message
houtworm
Apprentice
Apprentice


Joined: 08 Mar 2003
Posts: 288
Location: Den Haag, Netherlands

PostPosted: Wed Oct 13, 2004 12:59 am    Post subject: Reply with quote

distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version?
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
calle2003
n00b
n00b


Joined: 21 Oct 2004
Posts: 63

PostPosted: Thu Oct 21, 2004 8:36 pm    Post subject: Reply with quote

Hi all,

I found this thread because I'm not happy about old distfiles hanging around and wanted to delete them. I'm a bit concerned finding problems with the script with version numbers and so on, I'd rather have it delete exactly those files in DISTDIR that do not belong to an installed package. I think the approach should be to read the distfiles using the usual portage functionality and then delete all the rest. Unfortunately I couldn't find any possibilty to output the results of an ebuild's SRC_URI to the command line. Is there any?

Because of this, I did a small patch on portage.py (2.0.51);

Code:

--- src-2.0.51/portage.py       2004-10-21 21:52:26.753228088 +0200
+++ portage.py  2004-10-21 19:46:07.496450256 +0200
@@ -2210,6 +2210,7 @@
 
        validcommands = ["help","clean","prerm","postrm","preinst","postinst",
                        "config","setup","depend","fetch","digest",
+                       "distfiles",
                        "unpack","compile","test","install","rpm","qmerge","merge",
                        "package","unmerge", "manifest"]
 
@@ -2494,6 +2495,10 @@
        except:
                pass
 
+       if mydo=="distfiles":
+               print string.join(checkme,"\n")
+               return 0
+
        if not fetch(fetchme, mysettings, listonly=listonly, fetchonly=fetchonly):
                return 1


This adds a "distfiles" command to ebuild so that "ebuild <file> distfiles" prints all the distfiles of that package. It works with the usual portage syntax.

Second, I created a script to show and delete the obsolete files (distclean):

Code:

#!/bin/bash
# distclean utility
# Christian Schaefer
# License: public domain
# Use this script on your own risc. The author does
# not guarantue that doesn't destroy your system
# or is useful at all.

TMP1=tmp.distfiles.installed
TMP2=tmp.distfiles.obsolete
TMP3=tmp.distfiles.installed.tmp
DISTDIR=`portageq envvar DISTDIR`

function print_obsolete() {
        $PAGER $TMP2;
}

function delete_obsolete() {
        cat $TMP2 | xargs -r -l rm;
        rm $TMP1 $TMP2;
}

function print_statistics() {
        cat $TMP2 | xargs -r du --human-readable --total | tail -n 1;
}

function build_lists() {
# create list of distfiles for installed packages
# (distfiles patch to portage.py needed)
if [ ! -e "$TMP1" ]; then
        rm $TMP2 >/dev/nul 2>&1;
        qpkg -I -v --nocolor |
                xargs -l printf "=%s\n" |
                xargs -l equery which |
                xargs -i ebuild {} distfiles |
                sort | uniq > "$TMP3" ;
        mv "$TMP3" "$TMP1";
fi;

# create list of distfiles present on disc
# but not belonging to installed packages
if [ ! -e "$TMP2" ]; then
        find $DISTDIR -type f -printf "%f\n" |
                sort | uniq |
                comm -1 -3 "$TMP1" - |
                xargs -r -l printf "$DISTDIR/%s\n" > $TMP2 ;
fi;
}

function print_usage() {
        echo Utility to clean up the distfiles directory of portage
        echo Files not belonging to installed packages are considered
        echo obsolete and deleted.
        echo
        echo Usage: $0 option
        echo
        echo Options:
        echo
        echo "-p --pretend    print a list of obsolete files"
        echo "-s --size       print size of obsolete files"
        echo "--delete        delete obsolete files"
}

case $1 in
        ( -p | --pretend )
                build_lists;
                print_obsolete;
        ;;
        ( --delete )
                build_lists;
                delete_obsolete;
        ;;
        ( -s | --size )
                build_lists;
                print_statistics;
        ;;
        ( * )
                print_usage;
        ;;
esac


The script leaves two files in the current directory, if no delete is done:

tmp.distfiles.installed
tmp.distfiles.obsolete

They are reused on a second run (so you can delete after checking quite fast), but must be deleted manually when anything changes in the DISTDIR.

This is anything else but fast, in fact it takes several minutes to produce an output.

I would like to know your opinion on this approach. I have no knowledge of python and only very basic knowledge of shell scripts so I'm sure there are a lot of things that could be optimized. At least it works for me, an "emerge -ef world" after "distclean --delete" fetched nothing, but I've got 1.4 gigabytes of disk space back.

I do not recommend to use this script, it is just experimental!

Best regards,
Christian.
Back to top
View user's profile Send private message
abionescu
n00b
n00b


Joined: 11 Nov 2004
Posts: 2

PostPosted: Thu Nov 11, 2004 4:33 pm    Post subject: Reply with quote

You can use the following script to delete source files not used in your system:
Quote:

#!/bin/bash
echo "Building index file..."
rm index.file
echo "What software do you have installed?"
for installed in `find /var/db/pkg/ -name *.ebuild`
do
basename=`dirname $installed | xargs dirname | xargs basename`
filename=`dirname $installed | xargs basename`
for configs in `find "/usr/portage/"$basename -name "digest-"$filename -print`
do
echo "Adding..."$configs
cat $configs >> index.file 2>&1
done
done
echo "Comparing files..."
rm remove.file
for i in `ls /usr/portage/distfiles/*.*`
do
chksum_full=`md5sum $i`
chksum_new=${chksum_full:0:32}
no=`cat index.file | grep $chksum_new | wc -l`
if [[ $no == 0 ]]
then
echo "Remove"$'\t'$i
echo $i >> remove.file
else
echo "Keep"$'\t'$i
fi
done
echo "Cleaning up..."
rm index.file

which will create a list with files from /usr/portage/distfiles folder which has no reason to be there.

After you check the content of the file you can issue the command:
Quote:

cat remove.file | xargs rm

With some tunning in "index" creation part you can remove only the obsolete files (not found into portage)
Back to top
View user's profile Send private message
languste
n00b
n00b


Joined: 20 Dec 2004
Posts: 1

PostPosted: Mon Dec 20, 2004 1:05 am    Post subject: Reply with quote

use this script if you have problems with recent portage versions. it cleans your dist and package files.

Code:
#!/bin/bash
#
# filename: genclean
#
# Removes deprecated distfiles and packages of a gentoo system
#
# Tested with:
#              - portage 2.0.51-r3
#              - gentoolkit 0.2.0
#
# Shoud work with later versions as well
#
# v1.0, languste, Mon Dec 20 00:33:48 GMT 2004

# source needed vars
#
source_vars() {
        eval export `emerge info | egrep 'PORTDIR|DISTDIR|PKGDIR'`
}

# clean distfiles
#
distfiles_clean() {
        DIST_FILES=`find "$DISTDIR" -maxdepth 1 -type f -printf "%f\n" | sort`
        DIGEST_FILES=`find "$PORTDIR" "$PORTDIR_OVERLAY"  -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u`
        OLD_DIST_FILES=`comm -23 <(echo "$DIST_FILES") <(echo "$DIGEST_FILES")`

        remove_files "$DISTDIR" "$OLD_DIST_FILES"
}

# clean packages
#
packages_clean() {
        QPKG_FILES=`qpkg -I -v -nc | cut -d "/" -f2 | sort | sed -e 's/$/\.tbz2/g' -e 's,^,All/,g'`
        QPKG_LINKS=`qpkg -I -v -nc | sort | sed 's/$/\.tbz2/g'`

        PKG_FILES=`cd "$PKGDIR" && find ./All/* | cut -d "/" -f2- | sort`
        PKG_LINKS=`cd "$PKGDIR" && find .  -type l | cut -d "/" -f2- | sort`
       
        OLD_PKG_FILES=`comm -23 <(echo "$PKG_FILES") <(echo "$QPKG_FILES")`
        OLD_PKG_LINKS=`comm -23 <(echo "$PKG_LINKS") <(echo "$QPKG_LINKS")`

        remove_files "$PKGDIR" "$OLD_PKG_FILES $OLD_PKG_LINKS"
}

# remove files
#
remove_files() {
        DIR="$1"
        FILES="$2"

        if [ "x${FILES}" = "x" -o "x${FILES}" = "x " ]; then
                echo "no files to remove."
                exit
        fi

        echo -e "These files (and links) in "$DIR" will be removed:\n"
        echo "Size    Filename"
        echo "----    --------"

        cd "${DIR}"
        du -hc $FILES

        echo -en "\nRemove these files? (Type YES to continue): "
        read YES

        [ "x$YES" = "xYES" ] || exit

        echo "removing files..."
        rm `echo "$FILES"`
}

# main
#
if [ "$1" = "distfiles" ]; then
        source_vars
        distfiles_clean
elif [ "$1" = "packages" ]; then
        source_vars
        packages_clean
else
        echo -e "Usage: `basename $0` <distfiles|packages>\n"
        echo "This little script will remove unused \"distfiles\" or \"packages\"."
        echo "No files will be deleted whithout confirmation!"
        exit
fi


p.s. i'am a debian user playing with gentoo and i can't belive that there is no tool inside "gentoolkit" or wherever. handling this problem. in my 4 year old debian system there would be now terrabytes of old and deprecated stuff filling up my disk :?
Back to top
View user's profile Send private message
heikole
Tux's lil' helper
Tux's lil' helper


Joined: 04 Oct 2004
Posts: 148
Location: Berlin, Germany

PostPosted: Thu Dec 23, 2004 2:04 pm    Post subject: Reply with quote

houtworm wrote:
distcleaner worked just fine, but now it wants to delete ALL distfiles :-/
Perhaps the upgrade of portage (Portage 2.0.51_rc9) has something to do with it?
I use distcleaner v0.0.1 , perhaps there is a newer version?


Same problem for me. It seems that portage 2.0.51 is a little bit broken. distcleaner expects the call

Code:
src = pkg.get_env_var('SRC_URI')


to return some value in src, but there isn't anything in it. pkg.get_env_var is defined in class Package in
    /usr/lib/gentoolkit/pym/gentoolkit/gentoolkit.py

and called in distcleaner itself. Does somebody have any ideas how to fix this?
Back to top
View user's profile Send private message
skander
n00b
n00b


Joined: 30 Dec 2004
Posts: 12

PostPosted: Tue Jan 04, 2005 8:03 pm    Post subject: thanks robm! Reply with quote

Thanks for the great script!
Back to top
View user's profile Send private message
fistfullast33l
n00b
n00b


Joined: 26 May 2004
Posts: 26

PostPosted: Fri Jan 21, 2005 5:54 am    Post subject: Delete Everything? Reply with quote

Someone remind me why I can't just delete all the files in distfiles?

Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point.
Back to top
View user's profile Send private message
vonhelmet
l33t
l33t


Joined: 06 Apr 2004
Posts: 770
Location: Somewhere in a school

PostPosted: Fri Jan 21, 2005 9:46 am    Post subject: Re: Delete Everything? Reply with quote

fistfullast33l wrote:
Someone remind me why I can't just delete all the files in distfiles?

Is there a big problem with doing this? I don't mind using the bandwidth to download the files again at some point.


Yes, but the gentoo mirrors might mind you using up all their bandwidth.

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.
_________________
My blog
nvtuner software - enhance your AGP Geforce 6800 or 6200!
Back to top
View user's profile Send private message
fistfullast33l
n00b
n00b


Joined: 26 May 2004
Posts: 26

PostPosted: Fri Jan 21, 2005 1:41 pm    Post subject: Re: Delete Everything? Reply with quote

vonhelmet wrote:

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.


That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed.
Back to top
View user's profile Send private message
skander
n00b
n00b


Joined: 30 Dec 2004
Posts: 12

PostPosted: Fri Jan 21, 2005 1:46 pm    Post subject: Re: Delete Everything? Reply with quote

fistfullast33l wrote:
vonhelmet wrote:

Don't be so egocentric as to assume it's only your bandwidth - it cuts both ways.


That's a little harsh if you ask me. I don't think I was being egocentric, just uninformed.


It may have been a little harsh, but what did you take the "Don't sync more than once a day" message on the gentoo mirrors to mean?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Page 5 of 8

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum