Cleaning out stale distfiles

[vicaya]

not only shorter, but also probably faster too

run this (in one line) in your /usr/portage/distfiles or wherever you keep your distfiles (don't run it in any place you want to keep your files, you've been warned!

Code: Select all

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f

enjoy!

[Devport]

First I checked what your emerge command does and saw that it lists all source files required by currently installed packages. Then I dared to trust you and tried the whole thingy ... and it worked. Very nice. Thanks.

Something like this belongs into gentoolkit.

[Gentree]

well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !

I replaced the xarg rm -f with xarg ls -l and it listed every bloody thing in distfiles , old new redundant , the lot .

Now the title of this thread is

Cleaning out stale distfiles

Could you maybe post _exactly_ what this magic incantation is supposed to do for us?

What does it do that rm -f does not?

TIA

[TGL]

well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !

Ebuilds often get revision bumps (especially the ones of "big" packages: current Qt is -r8, and current Firefox is -r7...). And also, sometimes packages get masked and thus you have to downgrade them.

If you simply "rm -rf" your distfiles, then in both the above cases you will have to redownload some source files (many times in case of many revision bumps...). While you may not care because you have a good DSL connection, think a bit about Gentoo mirrors: upload bandwidth is expensive, finding companies or universities to mirror Gentoo distfiles is not easy, so wasting their bandwith by re-downloading a same file several times really is A Bad Thing©®.

[Gentree]

Although I haven't read all the posts in this forum, all the scripts I have seen seem really long and complicated.

Here is a really short one (although it may not be as quick as the others) which simply gets a list of all the distfiles used by the software in the world file by emerge -epf world and then deletes all distfiles not in this list.

It needs a bit of porting to get it to work on a non-default system.

Code: Select all

#!/bin/bash

emerge -epf world &> /tmp/distfilesAX432z1
ACCEPT_KEYWORDS="~x86" emerge -epf world &> /tmp/distfilesAX432z2

cd /usr/portage/distfiles

for DISTFILE in *; do
GREPRESULT=`grep $DISTFILE /tmp/distfilesAX432z1 /tmp/distfilesAX432z2`
        if ( [ -z "$GREPRESULT" ] )
                then rm $DISTFILE
        fi
done

I dont see the sense of the second emerge, emerge -epf should cover my whole installation, including any masked / unmasked tweeks I have added.

One big disadvantage here is that if a package was built with an ebuild that has been pulled from the tree you will also burn its distfile which is likely no longer on the mirrors either.

I think all these scripts are prone to this kind of issue which will hit you with an unanticipated exeption and you will spend hours trying to recover of debug what has happened.

I think a more reliable approach would be to set up the local machine as an ftp server, move all of distfiles to a new location and set GENTOO_MIRRORS= in make.conf to that local server.

emerge -ef world should then pull any and all distfiles currently required to rebuild the system back into distfiles and the moved files can all be deleted (or better burned to CD and kept for a rainy day)


There are posts on setting up gentoo as a local mirror which should give all the details needed to do this.

You would have to make sure emerge -ef world ran cleanly from the mirrors and that emerge -uDp world has nothing pending (ie you have a fully stable system) before starting , but that is needed before any clean out is done by any method.

[Gentree]

well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !

Ebuilds often get revision bumps (especially the ones of "big" packages: current Qt is -r8, and current Firefox is -r7...). And also, sometimes packages get masked and thus you have to downgrade them.

If you simply "rm -rf" your distfiles, then in both the above cases you will have to redownload some source files (many times in case of many revision bumps...). While you may not care because you have a good DSL connection, think a bit about Gentoo mirrors: upload bandwidth is expensive, finding companies or universities to mirror Gentoo distfiles is not easy, so wasting their bandwith by re-downloading a same file several times really is A Bad Thing©®.

you've complete missed what I was saying, I _DONT_ want to rm -rf but I cant see that the script does anything other than that.

As I posted , I got it to ls instead of rm and it listed every file, including the current ones. That's why I was asking for clarification on the what it is _supposed_ to do. Either I missed the point or the script is buggy.

[TGL]

you've complete missed what I was saying

Probably yes. Since you were not quoting any script in particular, my understanding was that you were talking about this whole thread, and that's what i've answered too.

[Gentree]

I dont quite see how you could have interpreted :

I replaced the xarg rm -f with xarg ls -l

as being a comment on the whole thread of seven pages rather than the two posts immediately above my post.

But if you just half read it and misunderstood "ce n'est pas grave".

Let's concentrate on make useful suggestions to this thread.

[TGL]

But if you just half read it and misunderstood "ce n'est pas grave".

More than half... Actually I've read that:

well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !

And that:

Now the title of this thread is

Cleaning out stale distfiles

Could you maybe post _exactly_ what this magic incantation is supposed to do for us?

What does it do that rm -f does not?

TIA

That was ~80% of your post... not that bad

Let's concentrate on make useful suggestions to this thread.

Like "try eclean, which comes with latest ~arch gentoolkit"?
You, lucky french spoker, can even read my not so short introduction to its features.

[Gentree]

Thanks for the link , I'll look into that and thanks for the doc.fr , a bit more digestable than the neolithic manpage format.

[tapted]

How about

Code: Select all

# mv /usr/portage/distfiles /var/www/localhost/htdocs 
# mkdir -m 0775 /usr/portage/distfiles
# chgrp portage /usr/portage/distfiles
# GENTOO_MIRRORS="http://localhost/" emerge -fe world
# rm -r /var/www/localhost/htdocs/distfiles

Of course, you will need a http server running...

And if you're short on disk space, you may want to do an occasional

Code: Select all

# for i in `ls /usr/portage/distfiles` ; do rm -fv "/var/www/localhost/htdocs/distfiles/$i" ; done

in another terminal.

[Gentree]

That's _exactly_ what I wanted, for some reason I thought it was more complicated.

Some thought needs to be given to packages with different slotted versions installed (like gimp-1.2.5 amd gimp2 and serveral vintages of kernel-sources files).

I think I'll burn my current distfiles to DVD to make sure I dont lose anything , then mount the dvd somewhere under htdocs and use your method.

Beautifully simple , that's very much.

[Gentree]

Works a treat!

I mounted my DVD on /var/www/localhost/htdocs/distfiles and its flying away copying back 488 files .

Nice one.

About time I did that it had got up to 4.2 gigs Now it's back down to 1.5

I had about 8 pkgs that dont come through but most of them will be due to my not having done emerge -ef world before moving distfiles.

Anyway 3G more free space and I can breath a bit easier.

Cheers

[ps2cho]

ok im confused here guys. I got a few questions.

Firstly, which script do i want to use here? I know sometimes reading the first few pages and scripts, some have errors and do not work. Which one is safe to use?

and then, how do i get it to work? Could you add a step-by-step guide on the first post showing how to get it to run?

For all us newbies

Thanks, ps2cho

[darkarmani]

my own code

Code: Select all

#!/bin/sh
cd /usr/portage/distfiles
rm *

works pretty good.

HaHa. I know you are just joking but the best part is that your little script can delete a lot of important files.

Suppose your CWD is ~ and you had already removed /usr/portage/distfiles. Now when you run your script the "cd" fails and all the files in your CWD are deleted. At that point just be glad it is not `rm -rf` and your CWD wasn't /.

[hielvc]

Well heres one for managing your packages. It uses the files listed in /var/db/pkg, which is the master database of the programes you have installed in your system. This should handel slots and such. To run it, name it, and run it useing either "p" or "-p" for pretend. If you agree with what its saying then as root run it, no switchs. and byby files
Download cleanpkg.sh.

Code: Select all

#!/bin/bash
# cleanpkg.sh a script for cleaning out your package directory. It uses /var/db/pkg
# to get the listing of all packages on your system then checks your
# packages directory against the list and anything no longer in /var/db
# is removed. It now 1-16-06 removes the broken links  in $PKGDIR/category.
# Authour Hiel Van Campen See gentoo forums  
source /etc/make.conf

function colors(){
#Set up colors
	NO=$'\x1b[0;0m'
	BR=$'\x1b[0;01m'
	RD=$'\x1b[31;01m' Rd=$'\x1b[00;31m'
	GR=$'\x1b[32;01m' Gr=$'\x1b[00;32m'
	YL=$'\x1b[33;01m' Yl=$'\x1b[00;33m'
	BL=$'\x1b[34;01m' Bl=$'\x1b[00;34m'
	FC=$'\x1b[35;01m' Fc=$'\x1b[00;35m'
	CY=$'\x1b[36;01m' Cy=$'\x1b[00;36m'
	COLUMNS=${COLUMNS:-80}
	spaces=$(for ((i=0;i<COLUMNS;++i)); do echo -n " "; done)
}

# Varibles
PRTND=""
CLEAN=""
bad_links=""
pkg=""

# Functions, most of em anyway
function PRNT_HLP(){
echo
cat <<END
	${Yl}Usage: Run "cleanpkg.sh or cleanpkg.sh -h /or h"  to see help 
	cleanpkg.sh -p /or p to see what it finds and will remove.
	cleanpkg.sh -c/or c to remove what was shown in pretend mode.
	You must be ${RD}"root"${Yl} to clean things up.${NO}
END
echo
exit
}

# empty files
clean(){
	cd $org_dir
	# if's to prevent errors from "rm"
	if [ -e pkglist ];then rm pkglist;fi
	if [ -e dblist ]; then 	rm dblist ;fi
	if [ -e category ]; then rm category rm_list ;fi
	unset pkg bad_links
}

#simple get_argvs function
function get_argv(){
# get opts "p" for pretend and 
if [[ "$1" == "" || "$1" == "h" ||  "$1" == "-h" ]]; then PRNT_HLP ;fi
if [[ "$1" == "p" || "$1" == "-p" ]]; then  PRTND="yes"; fi
if [[ "$1" == "c" || "$1" == "-c" ]];then	CLEAN="yes" ;fi
}

# get pkglist and dblist
function get_list(){
ls $PKGDIR/All > pkglist
# get data base list
for dir in $(ls /var/db/pkg); do
	ls /var/db/pkg/$dir >>dblist
done
#pkgs="$(wc -l pkglist)"
#dbfiles="$(wc -l dblist)"
#echo ${CY}"	$pkgs "
#echo "	$dbfiles"${NO}
}

# now compare dblist to pkglst and remove whats common
function dif_list(){
for p in $(< dblist); do
   	grep -v "$p" pkglist>tmp1
   	cat tmp1>pkglist
done
rm tmp1 dblist
}

# list of links
function link_list(){
cd $PKGDIR
	for d in $(ls );do
		if [ "$d" != "All" ]; then
			for z in $(ls $d);do
				if [ ! -e $d/$z ]; then
					if [ "$d" != "$cat"  ]; then
						echo "Category $d">>$org_dir/category
						cat="$d"
					fi	
					echo "  > $z">>$org_dir/category
					bad_links="yes"
					# file to hold links to be removed
					echo "$PKGDIR/$d/$z">>$org_dir/rm_list	
				fi
			done
		fi
	done
cd $org_dir
}

# the main function 8^)
function main(){
if [ "$PRTND" == "yes" ]; then
	# do package stuff 	
	if [[ -e pkglist &&  -s pkglist ]];then
		cat pkglist
		echo "${Gr}There are $(wc -l pkglist) tbz2's to remove${NO}"
	else 
		echo ${Gr}"No packages to remove"${NO};echo
	fi
	# do link stuff
	if [ "$bad_links" == "yes" ]; then
		cat category
		echo ${Gr}"The above links are broken and will be removed if you run this for real"${NO}
	else
		echo ${Gr}"no broken links to remove"${NO}
	fi
else    #chech UID to run for root
   	if [[ "$CLEAN" == "yes" && "$UID" == "0" ]] ; then
		for i in $(< pkglist); do
        	#echo "rm $PKGDIR/All/$i"
			rm $PKGDIR/All/$i
       	done
		for l in $(< rm_list); do
			#echo "rm $l"
			rm $l
		done
	else 
		if [[ "$UID" != "0" && "$PRTND" != "yes" ]];then
        	echo;echo ${Yl}"	You must be ${RD}ROOT${Yl} to remove files"
			echo "	To see files that would be removed"
			echo "	run useing p or -p." ${NO};echo
			exit
		fi
    fi
fi
}

# get present working directory
org_dir="$(echo $PWD)"
colors
get_argv $@
get_list
dif_list
link_list
main
cd "$(echo $org_dir)" ; unset org_dir
clean
exit 

EDIT 11-27-05 now sourceing /etc/make.conf to get your pkg directory and added clean() to remove files created when thru running
1-16-06 Added broken link removal. When the pkgs are removed it leaves a broken link in $PKGDIR/category. Doesnt hurt anything but
1-21-06 Cleanup and reorganised script. Added hep. use h or -h help, p or -p for pretend and c or -c.to clean. .....

[thrope]

Hi,

Just came to this thread as I'm looking to free up some space....
Think vicaya's one liner is fantastic... elegant, quick and simple...

Code: Select all

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f 

To explain to gentree what it is doing... emerge -epf world lists all the mirrors and files portage would try if it was downloading the system from scratch.
The first perl expression strips out the filenames from the urls, and put's each filename on its own line. Each filename appears about 10 times because of the different mirrors.
The sort -u sorts these in alphabetical order and also removes the duplicates.... so you now have a list of all the files needed for your system
Apending this with ls -f means you have a list of all files needed, followed by all the files there are.
then sort this again. Now in this list if a row appears twice, then it means it was both in the 'files needed' list and in the 'ls -f' list. If a row appears only once, it means it either isn't needed, or isn't there.
Uniq -c removes the duplicate rows, adding a count of the number of times a row appears and the second perl expression prints the file name, if the count in the first column is 1 and the file exists.
xargs then deletes these files...

It DOESN'T just delete all the files... in fact works very nicely... took only a few minutes for me to free up 1.5 GB!
[/code]

[Cintra]

Well heers one for managing your packages. It uses the files listed in /var/db/pkg which is the master database ? of the files you have installed in your system. This should handel slots and such. To run it name it and run it useing either "p" or "-p" for pretend. If you agree with what its saying then as root run it, no switchs. and byby files.

Code: Select all

#!/bin/bash
# test script

# empty files
:>pkglist
:>dblist

# get pkglist
ls /usr/portage/packages/All > pkglist
...

tried your script but I have

Code: Select all

PKGDIR="/mnt/hdb9/portage/packages"

so I changed a couple of lines and it gave me a nice long list

[hielvc]

I edited the script to source /etc/make.conf to pickup your PKGDIR automaticaly and it cleans up after its self now.

[Cintra]

Nice.. 'hielvc.sh' cleaned up 1 GB

[Centinul]

Hi,

Just came to this thread as I'm looking to free up some space....
Think vicaya's one liner is fantastic... elegant, quick and simple...

Code: Select all

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f 

To explain to gentree what it is doing... emerge -epf world lists all the mirrors and files portage would try if it was downloading the system from scratch.
The first perl expression strips out the filenames from the urls, and put's each filename on its own line. Each filename appears about 10 times because of the different mirrors.
The sort -u sorts these in alphabetical order and also removes the duplicates.... so you now have a list of all the files needed for your system
Apending this with ls -f means you have a list of all files needed, followed by all the files there are.
then sort this again. Now in this list if a row appears twice, then it means it was both in the 'files needed' list and in the 'ls -f' list. If a row appears only once, it means it either isn't needed, or isn't there.
Uniq -c removes the duplicate rows, adding a count of the number of times a row appears and the second perl expression prints the file name, if the count in the first column is 1 and the file exists.
xargs then deletes these files...

It DOESN'T just delete all the files... in fact works very nicely... took only a few minutes for me to free up 1.5 GB!
[/code]

I just wanted to say two things. First thanks for explaining what this elegant 2 liner does. Secondly, I want to thank the writer of it because I ran it, took about 30 seconds and I had 800MB freed up. Thanks!

[dpetka2001]

just found out about the existense of this topic...after reading the whole topic i think i'll give hielvc's script a try...or maybe you have another suggestion that might be better?? i would also like to make another question...is there any script that does the same with /usr/portage/packages aswell besides /usr/portage/distfiles because i would really like to remove some unneeded entries from there aswell if possible...thanks...

[wrc1944]

I just changed my portage directories to a non-standard setup on different partitions (see below), and yacleaner and distcleaner no longer worked no matter what I tried (editing the script files). Even ecleaner didn't work- I killed it after 15 minutes. Finally (after a lot of struggling), I saw vicaya's amazing one line perl solution above, and it worked flawlessly! I just did a cd /mnt/dump/distfiles as root, and copy/pasted it in at the prompt. Went from 405 to 354 distfiles in about 2 minutes. I was a little apprehensive (as I don't understand perl), but after reading thrope's great explanation, I went ahead.

BTW, I followed the info on this Gentoo doc page http://www.gentoo.org/doc/en/articles/p ... ing-p2.xml, so now my portage tree is on it's own partition /mnt/portage (ext2 with dir_index), /var & /tmp are on theirs (/mnt/rwstorage, with dir_index and data=writeback) with / symlinks, and distfiles is on another tuned ext3, "/mnt/dump" with no /usr/portage symlink- just the make.conf edit. Also had to make a new /etc/make.profile symlink to /mnt/portage/profiles/default-linux/x86/2005.1.

This setup is working very well and performance is outstanding.

-------------------------------------------------------------------------------------------------
# Portage Directories
# ===================
#
# Each of these settings controls an aspect of portage's storage and file
# system usage. If you change any of these, be sure it is available when
# you try to use portage. *** DO NOT INCLUDE A TRAILING "/" ***
#
# PORTAGE_TMPDIR is the location portage will use for compilations and
# temporary storage of data. This can get VERY large depending upon
# the application being installed.
PORTAGE_TMPDIR=/mnt/rwstorage/var/tmp
#
# PORTDIR is the location of the portage tree. This is the repository
# for all profile information as well as all ebuilds. This directory
# itself can reach 200M. WE DO NOT RECOMMEND that you change this.
PORTDIR=/mnt/portage
#
# DISTDIR is where all of the source code tarballs will be placed for
# emerges. The source code is maintained here unless you delete
# it. The entire repository of tarballs for gentoo is 9G. This is
# considerably more than any user will ever download. 2-3G is
# a large DISTDIR.
DISTDIR=/mnt/dump/distfiles
#
# PKGDIR is the location of binary packages that you can have created
# with '--buildpkg' or '-b' while emerging a package. This can get
# upto several hundred megs, or even a few gigs.
PKGDIR=/mnt/portage/packages
#
# PORTDIR_OVERLAY is a directory where local ebuilds may be stored without
# concern that they will be deleted by rsync updates. Default is not
# defined.
PORTDIR_OVERLAY=/usr/local/portage
#
# PORT_LOGDIR is the location where portage will store all the logs it
# creates from each individual merge. They are stored as YYMMDD-$PF.log
# in the directory specified. This is disabled until you enable it by
# providing a directory. Permissions will be modified as needed IF the
# directory exists, otherwise logging will be disabled.
PORT_LOGDIR=/mnt/rwstorage/var/log/portage

[hielvc]

wrc1944 could you try my cleanpkg.sh script? Im curious if it follows your $PKGDIR redirect. It should.

Just updated the cleanpkg.sh, or as Cintra calls it hell.sh .. Last night for some reason I started fthing of the setup of /usr/portage/packages and remebered that all the packaes are linked to $PKGDIR/category. So when ever a pkg is removed the broken links are left. It doesnt break anything at this time and the broken links look pretty in mc. theyre red, But I brokem so I added broken link removal. Also added a link where you can down load it


heres the link try 2 cleanpkg.sh

[wrc1944]

hielvc,
I'm on my gcc-4.1 test box right now updating to the latest gcc beta, and it's the standard portage setup, so I'll try your script in a day or two, next time I'm on the other box, and report back.

IIRC, I did try it yesterday on the non-standard box (among lots of others), but I think I might have still had my /etc/make.profile symlink messed up, and/or hadn't removed my /usr/portage symlink pointing to /mnt/portage yet before I edited my make.conf properly (that symlink scheme I tried first led me astray for a while).

Now that I think about it, yours might have partially worked, but since I had a messed up make.conf file at the time I didn't realize it probably would have worked. Right now, I can't remember if I had fixed the make.conf before or after I ran vicaya's one line solution. More later.