Cleaning out stale distfiles

[Cintra]

wrc1944 could you try my cleanpkg.sh script? Im curious if it follows your $PKGDIR redirect. It should.

Just updated the cleanpkg.sh, or as Cintra calls it hell.sh .. Last night for some reason I started fthing of the setup of /usr/portage/packages and remebered that all the packaes are linked to $PKGDIR/category. So when ever a pkg is removed the broken links are left. It doesnt break anything at this time and the broken links look pretty in mc. theyre red, But I brokem so I added broken link removal. Also added a link where you can down load it

heres the link try 2 cleanpkg.sh

thought it was time to run this again but the above link is not loading, so I copied cleanpkg.sh from above and ran it with '-h' first as usually do with things and it gave me an odd output

Code: Select all

# ./cleanpkg.sh -h
 375  375 7981 pkglist
 490  490 7979 dblist

so I did '-p' instead

p4pe ~ # ./cleanpkg.sh -p
 316  316 6694 pkglist
 490  490 7979 dblist
There are 0 pkglist tbz2's to remove
app-accessibility
app-admin
app-antivirus
app-arch
...
...
x11-misc
x11-terms
x11-themes
The above links are broken and will be removed if you run this for real

same story, so I ran it hoping to get rid of the 'broken links' but the same output appears again..

Mvh

[wrc1944]

OK- I'm back on the non-standard box, and I tried out the cleanpkg.sh script, but couldn't get it to work. I Put something in packages for your script to remove:

mymachine wrc # quickpkg glibc
* Building package for glibc-2.3.6-r2 ... [ ok ]

* Packages now in /mnt/portage/packages:
* glibc-2.3.6-r2: 11M

I looked to be sure, and there is a glibc-2.3.6-r2.tbz2 symlink in /mnt/portage/packages/sys-libs, pointing to glibc-2.3.6-r2.tbz2 in /mnt/portage/packages/All.

I then put cleanpkg.sh in /home/wrc, and made it executable, and su'd to root.

mymachine wrc # cd /home/wrc
mymachine wrc # sh cleanpkg.sh
1 1 20 pkglist
528 528 8815 dblist
grep: invalid option -- M
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.

mymachine wrc # ./cleanpkg.sh
1 1 20 pkglist
528 528 8815 dblist
grep: invalid option -- M
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.


No matter what I tried, I get the same output. Where am I going wrong?

I think I mentioned before my /var and /tmp are now in /mnt/rwstorage. I did this from the info on http://www.gentoo.org/doc/en/articles/p ... ing-p2.xml

EDIT:
Hmmmm... Should my "PKGDIR=/mnt/portage/packages" in make.conf be:

PKGDIR=/mnt/portage/packages/All

[Cintra]

mine is

Code: Select all

PKGDIR="/mnt/hdb9/portage/packages"

[wrc1944]

In my case, this time quickpkg created the ALL directory and the glibc symlink, while previously (at least in the last year or so) quickpkg has normally been just creating a regular tbz2 file in /usr/portage/packages. In fact, IIRC, I haven't even seen an "All" directory in packages for some time, so I sort of thought the "All" directory had been deprecated.

I don't know if this has anything to do with why I can't get cleanpkg.sh to work. But since I hardly ever use quickpkg anyway, except for something like right before I do a major gcc upgrade I make a binary of my working gcc, I just use rm -rf /mnt/portage/packages/* if I need to- which is almost never.

Distfiles is another matter, and before I went to a non-standard portage setup, I found yacleaner to be the quickest and easiest method for cleaning out the old uninstalled tars, and other stuff. Unfortunately, I can't get that to work now.

[hielvc]

SOrry I saw your emails last night. Sence the script worked for me at least at one point Im going to have to load abunch of old tbz2 and make links to them. It will probably be this evening or tommarow before I can see what the dang thing is doing.

[hielvc]

Ok I re-did the script. My set up in fstab is "/dev/hda5 /usr ext3 defaults 0 2" and it works. I also have a gcc-4.0.2 chroot where everything is on the one partition. It has 5 old tbz2 and 6 broken links according to th script. The output is a little diferent. In -p or p ,pretend, it list the tbz's to be removed then the category and then broken links listed under the category. Use "" , h, -h to see the simple help. p or -p for pretend and c or -c to clean.

Oh the link had better be working or I'm fireing myself as network "Mon"

[wrc1944]

Downloaded new version, got same result- or I'm doing it wrong.

mymachine wrc # sh cleanpkg.sh -p
grep: invalid option -- M
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
No packages to remove

no broken links to remove
mymachine wrc #

There is still the glibc as before to remove.

[hielvc]

wrc1944 I dont believe your prob is related to the script. Could you try running

Code: Select all

ps aux|grep "root"   <== match lines with root
ps aux|grep -v "root"  <==exclude anything with root in it.
t="root"
ps aux|grep "$t"   <== match lines with root
ps aux|grep -v "$t"   <== exclude lines with root

If those lines didnt error and they shouldnt , they are simple grep usage, then the script shouldnt give that grep error. If you do get an error with the test lines then I would try emergeing grep coreutils bash one at a time and test after each.

Oh if the test work and my script still errors then you might try the emerges.

I went so far as to download the script myself and I ran it. It worked but I saw a logic prob: I had no broken links but it found a bunch of old packages. I emerge sysnced and built 35 or packages today. The problem is that after removeing those pkgs I would have a bunch of broken links I need to retinker or maybe run the script twice.

[wrc1944]

OK - ran it as root, and got

mmachine wrc # ps aux|grep "root" <== match lines with root
bash: ==: No such file or directory
mymachine wrc # ps aux|grep -v "root" <==exclude anything with root in it.
bash: ==exclude: No such file or directory
mymachine wrc # t="root"
mymachine wrc # ps aux|grep "$t" <== match lines with root
bash: ==: No such file or directory
mymachine wrc # ps aux|grep -v "$t" <== exclude lines with root

I posted above that my portage directory is in another partition, and is referenced appropriately in my own make.conf, and everything related to emerge & portage is functioning OK. (apparently, this is making a difference, but please don't alter your script just because I mount portage in a non-standard manner). I'm sorry if this has caused confusion.

Additionally, I still don't understand why this script is necessary, unless one creates what would seem to be to an ordinary desktop user an inordinate amount of backup tarred binaries??

Please enlighten me.

[hielvc]

I have binpkgs enable in make conf. Which has saved my butt once or twice. Like distfiles the package directory will grow and grow in such a situation, upto 4 gigs once. My old method was to "rm ..../packages/*" and emerge world -e". Which worked but had a major week point, me! This way I learn more about bash and portage.

Now for your prob and youve got one. It looks to me like your bash needs to be emerged again, this reminds of the wierd coreutils prob I saw on the forum where emerge -s, equery and others would give errors like bash was bad but it turned out to be coreutils which didnt make sence but fixed his prob.

[wrc1944]

I'd sure like to help figure out what's going astray here, and it's probably related to my specific portage setup. However, in my case, I don't think it really matters too much, because I rarely need to clean out /portage/packages, wherever it might be located. But for others with non-standard portage trees on multiple partitions, I guess solving this will help.

On the bash point, I'm ~x86 from the start (2+ years), and never had problems with anything related to that.

In fact, I just did a complete tool chain & system/world rebuild on this gcc-3.4.5 box with no compiling problems, and no problems since (5 days). I do this on all my Gentoo boxes every 2-3 months. Wouldn't that eliminate any potential bash problems you mentioned?

The only other thing I can think of is that my particular and unique portage setup in make.conf must be conflicting with your script somehow, that I don't understand. I realize my setup is weird, because /var and /tmp are in a /mnt/rwstorage partition (as in the Gentoo doc). That's only an semi-educated guess, as I'm really an amateur at figuring out this kind of stuff, and certainly don't have your scripting knowledge. I'm a total noob at that.

[dbevfat]

I believe all these scripts delete file that were manually downloaded to distfiles (such as sun-jdk and similar packages). Something to be aware of.

Regards

[nihilistic]

I believe all these scripts delete file that were manually downloaded to distfiles (such as sun-jdk and similar packages). Something to be aware of.

Why do you care if you have the new ones?

[dbevfat]

Not sure I understand. What new ones?

What I meant was, that if you for example want to emerge sun-jdk-1.5.0, you have to:
- manually download the bin file and copy it into the distfiles directory,
- you have to run "emerge sun-jdk".

Now, if you run any of these scripts that clean up portage, the aforementioned file will also be removed, even though the package is still installed. And in case you want to re-emerge it, you'd have to download the file manually again. That's all I was saying, and there's nothing _really_ wrong with it.

It's just that the sun-jdk package could be considered to depend (from a user's perspective, not a real portage dependency) on that file, so maybe the file shouldn't be deleted in the first place. While keeping/deleting decision is arguable, the fact remains that the file is removed and you have to download it again, which is that "something to be aware of" thingy I tried to tell.

Regards

[hielvc]

dbevfat Files like java that you have to fetch, I keep on my /home partion. Scince I ocassionally get board and my fingers start typeing and an hour or so later my brain wakes up an engages, to find I've played another game of break my gentoo. So to cut down on the pain keep backups of things.

For those who dont know, I just finally installed it myself last night, portage-utils is a bunch tools for maitaining your system includeing eclaean which imitate
the scripts here including mine. To get it " emerge portage-utils "

[Cintra]

Code: Select all

[ Searching for file(s) eclean in *... ]
app-portage/gentoolkit-0.2.2_pre2 (/usr/bin/eclean)

[hielvc]

Though eclean-pkg leaves one broken link for whatever was the last pkg removed. Tsk tsk

[prophecy]

wouldn't it be prudent to just add a small function to emerge to remove the archive file when unmerging a package. Then none of this would be nessassary. Yhea, I probibly spelled that wrong... Well, of course you could have stale files in there reguardless, emerge crashes and the such, so then you would have to have a script go though and clean stuff up. I mean what was the original intention of keeping these files anyway. After you do an install you should not technically need them. It could be an option to remove the archive file after install in the make.conf file.

Justace

[Allix]

wouldn't it be prudent to just add a small function to emerge to remove the archive file when unmerging a package. Then none of this would be nessassary. Yhea, I probibly spelled that wrong... Well, of course you could have stale files in there reguardless, emerge crashes and the such, so then you would have to have a script go though and clean stuff up. I mean what was the original intention of keeping these files anyway. After you do an install you should not technically need them. It could be an option to remove the archive file after install in the make.conf file.

Justace

I could not agree more, freebsd has portsclean, something like that would be cool.
Also why i am at it , a option in emerge to select what you want to install when updating world would be handy , freebsd has an prefix that asks after each package is installed, a problem could occur if a package is needed as a dep and the user decides not to emerge it , a warning could come up saying if you don't emerge this your system will brake.

thanks

[RushPL]

I read the whole thread and no script other than the first one suits me at all. So does anybody have a ready to use modification of the first script in this thread that handles other files than .tgz/tar.gz/tar.bz2 and furthermore deletes other files that does not belong to portage anymore ? My HD on my server is quite limited so I need to be a bit more restrictive than leaving only portage-existant files, I think I will be quite fine with leaving only the newest versions of packages.
I am currently running the following combination cron.weekly:

Code: Select all

emerge --sync
/usr/bin/repcacheman
/var/cache/cleaner --nopretend

However it doesn't work perfectly for the reasons I provided.
That doesn't mean that I can't code it myself, it's just that my time is limited .. Hope you could help if you have free time on your own.
Cheers,
Rush

[cwng]

Hello,

after reading this thread, I have created a script from scratch using the idea of "emerge -efp world" to get the list of sources needed by the current world state, and comparing it with the files residing in distfiles, and removing those that do not appear to be needed by the current world state. Feel free to use/modify it as you see fit.

It is strongly recommended that you run with the -p option first to make sure nothing is amiss, and then run with the -r option (avoid emerge -efp world a 2nd time) to actually remove the files.

Code: Select all

#!/bin/bash

# variables
DISTFILESDIR="/usr/portage/distfiles"

# options
PRETEND=0
REGEN=1
EXCLUDEFILE="~/clean-dist.exclude"

# Temporary files
TMPF1=/tmp/clean-dist.worldfilelist
TMPF2=/tmp/clean-dist.sortedlist

# Display usage help and exit
function usage () {
        cat << EOF

This command cleans the distfiles by removing source archives
that are no longer needed.

Options:

  -p     Pretend mode.  Prints the file to be deleted, but do not delete.

  -r     Do not regenerate needed package list.  This script uses 'emerge -epf world'
         to regenerate the needed package list, so it is a pretty long process.
         If a previous package list is available, this option cause this script
         to omit the regeneration process.

  -x <FILE>
         Do not remove any files specified in the file <FILE>
         [Default: ~/clean-dist.exclude]

EOF
        exit
}

# output each file in package list
function extract-pkg-list () {
        for x in `cat ${TMPF1} | grep "http://"`; do
                echo ${x##*/}
        done
        for x in `cat ${TMPF1} | grep "ftp://"`; do
                echo ${x##*/}
        done
}

# generate the currently needed package list
function regen-pkg-list () {
        if (( ${REGEN} == 1 )) || [[ ! -e ${TMPF1} ]]; then
                emerge -epf world >& ${TMPF1}
                REGEN=1
        fi
        if (( ${REGEN} == 1 )) || [[ ! -e ${TMPF2} ]]; then
                extract-pkg-list | sort | uniq > ${TMPF2}
        fi
}

# compare each file in DISTFILESDIR with the currently needed
# package list, and output those that do not match
function get-remove-list () {
        for d in `ls ${DISTFILESDIR}`; do
                if [[ -d ${d} ]]; then
                        continue
                fi
                if [[ -f ${EXCLUDEFILE} ]]; then
                        grep ${d} ${EXCLUDEFILE} >& /dev/null
                        if (( ${?} == 0 )); then
                                continue
                        fi
                fi
                grep ${d} ${TMPF2} >& /dev/null
                if (( ${?} == 1 )); then
                        echo ${DISTFILESDIR}/${d}
                fi
        done
}

# Handle options
while (( $# > 0 )); do
        case ${1} in
        -p)
                PRETEND=1
                ;;
        -r)
                REGEN=0
                ;;
        -x)
                shift
                EXCLUDEFILE=${1}
                ;;
        *)
                usage
                ;;
        esac
        shift
done

# regenerate pkg list
regen-pkg-list

# perform cleaning
if (( ${PRETEND} == 1 )); then
        msg "This are files that will be removed: "
        get-remove-list
else
        get-remove-list | xargs rm -f
fi

This is basically similar to vicaya's one liner, but I don't understand perl and all those, so I implemented the basic idea (as explained by thrope) using pure bash script.

The script is pretty much self explanatory. You have to modify the variable DISTFILESDIR if you source are not residing at /usr/portage/distfiles.

One warning: even though I tried to test for directories and ignore them, the test doesn't seems to work. So directories (most notably cvs-src) will probably be listed for removal. So it is best if you run the script with the -p option, look at the output, and add in whatever files listed that you actually do not want to remove in the file ~/clean-dist.exclude. Then, run with the -r option (to avoid running "emerge -efp world" which is going to take ages).

The temporary list of files "/tmp/clean-dist.*" are not deleted by the script, in case someone need to debug it.

--

[Element Dave]

This script will clean out any tarballs in your /usr/portage/distfiles directory where a newer one is proven to exist. As the presence of a newer tarball for a given piece of software usually implies that you have emerged a newer version, it also implies that you don't need the tarball for the older version sitting around on your disk simply wasting space! No more ... they will be cleaned up.

Notes: this script runs in a "pretend" mode by default , where files that would be deleted are displayed, but not actually deleted. To clean out the files, pass on the --nopretend parameter and those old source files will be wiped up quicker than a flash! It only operates on the following types of file: .tar.gz, .tar.bz2, .tgz. Of course, I won't be held responsible if it doesn't work as expected!!! But I've tested it and it works just fine for me, and of course it will not delete files unless you explicitly tell it to. If you find it useful, do check back because (as usual) I will edit the post when I make improvements to the script. Code follows:

snipped...

Frankly, I see little to no value for the end-user to keep *any* distfiles, except perhaps in very special cases, such as being unfortunate enough to be stuck with an antiquated analog modem. Personally, the only reason I maintain any distfiles *at all* is to play nice with the Gentoo mirrors by minimizing my use of their generous bandwidth, which is something I'd hope everyone would do whenever possible. Point being: accidental deletion of distfiles should be the least of ones concerns.

If one can afford to spare a few GB of space, setting the portage FEATURES variable to include "ccache" and "buildpkg" (you must emerge the former) can dramatically reduce compilation time.

[skyPhyr]

hi cwng,

Really nice script - worked perfectly here - thanks.

It also left the directories in tact because it just uses rm rather than rm -r

Cheers,

Alan.

[PssD]

Thank You very much, cwng!

Cause of getting back into stoneage with a really slow connection, Your script is exactly what i was looking for.

I didn`t want to waste hardisk space but still want to use the old distfiles for deltup.

thanks!

[lyallp]

Code: Select all

emerge gentoolkit

Then use eclean