Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cleaning out stale distfiles
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
vicaya
n00b
n00b


Joined: 26 Jun 2004
Posts: 57

PostPosted: Mon Aug 22, 2005 5:46 pm    Post subject: How about this one liner? Reply with quote

not only shorter, but also probably faster too :)

run this (in one line) in your /usr/portage/distfiles or wherever you keep your distfiles (don't run it in any place you want to keep your files, you've been warned! ;)
Code:

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f

enjoy!
Back to top
View user's profile Send private message
Devport
Guru
Guru


Joined: 15 Dec 2004
Posts: 361

PostPosted: Sat Sep 10, 2005 2:25 pm    Post subject: Reply with quote

First I checked what your emerge command does and saw that it lists all source files required by currently installed packages. Then I dared to trust you and tried the whole thingy ... and it worked. Very nice. Thanks.

Something like this belongs into gentoolkit.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Tue Sep 20, 2005 12:07 pm    Post subject: Reply with quote

well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !

I replaced the xarg rm -f with xarg ls -l and it listed every bloody thing in distfiles , old new redundant , the lot .

Now the title of this thread is
Quote:
Cleaning out stale distfiles


Could you maybe post _exactly_ what this magic incantation is supposed to do for us?

What does it do that rm -f does not?

TIA 8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Tue Sep 20, 2005 12:25 pm    Post subject: Reply with quote

Gentree wrote:
well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !
Ebuilds often get revision bumps (especially the ones of "big" packages: current Qt is -r8, and current Firefox is -r7...). And also, sometimes packages get masked and thus you have to downgrade them.

If you simply "rm -rf" your distfiles, then in both the above cases you will have to redownload some source files (many times in case of many revision bumps...). While you may not care because you have a good DSL connection, think a bit about Gentoo mirrors: upload bandwidth is expensive, finding companies or universities to mirror Gentoo distfiles is not easy, so wasting their bandwith by re-downloading a same file several times really is A Bad Thing©®.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Tue Sep 20, 2005 12:36 pm    Post subject: Re: Another way of cleaning distfiles Reply with quote

gingekerr wrote:
Although I haven't read all the posts in this forum, all the scripts I have seen seem really long and complicated.

Here is a really short one (although it may not be as quick as the others) which simply gets a list of all the distfiles used by the software in the world file by emerge -epf world and then deletes all distfiles not in this list.

It needs a bit of porting to get it to work on a non-default system.

Code:

#!/bin/bash

emerge -epf world &> /tmp/distfilesAX432z1
ACCEPT_KEYWORDS="~x86" emerge -epf world &> /tmp/distfilesAX432z2

cd /usr/portage/distfiles

for DISTFILE in *; do
GREPRESULT=`grep $DISTFILE /tmp/distfilesAX432z1 /tmp/distfilesAX432z2`
        if ( [ -z "$GREPRESULT" ] )
                then rm $DISTFILE
        fi
done


I dont see the sense of the second emerge, emerge -epf should cover my whole installation, including any masked / unmasked tweeks I have added.

One big disadvantage here is that if a package was built with an ebuild that has been pulled from the tree you will also burn its distfile which is likely no longer on the mirrors either.

I think all these scripts are prone to this kind of issue which will hit you with an unanticipated exeption and you will spend hours trying to recover of debug what has happened.

I think a more reliable approach would be to set up the local machine as an ftp server, move all of distfiles to a new location and set GENTOO_MIRRORS= in make.conf to that local server.

emerge -ef world should then pull any and all distfiles currently required to rebuild the system back into distfiles and the moved files can all be deleted (or better burned to CD and kept for a rainy day)


There are posts on setting up gentoo as a local mirror which should give all the details needed to do this.

You would have to make sure emerge -ef world ran cleanly from the mirrors and that emerge -uDp world has nothing pending (ie you have a fully stable system) before starting , but that is needed before any clean out is done by any method.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Tue Sep 20, 2005 12:41 pm    Post subject: Reply with quote

TGL wrote:
Gentree wrote:
well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !
Ebuilds often get revision bumps (especially the ones of "big" packages: current Qt is -r8, and current Firefox is -r7...). And also, sometimes packages get masked and thus you have to downgrade them.

If you simply "rm -rf" your distfiles, then in both the above cases you will have to redownload some source files (many times in case of many revision bumps...). While you may not care because you have a good DSL connection, think a bit about Gentoo mirrors: upload bandwidth is expensive, finding companies or universities to mirror Gentoo distfiles is not easy, so wasting their bandwith by re-downloading a same file several times really is A Bad Thing©®.


you've complete missed what I was saying, I _DONT_ want to rm -rf but I cant see that the script does anything other than that.

As I posted , I got it to ls instead of rm and it listed every file, including the current ones. That's why I was asking for clarification on the what it is _supposed_ to do. Either I missed the point or the script is buggy.

:?
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Tue Sep 20, 2005 12:52 pm    Post subject: Reply with quote

Gentree wrote:
you've complete missed what I was saying
Probably yes. Since you were not quoting any script in particular, my understanding was that you were talking about this whole thread, and that's what i've answered too.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Tue Sep 20, 2005 1:09 pm    Post subject: Reply with quote

I dont quite see how you could have interpreted :
Quote:
I replaced the xarg rm -f with xarg ls -l
as being a comment on the whole thread of seven pages rather than the two posts immediately above my post.

But if you just half read it and misunderstood "ce n'est pas grave".

Let's concentrate on make useful suggestions to this thread.

:wink:
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Tue Sep 20, 2005 1:24 pm    Post subject: Reply with quote

Gentree wrote:
But if you just half read it and misunderstood "ce n'est pas grave".

More than half... Actually I've read that:
Gentree wrote:
well I dont have the time to decrypt that little lot but it looks like a very elaborate way to do rm -rf distfiles !
And that:
Gentree wrote:
Now the title of this thread is
Quote:
Cleaning out stale distfiles


Could you maybe post _exactly_ what this magic incantation is supposed to do for us?

What does it do that rm -f does not?

TIA 8)
That was ~80% of your post... not that bad :roll:

Quote:
Let's concentrate on make useful suggestions to this thread.

Like "try eclean, which comes with latest ~arch gentoolkit"?
You, lucky french spoker, can even read my not so short introduction to its features.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Tue Sep 20, 2005 1:46 pm    Post subject: Reply with quote

Thanks for the link , I'll look into that and thanks for the doc.fr , a bit more digestable than the neolithic manpage format.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
tapted
Tux's lil' helper
Tux's lil' helper


Joined: 02 Dec 2003
Posts: 122
Location: Sydney, Australia

PostPosted: Tue Oct 18, 2005 11:31 pm    Post subject: A simpler way...? Reply with quote

How about

Code:

# mv /usr/portage/distfiles /var/www/localhost/htdocs
# mkdir -m 0775 /usr/portage/distfiles
# chgrp portage /usr/portage/distfiles
# GENTOO_MIRRORS="http://localhost/" emerge -fe world
# rm -r /var/www/localhost/htdocs/distfiles


Of course, you will need a http server running...

And if you're short on disk space, you may want to do an occasional

Code:

# for i in `ls /usr/portage/distfiles` ; do rm -fv "/var/www/localhost/htdocs/distfiles/$i" ; done


in another terminal.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Wed Oct 19, 2005 12:00 am    Post subject: Reply with quote

That's _exactly_ what I wanted, for some reason I thought it was more complicated.

Some thought needs to be given to packages with different slotted versions installed (like gimp-1.2.5 amd gimp2 and serveral vintages of kernel-sources files).

I think I'll burn my current distfiles to DVD to make sure I dont lose anything , then mount the dvd somewhere under htdocs and use your method.

Beautifully simple , that's very much.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Wed Oct 19, 2005 2:16 am    Post subject: Reply with quote

Works a treat!

I mounted my DVD on /var/www/localhost/htdocs/distfiles and its flying away copying back 488 files .

Nice one.

About time I did that it had got up to 4.2 gigs :? Now it's back down to 1.5 :D

I had about 8 pkgs that dont come through but most of them will be due to my not having done emerge -ef world before moving distfiles.

Anyway 3G more free space and I can breath a bit easier.

Cheers 8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
ps2cho
Apprentice
Apprentice


Joined: 27 Jun 2005
Posts: 196

PostPosted: Sun Oct 23, 2005 3:42 pm    Post subject: Reply with quote

ok im confused here guys. I got a few questions.

Firstly, which script do i want to use here? I know sometimes reading the first few pages and scripts, some have errors and do not work. Which one is safe to use?

and then, how do i get it to work? Could you add a step-by-step guide on the first post showing how to get it to run?

For all us newbies :)

Thanks, ps2cho
_________________
Toshiba Satellite A45-S121 - 2.8GHz Celeron @3.2GHz // 512MB RAM // OnBoard VGA // WLAN
==
Athlon XP 2000+ @ 2.2GHz // 768MB PC2700 @ 177FSB // eVGA 6600GT // 60GB Maxtor
Back to top
View user's profile Send private message
darkarmani
n00b
n00b


Joined: 07 Apr 2005
Posts: 2

PostPosted: Mon Nov 14, 2005 7:32 pm    Post subject: Gracious Failure Reply with quote

stalynx wrote:
my own code

Code:
#!/bin/sh
cd /usr/portage/distfiles
rm *

works pretty good. :twisted:


HaHa. I know you are just joking but the best part is that your little script can delete a lot of important files.

Suppose your CWD is ~ and you had already removed /usr/portage/distfiles. Now when you run your script the "cd" fails and all the files in your CWD are deleted. At that point just be glad it is not `rm -rf` and your CWD wasn't /.

:D
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Wed Nov 16, 2005 2:22 am    Post subject: Reply with quote

Well heres one for managing your packages. It uses the files listed in /var/db/pkg, which is the master database of the programes you have installed in your system. This should handel slots and such. To run it, name it, and run it useing either "p" or "-p" for pretend. If you agree with what its saying then as root run it, no switchs. and byby files
Download cleanpkg.sh.
Code:

#!/bin/bash
# cleanpkg.sh a script for cleaning out your package directory. It uses /var/db/pkg
# to get the listing of all packages on your system then checks your
# packages directory against the list and anything no longer in /var/db
# is removed. It now 1-16-06 removes the broken links  in $PKGDIR/category.
# Authour Hiel Van Campen See gentoo forums 
source /etc/make.conf

function colors(){
#Set up colors
   NO=$'\x1b[0;0m'
   BR=$'\x1b[0;01m'
   RD=$'\x1b[31;01m' Rd=$'\x1b[00;31m'
   GR=$'\x1b[32;01m' Gr=$'\x1b[00;32m'
   YL=$'\x1b[33;01m' Yl=$'\x1b[00;33m'
   BL=$'\x1b[34;01m' Bl=$'\x1b[00;34m'
   FC=$'\x1b[35;01m' Fc=$'\x1b[00;35m'
   CY=$'\x1b[36;01m' Cy=$'\x1b[00;36m'
   COLUMNS=${COLUMNS:-80}
   spaces=$(for ((i=0;i<COLUMNS;++i)); do echo -n " "; done)
}

# Varibles
PRTND=""
CLEAN=""
bad_links=""
pkg=""

# Functions, most of em anyway
function PRNT_HLP(){
echo
cat <<END
   ${Yl}Usage: Run "cleanpkg.sh or cleanpkg.sh -h /or h"  to see help
   cleanpkg.sh -p /or p to see what it finds and will remove.
   cleanpkg.sh -c/or c to remove what was shown in pretend mode.
   You must be ${RD}"root"${Yl} to clean things up.${NO}
END
echo
exit
}

# empty files
clean(){
   cd $org_dir
   # if's to prevent errors from "rm"
   if [ -e pkglist ];then rm pkglist;fi
   if [ -e dblist ]; then    rm dblist ;fi
   if [ -e category ]; then rm category rm_list ;fi
   unset pkg bad_links
}

#simple get_argvs function
function get_argv(){
# get opts "p" for pretend and
if [[ "$1" == "" || "$1" == "h" ||  "$1" == "-h" ]]; then PRNT_HLP ;fi
if [[ "$1" == "p" || "$1" == "-p" ]]; then  PRTND="yes"; fi
if [[ "$1" == "c" || "$1" == "-c" ]];then   CLEAN="yes" ;fi
}

# get pkglist and dblist
function get_list(){
ls $PKGDIR/All > pkglist
# get data base list
for dir in $(ls /var/db/pkg); do
   ls /var/db/pkg/$dir >>dblist
done
#pkgs="$(wc -l pkglist)"
#dbfiles="$(wc -l dblist)"
#echo ${CY}"   $pkgs "
#echo "   $dbfiles"${NO}
}

# now compare dblist to pkglst and remove whats common
function dif_list(){
for p in $(< dblist); do
      grep -v "$p" pkglist>tmp1
      cat tmp1>pkglist
done
rm tmp1 dblist
}

# list of links
function link_list(){
cd $PKGDIR
   for d in $(ls );do
      if [ "$d" != "All" ]; then
         for z in $(ls $d);do
            if [ ! -e $d/$z ]; then
               if [ "$d" != "$cat"  ]; then
                  echo "Category $d">>$org_dir/category
                  cat="$d"
               fi   
               echo "  > $z">>$org_dir/category
               bad_links="yes"
               # file to hold links to be removed
               echo "$PKGDIR/$d/$z">>$org_dir/rm_list   
            fi
         done
      fi
   done
cd $org_dir
}

# the main function 8^)
function main(){
if [ "$PRTND" == "yes" ]; then
   # do package stuff    
   if [[ -e pkglist &&  -s pkglist ]];then
      cat pkglist
      echo "${Gr}There are $(wc -l pkglist) tbz2's to remove${NO}"
   else
      echo ${Gr}"No packages to remove"${NO};echo
   fi
   # do link stuff
   if [ "$bad_links" == "yes" ]; then
      cat category
      echo ${Gr}"The above links are broken and will be removed if you run this for real"${NO}
   else
      echo ${Gr}"no broken links to remove"${NO}
   fi
else    #chech UID to run for root
      if [[ "$CLEAN" == "yes" && "$UID" == "0" ]] ; then
      for i in $(< pkglist); do
           #echo "rm $PKGDIR/All/$i"
         rm $PKGDIR/All/$i
          done
      for l in $(< rm_list); do
         #echo "rm $l"
         rm $l
      done
   else
      if [[ "$UID" != "0" && "$PRTND" != "yes" ]];then
           echo;echo ${Yl}"   You must be ${RD}ROOT${Yl} to remove files"
         echo "   To see files that would be removed"
         echo "   run useing p or -p." ${NO};echo
         exit
      fi
    fi
fi
}

# get present working directory
org_dir="$(echo $PWD)"
colors
get_argv $@
get_list
dif_list
link_list
main
cd "$(echo $org_dir)" ; unset org_dir
clean
exit



EDIT 11-27-05 now sourceing /etc/make.conf to get your pkg directory and added clean() to remove files created when thru running
1-16-06 Added broken link removal. When the pkgs are removed it leaves a broken link in $PKGDIR/category. Doesnt hurt anything but
1-21-06 Cleanup and reorganised script. Added hep. use h or -h help, p or -p for pretend and c or -c.to clean. .....
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Fri Jan 20, 2006 5:43 pm; edited 4 times in total
Back to top
View user's profile Send private message
thrope
n00b
n00b


Joined: 03 Nov 2004
Posts: 66

PostPosted: Sun Nov 27, 2005 12:05 pm    Post subject: Reply with quote

Hi,

Just came to this thread as I'm looking to free up some space....
Think vicaya's one liner is fantastic... elegant, quick and simple...

Quote:
Code:

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f


To explain to gentree what it is doing... emerge -epf world lists all the mirrors and files portage would try if it was downloading the system from scratch.
The first perl expression strips out the filenames from the urls, and put's each filename on its own line. Each filename appears about 10 times because of the different mirrors.
The sort -u sorts these in alphabetical order and also removes the duplicates.... so you now have a list of all the files needed for your system
Apending this with ls -f means you have a list of all files needed, followed by all the files there are.
then sort this again. Now in this list if a row appears twice, then it means it was both in the 'files needed' list and in the 'ls -f' list. If a row appears only once, it means it either isn't needed, or isn't there.
Uniq -c removes the duplicate rows, adding a count of the number of times a row appears and the second perl expression prints the file name, if the count in the first column is 1 and the file exists.
xargs then deletes these files...

It DOESN'T just delete all the files... in fact works very nicely... took only a few minutes for me to free up 1.5 GB!
[/code]
Back to top
View user's profile Send private message
Cintra
Advocate
Advocate


Joined: 03 Apr 2004
Posts: 2111
Location: Norway

PostPosted: Sun Nov 27, 2005 1:33 pm    Post subject: Reply with quote

hielvc wrote:
Well heers one for managing your packages. It uses the files listed in /var/db/pkg which is the master database ? of the files you have installed in your system. This should handel slots and such. To run it name it and run it useing either "p" or "-p" for pretend. If you agree with what its saying then as root run it, no switchs. and byby files.
Code:
#!/bin/bash
# test script

# empty files
:>pkglist
:>dblist

# get pkglist
ls /usr/portage/packages/All > pkglist
...


tried your script but I have
Code:

PKGDIR="/mnt/hdb9/portage/packages"

so I changed a couple of lines and it gave me a nice long list
:-)
_________________
"I am not bound to please thee with my answers" W.S.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Sun Nov 27, 2005 6:26 pm    Post subject: Reply with quote

I edited the script to source /etc/make.conf to pickup your PKGDIR automaticaly and it cleans up after its self now.
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Thu Dec 08, 2005 5:35 am; edited 2 times in total
Back to top
View user's profile Send private message
Cintra
Advocate
Advocate


Joined: 03 Apr 2004
Posts: 2111
Location: Norway

PostPosted: Sun Nov 27, 2005 7:10 pm    Post subject: Reply with quote

Nice.. 'hielvc.sh' cleaned up 1 GB

:-)
_________________
"I am not bound to please thee with my answers" W.S.
Back to top
View user's profile Send private message
Centinul
Apprentice
Apprentice


Joined: 28 Jul 2005
Posts: 232

PostPosted: Thu Dec 22, 2005 5:10 am    Post subject: Reply with quote

thrope wrote:
Hi,

Just came to this thread as I'm looking to free up some space....
Think vicaya's one liner is fantastic... elegant, quick and simple...

Quote:
Code:

(emerge -epf world  2>&1 | perl -ne '$f=join("\n", m@\w://[^\s]+/([^\s]+)@g); print "$f\n" if $f' | sort -u; ls -f) | sort | uniq -c | perl -ane 'print "$F[1]\n" if $F[0]==1 && -f $F[1]' | xargs rm -f


To explain to gentree what it is doing... emerge -epf world lists all the mirrors and files portage would try if it was downloading the system from scratch.
The first perl expression strips out the filenames from the urls, and put's each filename on its own line. Each filename appears about 10 times because of the different mirrors.
The sort -u sorts these in alphabetical order and also removes the duplicates.... so you now have a list of all the files needed for your system
Apending this with ls -f means you have a list of all files needed, followed by all the files there are.
then sort this again. Now in this list if a row appears twice, then it means it was both in the 'files needed' list and in the 'ls -f' list. If a row appears only once, it means it either isn't needed, or isn't there.
Uniq -c removes the duplicate rows, adding a count of the number of times a row appears and the second perl expression prints the file name, if the count in the first column is 1 and the file exists.
xargs then deletes these files...

It DOESN'T just delete all the files... in fact works very nicely... took only a few minutes for me to free up 1.5 GB!
[/code]


I just wanted to say two things. First thanks for explaining what this elegant 2 liner does. Secondly, I want to thank the writer of it because I ran it, took about 30 seconds and I had 800MB freed up. Thanks!
Back to top
View user's profile Send private message
dpetka2001
l33t
l33t


Joined: 04 Mar 2005
Posts: 804

PostPosted: Tue Jan 10, 2006 2:21 pm    Post subject: Reply with quote

just found out about the existense of this topic...after reading the whole topic i think i'll give hielvc's script a try...or maybe you have another suggestion that might be better?? i would also like to make another question...is there any script that does the same with /usr/portage/packages aswell besides /usr/portage/distfiles because i would really like to remove some unneeded entries from there aswell if possible...thanks...
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3432
Location: Gainesville, Florida

PostPosted: Mon Jan 16, 2006 10:42 am    Post subject: Reply with quote

I just changed my portage directories to a non-standard setup on different partitions (see below), and yacleaner and distcleaner no longer worked no matter what I tried (editing the script files). Even ecleaner didn't work- I killed it after 15 minutes. Finally (after a lot of struggling), I saw vicaya's amazing one line perl solution above, and it worked flawlessly! I just did a cd /mnt/dump/distfiles as root, and copy/pasted it in at the prompt. Went from 405 to 354 distfiles in about 2 minutes. I was a little apprehensive (as I don't understand perl), but after reading thrope's great explanation, I went ahead.

BTW, I followed the info on this Gentoo doc page http://www.gentoo.org/doc/en/articles/partitioning-p2.xml, so now my portage tree is on it's own partition /mnt/portage (ext2 with dir_index), /var & /tmp are on theirs (/mnt/rwstorage, with dir_index and data=writeback) with / symlinks, and distfiles is on another tuned ext3, "/mnt/dump" with no /usr/portage symlink- just the make.conf edit. Also had to make a new /etc/make.profile symlink to /mnt/portage/profiles/default-linux/x86/2005.1.

This setup is working very well and performance is outstanding.

-------------------------------------------------------------------------------------------------
# Portage Directories
# ===================
#
# Each of these settings controls an aspect of portage's storage and file
# system usage. If you change any of these, be sure it is available when
# you try to use portage. *** DO NOT INCLUDE A TRAILING "/" ***
#
# PORTAGE_TMPDIR is the location portage will use for compilations and
# temporary storage of data. This can get VERY large depending upon
# the application being installed.
PORTAGE_TMPDIR=/mnt/rwstorage/var/tmp
#
# PORTDIR is the location of the portage tree. This is the repository
# for all profile information as well as all ebuilds. This directory
# itself can reach 200M. WE DO NOT RECOMMEND that you change this.
PORTDIR=/mnt/portage
#
# DISTDIR is where all of the source code tarballs will be placed for
# emerges. The source code is maintained here unless you delete
# it. The entire repository of tarballs for gentoo is 9G. This is
# considerably more than any user will ever download. 2-3G is
# a large DISTDIR.
DISTDIR=/mnt/dump/distfiles
#
# PKGDIR is the location of binary packages that you can have created
# with '--buildpkg' or '-b' while emerging a package. This can get
# upto several hundred megs, or even a few gigs.
PKGDIR=/mnt/portage/packages
#
# PORTDIR_OVERLAY is a directory where local ebuilds may be stored without
# concern that they will be deleted by rsync updates. Default is not
# defined.
PORTDIR_OVERLAY=/usr/local/portage
#
# PORT_LOGDIR is the location where portage will store all the logs it
# creates from each individual merge. They are stored as YYMMDD-$PF.log
# in the directory specified. This is disabled until you enable it by
# providing a directory. Permissions will be modified as needed IF the
# directory exists, otherwise logging will be disabled.
PORT_LOGDIR=/mnt/rwstorage/var/log/portage
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-13.2.1_p20230304
kernel-6.7.2 USE=experimental python3_11


Last edited by wrc1944 on Tue Jan 17, 2006 5:23 am; edited 1 time in total
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Jan 17, 2006 3:19 am    Post subject: Reply with quote

wrc1944 could you try my cleanpkg.sh script? Im curious if it follows your $PKGDIR redirect. It should.

Just updated the cleanpkg.sh, or as Cintra calls it hell.sh :lol:.. Last night for some reason I started fthing of the setup of /usr/portage/packages and remebered that all the packaes are linked to $PKGDIR/category. So when ever a pkg is removed the broken links are left. It doesnt break anything at this time and the broken links look pretty in mc. theyre red, But I brokem so I added broken link removal. Also added a link where you can down load it


heres the link try 2 cleanpkg.sh
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Tue Jan 17, 2006 6:04 am; edited 1 time in total
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3432
Location: Gainesville, Florida

PostPosted: Tue Jan 17, 2006 5:19 am    Post subject: Reply with quote

hielvc,
I'm on my gcc-4.1 test box right now updating to the latest gcc beta, and it's the standard portage setup, so I'll try your script in a day or two, next time I'm on the other box, and report back.

IIRC, I did try it yesterday on the non-standard box (among lots of others), but I think I might have still had my /etc/make.profile symlink messed up, and/or hadn't removed my /usr/portage symlink pointing to /mnt/portage yet before I edited my make.conf properly (that symlink scheme I tried first led me astray for a while).

Now that I think about it, yours might have partially worked, but since I had a messed up make.conf file at the time I didn't realize it probably would have worked. Right now, I can't remember if I had fixed the make.conf before or after I ran vicaya's one line solution. More later.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-13.2.1_p20230304
kernel-6.7.2 USE=experimental python3_11
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Page 7 of 8

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum