Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cleaning out stale distfiles
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
alexbuell
Guru
Guru


Joined: 18 Jul 2002
Posts: 471
Location: "Hemp"shire, UK

PostPosted: Sun Jul 27, 2003 6:40 pm    Post subject: Reply with quote

RoadRunner wrote:

It's just amazing how almost on every forum i'm in, there's a While E. Coyote nick =)


Beep beep!
_________________
Cheers,
Alex.

Linux - the best text adventure game ever.
Back to top
View user's profile Send private message
ghwomb
n00b
n00b


Joined: 04 Aug 2003
Posts: 1
Location: Kumla, Sweden

PostPosted: Mon Aug 04, 2003 4:33 pm    Post subject: Reply with quote

Just posted so I would get a mail if anything happened. Don't seem to be much activity here though. Perhaps the scripts is being tested the Debian way; lots of testing on uncommon systems and no releases.
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Mon Aug 04, 2003 4:43 pm    Post subject: Reply with quote

Eh, you could just click "Watch this topic for replies" in the lower left corner of the page.

A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball.
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
erik_swanson
Retired Dev
Retired Dev


Joined: 02 Feb 2003
Posts: 123
Location: Corvallis, OR USA

PostPosted: Mon Aug 04, 2003 6:38 pm    Post subject: Reply with quote

far wrote:
A problem I have found with my script (and probably with the scripts it was derived from) is that if an installed package no longer has a corresponding ebuild in portage, the script won't know which tarballs are associated with it. Therefore, they will get erased even though there may exist a newer ebuild which uses the same tarball.


Hence the reason I believe that it's better to only remove distfiles that are truly dead (no longer used by anything in portage).
This is what my one-liner does, which I'm reposting because I've improved it to auto-detect your portdir/distdir/overlays using 'emerge info.'

Code:
bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|")
Back to top
View user's profile Send private message
patrickfo
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jun 2002
Posts: 79
Location: France

PostPosted: Mon Aug 04, 2003 11:18 pm    Post subject: if this can help just two bash scripts that i use ... Reply with quote

i have 3 adsl gateway ( ipsec ). behind each there is two or more pc runing gentoo.
i choose to keep the sources files for each domain on a samba directory which is mounted at boot for every station.
just for the fun:
on the samba server in smb.conf :
Code:
...
[distfiles]
  comment = no comment ;)
  path = /my_sources_files_dir
  valid users = root
  read only = no
  create mask = 0700
  browseable = no

on each station:
a file ( smcred ) somewhere in etc:
Code:
username = root
password = afunnypassword

Code:
smbpassword -a root

you enter here your password (not necessary to be the same as root login!)
for security reasons:
Code:
chmod 0700 smcred

next i put in /etc/fstab
Code:
//servername/distfiles  /mnt/distfiles  smbfs credentials=/etc/smcred  0 0

then in /etc/make.conf on each station
Code:
DISTDIR=/mnt/distfiles

needless to say it is better to cp your files from your original distfile dir on the samba server BEFORE running unused_sf.sh, because missing files will be downloaded!!!

and now the scripts:
unused_sf.sh : create a list of sources files needed by each station in a directory named .cleandf/stationname/ which i put on the samba server in the distdir. ( i have modified the script that it can handle a single station with no distdir defined and tesded it just only once so be carefull!!!)
this script show also the packages that are installed and no mode in the portage tree, and download also files that are needed by installed packages and are missing ( i have found some patches for xfree more recent than te ones used for the original build on one or two stations and some other rewrited ( i suppose ) between the time of my compil and the time i run the script exemple: foomatic 3.0 )

expunge_sf.sh : list all the files and show the ones that are no more needed by any station and, file by file you can move them in a dir that you can later delete.
i am not a bash specialist, so be carefull! and also these scripts are very intensive time and resources consuming... and now the scripts
----------------
Code:
#!/bin/bash
# unusedsf: Find unused sources files in DISTDIR
   
### erreurs possibles
RPT_IS_NOT_A_DIR=65
UNKNOWN_OPTION=66

### mes variables

# dir in which we (euh... I ) put the files...
RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2 )

if [ ${#RPT} != 0 ]; then
   RPT="$RPT/.cleandf/$HOSTNAME"
else
   RPT="$HOME/.cleandf/$HOSTNAME"
fi

### gestion des options...

while getopts :d: OPTION ;
   do
      case $OPTION in
      d)
         RPT=$OPTARG
         if [[ -d $RPT ]]; then
            echo ">>> dir of files produced changed to $RPT!"
         fi
         ;;
      *)
         echo -e "\n!!! option $OPTARG unknown!\n"
         echo ">>> usage: $0 [-d <dir>]"
         echo -e "     -d directory\n"
         exit $UNKNOWN_OPTION
         ;;
      esac;
done

# creating DISTDIR/.cleandf/HOSTNAME if not exist
if [[ -e $RPT ]]; then
   if [[ -d $RPT ]]; then
      echo "$RPT found."
   else
      echo "$RPT must be a directory!"
      exit $RPT_IS_NOT_A_DIR
   fi
else
   mkdir -p $RPT
   echo "$RPT created."
fi

### do some cleaning...
for FICH in emerge_list qpkg qpkgV ;
   do
      if [[ -f $RPT/$FICH.txt ]]; then
         rm -f $RPT/$FICH.txt
      fi;
done

### création des fichiers de travail

echo -en "doing \"qpkg -nc -I\" ...\t\t"
qpkg -nc -I | sort > $RPT/qpkg.txt
echo "ok."
echo -en "doing \"qpkg -nc -I -v\" ...\t"
qpkg -nc -I -v | sort > $RPT/qpkgV.txt
echo "ok."
echo

for PACKAGE in $( cat $RPT/qpkg.txt );
   do
   for VERSION in $( grep $PACKAGE- $RPT/qpkgV.txt | cut -d '/' -f2 );
      do
      echo -e ">>> $PACKAGE/$VERSION"
      emerge --nospinner -f $PACKAGE/$VERSION.ebuild > $RPT/.temp
      if [ $? != 0 ]; then
      #packages not in portage tree, out of sync ( fixpackages!! )
         echo ">>> $PACKAGE/$VERSION output:" >> $RPT/ERREURS.txt
         cat $RPT/.temp >> $RPT/ERREURS.txt
         echo "============" >> $RPT/ERREURS.txt
      else
      #package that have the .ebuild file modif ( xfree-4.3. !!... )
         grep -i download $RPT/.temp 2>&1 >/dev/null
         if [ $? = 0 ]; then
            echo ">>> $PACKAGE/$VERSION should be rebuild:" >> $RPT/to_REBUILD.txt
            grep -i download $RPT/.temp >> $RPT/to_REBUILD.txt
            echo "============" >> $RPT/to_REBUILD.txt
         fi
      fi;
      cat $RPT/.temp >> $RPT/emerge_list.txt
   done ;
done


# reporting problems
clear
if [ -e $RPT/to_REBUILD.txt ]; then
   echo "Packages that should be rebuild:"
   cat $RPT/to_REBUILD.txt
   echo
else
   echo "Well, nothing to rebuild... ;-)"
fi

if [ -e $RPT/ERREURS.txt ]; then
   echo "Actions must be done on:"
   cat $RPT/ERREURS.txt
   echo
else
   echo "Very well, no errors found... ;-)"
fi
================
Code:
#!/bin/bash
# expunge_sf.sh: remove unecessaries files from DISTDIR

NOTHING_TO_DO=65
SAUVE_NOT_A_DIR=66

RPT=$(grep ^DISTDIR /etc/make.conf | cut -d "=" -f2)
SAUVE="/sauve/old_distfiles"
if [ ${#RPT} != 0 ]; then
   DISTDIR=$RPT
   RPT="$RPT/.cleandf"
else
   RPT="$HOME/.cleandf"
   DISTDIR="/usr/portage/distfiles"
fi

### gestion des options...

while getopts :d:s: OPTION ;
   do
      case $OPTION in
      d)
         RPT=$OPTARG
         ;;
      s)
         SAUVE=$OPTARG
         ;;
      *)
         echo -e "\n!!! option $OPTARG unknown!\n"
         echo ">>> usage: $0 [-d <dir>] -s <dir>"
         echo -e "     -d directory to put temporary files\n"
         echo -e "     -s directory to store backups\n"
         exit $UNKNOWN_OPTION
         ;;
      esac;
done

if [ -e $RPT ]; then
   echo "$RPT found."
else
   echo "You must run unused_sf BEFORE expunge_sf!!!"
   exit $NOTHING_TO_DO
fi
   
if [[ ( -e $SAUVE ) && ( -d $SAUVE ) ]]; then
   echo "$SAUVE found."
else   
   echo "You must supply a dir for backup files!!!"
   exit $SAUVE_NOT_A_DIR
fi

# some cleaning from previous use...

if [ -f $RPT/fichiers_inutiles.txt ]; then
   rm -rf $RPT/fichiers_inutiles.txt
fi

if [ -f $RPT/tous_fichiers.txt ]; then
   rm -rf $RPT/tous_fichiers.txt
fi;

# list of files in DISTDIR

station=$(ls $RPT)

for FICH in $(ls $DISTDIR --ignore="cvs-src") ;
   do
   echo $FICH
   used=0
   for ordi in $station;
      do
      grep $FICH $RPT/$ordi/emerge_list.txt >/dev/null 2>&1
      result=$?
      if [ $result = 0 ]; then
         used=1
      fi;

   done
   if [ $used -eq 1 ]; then
      echo "     $FICH" >> $RPT/tous_fichiers.txt
   else
      echo $FICH >> $RPT/fichiers_inutiles.txt
      echo "dead $FICH" >> $RPT/tous_fichiers.txt
   fi;
done

if [[ !( -e $RPT/fichiers_inutiles.txt) ]]; then
   echo "no files to move :( "
   exit 0
fi

resultat="hit control-C to break"

for FICH in $(cat $RPT/fichiers_inutiles.txt);
   do
   clear
   echo -e "$resultat\n"
   grep -A 15 -B3 --colour $FICH $RPT/tous_fichiers.txt

   echo -e "\n$FICH"
   echo -en "move $FICH to $SAUVE/: ? (y/n): \t"
   read reponse
   if [[ $reponse = "y" || $reponse = "Y" ]]; then
      mv $DISTDIR/$FICH $SAUVE/
      resultat="$FICH moved"
   else
      resultat="$FICH remain"
   fi;
done
============
good luck with them
any comment to ameliorate will be welcome, and injury must be directed to /dev/null ;-)))
Patrick
PS: the indent is gone away... try to recreat it or say me and i will send you this files...


Last edited by patrickfo on Fri Aug 08, 2003 9:54 pm; edited 2 times in total
Back to top
View user's profile Send private message
esprout
n00b
n00b


Joined: 09 Jan 2003
Posts: 22

PostPosted: Fri Aug 08, 2003 7:21 am    Post subject: Script for interactively processing PKGDIR Reply with quote

Ok, I figure I'll throw my script into the foray.

This one interactively processes the PKGDIR directory for old versions. Just run it without any arguments, and it will prompt you before deleting each file.

It'd be nice to have one script to process both PKGDIR and DISTDIR. Or maybe one of them already do, I missed it? So many scripts now.

I stole a couple lines/ideas from previous scripts, if something looks familiar :D
Code:

#!/usr/bin/perl

# Root check
if ($< != 0) {
        print "You must be root to run this script.\n";
        exit 0;
}

my $DISTDIR;
my $PKGDIR;

# Figure out Portage Directories
my @emergeinfo = `emerge info`;
foreach my $info (@emergeinfo) {
        if ($info =~ m/^DISTDIR=/) {
                ($DISTDIR=$info) =~ s/^DISTDIR="(.*)"\n$/$1/;
        } elsif ($info =~ m/^PKGDIR=/) {
                ($PKGDIR=$info) =~ s/^PKGDIR="(.*)"\n$/$1/;
        }
}

print "Processing files in $PKGDIR...\n";

# Process files in $PKGDIR
chdir("$PKGDIR");
my $old;
my $new;
my $candidate;

# Generate list of .tbz2 files to process
@FILES = `find . -name \*.tbz2 | sort`;

# Process list of files
foreach (@FILES) {
        # Only process symlinks to reduce user interaction
        next if ($_ =~ m|^./All/|);
        $_ =~ s|\n$||;
        $new = $_;
        $new =~ s|(\./.+/.+)-[\d]+.*|$1|;
        #print "new: $_\n";
        if ($old) {
                # new files might be same as old file
                if ("$old" eq "$new") {
                        my $input;
                        $candidate =~ s|.tbz2$||;
                        # strip .tbz2 extension and use perl sort
                        # to do a better guess of which files is older
                        $_ =~ s|.tbz2$||;
                        ($candidate, $_) = sort($candidate, $_);
                        # reappend .tbz2 extension
                        $candidate .= ".tbz2";
                        $_ .= ".tbz2";
                        #print "dup: $old, $new, $candidate\n";
                        # Figure out corresponding file in All directory
                        (my $allcandidate = $candidate) =~ s|^./.*/|./All/|;
                        # Ask user to confirm deletion
                        print "rm:   $allcandidate and\n      $candidate\nkeep: $_ [y(es)/n(o)/s(kip)]? : ";
                        ($input = (<STDIN>)) =~ s|\n$||;
                        # Delete away
                        if ("$input" eq "y") {
                                unlink($candidate, $allcandidate);
                                print "$candidate, $allcandidate deleted\n";
                                $old = $new;
                                $candidate = $_;
                        # Ask user for swapped deletion
                        } elsif ("$input" eq "n") {
                                (my $allcandidate = $_) =~ s|^./.*/|./All/|;
                                print "rm:   $allcandidate and\n      $_\nkeep: $candidate [y(es)/n(o)]? : ";
                                ($input = (<STDIN>)) =~ s|\n$||;
                                if ("$input" eq "y") {
                                        unlink($_, $allcandidate);
                                        print "$_, $allcandidate delete\n";
                                } else {
                                        $old = $new;
                                        $candidate = $_;
                                }
                        }
                } else {
                        #print "nodup: $old, $new, $candidate\n";
                        $old = $new;
                        $candidate = $_;
                }
        } else {
                #print "first round: $old, $new, $candidate\n";
                $old = $new;
                $candidate = $_;
        }
}


Last edited by esprout on Wed Aug 13, 2003 4:54 am; edited 1 time in total
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Fri Aug 08, 2003 5:39 pm    Post subject: Reply with quote

Please put your source code inside BBCode [code] blocks like so:

[code]10 PRINT "Hello, World!"[/code]

If you don't, you will lose the indentation and the code will be difficult to read.
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
patrickfo
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jun 2002
Posts: 79
Location: France

PostPosted: Mon Aug 11, 2003 6:00 am    Post subject: thank you Reply with quote

far wrote:
Please put your source code inside BBCode [code] blocks like so:

[code]10 PRINT "Hello, World!"[/code]

If you don't, you will lose the indentation and the code will be difficult to read.


thank you
i have corrected my post
Back to top
View user's profile Send private message
raptor
Apprentice
Apprentice


Joined: 20 Sep 2002
Posts: 171

PostPosted: Wed Aug 20, 2003 1:43 pm    Post subject: Size matters :") Reply with quote

small chages to original to show the old files size and total size that will be saved after stuff are deleted ...
example......
Would delete: ttmkfdir-3.0.4 [19 kB] in favour of ttmkfdir-3.0.9
Would delete: usbutils-0.11 [216 kB] in favour of usbutils-0.9
Would delete: vte-0.10.25 [759 kB] in favour of vte-0.10.26
Would delete: xscreensaver-4.05 [3133 kB] in favour of xscreensaver-4.06
Would delete: yelp-1.0.6 [478 kB] in favour of yelp-2.2.0
Would delete: zapping-0.6.4 [1058 kB] in favour of zapping-0.6.5
Would delete: zvbi-0.2.1 [486 kB] in favour of zvbi-0.2.3
Total size that will be reclaimed : 450038 kB

wow 450 MB not bad :")
===================================
#!/usr/bin/perl
use strict;

my $lastname = 0;
my $lastversion;
my $lastext;
my $totalSize;
my $lvSize;
my @stalefiles;
my @files;
........
if ($lastname && $name eq $lastname) {
if ($version gt $lastversion) {
$lvSize = (stat ("/usr/portage/distfiles/$name$version$lastext"))[7];
push (@stalefiles, [$name, $version, $lastversion, $lastext,$lvSize]);
}
}
......................

..........................
else
{
# Safe mode (default)!
print "\7*Pretend* mode, will only pretend to delete files.\nTo actually delete the files, reinvoke with the --nopretend parameter.\n\n";
foreach (@stalefiles) {
my ($name, $version, $lastversion, $ext, $lvSize) = @{$_};
$totalSize += $lvSize;
$lvSize = sprintf "%.2u kB", $lvSize/1024;
print "Would delete: $name$lastversion [$lvSize] in favour of $name$version\n";
}

print "Total size that will be reclaimed : " . (sprintf "%.2u kB", $totalSize/1024) . "\n";
}
1;
Back to top
View user's profile Send private message
SavageMindz
Tux's lil' helper
Tux's lil' helper


Joined: 28 Oct 2002
Posts: 87
Location: The bit of hell that has frozen over.

PostPosted: Sun Nov 23, 2003 7:01 am    Post subject: Reply with quote

erik.swanson wrote:
This is a "one-liner" I wrote to safely clean out obsolete distfiles that are no longer referenced by any ebuild in portage.


nice one liner, just cleared me out about 4gb or old distfiles, that was quite sorely needed :D . Modified your command so I could get a list of what it got rid of and it seems to have worked a charm.

cheers

Savage
Back to top
View user's profile Send private message
cbradney
Apprentice
Apprentice


Joined: 08 May 2003
Posts: 245
Location: Luxembourg

PostPosted: Tue Dec 16, 2003 6:09 pm    Post subject: Reply with quote

Got this one from Kerframil:
Code:

#!/usr/bin/env python
## distclean.py version 0.1 (20 Aug 2003,
## This is the first version with a version number)
##
## Removes source files for Gentoo
## packages that are no longer installed
## Use with '-p' (pretend) flag to just get a list of files
## that would be removed
##
## Copyright (c) 2003, Fredrik Arnerup (e97_far@e.kth.se)
## All rights reserved.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted provided that the following conditions are met:
##
##   * Redistributions of source code must retain the above copyright notice,
##     this list of conditions and the following disclaimer.
##
##   * Redistributions in binary form must reproduce the above copyright
##     notice, this list of conditions and the following disclaimer in the
##     documentation and/or other materials provided with the distribution.
##
## THIS SOFTWARE IS PROVIDED BY FREDRIK ARNERUP "AS IS" AND ANY
## EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
## WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
## DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
## FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
## DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
## SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
## CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
## LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
## OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
## DAMAGE.

import sys, os, os.path, getopt, portage

opt_p = 0
try:
    if len(sys.argv) > 1:
        opt_p = getopt.getopt(sys.argv[1:], 'p')[0][0][0] == '-p'
except getopt.GetoptError:
    pass

distdir = portage.config().environ()['DISTDIR']
print 'DISTDIR =', distdir
   
vartree = portage.db['/']['vartree']
packages = []
for name in vartree.getallnodes():
    packages.extend(vartree.dep_match(name))

files = {}
for package in packages:
    try:
        package_files = portage.portdb.aux_get(package, ['SRC_URI'])[0].split()
        package_files = [(url.split('/')[-1]) for url in package_files]
        for filename in package_files:
            files[filename] = 1
    except:
        print 'Failed to get file list for', package

if not files:
    sys.exit("No package files found.  This can't be right.\n")

try:
    list = portage.listdir(distdir)
except os.OSError:
    sys.exit('Failed to read ' + distdir)

size = 0; count = 0
for file in list:
    abs_file = distdir + '/' + file
    if (os.path.isfile(abs_file) and (not os.path.islink(abs_file))
        and (not file in files)):
        size += os.stat(abs_file).st_size
        count += 1
        if opt_p:
            print 'Would remove', abs_file
        else:
            try:
                os.remove(abs_file)
                print 'Removed', abs_file
            except OSError:
                print 'Failed to remove', abs_file


size /= 1048576  ## MB
print '%i files, total size: %i MB' % (count, size)
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Tue Dec 16, 2003 6:58 pm    Post subject: Reply with quote

Yes, that is the script from my webpage ...
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
cbradney
Apprentice
Apprentice


Joined: 08 May 2003
Posts: 245
Location: Luxembourg

PostPosted: Tue Dec 16, 2003 7:37 pm    Post subject: Reply with quote

Works wonderfully! Thank you!
Back to top
View user's profile Send private message
raptor
Apprentice
Apprentice


Joined: 20 Sep 2002
Posts: 171

PostPosted: Sun Jan 11, 2004 1:36 pm    Post subject: Re: mod for PORTAGE_OVERLAY, etc. Reply with quote

thanx for the scripts "robm" and others.. a small modification at the end of the script :
Now it will not delete the files but will move them in /usr/portage/olddist. This way we can restore if something goes wrong, and delete only when nececary....

here are the results :

Code:

$du -shc /usr/portage/distfiles/
2.6G    /usr/portage/distfiles
2.6G    total

$ ./distbuster
.....

$ du -shc /usr/portage/distfiles/
599M    /usr/portage/distfiles
599M    total
$ du -shc /usr/portage/olddist
2.0G    /usr/portage/olddist
2.0G    total




not bad...

Code:

opendir(DIR,$makeconf{"DISTDIR"})|| die "can't open ".$makeconf{"DISTDIR"};
#new line
mkdir "/usr/portage/olddist";
while ($file = readdir DIR) {
  next unless -f $makeconf{"DISTDIR"}."/$file";
  if  (! $files{$file}) {
    if ($opt_p) {
#      print "Would erase ".$makeconf{"DISTDIR"}."/$file\n";
      print "Would move  $makeconf{DISTDIR}/$file -> /usr/portage/olddist \n";
    }
    else {
#      unlink $makeconf{"DISTDIR"}."/$file";
       rename "$makeconf{DISTDIR}/$file", "/usr/portage/olddist/$file";
    }
  }
}
Back to top
View user's profile Send private message
raptor
Apprentice
Apprentice


Joined: 20 Sep 2002
Posts: 171

PostPosted: Sun Jan 11, 2004 1:47 pm    Post subject: not so good Reply with quote

As I used this script I found one big problem atleast for me...
I usualy use several gentoo machines and from time i'm rsync-ing their distfiles... Now the problem is that after I ran the modified-robm script all files that do not belong to packages i have installed are deleted/moved which is BAD.
The whole purpose for rsync-ing is that i do not need to dload files which are already dloaded on other machines...

Is there a way to preserve files that do not belong to already installed packges...!?!

i will apricate any help...

tia
Back to top
View user's profile Send private message
wolf31o2
Retired Dev
Retired Dev


Joined: 31 Jan 2003
Posts: 628
Location: Mountain View, CA

PostPosted: Mon Jan 26, 2004 6:04 pm    Post subject: Reply with quote

Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.

Now, if only it did PORT_OVERLAY also and was in python, it would be perfect. In fact, it would be good enough that I think a bug would definitely need to be opened for the portage team to include it in a future version of portage.
_________________
Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Mon Jan 26, 2004 8:20 pm    Post subject: Reply with quote

wolf31o2 wrote:
Use eric.swanson's one-liner, as it keeps files from ANY ebuild still in portage.

Now, if only it did PORT_OVERLAY also and was in python, it would be perfect.

We aim for perfection :-)
Code:
#!/usr/bin/env python

import os, os.path, portage

env = portage.config().environ()
distdir = env['DISTDIR']
portdir = env['PORTDIR']
overlay = env.get('PORTDIR_OVERLAY', '')
distfiles = portage.listdir(distdir)
keep = []

def visit(spam, dir, files):
    for file in files:
        if 'digest-' in file:
            path = os.path.join(dir, file)
            if not os.path.isfile(path): continue
            for line in open(path).readlines():
                try: keep.append(line.split()[2])
                except: pass

os.path.walk(portdir, visit, None)
if overlay: os.path.walk(overlay, visit, None)
remove = []
for file in distfiles:
    if file not in keep: remove.append(file)
for file in remove:
    file = os.path.join(distdir, file)
    if os.path.isfile(file):
        print file

Note: it only prints a list of files that you may remove. It is pretty slow.
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
wolf31o2
Retired Dev
Retired Dev


Joined: 31 Jan 2003
Posts: 628
Location: Mountain View, CA

PostPosted: Mon Jan 26, 2004 9:25 pm    Post subject: Reply with quote

Sweet... what would I have to change on the "print file" line to make it actually remove them?

Sorry, I know nothing of python.
_________________
Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org
Back to top
View user's profile Send private message
far
Guru
Guru


Joined: 10 Mar 2003
Posts: 394
Location: Stockholm, Sweden

PostPosted: Mon Jan 26, 2004 9:37 pm    Post subject: Reply with quote

wolf31o2 wrote:
Sweet... what would I have to change on the "print file" line to make it actually remove them?

Change it to "os.remove(file)"
_________________
The Porthole Portage Frontend
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Tue Jan 27, 2004 12:02 am    Post subject: Reply with quote

/me also likes this script, thanks. I took the permission to modify it a bit so that it uses the /var/cache/edb/dep files instead of the digests, which is faster. But it should give the same result in general, allthough there are some theoretical cases where it would not (for instance if you add an ebuild in your overlay, and then manually add its distfiles, and then don't use the ebuild, and then launch the cleanup script before your next sync, then your distfiles won't be referenced in the cache... well, who cares about such bugs?).
Code:
EDIT: removed the old code, see below for an updated version.

Also, I think you had a bug in your version in case of multiple overlays (they are allowed, at least in the .50_pre serie), but this would be easy to fix (for ov in string.split(overlay): ...).


Last edited by TGL on Wed Jan 28, 2004 9:00 pm; edited 1 time in total
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Wed Jan 28, 2004 8:58 pm    Post subject: Reply with quote

Added some colors to give it the Gentoo's touch. For cron jobs, this ugliness can be disable with the "-q" option. Also, if all you want is the raw list of files to delete (to pipe it to something else for instance), use "-p -q". Otherwise, a simple "-p" first to check that everything is ok, and then no option to actually clean the files.
Code:

EDIT: added --ask (-a) for per-file confirmation. Default is still to blindly delete everything.
EDIT 2: removed old code. You can now find the script in bug #33877... no need to make this thread longer than it already is.


Last edited by TGL on Mon Feb 09, 2004 11:22 pm; edited 2 times in total
Back to top
View user's profile Send private message
gebner
Tux's lil' helper
Tux's lil' helper


Joined: 01 Jan 2004
Posts: 100
Location: Wien, EU

PostPosted: Thu Feb 05, 2004 2:14 pm    Post subject: Reply with quote

erik.swanson wrote:
Code:
bash <(comm -23 <(find $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p') -maxdepth 1 -type f -printf %f\\n | sort) <(find $(emerge info | sed -n -e 's:PORTDIR.*="\(.*\)":\1/:p') -path '*/*/files/digest-*' | xargs awk '{print $3}' | sort -u) | sed -e "s|^|rm -i $(emerge info | sed -n -e 's:DISTDIR="\(.*\)":\1/:p')|")


Mine does practically the same, but is 5 secs faster :):
Code:
(eval export `emerge info | egrep 'PORTDIR|DISTDIR'`; find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' | xargs -n10 cut -d' ' -f3 | perl -ne 'chomp;$f{$_}=1;END{$d=$ENV{DISTDIR};opendir$h,$d; while($f=readdir$h){print"$d/$f\n"if!$f{$f}&&!-d"$d/$f"}}')


This shows only the files that are no longer necessary, if you want to actually delete them you'd have to append a '| xargs rm'.

Or in a bit more readable fashion (as I have it in /usr/local/bin/clean-distfiles):
Code:
#!/bin/sh
eval export `emerge info | egrep 'PORTDIR|DISTDIR'`

find $PORTDIR $PORTDIR_OVERLAY -path '*/*/files/digest-*' \
   | xargs -n10 cut -d' ' -f3 \
   | perl -ne 'chomp;
      $f{$_} = 1;
      END{
         $d = $ENV{DISTDIR};
         opendir $h, $d;
         while($f = readdir $h){
            print "$d/$f\n" if not $f{$f} and not -d "$d/$f";
         }
      }' \
   | if [ "$1" = "-d" ]; then
      xargs rm -v
   else
      cat
   fi


---
ChangeLog:
  • 1.1 added pretty-printed version and corrected old one not to delete directories
  • 1.0 initial post

_________________
clue = if 0 == 0 then clue else error "The poster didn't have any clue when he posted this."
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Fri Feb 06, 2004 1:13 am    Post subject: Reply with quote

Not bad, but still 8 times slower than mine :p
Back to top
View user's profile Send private message
wolf31o2
Retired Dev
Retired Dev


Joined: 31 Jan 2003
Posts: 628
Location: Mountain View, CA

PostPosted: Sun Feb 08, 2004 7:51 pm    Post subject: Reply with quote

TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit?
_________________
Ex-Gentoo Developer
Catalyst/Genkernel Development Lead
http://wolf31o2.org
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Sun Feb 08, 2004 8:01 pm    Post subject: Reply with quote

wolf31o2 wrote:
TGL: Have you considered submitting this to the portage team? Or at the very least to karltk for inclusion in gentoolkit?

I first thought i should ask karltk if he would be interrested, and then forgot about it. Thanks for the reminder, goto #gentoo-dev. :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Page 3 of 8

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum