Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cleaning out stale distfiles
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3432
Location: Gainesville, Florida

PostPosted: Tue Feb 15, 2005 10:41 pm    Post subject: Reply with quote

I just downloaded the latest version of distcleaner (I had used ealier versions before), and while it works great, I still get these errors. Is this normal? Is there any way to fix them? This is one one of my ~x86 installs. Where does the getfetchlist ebuild path not specified error come from? Could it be that the specific tar.bz2 file isn't found in distfiles, but it is still listed in another master list file somewhere?

Also, woundn't it be prudent not to run distclean without the -p option yet, seeing as how after an emerge -uD system I just updated glib, glibc, and xorg-x11? I was thinking waiting until the system had survived at least a reboot and another round of compiling first. I'm having no problems as of right before this most recent uD system. I've got a slow connection, so I don't want to have to redownload things if I need to revert.

Here's some of my info:
mymachine wrc # emerge --info
Portage 2.0.51-r15 (default-linux/x86/2004.0, gcc-3.4.3, glibc-2.3.4.20050125-r0, 2.6.10 i686)
=================================================================
System uname: 2.6.10 i686 AMD Athlon(tm) XP 1800+
Gentoo Base System version 1.6.9
Python: dev-lang/python-2.2.3-r1,dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb 10 2005, 18:45:11)]
dev-lang/python: 2.2.3-r1, 2.3.4-r1
sys-devel/autoconf: 2.59-r6, 2.13
sys-devel/automake: 1.7.9-r1, 1.8.5-r3, 1.5, 1.4_p6, 1.6.3, 1.9.4
sys-devel/binutils: 2.15.92.0.2-r2
sys-devel/libtool: 1.5.10-r4
virtual/os-headers: 2.4.19, 2.4.22-r1
ACCEPT_KEYWORDS="x86 ~x86"

----------------------------------------------------------------------------------------------------------
mymachine wrc # cd /home/wrc/distcleaner
mymachine distcleaner # ./distcleaner -p -a not_installed
error! can't found SRC_URI for app-admin/addpatches-0.2
!!! aux_get(): ebuild path for 'app-crypt/cryptplug-0.3.16' not specified:
!!! None
getfetchlist(): aux_get() error reading app-crypt/cryptplug-0.3.16; aborting.
!!! aux_get(): ebuild path for 'app-editors/gedit-2.6.1' not specified:
!!! None
getfetchlist(): aux_get() error reading app-editors/gedit-2.6.1; aborting.
!!! aux_get(): ebuild path for 'app-text/aspell-0.50.5' not specified:
!!! None
getfetchlist(): aux_get() error reading app-text/aspell-0.50.5; aborting.
!!! aux_get(): ebuild path for 'app-text/gpdf-0.131' not specified:
!!! None
getfetchlist(): aux_get() error reading app-text/gpdf-0.131; aborting.
!!! aux_get(): ebuild path for 'dev-java/blackdown-jdk-1.4.1' not specified:
!!! None
getfetchlist(): aux_get() error reading dev-java/blackdown-jdk-1.4.1; aborting.
!!! aux_get(): ebuild path for 'dev-lang/python-2.2.3-r1' not specified:
!!! None
getfetchlist(): aux_get() error reading dev-lang/python-2.2.3-r1; aborting.
!!! aux_get(): ebuild path for 'dev-libs/glib-2.6.2' not specified:
!!! None
getfetchlist(): aux_get() error reading dev-libs/glib-2.6.2; aborting.
!!! aux_get(): ebuild path for 'dev-libs/atk-1.2.3' not specified:
!!! None
getfetchlist(): aux_get() error reading dev-libs/atk-1.2.3; aborting.
error! can't found SRC_URI for kde-base/kde-3.2.3
error! can't found SRC_URI for kde-base/kde-3.3.2
!!! aux_get(): ebuild path for 'kde-base/kdegraphics-3.2.3' not specified:
!!! None
getfetchlist(): aux_get() error reading kde-base/kdegraphics-3.2.3; aborting.
error! can't found SRC_URI for kde-base/kde-env-3-r3
!!! aux_get(): ebuild path for 'kde-base/kdelibs-3.2.3' not specified:
!!! None
getfetchlist(): aux_get() error reading kde-base/kdelibs-3.2.3; aborting.
!!! aux_get(): ebuild path for 'kde-base/kdebase-3.2.3' not specified:
!!! None
getfetchlist(): aux_get() error reading kde-base/kdebase-3.2.3; aborting.
!!! aux_get(): ebuild path for 'media-libs/xine-lib-1_rc5-r2' not specified:
!!! None
getfetchlist(): aux_get() error reading media-libs/xine-lib-1_rc5-r2; aborting.
!!! aux_get(): ebuild path for 'media-libs/alsa-driver-1.0.4-r1' not specified:
!!! None
getfetchlist(): aux_get() error reading media-libs/alsa-driver-1.0.4-r1; aborting.
!!! aux_get(): ebuild path for 'media-libs/gdk-pixbuf-0.21.0' not specified:
!!! None
getfetchlist(): aux_get() error reading media-libs/gdk-pixbuf-0.21.0; aborting.
!!! aux_get(): ebuild path for 'media-sound/xmms-1.2.10-r10' not specified:
!!! None
getfetchlist(): aux_get() error reading media-sound/xmms-1.2.10-r10; aborting.
!!! aux_get(): ebuild path for 'net-dialup/ppp-2.4.2-r2' not specified:
!!! None
getfetchlist(): aux_get() error reading net-dialup/ppp-2.4.2-r2; aborting.
!!! aux_get(): ebuild path for 'net-ftp/gftp-2.0.18' not specified:
!!! None
getfetchlist(): aux_get() error reading net-ftp/gftp-2.0.18; aborting.
error! can't found SRC_URI for net-mail/mailbase-0.00-r6
!!! aux_get(): ebuild path for 'net-www/epiphany-1.2.5' not specified:
!!! None
getfetchlist(): aux_get() error reading net-www/epiphany-1.2.5; aborting.
error! can't found SRC_URI for sys-apps/cronbase-0.3.1
error! can't found SRC_URI for sys-devel/autoconf-wrapper-2-r1
error! can't found SRC_URI for sys-devel/binutils-config-1.8
error! can't found SRC_URI for sys-devel/automake-wrapper-1-r1
error! can't found SRC_URI for sys-devel/gcc-config-1.3.10-r1
!!! aux_get(): ebuild path for 'sys-kernel/linux-headers-2.4.19' not specified:
!!! None
getfetchlist(): aux_get() error reading sys-kernel/linux-headers-2.4.19; aborting.
!!! aux_get(): ebuild path for 'sys-kernel/gentoo-sources-2.4.20-r5' not specified:
!!! None
getfetchlist(): aux_get() error reading sys-kernel/gentoo-sources-2.4.20-r5; aborting.
!!! aux_get(): ebuild path for 'sys-libs/db-3.2.9-r2' not specified:
!!! None
getfetchlist(): aux_get() error reading sys-libs/db-3.2.9-r2; aborting.
!!! aux_get(): ebuild path for 'x11-libs/pango-1.2.1-r1' not specified:
!!! None
getfetchlist(): aux_get() error reading x11-libs/pango-1.2.1-r1; aborting.
error! can't found SRC_URI for xfce-base/xfce4-4.2.0
gpm-1.20.1-patches-1.1.tar.bz2 [7 kB]
gftp-2.0.17.tar.bz2 [1199 kB]
xorg-x11-6.8.1.902.tar.bz2 [43103 kB]
glibc-2.3.4-branch-update-20041102.patch.bz2 [2963 kB]
xorg-x11-6.8.1.904-patches-0.1.3.tar.bz2 [156 kB]
glibc-manpages-2.3.4.tar.bz2 [14 kB]
pam-0.77-patches-1.4.tar.bz2 [112 kB]
glib-2.6.2.tar.bz2 [2288 kB]
xorg-x11-6.8.1.904.tar.bz2 [43105 kB]
shadow-4.0.6.tar.bz2 [992 kB]
gentoo_ice-xmms-0.2.tar.bz2 [50 kB]
glibc-2.3.3.tar.bz2 [12986 kB]
xorg-x11-6.8.1.904-files-0.2.tar.bz2 [9 kB]
gtk+-2.6.1.tar.bz2 [10811 kB]
e2fsprogs-1.35.tar.gz [3078 kB]
gftp-2.0.18.tar.bz2 [1342 kB]
glext.h-20040830.bz2 [30 kB]
net-tools-1.60-gentoo-extra-1.tar.bz2 [10 kB]
xmms-1.2.10.tar.bz2 [2361 kB]
gpm-1.20.1-patches-1.2.tar.bz2 [7 kB]
Mail-SpamAssassin-3.0.2.tar.bz2 [826 kB]
glibc-infopages-2.3.4.tar.bz2 [1147 kB]
glib-2.6.1.tar.bz2 [2199 kB]
gnomexmms.xpm [7 kB]

Total size saved: 128815 kB (24 files)
mymachine distcleaner #
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-13.2.1_p20230304
kernel-6.7.2 USE=experimental python3_11
Back to top
View user's profile Send private message
sandymc
n00b
n00b


Joined: 17 Aug 2003
Posts: 2

PostPosted: Wed Mar 02, 2005 2:05 pm    Post subject: Reply with quote

This is yet another tool for cleaning and maintaining the distfiles directory. Although there are others that are quicker and smaller in size, this does some things the various other versions on the forum don't do:
  • Keeps all source files mentioned in portage regardless of whether they are installed or not, version numbers, masking, etc
  • Obtains files locations from portage
  • Can move stale files to another directory rather than deleting them
  • Can copy required files from another location to distfiles, e.g., if you only have intermittent access to another machine, or want to copy from a DVD, etc
  • Extensive "pretend" ability
  • Written in Python

For instructions, run with -h
Code:

#! /usr/bin/env python
# distmaint.py -- Portage distfiles cleaning and management tool
# Version 0.1
# Copyright 2005 Sandy McGuffog.
# Distributed under the GNU Public License v2
# Comments/bugs to mcguffogl@bigfoot.com

import sys
import getopt
import os
import os.path
import string
import shutil
import glob
import portage

# ===========================================================================
# If Your Portage installation is non-standard, alter these.......
# Note this only aplies if we can't find location from the portage settings
# ===========================================================================
sPortage = '/usr/portage'
sOverlay = '/usr/local/portage'
sDirstFiles = '/usr/portage/distfiles'
sDigestDirectory = '/files'
# ===========================================================================
# End of area to alter.......
# ===========================================================================

# Ugly, nasty global variables.....
bVerbose = False
bPretend = False

def usage():
   print 'distmaint V0.1'
   print 'Usage: distmaint [OPTIONS] or python distmaint [OPTIONS]'
   print 'Scan the portage database for source files required for any and'
   print 'all ebuilds, then either (a) delete orphan (non-required files)'
   print 'from portage/distfiles or (b) move ophan files to another directory'
   print 'or (c) scan for and get required files from another location'
   print
   print 'Mandatory arguments to long options are mandatory for short options too.'
   print '  -d, --delete             delete orphan files in portage/distfiles'
   print '  -h, --help               display this help and exit'
   print '  -v, --verbose            explain what is being done. In (much) detail.'
   print '  -p, --pretend            instead of actually deleteing or moving files,'
   print '                           just display what would be done'
   print '  -n, --number=N           limit the number of files moved/deleted to N'
   print '  -g, --get=DIRECTORY      scan DIRECTORY for required files, and copy'
   print '                           to portage/distfiles if found'
   print '  -m, --move=DIRECTORY     move orphan files to DIRECTORY rather than'
   print '                           delete them'
   print
   print 'Examples:'
   print '  distmaint'
   print '    Scan all ebuild files, build a list of required source files for'
   print '    all ebuilds, and compare that list to what is contained in distfiles'
   print '  distmaint -d'
   print '    Deletes all orphan files in portage/distfiles'
   print '  distmaint -dpn 10'
   print '    Outputs the names of the orphan files in portage/distfiles that would'
   print '    be deleted, limiting the number to 10'
   print '  distmaint -m /home/samba/public -n 10'
   print '    Moves the first 10 orphan files in portage/distfiles to'
   print '    the /home/samba/public directory'
   print '  distmaint -g "/mnt/WinXP/Documents and Settings/All Users/Documents/Programs/Gentoo/distfiles"'
   print '    Copy all files required by any ebuild that are available in the'
   print '    /mnt/WinXP/..... directory to the portage distfiles directory'
   print
   
def IOError(sMessage):
   print '*** File I/O error occured '+sMessage+'; may be a permisioning problem'
   sys.exit(2)

def addFileList(sDirectory, dList):
#   We could us glob.glob('/usr/portage/*/*/files/digest-*') here, it would be
#   quicker, but walk is more robust versus directory structure changes
   for root, dirs, files in os.walk(sDirectory):
      for individualFile in files:
         if individualFile[0:7] == 'digest-':
            try:
               fInputFile = open(os.path.join(root, individualFile), 'r')
               bDone = False
               while not bDone:   
                  sLine = fInputFile.readline()
                  if not sLine:
                     bDone = True
                  else:
                     tokens = string.split(sLine)
                     if len(tokens) == 4 and tokens[0] == 'MD5':
                        if bVerbose:
                           print 'Adding file', tokens[2], 'length', tokens[3], 'from', root[(len(sDirectory)+1):(len(root) - len(sDigestDirectory))], 'to wish list'
                        dList[tokens[2]] = root[(len(sPortage)+1):(len(root) - len(sDigestDirectory))]
            except:
               IOError('reading file '+os.path.join(root, individualFile))

def main():
   #Need to declare these global here, else python just creates local variables instead of using the globals....
   global bVerbose
   global bPretend

   bVerbose = False
   bDelete = False
   bMove = False
   bPretend = False
   bGet = False
   bLimit = False
   iLimit = 1000000000
   iLimitStore = iLimit
   sMoveDir = ''

   try:
      lOpts, lArgs = getopt.getopt(sys.argv[1:], "dhvpn:g:m:", ["help", "delete", "pretend", "number=", "get=", "move="])
   except (getopt.GetoptError):
      # print help information and exit:
      usage()
      sys.exit(2)
   for sOption, sArgument in lOpts:
      if sOption == "-v":
         bVerbose = True
      if sOption in ("-h", "--help"):
         usage()
         sys.exit()
      if sOption in ("-d", "--delete"):
         bDelete = True
      if sOption in ("-p", "--pretend"):
         bPretend = True
      if sOption in ("-g", "--get"):
         bGet = True
         sGetDirectory = sArgument;
         if not os.path.isdir(sGetDirectory):
            print sGetDirectory, 'is not an existing directory'
            usage()
            sys.exit(2)
      if sOption in ("-n", "--number"):
         bLimit = True
         iLimit = int(sArgument);
         iLimitStore = iLimit
      if sOption in ("-m", "--move"):
         bMove = True
         sMoveDir = sArgument;
         if not os.path.isdir(sMoveDir):
            print sMoveDir, 'is not an existing directory'
            usage()
            sys.exit(2)
   
   if bLimit:
      print '*** Limiting number of files moved, copied or deleted to', iLimit
      
   if bDelete and bPretend:
      print '*** Note, pretend overrides delete; no files will be deleted; to delete run without the  -p'
            
   if bMove and bPretend:
      print '*** Note, pretend overrides move; no files will be moved; to move run without the  -p'
      
      
   cPortdir = portage.settings['PORTDIR']
   cDistdir = portage.settings['DISTDIR']
   cOverlaydir = portage.settings['PORTDIR_OVERLAY']
   
   if len(cPortdir) > 2 and len(cDistdir) > 2:
      #assume we got good data from portage
      sPortage = cPortdir
      sOverlay = cOverlaydir
      sDirstFiles = cDistdir
   else:
      print '*** Warning - could not get file locations from Portage - using defaults:'
      print 'PORTDIR =', sPortage
      print 'DISTDIR =', sDirstFiles
      print 'PORTDIR_OVERLAY =', sOverlay

   #sPortage = '/usr/portage/dev-java'
   
   dWishList = {}
   print 'Searching Portage for files to add to the wish list; this may take several minutes.............'
   addFileList(sPortage, dWishList)
   if not sOverlay == '':
      addFileList(sOverlay, dWishList)
   
   print 'Total of', len(dWishList), 'files added to the wishlist'
   
   print 'Searching distfiles for existing files.............'
   lSortedDistFiles = glob.glob(os.path.join(sDirstFiles, '*'))
   lSortedDistFiles.sort()
   
   if bGet:
      print 'Checking for files to get'
      iTotalCopied = 0
      iTotalFiles = 0
      for root, dirs, files in os.walk(sGetDirectory):
         for individualFile in files:
            iTotalFiles = iTotalFiles + 1
            try:
               if (iLimit > 0 or not bLimit) and individualFile in dWishList and not (os.path.join(sDirstFiles, individualFile)) in lSortedDistFiles:
                  iLimit = iLimit - 1
                  if bPretend:
                     try:
                        if os.access(os.path.join(root, individualFile), os.R_OK):
                           (st_mode, st_ino, st_dev, st_nlink, st_uid, st_gid, st_size, st_atime , st_mtime, st_ctime) = os.stat(os.path.join(root, individualFile))
                           iTotalCopied = iTotalCopied + st_size
                     except:
                        IOError('attempting to access file '+sExistingFile)
                     print 'Would copy', os.path.join(root, individualFile), 'to', sDirstFiles, 'for package', dWishList[individualFile]
                  else:
                     #Copy2 preserves dates and times
                     shutil.copy2(os.path.join(root, individualFile), sDirstFiles)
                     print 'Copied', os.path.join(root, individualFile), 'to', sDirstFiles
            except:
               IOError('attempting to copy file '+os.path.join(root, individualFile)+' to '+sDirstFiles)
      if bPretend:
         print iTotalCopied/1024, 'KB in',iLimitStore - iLimit, 'files out of a total of', iTotalFiles,'files considered would be copied from', sGetDirectory
               
   else:
      # Here its either move or delete
      print 'Matching the wish list to the existing files.............'
      iFound = 0
      iNotFound = 0
      iTotalDeleted = 0
      for sExistingFile in lSortedDistFiles:
         (sHead, sTail) = os.path.split(sExistingFile)
         if os.path.isdir(sExistingFile):
            if bVerbose:
               print sTail, 'is a directory - ignoring'
         elif sTail in dWishList:
            if bVerbose:
               print 'Found', sTail, 'in wishlist'
            iFound = iFound+1
         else:
            if bVerbose:
               print 'Did not find', sTail, 'in wishlist'
            if iLimit > 0 or not bLimit:
               iLimit = iLimit - 1
               if bPretend:
                  try:
                     if os.access(sExistingFile, os.R_OK):
                        (st_mode, st_ino, st_dev, st_nlink, st_uid, st_gid, st_size, st_atime , st_mtime, st_ctime) = os.stat(sExistingFile)
                        iTotalDeleted = iTotalDeleted + st_size
                  except:
                     IOError('attempting to access file '+sExistingFile)
                  if bDelete:
                     print 'Would delete', sExistingFile
                  elif bMove:
                     print 'Would move', sExistingFile, 'to', os.path.join(sMoveDir, sTail)
               else:
                  if bDelete:
                     try:
                        os.remove(sExistingFile)
                     except:
                        IOError('attempting to delete file '+sExistingFile)
                     print 'Deleted', sExistingFile
                  elif bMove:
                     try:
                        shutil.move(sExistingFile, os.path.join(sMoveDir, sTail))
                     except:
                        IOError('attempting to move file '+sExistingFile+' to '+os.path.join(sMoveDir, sTail))
                     print 'Moved', sExistingFile, 'to', os.path.join(sMoveDir, sTail)
            iNotFound = iNotFound+1
      
      print 'Found',iFound, 'required files and', iNotFound, 'orphan files'
      if bPretend:
         print iTotalDeleted/1024, 'KB could be freed up from the', iLimitStore - iLimit, 'files considered for deletion or movement'
   print
   
if __name__ == '__main__':
   main()
   
Back to top
View user's profile Send private message
soulwarrior
Guru
Guru


Joined: 21 Oct 2002
Posts: 331

PostPosted: Wed Mar 02, 2005 3:47 pm    Post subject: Reply with quote

sandymc wrote:
This is yet another tool for cleaning and maintaining the distfiles directory. Although there are others that are quicker and smaller in size, this does some things the various other versions on the forum don't do:


Thanks, will try it out :)
This distfiles-clean-tool tool does nicely complement the http-replicator.

Small note: some lines are broken into two as they were too width for the forum I suppose.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3432
Location: Gainesville, Florida

PostPosted: Wed Mar 02, 2005 4:35 pm    Post subject: Reply with quote

I pasted what was in the forum into a kedit file without word-wrap. Are you referring to these 3 lines, or is this incorrect? I'd like to try this out- is there anywhere we can download the correctly formatted file, or should we just try editing these lines in a new text file, as shown below?


print 'Adding file', tokens[2], 'length', tokens[3], 'from', root[(len(sDirectory)+1):(len(root) - len(sDigestDirectory))], 'to wish list'

(st_mode, st_ino, st_dev, st_nlink, st_uid, st_gid, st_size, st_atime , st_mtime, st_ctime) = os.stat(os.path.join(root, individualFile))

print iTotalCopied/1024, 'KB in',iLimitStore - iLimit, 'files out of a total of', iTotalFiles,'files considered would be copied from', sGetDirectory
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-13.2.1_p20230304
kernel-6.7.2 USE=experimental python3_11
Back to top
View user's profile Send private message
gspir2004
n00b
n00b


Joined: 30 Sep 2004
Posts: 8

PostPosted: Wed Mar 02, 2005 10:43 pm    Post subject: why to do it the hard way? ...use the emerge command... Reply with quote

the -f arg of the emerge command can do it easy...
Linux is so flexibale....!!!!!
say that you have 2 computers , a host and a server.the server is updated and full of distfiles and the host wants too be updated from the server and has no distfiles at all.......this was my first idea.then i NANOed the /etc/make.conf.example and the idea hit me!!! Why don't i use the wget command in the fenchonly line to do the job for me(make a directory and copy all the distfiles there,erase the original distfiles in the /usr/portage/distfiles/ and THEN ask the wget command throw make.conf throw emerge command to "download" (way of talking) all the sources i had emerged earllyer,FROM the new distfiles directory that i created (backup way of talking)?
THEN i tryed and i trayed but ....nothing (man wget about parameters to add in make.conf line e.t.c. e.t.c. e.t.c.)

and then an other idea came too me!!!!! (((("""""!!!!!! USE THE .......GENTOO_MIRRORS="<your_mirror_here> http://distfiles.gentoo.org http://www...... YOU A...OLE!!!!!!!!!"""""""""")))))))):lol:

so I DID and IT WORKED!!!!!!!!!!!!!!!!!!!!!!
WHAT A NICE COMMAND >>>>>>>>>>EMERGE<<<<<<<<<<<<
make a dir
copy distfiles
erase original files
and "<your_mirror_here>" ............ the new directory!!!!!!!!!!!!!!!!!
emerge -uDefqv --newuse world (download all sources to update later the world packages from scratch with newuse and deep and update and lot off mesages while download.....e.t.c.)


AND THE output is THIS>>>>>
localhost>emerge -uDefqv --newuse world

.
.
.
.
.
.
.

Local mirror has file: module-init-tools-3.1.tar.bz2
Local mirror has file: modutils-2.4.27.tar.bz2
>>> Previously fetched file: modutils-2.4.27.tar.bz2 size ;-)
>>> Previously fetched file: modutils-2.4.27.tar.bz2 MD5 ;-)
>>> Previously fetched file: module-init-tools-3.1.tar.bz2 size ;-)
>>> Previously fetched file: module-init-tools-3.1.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) module-init-tools-3.1.tar.bz2
>>> md5 src_uri ;-) modutils-2.4.27.tar.bz2
>>> emerge (85 of 92) sys-apps/tcp-wrappers-7.6-r8 to /
Local mirror has file: tcp_wrappers_7.6.tar.gz
Local mirror has file: tcp-wrappers-7.6-r8-patches.tar.bz2
>>> Previously fetched file: tcp-wrappers-7.6-r8-patches.tar.bz2 size ;-)
>>> Previously fetched file: tcp-wrappers-7.6-r8-patches.tar.bz2 MD5 ;-)
>>> Previously fetched file: tcp_wrappers_7.6.tar.gz size ;-)
>>> Previously fetched file: tcp_wrappers_7.6.tar.gz MD5 ;-)
>>> md5 src_uri ;-) tcp_wrappers_7.6.tar.gz
>>> md5 src_uri ;-) tcp-wrappers-7.6-r8-patches.tar.bz2
>>> emerge (86 of 92) app-crypt/mit-krb5-1.3.6-r1 to /
Local mirror has file: krb5-1.3.6-signed.tar
>>> Previously fetched file: krb5-1.3.6-signed.tar size ;-)
>>> Previously fetched file: krb5-1.3.6-signed.tar MD5 ;-)
>>> md5 src_uri ;-) krb5-1.3.6-signed.tar
>>> emerge (87 of 92) net-misc/openssh-3.9_p1-r1 to /
Local mirror has file: openssh-3.9p1.tar.gz
>>> Previously fetched file: openssh-3.9p1.tar.gz size ;-)
>>> Previously fetched file: openssh-3.9p1.tar.gz MD5 ;-)
>>> md5 src_uri ;-) openssh-3.9p1.tar.gz
>>> emerge (88 of 92) app-crypt/hashalot-0.3 to /
Local mirror has file: hashalot-0.3.tar.gz
>>> Previously fetched file: hashalot-0.3.tar.gz size ;-)
>>> Previously fetched file: hashalot-0.3.tar.gz MD5 ;-)
>>> md5 src_uri ;-) hashalot-0.3.tar.gz
>>> emerge (89 of 92) sys-apps/util-linux-2.12q to /
Local mirror has file: util-linux-2.12q.tar.gz
Local mirror has file: util-linux-2.12p.diff.bz2
>>> Previously fetched file: util-linux-2.12p.diff.bz2 size ;-)
>>> Previously fetched file: util-linux-2.12p.diff.bz2 MD5 ;-)
>>> Previously fetched file: util-linux-2.12q.tar.gz size ;-)
>>> Previously fetched file: util-linux-2.12q.tar.gz MD5 ;-)
>>> md5 src_uri ;-) util-linux-2.12q.tar.gz
>>> md5 src_uri ;-) util-linux-2.12p.diff.bz2
>>> emerge (90 of 92) sys-libs/pwdb-0.62 to /
Local mirror has file: pwdb-0.62.tar.gz
>>> Previously fetched file: pwdb-0.62.tar.gz size ;-)
>>> Previously fetched file: pwdb-0.62.tar.gz MD5 ;-)
>>> md5 src_uri ;-) pwdb-0.62.tar.gz
>>> emerge (91 of 92) sys-apps/hdparm-5.9 to /
Local mirror has file: hdparm-5.9.tar.gz
>>> Previously fetched file: hdparm-5.9.tar.gz size ;-)
>>> Previously fetched file: hdparm-5.9.tar.gz MD5 ;-)
>>> md5 src_uri ;-) hdparm-5.9.tar.gz
>>> emerge (92 of 92) sys-devel/bin86-0.16.17 to /
Local mirror has file: bin86-0.16.17.tar.gz
>>> Previously fetched file: bin86-0.16.17.tar.gz size ;-)
>>> Previously fetched file: bin86-0.16.17.tar.gz MD5 ;-)
>>> md5 src_uri ;-) bin86-0.16.17.tar.gz
.
.
.
.
.
.
.
.


THE EMERGE COMMAND COPIED THE NESESERY FILES FROM DISTFILES BACKUP DIRECTORY TO THE /usr/portage/distfiles......XA-XA XA-XA
at the end the backup dir is 7GB and the proper dir is 700MB!!!!!!!!!!!!!!!!!!!!!!!!!:P


https://forums.gentoo.org/viewtopic-t-302818.html
Back to top
View user's profile Send private message
leks
n00b
n00b


Joined: 11 Oct 2004
Posts: 40

PostPosted: Mon Mar 07, 2005 12:06 am    Post subject: Re: why to do it the hard way? ...use the emerge command... Reply with quote

gspir2004 wrote:
<lots of text>


man you, drunk? :|
_________________
FreeBSD is neat
Back to top
View user's profile Send private message
alkan
Guru
Guru


Joined: 06 Aug 2004
Posts: 385
Location: kasimlar yaylasi

PostPosted: Tue Apr 12, 2005 1:42 am    Post subject: Reply with quote

Here is the one use for cleaning stale dist files. Sorry it is in bash script (I am not well experienced in phyton). Although it can be converted into phyton easly.

It is possibly a tad (or more) faster than other scripts. It also doesn't suffer from misdetections since distfiles are extracted from pkg database.
Code:

#!/bin/bash
#distclean

case $1 in
   -p | --pretend)
      remove="no"
      ;;
   -a | --ask)
      remove="ask"
      ;;
   "")
      remove="yes"
      ;;
   -h | --help | *)
      echo -e "distclean : prune stale distfiles"
      echo -e ""
      echo -e "Usage:"
      echo -e "distclean -p|--pretend\t: do not delete"
      echo -e "distclean -a|--ask    \t: ask for confirmation"
      echo -e "distclean -h|--help   \t: this help"
      echo -e "distclean             \t: clean all stale distfiles"
      echo -e ""
      echo -e ""
      exit
      ;;
esac
   
#db and distfile settings
echo "Reading Portage environment..."
VDB_PATH=$(portageq vdb_path)
DISTDIR=$(portageq distdir)

#if Control-C pressed restore distfile perms
function restoreperm(){
   echo -e "\n\nExiting due to signal.\n\n"
   chmod -t $DISTDIR/*
   exit 1
}
trap restoreperm SIGHUP SIGINT SIGQUIT SIGTERM SIGKILL

#mark all as stale
chmod +t $DISTDIR/*

#reverse perms for current package distfiles
echo "Reading package database..."
ENV=`find $VDB_PATH -type f -iname 'environment.bz2'`
for x in $ENV ; do
   KEEP=`bzcat ${x} | sed -n '1s/A=//gp'`
   for y in ${KEEP//\'/} ; do
      [ -e "${DISTDIR}/${y}" ] && chmod -t ${DISTDIR}/${y}
   done
done

#do the stuff
case $remove in
   yes)
      find $DISTDIR -perm +1000 -type f -maxdepth 1 \
      -printf "\\033[32m<<\\033[00m  %f  \n" \
      -exec rm -f  \{\} \;
      echo -e "\nDone.\n\n"
   ;;
   ask)
      find $DISTDIR -perm +1000 -type f -maxdepth 1 \
      -printf "[ \\033[32m%6k kB\\033[00m ] " \
      -ok rm -f  \{\} \;
      echo -e "\nDone.\n\n"
   ;;
   no | *)
      find $DISTDIR -perm +1000 -type f -maxdepth 1 \
      -printf "[ \\033[32m%6k kB\\033[00m ] %f\n"
   ;;
esac

#restore remaining file perms
chmod -t $DISTDIR/*
Back to top
View user's profile Send private message
stalynx
Apprentice
Apprentice


Joined: 03 Oct 2002
Posts: 162

PostPosted: Tue Apr 12, 2005 1:48 am    Post subject: Reply with quote

my own code

Code:
#!/bin/sh
cd /usr/portage/distfiles
rm *

works pretty good. :twisted:
Back to top
View user's profile Send private message
alkan
Guru
Guru


Joined: 06 Aug 2004
Posts: 385
Location: kasimlar yaylasi

PostPosted: Tue Apr 12, 2005 4:13 am    Post subject: Reply with quote

stalynx wrote:
my own code

Code:
#!/bin/sh
cd /usr/portage/distfiles
rm *

works pretty good. :twisted:


you haven't tried with a 56k modem where speed rarely tops a whopping 10k and pay half hour of your salaray for one minute phone connection. have you? It is about efficiency.
Back to top
View user's profile Send private message
stalynx
Apprentice
Apprentice


Joined: 03 Oct 2002
Posts: 162

PostPosted: Tue Apr 12, 2005 6:32 am    Post subject: Reply with quote

Code:
You haven't tried with a 56k modem where speed rarely tops a whopping 10k and pay half hour of your salaray for one minute phone connection. have you? It is about efficiency.


Nah not in a long time. But I understand and I'm just joking around. :wink:
Back to top
View user's profile Send private message
outspoken
Guru
Guru


Joined: 14 Feb 2004
Posts: 464
Location: orlando, fl

PostPosted: Fri Apr 22, 2005 9:21 pm    Post subject: Reply with quote

alkan wrote:
stalynx wrote:
my own code

Code:
#!/bin/sh
cd /usr/portage/distfiles
rm *

works pretty good. :twisted:


you haven't tried with a 56k modem where speed rarely tops a whopping 10k and pay half hour of your salaray for one minute phone connection. have you? It is about efficiency.


what does a modem and money have to do with running a remote script?
there are too many ways to list how to get it done without "watching" it. the first, and most obvious - SCREEN. run your command/script in a screen, come back in a little while and check up on it - or get a note via email, or time it, or whatever.
Back to top
View user's profile Send private message
alkan
Guru
Guru


Joined: 06 Aug 2004
Posts: 385
Location: kasimlar yaylasi

PostPosted: Sat Apr 23, 2005 4:05 am    Post subject: Reply with quote

How am i supposed to compile it next time if i don't have the source tar. The question is what is screen has to do with all this.

Last edited by alkan on Sat Apr 23, 2005 5:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 23, 2005 7:40 am    Post subject: Reply with quote

heheheh.
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
dkey
n00b
n00b


Joined: 11 May 2005
Posts: 25

PostPosted: Wed May 11, 2005 10:16 pm    Post subject: Reply with quote

@sandymc
do I must supply any options on first start? I did a
./distmaint.py -d (before -p)
but now ALL my /usr/portage/distfiles/ are away, no worry, my connection is good enought ;)

but I would like to use this script in future...so let me know if I did something wrong?

thx,
dkey
Back to top
View user's profile Send private message
patrickfo
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jun 2002
Posts: 79
Location: France

PostPosted: Wed Jun 22, 2005 11:15 am    Post subject: i wrote some utilities... Reply with quote

updated: scripts normaly must work now ;)
Here are somes utilities i wrote to maintain my gentoo boxes, use them at you own risk ;)
I wrote them because i wanted that all my boxes are identicals and my gentoo distri are like freshly installed ;)
if you find errors or enancements to do, please email me Patrick.Fourniols@wanadoo.fr
and sorry for my very bad english, i speak better french ( i hope ;) )

cleandu.sh:
find ebuilds not updated and needing new sources files, new version of patches in gcc ebuild for exemple ;)
if you have many boxes and only one distfiles directory, run it on every boxes and then run del_oldfiles.sh to
erase unneeded sources files to rebuild everything on any boxe or install new ones with others needs or
processors.

del_oldfiles.sh:

see above

find-wrongs-ebuild.sh:
find ebuilds losts in time ( no more in portage tree or portdir_overlay tree ).
Normaly first to be run ( if cleandu.sh list all sources neededs, then there are somes, one day cleandu.sh
will do that for you ;) ) and then update or unmerge manualy them.

find-pkg-not-updated.sh:
find ebuilds not in world file ( dependency needed ) and not updated since installed.
remove the -p option in /tmp/ebuild-updates.sh to update them, run it and then do what you can with pb
( yes the word exist and you will notice that ;) )
CAUTION: save your /usr/lib/portage/world file first if you don't want to manualy edit it ;)

find-ebuild-needing-sources.sh:
some sources files have obscures names and you can't find from which ebuilds they are or they are in many
ebuilds and you don't notice that first ;) ( exemple: gentoo patch for xmms also in plugins ebuild ;) )
work only on files sources used.

find-duplicate.sh:
supply old fonction like qpkg --dups -v

TO DO: some cleanings in directories used and many, many more... ;)

1) cleandu.sh
Code:
#!/bin/bash
# cleandu V2.0 - pur jus de chaussettes ;)
# compare les fichiers nécessaires à la compilation des packages installés
# à la liste de ceux qui ont été utilisés pour les compiler.
# sert aussi, en réseau, pour nettoyer les fichiers sources inutiles
# s'il y a un répertoire distifiles partagé.

echo -e "\n>>> Initialising necessary things..."
emerge info > /tmp/info.txt
DISTDIR=$(grep ^DISTDIR= /tmp/info.txt | cut -d'"' -f2)
PORTDIR=$(grep ^PORTDIR= /tmp/info.txt | cut -d'"' -f2)
PORTTMP=$(grep ^PORTAGE_TMPDIR= /tmp/info.txt | cut -d'"' -f2)
RPT=$DISTDIR/.cleandu/$HOSTNAME
TMP=$PORTTMP/cleandu.tmp
rm -rf /tmp/info.txt

[ ! -e $RPT ] && mkdir -p $RPT
[ ! -d $RPT ] && \
   echo "error: !!! $RPT MUST be a dir !!!" \
   && exit 0

[ ! -e $TMP ] && mkdir -p $TMP
[ ! -d $TMP ] && \
   echo "error: !!! $TMP MUST be a dir !!!" \
   && exit 0

rm -rf $RPT/* $TMP/*

# liste des ebuilds pour les packages installés.
echo -e "\n>>> Creating packages lists..."
ls /var/db/pkg/*/* -d --color=none | sed "s_/var/db/pkg/__" | sort > $TMP/qpkgv.txt

# liste des sources nécessaires à la compilation.
echo -e "\n>>> Building list of sources needed now to build packages installed..."
echo -n "emerge -pf " > $TMP/pfetch.sh
for fich in $(cat $TMP/qpkgv.txt)
   do
      echo -n "=$fich " >> $TMP/pfetch.sh
   done
echo -n " > $TMP/files_tmp.txt 2>&1" >> $TMP/pfetch.sh
/bin/bash $TMP/pfetch.sh
cat $TMP/files_tmp.txt | cut -d' ' -f1 | gawk -F'/' '{ if (NF > 1) {print $NF} }' | sort -u > $RPT/files_needed.txt

# liste des sources utilisées.
echo -e "\n>>> Building list of sources used to build packages on this gentoo box..."
bzcat /var/db/pkg/*/*/environment.bz2 | grep ^A= | cut -d'=' -f2 | cut -d"'" -f2 | gawk -F' ' '{ for (n=1; n <=NF; n++ ) print $n }' | sort -u > $RPT/files_from_env.txt

echo -e "\n>>> Working on files...\n"
echo "Left files used - Right files needed"
diff $RPT/files_from_env.txt $RPT/files_needed.txt --side-by-side --suppress-common-lines


2) del_oldfiles.sh
Code:
#!/bin/bash

echo -e "\n>>> Initialisings necessary things..."
emerge info > /tmp/info.txt
DISTDIR=$(grep ^DISTDIR= /tmp/info.txt | cut -d '"' -f 2)
PORTDIR=$(grep ^PORTDIR= /tmp/info.txt | cut -d '"' -f 2)
PORTTMP=$(grep ^PORTAGE_TMPDIR= /tmp/info.txt | cut -d '"' -f 2)
RPT=$DISTDIR/.cleandu
TMP=$PORTTMP/cleandu.tmp
rm -rf /tmp/info.txt

[ ! -d $RPT ] && \
   echo "error: !!! $RPT you MUST run cleandu before running del_oldfiles !!!" \
   && exit 0

rm -rf $RPT/dist.txt $RPT/needed.txt

for fich in $(ls $RPT)
   do
      [ -d $RPT/$fich ] && \
         [ -e $RPT/$fich/files_from_env.txt ] && \
         cat $RPT/$fich/files_from_env.txt >> $RPT/files_needed.txt
done

cat $RPT/files_needed.txt | sort -u >> $RPT/needed.txt

echo -e "\n>>> Building distfiles list..."
ls -I cvs-src $DISTDIR | sort > $RPT/dist.txt

rm -rf $RPT/files_*

for fich in $(cat $RPT/dist.txt)
do
   grep $fich $RPT/needed.txt 2>&1 >/dev/null
   if [[ $? != 0 && -e $RPT/../$fich ]]; then
      echo -en "\n$fich not needed! Delete ? (y/n): \t"
      read y
      if [[ $y = "y" || $y = "Y" ]]; then
         rm -rf $RPT/../$fich
         echo "$fich removed..."
      else
         echo "$fich remain..."
      fi;
   fi;
done


3) find-ebuilds-needing-sources.sh
Code:
#!/bin/bash

l_sources=$1

[[ $l_sources == "" ]] && echo "usage: $0 \"file1 file2 ...\"" && exit 0

ls /var/db/pkg/*/* -d --color=none | sed "s_/var/db/pkg/__" | sort > /tmp/ebuilds_list_full.txt

echo ">> Finding ebuilds..."
for src in $l_sources
do
   for pkg in $(cat /tmp/ebuilds_list_full.txt)
   do
      bzcat /var/db/pkg/$pkg/environment.bz2 | grep -q $src
      [ $? == 0 ] && echo "$src in $pkg"
   done
done


4) find-pkg-not-updated.sh
Code:
#!/bin/bash

ls /var/db/pkg/*/* -d --color=none | sed "s_/var/db/pkg/__" | sort > /tmp/ebuilds_list_full.txt

cat /tmp/ebuilds_list_full.txt | sed "s/-[0-9].*//" > /tmp/liste_packages.txt

echo -n "emerge -up " > /tmp/ebuild_update.sh

for pkg in $(cat /tmp/liste_packages.txt)
   do
      echo -n "$pkg " >> /tmp/ebuild_update.sh
   done
echo ">> Update pretend... ;)"
sh /tmp/ebuild_update.sh


5) find-wrongs-ebuilds.sh
Code:
#!/bin/bash

PORTDIR=$(emerge info | grep ^PORTDIR= | cut -d'"' -f2)
PORTDIR_OVERLAY=$(emerge info | grep ^PORTDIR_OVERLAY= | cut -d'"' -f2)
MY_PATH="$PORTDIR $PORTDIR_OVERLAY"

#some cleaning ;)
[ -f /tmp/ebuilds_wrongs_list.txt ] && rm -f /tmp/ebuilds_wrongs_list.txt

ls /var/db/pkg/*/* -d --color=none | sed "s_/var/db/pkg/__" | sort > /tmp/ebuilds_list_full.txt

echo "Searching for wrongs ebuilds..."
for ligne in $(cat /tmp/ebuilds_list_full.txt)
do
   status=1
   for ac_dir in $MY_PATH
   do
      pkg=$(echo $ligne | sed "s/-[0-9].*//")
      ebld=$(echo $ligne | cut -d'/' -f2)
      [ -f $ac_dir/$pkg/$ebld.ebuild ] && status=0
   done
   [ $status != 0 ] && echo -n "$ligne " >> /tmp/ebuilds_wrongs_list.txt
done

#report result ;)
if [ -f /tmp/ebuilds_wrongs_list.txt ] ; then
   echo "These ebuilds are not in portage tree: try to update them manualy or unmerge them..."
   for ligne in $(cat /tmp/ebuilds_wrongs_list.txt)
      do
         echo $ligne
   done
else
   echo "All ebuilds are OK ;)"
fi


6) find-duplicates.sh
Code:
#!/bin/bash

old=""
in_dupli=0
dupli=0

ls /var/db/pkg/*/* -d --color=none | sed "s_/var/db/pkg/__" | sort > /tmp/ebuilds_list_full.txt

cat /tmp/ebuilds_list_full.txt | sed "s/-[0-9].*//" | sort > /tmp/liste_packages.txt

for pkg in $(cat /tmp/liste_packages.txt)
   do
      if [[ $pkg == $old ]]; then
         dupli=1
         in_dupli=1
      else
         dupli=0
      fi
      if [[ $in_dupli == 1 ]] && [[ $dupli == 0 ]]; then
         grep $old-[0-9] /tmp/ebuilds_list_full.txt
         in_dupli=0
      fi
      old=$pkg
   done
Back to top
View user's profile Send private message
alonbl
n00b
n00b


Joined: 30 Jun 2005
Posts: 57

PostPosted: Thu Jun 30, 2005 4:03 pm    Post subject: Reply with quote

There is a bug trace for this issue at https://bugs.gentoo.org/show_bug.cgi?id=33877
Back to top
View user's profile Send private message
andrewd18
Guru
Guru


Joined: 11 Apr 2004
Posts: 364
Location: Wisconsin, USA

PostPosted: Sun Jul 03, 2005 7:39 pm    Post subject: Reply with quote

alkan wrote:
Here is the one use for cleaning stale dist files. Sorry it is in bash script (I am not well experienced in phyton). Although it can be converted into phyton easly.

...


Great program, thanks!

~~ Andrew D.
_________________
Keep Your Toolchain Stable! - emwrap.sh

There's no place like ::1
Back to top
View user's profile Send private message
tacvbo
n00b
n00b


Joined: 28 May 2003
Posts: 39
Location: México City

PostPosted: Wed Jul 06, 2005 10:03 pm    Post subject: Reply with quote

Wiiiii!!! Another unsupported and unoffical Gentoo cleaner...

...but, with new features!: :D

    * Clean old distfiles in the $DISTDIR directory (/usr/portage/distfiles)
    * Clean old binary packages files in $PKGDIR directory (/usr/portage/packages)
    * Clean old dirs and files in $WORKDIR directory (/var/tmp/portage)
    * Clean old portage's log files in $PORT_LOGDIR (/var/log/portage)


See it's own forums thread here: https://forums.gentoo.org/viewtopic-t-337074.html

Grab it here: yacleaner repository and please post your comments, patches and suggestions. :-)

Code:

yacleaner-0.3: please tell me what to do.

Usage:
    yacleaner-0.3 [ options ] [ action ] < dist | binpkg | worktmp | log | all >
    yacleaner-0.3 --help

Options: [ --ask | --nocolor | --pretend | --verbose ]

Actions:  [ --delete ]


    For more help try 'yacleaner-0.3 --help'
Back to top
View user's profile Send private message
gentuse
Tux's lil' helper
Tux's lil' helper


Joined: 15 Aug 2002
Posts: 75
Location: Seattle

PostPosted: Fri Jul 15, 2005 6:15 pm    Post subject: Script to remove obsolete files in /usr/portage/distfiles Reply with quote

I'm sure this has been brought up before, and I'm sure there's a better way to do this, but here's something I threw together today. Let me know if you can see any ways to improve it.

This script will remove any files in /usr/portage/distfiles that aren't referenced in any digest-* files for packages.

Code:
#!/bin/sh

filesneeded=`find /usr/portage/ -name digest-\* -type f | xargs awk '{print $3}' | sort -u`

find /usr/portage/distfiles/ -type f -maxdepth 1 -printf "%f\n" | while read a
do
  if [[ ! "${filesneeded}" =~ "${a}" ]]
  then
    echo /usr/portage/distfiles/${a}
  fi
done | xargs rm -f


The magic is the "=~" operator, as discussed here.
Back to top
View user's profile Send private message
gentuse
Tux's lil' helper
Tux's lil' helper


Joined: 15 Aug 2002
Posts: 75
Location: Seattle

PostPosted: Fri Jul 15, 2005 7:50 pm    Post subject: Reply with quote

This seems faster, but uses temporary files.

Code:
#!/bin/sh

f=/tmp/filesneeded.txt
e=/tmp/existingfiles.txt

find /usr/portage/ -name digest-\* -type f | xargs awk '{print $3}' | sort -u > ${f}
find /usr/portage/distfiles/ -type f -maxdepth 1 -printf "%f\n" | sort > ${e}

comm -2 -3 ${e} ${f} | sed 's%^%/usr/portage/distfiles/%' | xargs rm -f
rm -f ${e} ${f}
Back to top
View user's profile Send private message
kallamej
Administrator
Administrator


Joined: 27 Jun 2003
Posts: 4975
Location: Gothenburg, Sweden

PostPosted: Fri Jul 15, 2005 10:59 pm    Post subject: Reply with quote

Merged https://forums.gentoo.org/viewtopic-p-2576280.html#2576280 to this thread.
_________________
Please read our FAQ Forum, it answers many of your questions.
irc: #gentoo-forums on irc.libera.chat
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Tue Jul 19, 2005 5:18 am    Post subject: distdir cleaners Reply with quote

Ok here are my 5 cents::)), i have written this some time ago, but there was not much interest, the original posting is here
https://forums.gentoo.org/viewtopic-t-254565-highlight-.html
It doesn't need portage patching anymore, because the bug was corrected ::))) Essentially it is not that user friendly, you'll need to define two variable in /etc/make.conf FILES_NOT_NEEDED and FILES_PROBABLY_NEEDED.
I define them as
Code:
FILES_PROBABLY_NEEDED=/var/tmp/probably_needed
FILES_NOT_NEEDED=/var/tmp/not_needed

And it will move all files, which doesn't have corresponding ebuilds in $FILES_NOT_NEEDED and all not installed in $FILES_PROBABLY_NEEDED. leaving only the files, which are needed fo emerge -e world. It uses portage, that's why if you say LINGUAS="sth" it will take in account this also and so on:::))))
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
luna80
Veteran
Veteran


Joined: 01 Feb 2004
Posts: 1545
Location: switzerland

PostPosted: Fri Jul 22, 2005 3:32 am    Post subject: Reply with quote

thanks a lot kerframil, your script is very usefull 8)
_________________
*** !f j00 c4n r34D tH15 tH3N j00 n33D t0 l0g Off ***
Back to top
View user's profile Send private message
Tsuna
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jan 2005
Posts: 147

PostPosted: Sat Aug 06, 2005 6:05 pm    Post subject: Reply with quote

the original script posted in the 1st post is buggy. It will badly detect outdated ebuilds when numeric comparison between there versions fails for exemple if you have gnome-2.10.0 and gnome-2.8.10, it will delete 2.10.0 because 2.8.10 is numerically greater than 2.10.0
I made a simple workaround for this, but since I don't know much about programming in Perl, I made it in C

Here is the source of the C part of it:

Code:
/**
 * syntax: ./vercompare <v1> <v2>
 * Returns 1 if v1 is a newer version number than v2, 2 if v2 is a newer
 * version than v1, 0 otherwise (both should be equivalant)
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <ctype.h>

/**
 * Returns 1 if str is made only of digit, 0 otherwise
 */
static int isDigit(const char *str);
/**
 * Returns 1 if str is made only of alpha characters, 0 otherwise
 */
static int isAlpha(const char *str);
/**
 * Returns 1 if v1 > v2, 2 if v2 > v1, 0 otherwise. Comparison is not a simple
 * numeric comparison, it's about version numbering
 */
static int vercmp(char *v1, char *v2);

static int isDigit(const char *str) {
    size_t i = 0;
    while(str[i] != '\0') {
        if(!isdigit(str[i])) {
            return 0;
        }
        i++;
    }
    return 1;
}

static int isAlpha(const char *str) {
    size_t i = 0;
    while(str[i] != '\0') {
        if(!isalpha(str[i])) {
            return 0;
        }
        i++;
    }
    return 1;
}

static int vercmp(char *v1, char *v2) {
    char *b1, *b2; /* tmp buffers */
    size_t maxlen, p1, p2;
    long a1, a2;
   
#ifndef NDEBUG
    printf("v1=%s v2=%s\n", v1, v2);
#endif

    maxlen = strlen(v1) > strlen(v2) ? strlen(v1) : strlen(v2);
   
    b1 = malloc(maxlen + 1); /* this way, we won't bother to realloc */
    assert(b1 != NULL);
    b2 = malloc(maxlen + 1);
    assert(b2 != NULL);

    while(v1[0] != '\0' && v2[0] != '\0') {
        /* read some numbers from v1 and v2 */
        p1 = strspn(v1, "0123456789");
        p2 = strspn(v2, "0123456789");

        /* if we didn't find anything number, let's just take one char */
        if(p1 == 0)
            p1 = 1;
        if(p2 == 0)
            p2 = 1;

        strncpy(b1, v1, p1);
        strncpy(b2, v2, p2);
        /* make sure the tmp buffers are NULL terminated once we filled them */
        b1[p1 + 1] = '\0';
        b2[p2 + 1] = '\0';
       
#ifndef NDEBUG
        printf("token v1=%s token v2=%s\n", b1, b2);
#endif

        if(isDigit(b1) && isDigit(b2)) {
            a1 = atol(b1);
            a2 = atol(b2);
            if(a1 > a2) {
#ifndef NDEBUG
                puts(">>> v1 > v2 !");
#endif
                free(b1);
                free(b2);
                return 1;
            } else if(a1 < a2) {
#ifndef NDEBUG
                puts(">>> (i) v1 < v2");
#endif
                free(b1);
                free(b2);
                return 2;
            } else {
#ifndef NDEBUG
                puts("... (i) v1 =? v2");
#endif
            }
        } else if(isAlpha(b1) && isAlpha(b2)) {
            a1 = strcmp(b1, b2);
            if(a1 > 0) {
#ifndef NDEBUG
                puts(">>> (s) v1 > v2");
#endif
                free(b1);
                free(b2);
                return 1;
            } else if(a1 < 0) {
#ifndef NDEBUG
                puts(">>> (s) v1 < v2");
#endif
                free(b1);
                free(b2);
                return 2;
            }
        } else {
#ifndef NDEBUG
            puts("... (s) v1 =? v2");
#endif
        }

        v1 += p1;
        v2 += p2;
    } /* end of while */
   
    free(b1);
    free(b2);
    /* if we got here it's because we reached the end of v1 or v2 */
    if(v1[0] == '\0') {
        if(v2[0] == '\0') {
            return 0;
        } else {
#ifndef NDEBUG
            puts(">>> (l) v1 < v2");
#endif
            return 2;
        }
    } else { /* ==> v2[0] == '\0' && v1[0] != '\0' */
#ifndef NDEBUG
            puts(">>> (l) v1 > v2");
#endif
            return 1;
    }
}

int main(int argc, char *argv[]) {
    int rv;
   
    if(argc != 3)
        return 0;

    rv = vercmp(argv[1], argv[2]);
   
#ifndef NDEBUG
    if(rv == 1) {
        printf("%s > %s\n", argv[1], argv[2]);
    } else if(rv == 2) {
        printf("%s < %s\n", argv[1], argv[2]);
    } else {
        printf("%s ?= %s ??\n", argv[1], argv[2]);
    }
#endif
   
    return rv;
}


Put this code in vercompare.c and compile with the following command:
Code:
$ gcc -pedantic -ansi -W -Wall -Wfloat-equal -Waggregate-return -Wcast-align -Wcast-qual -Wnested-externs -Wpointer-arith -Wundef -Wshadow -Wbad-function-cast -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations -Wconversion -Winline -DNDEBUG -o vercompare vercompare.c
$ strip vercompare


Then put the executable file vercompare in the same directory where the Perl script is (let's suppose the script is named cleandistfile.pl) and patch the Perl script with the following diff:
Code:
--- cleandistfiles.pl   2005-07-28 23:04:46.000000000 +0200
+++ cleandistfiles.pl.old  2005-08-06 19:51:56.000000000 +0200
@@ -52,14 +39,9 @@
       $version = $2;
       next if ($maskedfiles{$name}); # Ignore "masked" files
       if ($lastname && $name eq $lastname) {
-         if ($version gt $lastversion) {
-            push (@stalefiles, [$name, $version, $lastversion, $lastext]);
+         system './vercompare', $version, $lastversion;
+        if ($? >> 8 == 1) {
+           push (@stalefiles, [$name, $version, $lastversion, $lastext]);
+         } elsif($? >> 8 == 2) {
+           push (@stalefiles, [$name, $lastversion, $version, $lastext]);
           }
       }
       $lastname = $name;
       $lastversion = $version;


thats all :)
I haven't reviewed all the other scripts proposed in this topic, some seem to be rather interesting :)

edit: patched a stupid memory leak, thanks splint!
_________________
I am not sending subliminal messages
Back to top
View user's profile Send private message
gingekerr
n00b
n00b


Joined: 12 Aug 2005
Posts: 4
Location: Reigate/Cambridge, UK

PostPosted: Fri Aug 12, 2005 10:59 am    Post subject: Another way of cleaning distfiles Reply with quote

Although I haven't read all the posts in this forum, all the scripts I have seen seem really long and complicated.

Here is a really short one (although it may not be as quick as the others) which simply gets a list of all the distfiles used by the software in the world file by emerge -epf world and then deletes all distfiles not in this list.

It needs a bit of porting to get it to work on a non-default system.

Code:

#!/bin/bash

emerge -epf world &> /tmp/distfilesAX432z1
ACCEPT_KEYWORDS="~x86" emerge -epf world &> /tmp/distfilesAX432z2

cd /usr/portage/distfiles

for DISTFILE in *; do
GREPRESULT=`grep $DISTFILE /tmp/distfilesAX432z1 /tmp/distfilesAX432z2`
        if ( [ -z "$GREPRESULT" ] )
                then rm $DISTFILE
        fi
done


not bad for 9 lines of code?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Page 6 of 8

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum