Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Another clean distfiles script (clean-dist.sh)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
yzg
Apprentice
Apprentice


Joined: 18 Jun 2005
Posts: 285

PostPosted: Mon Jan 19, 2009 2:49 am    Post subject: Another clean distfiles script (clean-dist.sh) Reply with quote

I have been using this script for sometime and I thought that it could be useful to other people as well. It finds all the files that would be downloaded
if you emerge all the packages on your computer, then compares them with the files in the distfiles directory and removes the ones which are not required.


Code:

#!/bin/bash

#DISCLAIMER this script is provided "AS-IS" and your use is at your own risk

za=$(mktemp)
zz=$(mktemp)
z1=$(mktemp)
z2=$(mktemp)

echo list all installed packages ...
equery -q list | sed '{s/^/=/}' > $za

if [[ $? -ne "0" ]]
then
        echo ========================================
        echo ERROR
        cat $za
        echo ========================================
        exit
fi

echo list all URLs to be downloaded by them ...
emerge -pf `cat $za` > $zz

if [[ $? -ne "0" ]]
then
        echo ========================================
        echo ERROR
        cat $zz
        echo ========================================
        exit
fi

echo convert URLs to unique files ...
sed -r '/^(ht|f)tp:\/\//{s/ +/\n/g;p};d' $zz | sed '{s/\/*[^\/]*\///g;p};d' | sort | uniq > $z1

echo list stored files ...
ls -1 /usr/portage/distfiles/ | grep -v "lost+found" | sort > $z2


array1=(`cat $z1`)
array2=(`cat $z2`)
#echo Two first elements: ${array1[0]} ${array2[1]}

echo find files to remove ...
let n1=${#array1[*]}
let n2=${#array2[*]}

for ((j = 0; j < n2; j++));
do
        for ((i = 0; i < n1; i++));
        do
                if [ ${array1[$i]} = ${array2[$j]} ];
                then
                        unset array2[$j]
                        break;
                fi
        done
done

#echo Two first elements: ${array2[0]} ${array2[1]}

array2=( "${array2[@]}" )
#echo Two first elements: ${array2[0]} ${array2[1]}

let n3=${#array2[*]}

let ts=0
for ((j = 0; j < n3; j++));
do
    echo $j ${array2[$j]}
    let ts=ts+$(stat -c%s /usr/portage/distfiles/${array2[$j]})
done
let ts=ts/1000000

echo
echo "Required files : $n1"
echo "Stored   files : $n2"
echo "Files to remove: $n3"
echo "Space to free  : $ts MB"

if [[ $n3 -ne 0 ]]
then

echo
echo "continue (y/N)? "
read tmp
if [[ "$tmp" = "y" ]]
then
        echo
        echo removing the files ...
        for ((j = 0; j < n3; j++));
        do
                rm /usr/portage/distfiles/${array2[$j]}
        done

        echo files were removed.
else
        echo
        echo nothing was done.
fi

else

        echo
        echo nothing to do
fi

echo clean temporary files ...
rm $za
rm $zz
rm $z1
rm $z2

echo
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Mon Jan 19, 2009 1:01 pm    Post subject: Reply with quote

what exactly is the purpose of this?
doesn't eclean have an option that does what you want?
Back to top
View user's profile Send private message
Dralnu
Veteran
Veteran


Joined: 24 May 2006
Posts: 1919

PostPosted: Wed Feb 18, 2009 11:14 am    Post subject: Reply with quote

alex.blackbit wrote:
what exactly is the purpose of this?
doesn't eclean have an option that does what you want?


eclean-dist probably does this. Nice work, though. Its amazing how many tools gentoo has in the tree that people can easily replace with a short bash script.
_________________
The day Microsoft makes a product that doesn't suck, is the day they make a vacuum cleaner.
Back to top
View user's profile Send private message
toralf
Advocate
Advocate


Joined: 01 Feb 2004
Posts: 2954
Location: Hamburg/Germany

PostPosted: Wed Feb 18, 2009 12:34 pm    Post subject: Reply with quote

Dralnu wrote:
Nice work, though. Its amazing how many tools gentoo has in the tree that people can easily replace with a short bash script.
b/c yzg obviously already have emerged app-portage/gentoolkit I don't see an advantage of writing an extra script for the functionality provided by the command line
Code:
eclean -d distfiles
Back to top
View user's profile Send private message
Mimamau
Apprentice
Apprentice


Joined: 11 Jun 2002
Posts: 150
Location: Germany

PostPosted: Wed Feb 18, 2009 12:38 pm    Post subject: Reply with quote

Why keeping all the sources of installes packages?

From time to time I just do 'rm /usr/portage/distfiles/*'.
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Wed Feb 18, 2009 1:18 pm    Post subject: Reply with quote

i believe it is a normal step to rm the content of the distfiles directory.
the only problem i see here (since years) is that there is just one directory for regularly downloaded sources through portage and fetch-restricted files that were put there.
everything portage had downloaded can be removed, because it is easy to re-get them by just calling portage.
the fetch-restricted files have to be backup-ed in a seperate place or they will be gone after rm-ing.
IMHO eclean lacks a feature that removes everything except fetch-restricted files.
there is a parameter of eclean that keeps fetch-restricted files (-f), but it works only in combination with -d, which keeps "the minimum for a reinstallation". there really should be an option that removes EVERYTHING except fetch-restricted files.
comments?
Back to top
View user's profile Send private message
Mimamau
Apprentice
Apprentice


Joined: 11 Jun 2002
Posts: 150
Location: Germany

PostPosted: Wed Feb 18, 2009 3:26 pm    Post subject: Reply with quote

Even most fetch restricted files can be downloaded again... :)
As you have to download them by hand and then move to distfiles, you could also copy them to your backup directory during this step, wouldnt be much more effort.

But generally I think this could be usefull.
Back to top
View user's profile Send private message
alex.blackbit
Advocate
Advocate


Joined: 26 Jul 2005
Posts: 2397

PostPosted: Wed Feb 18, 2009 4:00 pm    Post subject: Reply with quote

this is what i actually do at the moment.
if somebody is interested, here is the script:
Code:
#!/bin/sh

ARCHIVE_PATH="/mnt/archive/distfiles_archive"
CP_TO_DISTFILES="install -v -o portage -g portage -m 0664 $@ $(portageq distdir)"

if [ -e ${ARCHIVE_PATH}/.archive ];
then
        cp -iv $@ ${ARCHIVE_PATH}
else
        echo "distfiles archive not found ;( file(s) will _NOT_ be archived !!!"
fi

if [ $(id -u) -eq 0 ];
then
        exec ${CP_TO_DISTFILES}
else
        exec sudo ${CP_TO_DISTFILES}
fi

since i download such files with my normal user account i have to somehow transfer those files to /usr/portage/distfiles and adjust the permissions. well, actually, this wouldn't be strictly necessary, but that way it's really clean.

but anyway, i still think that eclean should have such a feature.
anybody with some python skills reading this thread?!?!
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4794

PostPosted: Sat Feb 21, 2009 8:06 am    Post subject: Reply with quote

eclean has several problems, the main one being that it only works heuristically. Since it does not take useflags into account, it cannot detect reliably which files are needed for emerge -e @world @system. For this reason, I always recommend to use the trickyfetch script.

That script is not only more reliable but also supports the special treatment of restricted files you asked for:
Just move the fetch restricted files to the ${DISTDIR}/.restricted directory (or move them to another directory and refer to them by symbolic links).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum