| View previous topic :: View next topic |
| Author |
Message |
niranjana_km
n00b
Joined: 15 Mar 2014
Posts: 14
|
 Posted: Wed Dec 31, 2014 8:22 am Post subject: Keeping /usr/portage/distfiles tidy |
 |
|
Keeping /usr/portage/distfiles tidy
When we keep updating gentoo, over a time the source packages accumulate in /usr/portage/distfiles. This consumes a lot of disk space because of source packages of old versions of installed programs still sitting there. To clean it manually is tedious as there are thousands of packages. And deleting all the source packages in it is not a wise thing as they are required when rebuilding something due to dependency or USE change. Here is a handy script which lists the source packages used for current installation and those packages not used (due to an update) but still sitting in /usr/portage/distfiles. After listing here is another script which can move the 'not used' packages into a directory, which you can then move it to your storage devices for backup (for any future needs) or delete it. The listing script is as follows,
| Code: |
#!/bin/bash
# Old package source files lister for Gentoo.
# The package database location: "/var/db/pkg"
bzcat -k /var/db/pkg/*/*/environment.bz2 | grep "declare -x A=" >Adistlist
bzcat -k /var/db/pkg/*/*/environment.bz2 | grep "declare -x AA=" >AAdistlist
sed -e "s/declare -x A=\"//g" -e "s/\"//g" -e "s/ /\n/g" Adistlist >installed
sed -e "s/declare -x AA=\"//g" -e "s/\"//g" -e "s/ /\n/g" AAdistlist >>installed
sort -u installed >xinstalled
mv xinstalled installed
ls -1 /usr/portage/distfiles >allfiles
cp allfiles uninstalled
for i in $(cat installed)
do
sed --in-place -e "s/$i//1" uninstalled
done
cat uninstalled | tr -s '\n' >xuninstalled
mv xuninstalled uninstalled
# EOF
|
And the moving script is,
| Code: |
#!/bin/bash
# Old package source files cleaner for Gentoo.
# You need not to run this script as root. Because, "/usr/portage/distfiles"
# directory is owned by `portage` and belongs to `portage` group. It has read/write
# permission for all the group members and if you are a member of the `portage` group you can also run this script.
distdir="/usr/portage/distfiles"
listdir="./" # Directory where the above prepared list is sitting.
mvtodir="/usr/portage/distfiles/duninstalled-$(date +%Y%m%d)"
cd $distdir
mkdir $mvtodir
for i in $(cat $listdir/uninstalled)
do
mv -t $mvtodir $i
done
## Alternate method:
## First, the name of the source files needed in the current installation are
## listed, sorted and uniquified using the script 'clean-distfiles/list-distfiles.sh'.
## Then the source files listed are moved to a directory named 'installed' and
## whatever remains is moved to a directory named 'uninstalled'. Then the contents
## of the directory 'installed' are moved back into the '/usr/portage/distfiles/' directory.
## Then the directory 'uninstalled' is moved to external harddisk with date for backup.
# for i in $(cat installed)
# do
# mv -t dinstalled $i
# done
#
# for i in *
# do
# mv -t duninstalled $i
# done
# EOF
|
| Quote: |
Note: Not tested with -9999 ebuilds i.e the program source fetched with a revision control system like git, subversion, bazar etc...
|
Happy new year Gentoo. Love you.... |
|
| Back to top |
|
 |
charles17
Advocate
Joined: 02 Mar 2008
Posts: 3686
|
| Posted: Wed Dec 31, 2014 10:51 am Post subject: |
|
|
| What is the point of not using eclean-dist? |
|
| Back to top |
|
|
Clad in Sky
l33t
Joined: 04 May 2007
Posts: 895
Location: Germany
|
| Posted: Wed Dec 31, 2014 11:17 am Post subject: |
|
|
Probably OP wasn't aware of this. Neither was I. So thanks to OP and to you for pointing me at this.
_________________
Kali Ma
Now it's autumn of the aeons
Dance with your sword
Now it's time for the harvest |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55196
Location: 56N 3W
|
| Posted: Wed Dec 31, 2014 11:43 am Post subject: |
|
|
Clad in Sky,
eclean works for packages too.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
niranjana_km
n00b
Joined: 15 Mar 2014
Posts: 14
|
| Posted: Thu Jan 01, 2015 8:12 am Post subject: |
|
|
| Ya, I was not aware of 'eclean', thanks charles17 for pointing it.. |
|
| Back to top |
|
|
niranjana_km
n00b
Joined: 15 Mar 2014
Posts: 14
|
|
| Back to top |
|
|
charles17
Advocate
Joined: 02 Mar 2008
Posts: 3686
|
| Posted: Sat Jan 03, 2015 9:19 am Post subject: |
|
|
| niranjana_km wrote: | | Ya, I was not aware of 'eclean', thanks charles17 for pointing it.. |
Another nice tool you should try is | Quote: | | $ eix-test-obsolete |
|
|
| Back to top |
|
|
niranjana_km
n00b
Joined: 15 Mar 2014
Posts: 14
|
| Posted: Wed Jan 28, 2015 5:55 am Post subject: |
|
|
| Yes, since it uses cache the search is very quick and nice. I started using eix instead of equery. Thanks... |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Jan 28, 2015 7:17 am Post subject: |
|
|
I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files.
What I use is the trickyfetch script from the mv overlay and its "obsolete" command: This moves all your sourcefiles to $DISTDIR/.obsolete and causes portage to fetch all files of all of your packages (according to your useflags); "fetching" means first to move from $DISTDIR/.obsolete (and some other directories), and only if it is not found to fallback to wget (or whatever you want to use as FETCHCOMMAND). Hence, in the end, all your "unneeded" files remain in $DISTDIR/.obsolete, and you can decide whether to clean them all or keep some of them. (And BTW you also realize if there were file bumps without a revision bump so that you are guaranteed to rebuild all of your packages without internet access.) |
|
| Back to top |
|
|
szczerb
Veteran
Joined: 24 Feb 2007
Posts: 1709
Location: Poland => Lodz
|
| Posted: Wed Jan 28, 2015 10:40 am Post subject: |
|
|
I'm just using this: | Code: | | find /usr/portage/distfiles -maxdepth 1 -type f -mtime +30 -exec rm -v {} \; |
|
|
| Back to top |
|
|
cwr
Veteran
Joined: 17 Dec 2005
Posts: 1969
|
| Posted: Wed Jan 28, 2015 1:48 pm Post subject: |
|
|
| szczerb wrote: | I'm just using this: | Code: | | find /usr/portage/distfiles -maxdepth 1 -type f -mtime +30 -exec rm -v {} \; |
|
I'm surprised that that works - a lot of my portage files are dated before 2010.
Will |
|
| Back to top |
|
|
niranjana_km
n00b
Joined: 15 Mar 2014
Posts: 14
|
| Posted: Thu Jan 29, 2015 7:27 am Post subject: |
|
|
The trickyfetch script method seems to be little inefficient. I have not used it, but its above description by "mv" made me to think so. How much time it takes to complete it?
And the method used by "szczerb" doesn't guarantee retention of all the source packages used in present installation. |
|
| Back to top |
|
|
charles17
Advocate
Joined: 02 Mar 2008
Posts: 3686
|
| Posted: Thu Jan 29, 2015 8:56 am Post subject: |
|
|
| mv wrote: | | I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files. |
Could you give an example of such file(s) being kept because of an non-honored USE flag?
Me thinks it's worth filing a bug to get eclean improved. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Jan 29, 2015 9:15 am Post subject: |
|
|
| charles17 wrote: | | mv wrote: | | I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files. |
Could you give an example of such file(s) being kept because of an non-honored USE flag?
Me thinks it's worth filing a bug to get eclean improved. |
Examples depend of course on your useflags. For instance, many language files of firefox are only downloaded if certain linguas_...-USE-Flags are set.
This is not a "bug" which can be easily fixed: You need knowledge about the whole user configuration (not only make.conf and /usr/portage/package.use but also selected profile(s), USE-expand variables etc.) to get the currently active USE-flag setting.
Moreover, there are probably some files which you never want to be removed, even if you do not have them installed currently (e.g. because of dead upstream or fetch-restrictions or whatever). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Jan 29, 2015 9:27 am Post subject: |
|
|
| niranjana_km wrote: | | The trickyfetch script method seems to be little inefficient. I have not used it, but its above description by "mv" made me to think so. How much time it takes to complete it? |
The overhead over portage is almost negligible (except for the texlive packages which you can easily manually move out from .obsolete if you do not want to check them, too): The needed time is essentially that of | Code: | | emerge -efD --with=bdeps=y @world |
On my system (athlon, ~1200 packages, including full texlive) it takes around 10 minutes, because one cannot configure portage to omit doing the checksums for every file. On the other hand, only this way it can be guaranteed that there have not been any changes in the portage tree concerning your downloaded files.
I do not cleanup after every emerge --sync but only e.g. before I make a backup of my main system (which I do less frequently), so these 10 minutes or so are not very disturbing.
The advantage of all this is that you can influence a lot manually: For instance, if you use the same DISTDIR for several machines/chroots (with different configurations), you just have to call the above emerge command on every machine/chroot, and then the files remaining in .obsolete are those used by neither.
Or if you want to keep e.g. the sources for adobe-flash for all architectures, although you currently have installed only amd64, you just call | Code: | | emerge -1FO adobe-flash |
before cleaning up your .obsolete directory. Moreover, I prefer to keep some source-files (e.g. for games or other optional packages which I install on only a few systems) in separate directories: The trickyfetch script can generate corresponding symlinks and just moves out these symlinks from .save.* directories if needed.
There is also a .restricted directory in which I put symlinks to files with fetch restrictions: The trickyfetch script takes care about moving them out, so that you do not run into errors about nonexisting sources for fetch-restricted packages. |
|
| Back to top |
|
|
|
Installing Gentoo
