| View previous topic :: View next topic |
| Author |
Message |
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
 Posted: Mon Feb 28, 2005 7:05 am Post subject: HOWTO: Clean your /etc/portage/package.* files |
 |
|
Clean out your package.keywords, package.unmask, package.use, package.mask, package.cflags, package.cxxflags and package.ldflags files (automatically, of course)
If you're anything like me your package files may be rather unorganized from all the testing of very experimental packages and such. In order to remedy this problem I wrote a little (well it was little to begin with) bash script that checks every package in these files to see if without being there the same version would still be emerged with the same use flags (in the case of the package.use checking). It also checks for some other things like redundant flags, not-installed packages, duplicates - things like that.
Download
You can download the script here.
...or (for the truly lazy) copy and paste as root:
| Code: | wget http://users.wpi.edu/~chanin/scripts/etcportclean
chmod +x etcportclean
mv etcportclean /usr/bin/ |
Usage
Before using it it would be a good idea to go over --help (so you really should), some of the options are very useful.
Otherwise, just run it as root.
Errata
package.cflags, package.cxxflags, and package.ldflags are not standard, if you just create the files they won't do anything. This script works with the format described by thebigslide here (you can also get the needed files there as well).
It works for me, nonetheless, it has not had much offical testing (though many people have had success with it, it seems). I would suggest that you backup your package.* files before running it. On this same tangent, as is stated in the file, it comes with no warranty. However, if it doesn't work for you, and you can give me some information about why, please tell me. I'll happily make an attempt at fixing it.
Also, it doesn't actually interface with the portage backend. What this means is that it is very slow. On my Athlon64 3200+ it's not so bad, however I cannot vouch for how long it will take on a Pentium 1 or something similar.
Finally, I would urge you to take the results with a grain of salt. It generally seems to be correct, but if it's telling you that nothing in package.use is needed, you've probably encountered a bug.
| Quote: | Change Log
3.11 Added per-line duplicate checking for all files. Added not-existing category check for package.*flags files.
3.10 Made not installed package checking for keywords, unmask and mask work better. Fixed a bug where package.use scanning would claim flags were not needed when they were due to a package being in emerge -puD world.
3.09 Fixed two small bugs with non-command-line control. Thanks to teh-steve for pointing out one of them.
3.08 Fixed a tiny bug. Thanks again to torne.
3.07 Squashed the last of torne's bugs, hopefully. Also, changed it so that keep works in reverse - now all lines above the 'Keep' line are ignored for normal checks.
3.0[456] Fixing the bugs mentioned by torne.
3.03 Fixed the bugs mentioned by lookitsme and torne. Added duplicate flag checking for categories in package.*flags. Fixed a bug with duplicate flag checking for package.use/*flags.
3.02 Fixed a few bugs with duplicate checking, fixed a bug about reporting incorrect offending lines, updated help quite a bit.
3.01 Added checking for package.ldflags, fixed a problem with sometimes getting confused over packages that had updates, fixed checking for --param flag (for package.cflags/cxxflags), fixed checking for duplicate flags (it failed in a few rare, odd cases), updated help text a bit.
3.00 Added support for package.cflags and package.cxxflags. Addded an automatic update feature, use -u. Reorganized a bunch of code. Fixed a bunch of bugs. Updated documentation/help. Added a 'very verbose' option, access with -V.
2.54 Fixed a bug that was causing the slot check to fail far more packages than it should've been.
2.53 The script will now report when it comes accross a package that doesn't seem to exist; for all checks.
2.52 Made script work better with slotted packages that had more than one slot filled (amazingly enough, without any extra calls to emerge.) A couple of unnoticable changes/fixes.
2.51 Fixed a minor bug. Added -v (verbose) command-line option. Cleaned up some of the code a bit.
2.50 New command-line arguements, and command-line arguement parsing code. Updated --help. Rewrote many error messages. Rewrote many comments. Removed normal package.use scanning (advanced is default now.) Implemented package.use duplicate flag checking.
2.40 Fixed bug mentioned by zsoltika. Made 'Keep' feature work again. Implemented duplicate checking for package.(^use), and use flag concatenation for package.use, many thanks to thebigslide for that idea. Normal package.use scanning is now deprecated, it's kind of dangerous and not very good code.
2.32 Fixed some minor bugs.
2.31 Made advanced package.use scanning significantly faster. Fixed a couple of bugs with command-line arguements.
2.30 Made package.keywords, package.unmask, and package.mask scans significantly faster. Fixed a logic bug in advanced package.use scanning (it wasn't finding everything it could, and was overall very incorrect.) Added a check to make sure 'emerge -puD world' returns output like the script is looking for. Added command arguements to turn correct 'emerge -puD world' checking on or off. Updated --help.
2.20 Made normal package.use scan faster. Made advanced package.use scanning much (about 2 times) faster. Made package.use functions choke less if they encounter masked packages.
2.11 Made help screen more helpful (thanks for pointing that out teh-steve.) Added command-line options for turning the checks on and off.
2.10 Made advanced use flag checking report if it encounters a package that is not installed, instead of just saying that all its use flags were not necessary. Made script compatible with packages that are blockers, have fetch restrictions, et cetera; as always, may still be buggy.
2.00 Major rewrite of some of the code, made many more things into functions, a lot cleaner; added and reorganized some comments. Added feature to check individual use flags for package.use, will only list them however, not remove them. Access with 'advance' command arguement. Added automatic file updating (will not prompt if you want to remove the entries) - access with 'yes' command arguement. Changed name from gotostable to etcportclean, in order to better reflect what the program does in an equally esoteric manner.
1.13 Added feature to check package.mask.
1.12 Made it so that if there is a line with 'Keep' on it in any of the files gotostable will not consider any of the packages below that line. Be they unnecessary or not installed.
1.11 Fixed the bug mentioned by flybynite in the second post. (Would not remove entry if it was the last one in the file.)
1.10 Added feature to check package.use. Added checking, in any of the files, for packages that have entries, but that are not currently installed. Made it more obvious when something went wrong with removing the unnecessary entries.
1.00 First release, checked package.keywords and package.unmask |
Todo
etcportclean badly needs to be rewritten to minimize pointless or redundant checks and speed up general program execution. I may get around to this some day.
Last edited by SoTired on Thu Oct 06, 2005 5:27 am; edited 55 times in total |
|
| Back to top |
|
 |
anderlin
Tux's lil' helper
Joined: 29 Nov 2003
Posts: 149
Location: Trondheim, Norway
|
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Mon Feb 28, 2005 10:12 am Post subject: |
|
|
Nice Idea!!!
Keywords worked for me, removed about 7 packages I didn't need anymore!
Unmask didn't work for me. Didn't have time to help debug. Something might need to be escaped?
| Code: |
gate1 root # '/root/gotostable'
Check package.keywords for needless keywords? (yes/no)
no
Check package.unmask for needlessly unmasked packages? (yes/no)
yes
Checking package.unmask...
The following package can be removed from package.unmask: net-www/mozilla
Remove the not needed entries? (yes/no)
yes
Done.
gate1 root # cat /etc/portage/package.unmask
>=net-www/mozilla-1.6
gate1 root # ls -l /etc/portage/package.unmask
-rw-r--r-- 1 root root 22 Feb 28 04:00 /etc/portage/package.unmask
gate1 root #
|
|
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Mon Feb 28, 2005 9:12 pm Post subject: |
|
|
| flybynite wrote: |
Unmask didn't work for me. Didn't have time to help debug. Something might need to be escaped?
|
I think I've got all the escaping done. There're a fair number of hacks in there to escape stuff (and a bunch more to restore backups in case something goes wrong - all in all, the delete unnecessary packages ought to be very safe.)
Anyways, it was the restore backup code that was causing the problem. It made sure that the final file was not 0 bytes long, assuming that something went horribly wrong with the deleting "unnecessary" entries from it. However, seeing how you only had one entry in the file it was restoring the backup when it shouldn't have been. It's been fixed in v1.11. |
|
| Back to top |
|
|
teh-steve
n00b
Joined: 28 Jan 2005
Posts: 27
Location: Worcester
|
| Posted: Tue Mar 01, 2005 5:48 am Post subject: SoTired Smells |
|
|
The first part of the etcportclean (the one that checks package.keywords) seems to run slower than it did with gotostable. I commented out a lot of each script so that they would run only the first part without prompting me and then times them. Here are the results:
| Code: |
root /home/steve # time etcportclean
Checking package.keywords...
There is nothing to remove.
Done.
real 0m7.022s
user 0m5.999s
sys 0m0.882s
root /home/steve # time gotostable
Checking package.keywords...
There is nothing to remove.
Done.
real 0m6.994s
user 0m5.988s
sys 0m0.864s
|
They're pretty much the same but I also didn't have any packages to remove. It might get worse or better with useless packages. I'll test them with a bunch of useless keywords later.
Anyway, this output is very vague and sometimes doesn't make much sense. Besides that, the first line wraps on an 80-character terminal. Stupid last character....
If auto equals 'auto' there is no confirmation to remove entries from the files.
If auto equals 'advance' advanced use-flag checking will be used.
I demand optimization!
Edit: I tested each script again in the same fashion, this time using auto with etcportclean to avoid prompting, and intentionally put ten useless entries into my package.keywords file each time. Here are the results:
| Code: |
root /home/steve # time etcportclean auto
Checking package.keywords...
dev-util/ccache in package.keywords is unnecessary.
app-office/openoffice in package.keywords is unnecessary.
sys-apps/eject is in package.keywords but is not installed.
x11-libs/qt is in package.keywords but is not installed.
net-dns/libidn is in package.keywords but is not installed.
net-nds/portmap is in package.keywords but is not installed.
app-admin/fam is in package.keywords but is not installed.
dev-libs/libpcre is in package.keywords but is not installed.
dev-libs/libgpg-error is in package.keywords but is not installed.
dev-libs/libgcrypt is in package.keywords but is not installed.
Unnecessary entries successfully removed.
Done.
real 0m13.513s
user 0m11.648s
sys 0m1.647s
root /home/steve # time gotostable
Checking package.keywords...
dev-util/ccache in package.keywords is unnecessary.
app-office/openoffice in package.keywords is unnecessary.
sys-apps/eject is in package.keywords but is not installed.
x11-libs/qt is in package.keywords but is not installed.
net-dns/libidn is in package.keywords but is not installed.
net-nds/portmap is in package.keywords but is not installed.
app-admin/fam is in package.keywords but is not installed.
dev-libs/libpcre is in package.keywords but is not installed.
dev-libs/libgpg-error is in package.keywords but is not installed.
dev-libs/libgcrypt is in package.keywords but is not installed.
Unnecessary entries successfully removed.
Done.
real 0m13.565s
user 0m11.617s
sys 0m1.690s
|
This time etcportclean was a bit faster (though it did take more user time). I suppose they're equally slow... 
Again, I demand optimization! |
|
| Back to top |
|
|
teh-steve
n00b
Joined: 28 Jan 2005
Posts: 27
Location: Worcester
|
| Posted: Tue Mar 01, 2005 9:29 pm Post subject: |
|
|
| Awesome. The new version runs much faster for me. I don't have time to bench it now but I will make an attempt to do so later tonight. Thanks! |
|
| Back to top |
|
|
ecoffey
Tux's lil' helper
Joined: 16 Jun 2004
Posts: 122
Location: Colorado
|
| Posted: Tue Mar 01, 2005 9:55 pm Post subject: |
|
|
thats a handy little utility
good job  |
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Wed Mar 02, 2005 10:24 pm Post subject: |
|
|
Thanks for the comments.
Any more feature or bug fix type requests?
I don't think the package.(keywords|unmask|mask) scanning is going to get much faster. And package.use definitely won't. Advanced package.use on the other hand may, if I find some free time. So no need for more requests about those things.
Mainly bugs though - there really must be some. Just looking through the changelog I find a bunch of major ones most releases (not handling blockers or fetch restricted packages, package.use scanning saying everything was not needed - that one's actually been fixed twice), so has anyone aside from flybynite found any yet? |
|
| Back to top |
|
|
Earthwings
Bodhisattva
Joined: 14 Apr 2003
Posts: 7753
Location: Germany
|
|
| Back to top |
|
|
thebigslide
l33t
Joined: 23 Dec 2004
Posts: 792
Location: under a car or on top of a keyboard
|
| Posted: Thu Mar 03, 2005 10:06 am Post subject: |
|
|
Interesting... I might write up something to combine the USE flags of identicle packages and also to parse package.cflags, cxxflags, and features for to hack into this... biab
Back:
| Code: | cleanfile() {
local PKGLIST USEFLAGS I J
PKGLIST=`cat /etc/portage/package.${1} | cut -d " " -f1 | sort -u`
for I in $PKGLIST
do
USEFLAGS=`grep "$I" /etc/portage/package.${1} | cut -d " " -f2-`
echo -n "$I " >> /etc/portage/package.${1}.temp
for J in $USEFLAGS;
do
echo "$J" >> /etc/portage/package.${1}.temp
done
echo "" >> /etc/portage/package.${1}.temp
done
} |
call "cleanfile use" to generate a package.use.temp that has all the duplicate packages removed with the use slammed onto one line. It doesn't check for countering or redundant flags, tho. Gimmie a few minutes for that...
OK: If you want it to eliminate duplicate USE flags: | Code: | cleanfile() {
local PKGLIST USEFLAGS TEMPUSE ANTIFLAG DUPTEST I J
PKGLIST=`cat /etc/portage/package.${1} | cut -d " " -f1 | sort -u`
for I in $PKGLIST
do
USEFLAGS=`grep "$I" /etc/portage/package.${1} | cut -d " " -f2-`
echo -n "$I " >> /etc/portage/package.${1}.temp
rm -f /etc/portage/.TEMP &>/dev/null
for J in $USEFLAGS;
do
echo $J >> /etc/portage/.TEMP
if [ `echo $J | cut -c 1` = "-" ]; then
ANTIFLAG=`echo $J | cut -d "-" -f2`
DUPTEST=`echo $USEFLAGS | grep " $ANTIFLAG"`
if [ "$DUPTEST" != "" ]; then
echo "WARNING: $I USEs both $J and $ANTIFLAG!!!"
fi
fi
done
echo `sort -u < /etc/portage/.TEMP` >> /etc/portage/package.${1}.temp
rm -f /etc/portage/.TEMP &>/dev/null
done
} |
If you have -flag and flag in /etc/portage/package.use for the same package, it produces a warning
This script parsed my package.use in less than 1 second. Maybe you could work that into your script?
Last edited by thebigslide on Thu Mar 03, 2005 11:17 am; edited 3 times in total |
|
| Back to top |
|
|
zsoltika
l33t
Joined: 13 Nov 2003
Posts: 634
Location: Budapest, Hungary
|
| Posted: Thu Mar 03, 2005 11:00 am Post subject: |
|
|
Gave it a try, and it workedd as expected, so thanks for your work.
Just one feature request/bug report:
It showed that: | Code: | Checking package.mask...
x11-libs/fox in package.mask is unnecessary. |
But my package.mask shows: | Code: | # ez a fox kiuti az xfe-t...
# >=x11-libs/fox-1.0.90 |
(The first lin is in hungarian )
IMHO it should ignore #-ed lines (maybe by depending on some command line option...
Zsoltika |
|
| Back to top |
|
|
provicemo
Apprentice
Joined: 12 Dec 2003
Posts: 201
Location: Detroit, MI
|
| Posted: Thu Mar 03, 2005 11:54 pm Post subject: |
|
|
Very nice script.
Thank you |
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Fri Mar 04, 2005 8:54 pm Post subject: |
|
|
Thanks for the bug report, feature idea, and, of course, compliments - bug is fixed in 2.40 and the feature is (somewhat) implemented.
I'm hesitant about incorporating package.c(xx)*flags into the script - it doesn't seem like there is a real standard for how the files will look, and furthermore there's no way to determine if the flags in it are necessary or not. Nevertheless, perhaps some scanning for duplicates or something is in order, time will tell.
Last edited by SoTired on Sun Mar 20, 2005 7:44 am; edited 1 time in total |
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Sun Mar 20, 2005 7:43 am Post subject: |
|
|
Nevermind, problem solved.
Last edited by SoTired on Fri Mar 25, 2005 2:34 am; edited 1 time in total |
|
| Back to top |
|
|
xchris
Advocate
Joined: 10 Jul 2003
Posts: 2824
|
| Posted: Sun Mar 20, 2005 10:13 am Post subject: |
|
|
really handy script!
Thank you!
_________________
while True:Gentoo() |
|
| Back to top |
|
|
pjv
Guru
Joined: 02 Jul 2003
Posts: 353
Location: Belgium
|
| Posted: Mon Apr 11, 2005 9:16 pm Post subject: |
|
|
| very nice, exactly what i was looking for. |
|
| Back to top |
|
|
gentoo_lan
l33t
Joined: 08 Sep 2004
Posts: 891
Location: Charles Town, WV
|
| Posted: Mon Apr 11, 2005 10:58 pm Post subject: |
|
|
Very handy. Thanks.
[EDIT] I only found one error it tried to take sys-libs/db out of my package.keywords which was needed by noatun. Other than that though it worked flawlessly. Thanks again. |
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Tue Apr 12, 2005 9:01 pm Post subject: |
|
|
| gentoo_lan: I think the bug you're refering to may be fixed in 3.01 - not totally sure. If it isn't any chance you could give some more information? Like what 'emerge -p sys-libs/db' returns without its keyword set? |
|
| Back to top |
|
|
gentoo_lan
l33t
Joined: 08 Sep 2004
Posts: 891
Location: Charles Town, WV
|
| Posted: Tue Apr 12, 2005 9:18 pm Post subject: |
|
|
| It has been fixed. Thanks for the excellent script and keep up the good work. |
|
| Back to top |
|
|
lookitsme
n00b
Joined: 06 Nov 2003
Posts: 48
Location: Kuala Lumpur, Malaysia
|
| Posted: Sat Jun 04, 2005 11:48 am Post subject: Error |
|
|
I'm getting this error:
| Code: |
expr: syntax error
/usr/bin/etcportclean: line 653: [: too many arguments
|
And I can't seem to figure out why. Oh, getting it on two different machines btw... |
|
| Back to top |
|
|
dfy
Tux's lil' helper
Joined: 05 Apr 2004
Posts: 79
|
| Posted: Sun Jun 05, 2005 4:09 pm Post subject: |
|
|
| This is excellent. Exactly what I've been looking for - thank you very much! |
|
| Back to top |
|
|
torne
n00b
Joined: 22 Jan 2004
Posts: 65
Location: Cambridge, UK
|
| Posted: Sun Jul 31, 2005 1:56 pm Post subject: |
|
|
Works nicely, but it doesn't parse package.keywords lines of the form:
~dev-php/jpgraph-1.17 ~x86
i.e. versioned keywords. Almost all my keywords are of this form, as I don't generally want any arbitrary future ~arch version of a package, just a newer version than the current arch. A number of them are no longer relevant as the referred-to version is now stable, but I don't have the patience to go through the list and check
So, err, consider this a feature request? |
|
| Back to top |
|
|
SoTired
Apprentice
Joined: 19 May 2004
Posts: 174
|
| Posted: Sun Jul 31, 2005 9:36 pm Post subject: |
|
|
Sorry for taking so long with a few of these bugs, but 3.03 has now been released.
| lookitsme wrote: | I'm getting this error:
| Code: |
expr: syntax error
/usr/bin/etcportclean: line 653: [: too many arguments
|
And I can't seem to figure out why. Oh, getting it on two different machines btw... |
No idea what's up with that, your expr seems to be crazy. Anyways, in 3.03 I changed it to use two ifs rather than an expr. So this should be fixed.
| torne wrote: | Works nicely, but it doesn't parse package.keywords lines of the form:
~dev-php/jpgraph-1.17 ~x86
i.e. versioned keywords. Almost all my keywords are of this form, as I don't generally want any arbitrary future ~arch version of a package, just a newer version than the current arch. A number of them are no longer relevant as the referred-to version is now stable, but I don't have the patience to go through the list and check
So, err, consider this a feature request? |
Fixed.
| dfy wrote: | | This is excellent. Exactly what I've been looking for - thank you very much! |
Thanks for the compliments.  |
|
| Back to top |
|
|
torne
n00b
Joined: 22 Jan 2004
Posts: 65
Location: Cambridge, UK
|
| Posted: Mon Aug 01, 2005 12:46 am Post subject: |
|
|
| It still doesn't do the right thing for versioned keywords - it noticed that one of them referred to a package that wasn't installed and removed it, but it didn't notice several others that are also not installed, and it didn't notice any of the ones where the version referred to is now marked stable. |
|
| Back to top |
|
|
Gatak
Apprentice
Joined: 04 Jan 2004
Posts: 174
|
| Posted: Mon Aug 01, 2005 12:56 am Post subject: |
|
|
| torne wrote: | | It still doesn't do the right thing for versioned keywords - it noticed that one of them referred to a package that wasn't installed and removed it, but it didn't notice several others that are also not installed, and it didn't notice any of the ones where the version referred to is now marked stable. |
What do you mean with versioned keywords? The example ~dev-php/jpgraph-1.17 ~x86 is wrong. It should be =dev-php/jpgraph-1.17 ~x86 or >=dev-php/jpgraph-1.17 ~x86 or <=dev-php/jpgraph-1.17 ~x86 |
|
| Back to top |
|
|
|
Unsupported Software
