Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Clean your /etc/portage/package.* files
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Mon Feb 28, 2005 7:05 am    Post subject: HOWTO: Clean your /etc/portage/package.* files Reply with quote

Clean out your package.keywords, package.unmask, package.use, package.mask, package.cflags, package.cxxflags and package.ldflags files (automatically, of course)
If you're anything like me your package files may be rather unorganized from all the testing of very experimental packages and such. In order to remedy this problem I wrote a little (well it was little to begin with) bash script that checks every package in these files to see if without being there the same version would still be emerged with the same use flags (in the case of the package.use checking). It also checks for some other things like redundant flags, not-installed packages, duplicates - things like that.

Download
You can download the script here.
...or (for the truly lazy) copy and paste as root:
Code:
wget http://users.wpi.edu/~chanin/scripts/etcportclean
chmod +x etcportclean
mv etcportclean /usr/bin/


Usage
Before using it it would be a good idea to go over --help (so you really should), some of the options are very useful.
Otherwise, just run it as root.

Errata
package.cflags, package.cxxflags, and package.ldflags are not standard, if you just create the files they won't do anything. This script works with the format described by thebigslide here (you can also get the needed files there as well).

It works for me, nonetheless, it has not had much offical testing (though many people have had success with it, it seems). I would suggest that you backup your package.* files before running it. On this same tangent, as is stated in the file, it comes with no warranty. However, if it doesn't work for you, and you can give me some information about why, please tell me. I'll happily make an attempt at fixing it.

Also, it doesn't actually interface with the portage backend. What this means is that it is very slow. On my Athlon64 3200+ it's not so bad, however I cannot vouch for how long it will take on a Pentium 1 or something similar.

Finally, I would urge you to take the results with a grain of salt. It generally seems to be correct, but if it's telling you that nothing in package.use is needed, you've probably encountered a bug.

Quote:
Change Log
3.11 Added per-line duplicate checking for all files. Added not-existing category check for package.*flags files.
3.10 Made not installed package checking for keywords, unmask and mask work better. Fixed a bug where package.use scanning would claim flags were not needed when they were due to a package being in emerge -puD world.
3.09 Fixed two small bugs with non-command-line control. Thanks to teh-steve for pointing out one of them.
3.08 Fixed a tiny bug. Thanks again to torne.
3.07 Squashed the last of torne's bugs, hopefully. Also, changed it so that keep works in reverse - now all lines above the 'Keep' line are ignored for normal checks.
3.0[456] Fixing the bugs mentioned by torne.
3.03 Fixed the bugs mentioned by lookitsme and torne. Added duplicate flag checking for categories in package.*flags. Fixed a bug with duplicate flag checking for package.use/*flags.
3.02 Fixed a few bugs with duplicate checking, fixed a bug about reporting incorrect offending lines, updated help quite a bit.
3.01 Added checking for package.ldflags, fixed a problem with sometimes getting confused over packages that had updates, fixed checking for --param flag (for package.cflags/cxxflags), fixed checking for duplicate flags (it failed in a few rare, odd cases), updated help text a bit.
3.00 Added support for package.cflags and package.cxxflags. Addded an automatic update feature, use -u. Reorganized a bunch of code. Fixed a bunch of bugs. Updated documentation/help. Added a 'very verbose' option, access with -V.
2.54 Fixed a bug that was causing the slot check to fail far more packages than it should've been.
2.53 The script will now report when it comes accross a package that doesn't seem to exist; for all checks.
2.52 Made script work better with slotted packages that had more than one slot filled (amazingly enough, without any extra calls to emerge.) A couple of unnoticable changes/fixes.
2.51 Fixed a minor bug. Added -v (verbose) command-line option. Cleaned up some of the code a bit.
2.50 New command-line arguements, and command-line arguement parsing code. Updated --help. Rewrote many error messages. Rewrote many comments. Removed normal package.use scanning (advanced is default now.) Implemented package.use duplicate flag checking.
2.40 Fixed bug mentioned by zsoltika. Made 'Keep' feature work again. Implemented duplicate checking for package.(^use), and use flag concatenation for package.use, many thanks to thebigslide for that idea. Normal package.use scanning is now deprecated, it's kind of dangerous and not very good code.
2.32 Fixed some minor bugs.
2.31 Made advanced package.use scanning significantly faster. Fixed a couple of bugs with command-line arguements.
2.30 Made package.keywords, package.unmask, and package.mask scans significantly faster. Fixed a logic bug in advanced package.use scanning (it wasn't finding everything it could, and was overall very incorrect.) Added a check to make sure 'emerge -puD world' returns output like the script is looking for. Added command arguements to turn correct 'emerge -puD world' checking on or off. Updated --help.
2.20 Made normal package.use scan faster. Made advanced package.use scanning much (about 2 times) faster. Made package.use functions choke less if they encounter masked packages.
2.11 Made help screen more helpful (thanks for pointing that out teh-steve.) Added command-line options for turning the checks on and off.
2.10 Made advanced use flag checking report if it encounters a package that is not installed, instead of just saying that all its use flags were not necessary. Made script compatible with packages that are blockers, have fetch restrictions, et cetera; as always, may still be buggy.
2.00 Major rewrite of some of the code, made many more things into functions, a lot cleaner; added and reorganized some comments. Added feature to check individual use flags for package.use, will only list them however, not remove them. Access with 'advance' command arguement. Added automatic file updating (will not prompt if you want to remove the entries) - access with 'yes' command arguement. Changed name from gotostable to etcportclean, in order to better reflect what the program does in an equally esoteric manner.
1.13 Added feature to check package.mask.
1.12 Made it so that if there is a line with 'Keep' on it in any of the files gotostable will not consider any of the packages below that line. Be they unnecessary or not installed.
1.11 Fixed the bug mentioned by flybynite in the second post. (Would not remove entry if it was the last one in the file.)
1.10 Added feature to check package.use. Added checking, in any of the files, for packages that have entries, but that are not currently installed. Made it more obvious when something went wrong with removing the unnecessary entries.
1.00 First release, checked package.keywords and package.unmask


Todo
etcportclean badly needs to be rewritten to minimize pointless or redundant checks and speed up general program execution. I may get around to this some day.


Last edited by SoTired on Thu Oct 06, 2005 5:27 am; edited 55 times in total
Back to top
View user's profile Send private message
anderlin
Tux's lil' helper
Tux's lil' helper


Joined: 29 Nov 2003
Posts: 149
Location: Trondheim, Norway

PostPosted: Mon Feb 28, 2005 7:46 am    Post subject: Reply with quote

Thank you very much! Worked great for me.
_________________
http://anderlin.dyndns.org/
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Mon Feb 28, 2005 10:12 am    Post subject: Reply with quote

Nice Idea!!!

Keywords worked for me, removed about 7 packages I didn't need anymore!

Unmask didn't work for me. Didn't have time to help debug. Something might need to be escaped?



Code:

gate1 root # '/root/gotostable'
Check package.keywords for needless keywords? (yes/no)
no
Check package.unmask for needlessly unmasked packages? (yes/no)
yes

Checking package.unmask...
The following package can be removed from package.unmask: net-www/mozilla
Remove the not needed entries? (yes/no)
yes

Done.
gate1 root # cat /etc/portage/package.unmask
>=net-www/mozilla-1.6
gate1 root # ls -l /etc/portage/package.unmask
-rw-r--r--  1 root root 22 Feb 28 04:00 /etc/portage/package.unmask
gate1 root #               
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Mon Feb 28, 2005 9:12 pm    Post subject: Reply with quote

flybynite wrote:

Unmask didn't work for me. Didn't have time to help debug. Something might need to be escaped?


I think I've got all the escaping done. There're a fair number of hacks in there to escape stuff (and a bunch more to restore backups in case something goes wrong - all in all, the delete unnecessary packages ought to be very safe.)

Anyways, it was the restore backup code that was causing the problem. It made sure that the final file was not 0 bytes long, assuming that something went horribly wrong with the deleting "unnecessary" entries from it. However, seeing how you only had one entry in the file it was restoring the backup when it shouldn't have been. It's been fixed in v1.11.
Back to top
View user's profile Send private message
teh-steve
n00b
n00b


Joined: 28 Jan 2005
Posts: 27
Location: Worcester

PostPosted: Tue Mar 01, 2005 5:48 am    Post subject: SoTired Smells Reply with quote

The first part of the etcportclean (the one that checks package.keywords) seems to run slower than it did with gotostable. I commented out a lot of each script so that they would run only the first part without prompting me and then times them. Here are the results:

Code:

root /home/steve # time etcportclean

Checking package.keywords...
There is nothing to remove.

Done.

real    0m7.022s
user    0m5.999s
sys     0m0.882s


root /home/steve # time gotostable

Checking package.keywords...
There is nothing to remove.

Done.

real    0m6.994s
user    0m5.988s
sys     0m0.864s


They're pretty much the same but I also didn't have any packages to remove. It might get worse or better with useless packages. I'll test them with a bunch of useless keywords later.

Anyway, this output is very vague and sometimes doesn't make much sense. Besides that, the first line wraps on an 80-character terminal. Stupid last character....

If auto equals 'auto' there is no confirmation to remove entries from the files.
If auto equals 'advance' advanced use-flag checking will be used.


I demand optimization!

Edit: I tested each script again in the same fashion, this time using auto with etcportclean to avoid prompting, and intentionally put ten useless entries into my package.keywords file each time. Here are the results:

Code:

root /home/steve # time etcportclean auto

Checking package.keywords...
dev-util/ccache in package.keywords is unnecessary.
app-office/openoffice in package.keywords is unnecessary.
sys-apps/eject is in package.keywords but is not installed.
x11-libs/qt is in package.keywords but is not installed.
net-dns/libidn is in package.keywords but is not installed.
net-nds/portmap is in package.keywords but is not installed.
app-admin/fam is in package.keywords but is not installed.
dev-libs/libpcre is in package.keywords but is not installed.
dev-libs/libgpg-error is in package.keywords but is not installed.
dev-libs/libgcrypt is in package.keywords but is not installed.
Unnecessary entries successfully removed.

Done.

real    0m13.513s
user    0m11.648s
sys     0m1.647s


root /home/steve # time gotostable

Checking package.keywords...
dev-util/ccache in package.keywords is unnecessary.
app-office/openoffice in package.keywords is unnecessary.
sys-apps/eject is in package.keywords but is not installed.
x11-libs/qt is in package.keywords but is not installed.
net-dns/libidn is in package.keywords but is not installed.
net-nds/portmap is in package.keywords but is not installed.
app-admin/fam is in package.keywords but is not installed.
dev-libs/libpcre is in package.keywords but is not installed.
dev-libs/libgpg-error is in package.keywords but is not installed.
dev-libs/libgcrypt is in package.keywords but is not installed.
Unnecessary entries successfully removed.

Done.

real    0m13.565s
user    0m11.617s
sys     0m1.690s


This time etcportclean was a bit faster (though it did take more user time). I suppose they're equally slow... ;)

Again, I demand optimization!
Back to top
View user's profile Send private message
teh-steve
n00b
n00b


Joined: 28 Jan 2005
Posts: 27
Location: Worcester

PostPosted: Tue Mar 01, 2005 9:29 pm    Post subject: Reply with quote

Awesome. The new version runs much faster for me. I don't have time to bench it now but I will make an attempt to do so later tonight. Thanks!
Back to top
View user's profile Send private message
ecoffey
Tux's lil' helper
Tux's lil' helper


Joined: 16 Jun 2004
Posts: 122
Location: Colorado

PostPosted: Tue Mar 01, 2005 9:55 pm    Post subject: Reply with quote

thats a handy little utility

good job :-)
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Wed Mar 02, 2005 10:24 pm    Post subject: Reply with quote

Thanks for the comments. :)

Any more feature or bug fix type requests?

I don't think the package.(keywords|unmask|mask) scanning is going to get much faster. And package.use definitely won't. Advanced package.use on the other hand may, if I find some free time. So no need for more requests about those things.

Mainly bugs though - there really must be some. Just looking through the changelog I find a bunch of major ones most releases (not handling blockers or fetch restricted packages, package.use scanning saying everything was not needed - that one's actually been fixed twice), so has anyone aside from flybynite found any yet?
Back to top
View user's profile Send private message
Earthwings
Administrator
Administrator


Joined: 14 Apr 2003
Posts: 7751
Location: Karlsruhe, Germany

PostPosted: Thu Mar 03, 2005 7:11 am    Post subject: Reply with quote

Splitted from http://forums.gentoo.org/viewtopic-t-3011.html where it got merged in accident.
Back to top
View user's profile Send private message
thebigslide
l33t
l33t


Joined: 23 Dec 2004
Posts: 787
Location: under a car or on top of a keyboard

PostPosted: Thu Mar 03, 2005 10:06 am    Post subject: Reply with quote

Interesting... I might write up something to combine the USE flags of identicle packages and also to parse package.cflags, cxxflags, and features for to hack into this... biab

Back:

Code:
cleanfile() {
local PKGLIST USEFLAGS I J
PKGLIST=`cat /etc/portage/package.${1} | cut -d " " -f1 | sort -u`
for I in $PKGLIST
do
   USEFLAGS=`grep "$I" /etc/portage/package.${1} | cut -d " " -f2-`
   echo -n "$I " >> /etc/portage/package.${1}.temp
   for J in $USEFLAGS;
   do
      echo "$J" >> /etc/portage/package.${1}.temp
   done
   echo "" >> /etc/portage/package.${1}.temp
done
}

call "cleanfile use" to generate a package.use.temp that has all the duplicate packages removed with the use slammed onto one line. It doesn't check for countering or redundant flags, tho. Gimmie a few minutes for that...

OK: If you want it to eliminate duplicate USE flags:
Code:
cleanfile() {
local PKGLIST USEFLAGS TEMPUSE ANTIFLAG DUPTEST I J

PKGLIST=`cat /etc/portage/package.${1} | cut -d " " -f1 | sort -u`
for I in $PKGLIST
do
        USEFLAGS=`grep "$I" /etc/portage/package.${1} | cut -d " " -f2-`
        echo -n "$I " >> /etc/portage/package.${1}.temp
        rm -f /etc/portage/.TEMP &>/dev/null
        for J in $USEFLAGS;
        do
                echo $J >> /etc/portage/.TEMP
                if [ `echo $J | cut -c 1` = "-" ]; then
                        ANTIFLAG=`echo $J | cut -d "-" -f2`
                        DUPTEST=`echo $USEFLAGS | grep " $ANTIFLAG"`
                        if [ "$DUPTEST" != "" ]; then
                                echo "WARNING: $I USEs both $J and $ANTIFLAG!!!"
                        fi
                fi
        done
        echo `sort -u < /etc/portage/.TEMP` >> /etc/portage/package.${1}.temp
        rm -f /etc/portage/.TEMP &>/dev/null
done
}
If you have -flag and flag in /etc/portage/package.use for the same package, it produces a warning
This script parsed my package.use in less than 1 second. Maybe you could work that into your script?
_________________
/etc/portage/package.c{,xx}flags and package.features!
All posts released under the GPL and the Creative Commons license


Last edited by thebigslide on Thu Mar 03, 2005 11:17 am; edited 3 times in total
Back to top
View user's profile Send private message
zsoltika
l33t
l33t


Joined: 13 Nov 2003
Posts: 634
Location: Budapest, Hungary

PostPosted: Thu Mar 03, 2005 11:00 am    Post subject: Reply with quote

Gave it a try, and it workedd as expected, so thanks for your work.
Just one feature request/bug report:
It showed that:
Code:
Checking package.mask...
x11-libs/fox in package.mask is unnecessary.

But my package.mask shows:
Code:
# ez a fox kiuti az xfe-t...
# >=x11-libs/fox-1.0.90

(The first lin is in hungarian :) )
IMHO it should ignore #-ed lines (maybe by depending on some command line option...

Zsoltika
Back to top
View user's profile Send private message
provicemo
Apprentice
Apprentice


Joined: 12 Dec 2003
Posts: 201
Location: Detroit, MI

PostPosted: Thu Mar 03, 2005 11:54 pm    Post subject: Reply with quote

Very nice script.
Thank you
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Fri Mar 04, 2005 8:54 pm    Post subject: Reply with quote

Thanks for the bug report, feature idea, and, of course, compliments - bug is fixed in 2.40 and the feature is (somewhat) implemented.

I'm hesitant about incorporating package.c(xx)*flags into the script - it doesn't seem like there is a real standard for how the files will look, and furthermore there's no way to determine if the flags in it are necessary or not. Nevertheless, perhaps some scanning for duplicates or something is in order, time will tell.


Last edited by SoTired on Sun Mar 20, 2005 7:44 am; edited 1 time in total
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Sun Mar 20, 2005 7:43 am    Post subject: Reply with quote

Nevermind, problem solved.

Last edited by SoTired on Fri Mar 25, 2005 2:34 am; edited 1 time in total
Back to top
View user's profile Send private message
xchris
Advocate
Advocate


Joined: 10 Jul 2003
Posts: 2824
Location: 45.488291,9.186094

PostPosted: Sun Mar 20, 2005 10:13 am    Post subject: Reply with quote

really handy script!
Thank you!
_________________
while True:Gentoo()
Back to top
View user's profile Send private message
pjv
Guru
Guru


Joined: 02 Jul 2003
Posts: 353
Location: Belgium

PostPosted: Mon Apr 11, 2005 9:16 pm    Post subject: Reply with quote

very nice, exactly what i was looking for.
Back to top
View user's profile Send private message
gentoo_lan
l33t
l33t


Joined: 08 Sep 2004
Posts: 889
Location: Denver, CO

PostPosted: Mon Apr 11, 2005 10:58 pm    Post subject: Reply with quote

Very handy. Thanks.

[EDIT] I only found one error it tried to take sys-libs/db out of my package.keywords which was needed by noatun. Other than that though it worked flawlessly. Thanks again.
_________________
Registered Linux user# 375038.
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Tue Apr 12, 2005 9:01 pm    Post subject: Reply with quote

gentoo_lan: I think the bug you're refering to may be fixed in 3.01 - not totally sure. If it isn't any chance you could give some more information? Like what 'emerge -p sys-libs/db' returns without its keyword set?
Back to top
View user's profile Send private message
gentoo_lan
l33t
l33t


Joined: 08 Sep 2004
Posts: 889
Location: Denver, CO

PostPosted: Tue Apr 12, 2005 9:18 pm    Post subject: Reply with quote

It has been fixed. Thanks for the excellent script and keep up the good work.
_________________
Registered Linux user# 375038.
Back to top
View user's profile Send private message
lookitsme
n00b
n00b


Joined: 06 Nov 2003
Posts: 48
Location: Kuala Lumpur, Malaysia

PostPosted: Sat Jun 04, 2005 11:48 am    Post subject: Error Reply with quote

I'm getting this error:
Code:

expr: syntax error
/usr/bin/etcportclean: line 653: [: too many arguments

And I can't seem to figure out why. Oh, getting it on two different machines btw...
Back to top
View user's profile Send private message
dfy
Tux's lil' helper
Tux's lil' helper


Joined: 05 Apr 2004
Posts: 79

PostPosted: Sun Jun 05, 2005 4:09 pm    Post subject: Reply with quote

This is excellent. Exactly what I've been looking for - thank you very much!
Back to top
View user's profile Send private message
torne
n00b
n00b


Joined: 22 Jan 2004
Posts: 65
Location: Cambridge, UK

PostPosted: Sun Jul 31, 2005 1:56 pm    Post subject: Reply with quote

Works nicely, but it doesn't parse package.keywords lines of the form:
~dev-php/jpgraph-1.17 ~x86
i.e. versioned keywords. Almost all my keywords are of this form, as I don't generally want any arbitrary future ~arch version of a package, just a newer version than the current arch. A number of them are no longer relevant as the referred-to version is now stable, but I don't have the patience to go through the list and check ;)

So, err, consider this a feature request?
Back to top
View user's profile Send private message
SoTired
Apprentice
Apprentice


Joined: 19 May 2004
Posts: 173

PostPosted: Sun Jul 31, 2005 9:36 pm    Post subject: Reply with quote

Sorry for taking so long with a few of these bugs, but 3.03 has now been released.

lookitsme wrote:
I'm getting this error:
Code:

expr: syntax error
/usr/bin/etcportclean: line 653: [: too many arguments

And I can't seem to figure out why. Oh, getting it on two different machines btw...


No idea what's up with that, your expr seems to be crazy. Anyways, in 3.03 I changed it to use two ifs rather than an expr. So this should be fixed.

torne wrote:
Works nicely, but it doesn't parse package.keywords lines of the form:
~dev-php/jpgraph-1.17 ~x86
i.e. versioned keywords. Almost all my keywords are of this form, as I don't generally want any arbitrary future ~arch version of a package, just a newer version than the current arch. A number of them are no longer relevant as the referred-to version is now stable, but I don't have the patience to go through the list and check ;)

So, err, consider this a feature request?


Fixed.

dfy wrote:
This is excellent. Exactly what I've been looking for - thank you very much!


Thanks for the compliments. 8)
Back to top
View user's profile Send private message
torne
n00b
n00b


Joined: 22 Jan 2004
Posts: 65
Location: Cambridge, UK

PostPosted: Mon Aug 01, 2005 12:46 am    Post subject: Reply with quote

It still doesn't do the right thing for versioned keywords - it noticed that one of them referred to a package that wasn't installed and removed it, but it didn't notice several others that are also not installed, and it didn't notice any of the ones where the version referred to is now marked stable.
Back to top
View user's profile Send private message
Gatak
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jan 2004
Posts: 99

PostPosted: Mon Aug 01, 2005 12:56 am    Post subject: Reply with quote

torne wrote:
It still doesn't do the right thing for versioned keywords - it noticed that one of them referred to a package that wasn't installed and removed it, but it didn't notice several others that are also not installed, and it didn't notice any of the ones where the version referred to is now marked stable.
What do you mean with versioned keywords? The example ~dev-php/jpgraph-1.17 ~x86 is wrong. It should be =dev-php/jpgraph-1.17 ~x86 or >=dev-php/jpgraph-1.17 ~x86 or <=dev-php/jpgraph-1.17 ~x86
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum