Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How can I find and remove non OSI licensed packages?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Mon Oct 13, 2003 10:46 pm    Post subject: How can I find and remove non OSI licensed packages? Reply with quote

I would like to remove all the packages on my system that aren't covered under an OSI approved license, since I want my system to be 100% OSS.
Is there any easy way to do this?


Last edited by hbmartin on Sat Mar 12, 2011 12:48 am; edited 1 time in total
Back to top
View user's profile Send private message
nephros
Advocate
Advocate


Joined: 07 Feb 2003
Posts: 2138
Location: Graz, Austria (Europe - no kangaroos.)

PostPosted: Mon Oct 13, 2003 11:57 pm    Post subject: Reply with quote

I don't think there is an automated way to do it, but there should be files like this:
/var/db/pkg/pkg-category/pkg-name/LICENSE
which contain the names of the licenses of all installed packages. Using some find & sed magic you should be able to make a list of all non-free packages installed.
You can find all possible licenses in /usr/portage/licenses/

hope that helps.
_________________
Please put [SOLVED] in your topic if you are a moron.
Back to top
View user's profile Send private message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Wed Jan 21, 2004 12:06 am    Post subject: Reply with quote

I don't know enough about how portage works (and I don't know Python, just Perl) but would anyone know how to hack something the selected all my installed packages and then sorted them be license?

Last edited by hbmartin on Sat Mar 12, 2011 12:48 am; edited 1 time in total
Back to top
View user's profile Send private message
sumin k'adra
n00b
n00b


Joined: 20 Sep 2003
Posts: 65
Location: santa fe, nm

PostPosted: Wed Jan 21, 2004 7:06 am    Post subject: Reply with quote

in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

-sk
Back to top
View user's profile Send private message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Wed Jan 21, 2004 2:32 pm    Post subject: Reply with quote

sumin k'adra wrote:
in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

-sk

That still won't help me with software that's already installed.


Last edited by hbmartin on Sat Mar 12, 2011 12:49 am; edited 1 time in total
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9013
Location: beyond the rim

PostPosted: Wed Jan 21, 2004 6:18 pm    Post subject: Reply with quote

sumin k'adra wrote:
in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

Where did you get this information ? 2.0.50 has many new features, but not that one.

Ok, I've written a small script to check your installed packages for licenses you don't want, you can get it at http://gentoo.devel-net.org/portage/license-checker . It is completely unofficial atm, so don't file bugs about it, if you find something odd either report it to me directly or ignore it :wink:
Oh, I'm not going to provide any list for OSI or GNU compatible licenses as you should know what you're looking for.
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Wed Jan 21, 2004 7:11 pm    Post subject: Reply with quote

Edit: The latest version of the script can be found here:

http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

I've written a little script that will go through your packages and will get rid of those with unwanted licenses.

Just add the licenses you're happy with in the oklicenses variable. The script will then do a pretend unmerge for those packages with unacceptable licenses. Change the script to take out the p option of emerge only when you're sure you wont break your system.

Code:
#!/bin/bash
# Program to find licenses of installed packages
                                                                                                                             
# Space separated list of acceptable licenses. See /usr/portage/licenses
oklicenses="GPL-1 GPL-2 as-is"
                                                                                                                             
pkgdir=/var/db/pkg
filelist=`mktemp`
                                                                                                                             
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
                                                                                                                             
sort $filelist >> ${filelist}.sorted
                                                                                                                             
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
                                                                                                                             
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -f2- -d/ >> ${filelist}.bad
                                                                                                                             
# Remove the p at your own peril! Check it with the p first just in case you missed something
for i in `cat ${filelist}.bad`; do
        emerge -pC =${i}
done
                                                                                                                             
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*


Sometimes scripts get errors when cut+pasted from webpages so you can also get this version of the script here:

http://gentoo.kicks-ass.org/scripts/licensegrep-0.1.sh
_________________
Search | Read | Answer | Report | Strip


Last edited by tomk on Thu Apr 01, 2004 9:58 pm; edited 3 times in total
Back to top
View user's profile Send private message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Thu Jan 22, 2004 4:04 am    Post subject: Reply with quote

Thanks a bunch tomk :-D
I'm having fun hacking on it.
Would you mind explaining the diff line to me?

Maybe you should submit it to be part of the gentoolkit :)


Last edited by hbmartin on Sat Mar 12, 2011 12:48 am; edited 1 time in total
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Thu Jan 22, 2004 1:13 pm    Post subject: Reply with quote

No probs, I had fun hacking it too.

When it gets to the diff you've got two lists, sorted and good. Both are in the format:

License categogy/package-version

The diff then sees what's in sorted but not in good, i.e all the bad stuff. The first grep just gets rid of the name of the file being grepped. The second grep gets the bad lines and the cut removes the license and category. I was going to leave the category, but some packages have more than one license so had to cut it somewhere else.

I'd have to tidy it up a bit before I submitted it. The main problem being that it does each emerge seperately. If I can change it so it does an emerge -pC of everything, so you get a nice list and/or an easier way of removing packages from the list. Maybe if it was reversed so you specify which licenses you don't like rather than the ones you like.

I'll play around with it a bit more
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Thu Jan 22, 2004 3:59 pm    Post subject: Reply with quote

I prefer being able to specify which licenses I like that having to dig through each license.
I'm having a problem, when I set the oklicenses to
Code:
oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON Sudo W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3"

Things like mod_perl and xscreensaver are showing up in /tmp/tmpXX.bad
Any ideas what might be going wrong?
Besides that I haven't changed the script, except to comment out the emerge loop and the rm tmp files.


Last edited by hbmartin on Sat Mar 12, 2011 12:49 am; edited 1 time in total
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Thu Jan 22, 2004 4:35 pm    Post subject: Reply with quote

Edit: The latest version of the script can be found here:

http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

OK found the problem, the good list needs sorting before it's diffed against the sorted list. I've also cleaned it up a bit so that it does all the emerges at once, gives you a chance to look at the bad list and will do a real unmerge if you tell it to. This version (0.2) should work ok, as before you can download it:

WARNING: If you tell it to this version will unmerge packages, some of which may be vital to your system. Please check the bad list before unmerging anything

http://gentoo.kicks-ass.org/scripts/licensegrep-0.2.sh

Code:
#!/bin/bash
# Program to find and remove packages with unwanted licenses
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
oklicenses="GPL-1 GPL-2 as-is BSD"
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-pC"
 
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        less ${filelist}.bad
fi
 
echo "Pretend unmerge of packages:"
emerge $emergeopts $emergefiles
 
read -n1 -p "Continue with real unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo && exit 0
fi
 
emergeopts="-C"
 
emerge $emergeopts $emergefiles
 
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*

_________________
Search | Read | Answer | Report | Strip


Last edited by tomk on Thu Apr 01, 2004 10:00 pm; edited 2 times in total
Back to top
View user's profile Send private message
hbmartin
Guru
Guru


Joined: 12 Sep 2003
Posts: 386
Location: Home is where the boxen are

PostPosted: Thu Jan 22, 2004 8:09 pm    Post subject: Reply with quote

I just unmerged a bunch of packages with 0.2, it works great!
Nice interface, it's alot easier.
Here's my code for .21

Code:

#!/bin/bash
# Program to find and remove packages with unwanted licenses
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
#OSI Compatible (+ as-is & public-domain)
#I couldn't find some of the OSI licenses in /usr/portage/licenses
#http://www.opensource.org/licenses/
 
oklicenses="as-is public-domain Academic Apache-1.1 Apple Artistic Artistic-2 BSD GPL-1 GPL-2 LGPL-2 LGPL-2.1 PLAN9 IBM MIT MPL-1.0 MPL-1.1 nethack OpenSoftware PHP PYTHON PSF-2.1.1 PSF-2.2 QPL QPL-1.0 Sleepycat SISSL-1.1 SPL ncsa-1.3 W3C wxWinLL-3 ZLIB"
 
#We should probably do OSI compatible + whatever Gentoo needs
 
 
#My system's
#Hopefully I'll narrow this down, since most of these licenses only cover one or two pacakges
#oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3 CRACKLIB DB"
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-C"
 
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
badpackages=`wc -l ${filelist}.bad | cut -f1 -d\ `
if [ $badpackages = '0' ]; then
    echo "No packages found outside of license parameters"; exit 2;
else
    echo "Found ${badpackages} packages outside of license parameters";
fi
 
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        less ${filelist}.bad
fi
 
echo "Pretend unmerge of packages:"
emerge -p $emergeopts $emergefiles
 
read -n1 -p "Continue with real unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo; exit 1
fi
 
 
emerge $emergeopts $emergefiles
 
cp ${filelist}.bad /var/log/unmerged.license
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*
 
exit 0



http://home.earthlink.net/~cocoadev/licensegrep.sh

Changes:
oklicenses is OSI by default (see the comments)
emergeopts only has to be specifed once
checking to see it there are no bad packages
s/echo && exit 0/echo; exit0/
exit codes (0 = sucess, 1 = user canceled unmerge, 2 = no bad packages found)
List of unmerged pacakges copied to /var/log/unmerged.license
Can we get this into CVS somewhere? I love working with CVS (and I think we might need it swoon), so I'd be happy to get a project from sf.net
Actually, nevermind. I want to be able to blame someone's code if this messes up my system ;)

I assume you're going to GPL this?


Last edited by hbmartin on Sat Mar 12, 2011 12:49 am; edited 1 time in total
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Fri Jan 23, 2004 4:27 pm    Post subject: Reply with quote

hbmartin wrote:
I assume you're going to GPL this?


Yep GPL, otherwise it would probably delete itself :)
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
maxcow
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jul 2003
Posts: 126

PostPosted: Sat Jan 24, 2004 1:51 am    Post subject: Reply with quote

maybe Vim's license, bzip2 license, Xfree license and free documentation license should also be included as default too.
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Fri Feb 20, 2004 12:17 pm    Post subject: Reply with quote

Me and hbmartin have been working on this script, making quite a few improvements. It's still a work in progress, but it's usable. You are now given a list of the packages which have incompatable licenses, giving you the option to keep certain packages or licenses.

We've also renamed the script to licenses2kill, you can get it here:


http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

Here's the code:

Code:
#!/bin/bash
# $Id: licenses2kill.sh,v 1.6 2004/01/30 19:57:06 tom Exp $
# Program to find and remove packages with unwanted licenses
# Copyright (c) Tom Knight (tomk@forums.gentoo.org)
#               Harold Martin (cocoadev@earthlink.net)
# This program is distributed under the terms of GPL version 2.
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
#OSI Compatible (+ as-is & public-domain)
#I couldn't find some of the OSI licenses in /usr/portage/licenses
#http://www.opensource.org/licenses/
 
oklicenses="as-is public-domain Academic Apache-1.1 Apple Artistic Artistic-2 BSD GPL-1 GPL-2 LGPL-2 LGPL-2.1 PLAN9 IBM MIT MPL-1.0 MPL-1.1 nethack OpenSoftware PHP PYTHON PSF-2.1.1 PSF-2.2 QPL QPL-1.0 Sleepycat SISSL-1.1 SPL ncsa-1.3 W3C wxWinLL-3 ZLIB"
 
#We should probably do OSI compatible + whatever Gentoo needs
 
 
#My system's
#Hopefully I'll narrow this down, since most of these licenses only cover one or two pacakges
#oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3 CRACKLIB DB"
 
cleanup ()
  {
   rm -f ${filelist}*
  }
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-C"
logfile=/var/log/unmerged.license
 
echo "Searching licenses"
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
badpackages=`wc -l ${filelist}.bad | awk '{print $1}'`
if [[ $badpackages = '0' ]]; then
    echo "No packages found outside of license parameters"; cleanup; exit 2;
else
    echo "Found ${badpackages} packages outside of license parameters";
fi
 
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        while :; do
                nl -s ") " ${filelist}.bad
                read -p "Enter the number of a package you wish to keep, or d to continue " keep_package_no
                if [[ ${keep_package_no} != 'd' ]]; then
                        if (( ${keep_package_no} > 0 )) && (( ${keep_package_no} <= ${badpackages} )); then
                                keep_package=`head -n ${keep_package_no} ${filelist}.bad | tail -n1 | cut -f2- -d/`
                                keep_license=`head -n ${keep_package_no} ${filelist}.bad | tail -n1 | cut -f1 -d\ `
                                echo "Keeping ${keep_package} package"
 
                                badleft=`wc -l ${filelist}.bad | awk '{print $1}'`
                                declare -i headn
                                declare -i tailn
                                headn=${keep_package_no}-1
                                tailn=${badleft}-${keep_package_no}
                                head -n ${headn} ${filelist}.bad > ${filelist}.tmp && \
                                tail -n ${tailn} ${filelist}.bad >> ${filelist}.tmp && \
                                mv ${filelist}.{tmp,bad}
                                 
                                # Maybe this should add the license to oklicenses for future use?
                                read -n1 -p "Keep all packages with ${keep_license} license? " keep_all
                                echo
                                if [[ $keep_all = 'y' ]]; then
                                        echo "Keeping all ${keep_license} packages"
                                        grep -v ^${keep_license} ${filelist}.bad > ${filelist}.tmp && \
                                        mv ${filelist}.{tmp,bad}
                                fi
                        else
                                echo "Number out of range"
                        fi
                else
                        break
                fi
        done
fi
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
read -n1 -p "Continue with unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo; cleanup; exit 1
fi
 
 
emerge $emergeopts $emergefiles
 
echo "Packages removed by licenses2kill" >> $logfile
echo `date +%c` >> $logfile
echo >> $logfile
cat ${filelist}.bad >> $logfile
echo >> $logfile
 
cleanup
 
exit 0

_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
HolyCoitus
n00b
n00b


Joined: 16 Jun 2003
Posts: 21
Location: Fort Wayne, Indiana

PostPosted: Tue Jul 20, 2004 6:39 pm    Post subject: Reply with quote

Very nice script so far. I changed a few things to my likings, so I might as well post them back to the forums. Going to do code snippets so that the thread doesn't get too confusing with pasted code.

http://mywebpages.comcast.net/HolyCoitus/licenses2kill.sh

Changes:
    Added output so you can have flat text files to view afterwards if you so choose.
    Added a dump of the licenses that are not on the list, so you can get an idea much easier of what they are
    Added a check at the end to see if you are root or in the portage group before attempting to unmerge anything


Code:
# Blank for no logging, otherwise it will output to the file you specify
licenselog=""
packagelog=""


Code:
# separate the licenses from the packages that they associate with, and then save the variable for later use
badlicenses=`cat "${filelist}.bad" | cut -f1 -d\  | sort -u | tee  ${filelist}.liccrop | wc -l`
echo "There are $badlicenses licenses outside of parameters"

read -n1 -p "Do you want to view the list of licenses that are being rejected? (y/n)" licrep

echo

if [[ $licrep = 'y' ]];
then
   if [[ $licenselog != "" ]]
   then
      nl -s ") " ${filelist}.liccrop | tee $licenselog
   else
      nl -s ") " ${filelist}.liccrop
   fi
fi


Code:
      if [[ $packagelog != "" ]]
      then
         nl -s ") " ${filelist}.bad | tee $packagelog
      else
         nl -s ") " ${filelist}.bad
      fi


Code:
# Check to see if the user is root or in the portage group before you try to unmerge things.
if [[ `/usr/bin/whoami` = 'root' || `groups | grep -c "portage"` -ge 1 ]]
then

_________________
Celebrate this chance to be alive and breathing!
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Fri Oct 08, 2004 5:56 pm    Post subject: Reply with quote

Seems I missed this when you first posted. I like the changes you've made, especially dumping the list of licenses.
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Fri Oct 08, 2004 6:01 pm    Post subject: Reply with quote

Moved from Other Things Gentoo.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum