How can I find and remove non OSI licensed packages?

Message

hbmartin · Post by **hbmartin** » Mon Oct 13, 2003 10:46 pm

I would like to remove all the packages on my system that aren't covered under an OSI approved license, since I want my system to be 100% OSS.
Is there any easy way to do this?

nephros · Post by **nephros** » Mon Oct 13, 2003 11:57 pm

I don't think there is an automated way to do it, but there should be files like this:
/var/db/pkg/pkg-category/pkg-name/LICENSE
which contain the names of the licenses of all installed packages. Using some find & sed magic you should be able to make a list of all non-free packages installed.
You can find all possible licenses in /usr/portage/licenses/

hope that helps.

hbmartin · Post by **hbmartin** » Wed Jan 21, 2004 12:06 am

I don't know enough about how portage works (and I don't know Python, just Perl) but would anyone know how to hack something the selected all my installed packages and then sorted them be license?

sumin k'adra · Post by **sumin k'adra** » Wed Jan 21, 2004 7:06 am

in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

-sk

hbmartin · Post by **hbmartin** » Wed Jan 21, 2004 2:32 pm

sumin k'adra wrote:in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

-sk

That still won't help me with software that's already installed.

Genone · Post by **Genone** » Wed Jan 21, 2004 6:18 pm

sumin k'adra wrote:in the next version of portage (2.0.50) you should be able to select which licenses you find acceptable in make.conf. Try searching the forums for the thread on it.

Where did you get this information ? 2.0.50 has many new features, but not that one.

Ok, I've written a small script to check your installed packages for licenses you don't want, you can get it at http://gentoo.devel-net.org/portage/license-checker . It is completely unofficial atm, so don't file bugs about it, if you find something odd either report it to me directly or ignore it

Oh, I'm not going to provide any list for OSI or GNU compatible licenses as you should know what you're looking for.

tomk · Post by **tomk** » Wed Jan 21, 2004 7:11 pm

Edit: The latest version of the script can be found here:

http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

I've written a little script that will go through your packages and will get rid of those with unwanted licenses.

Just add the licenses you're happy with in the oklicenses variable. The script will then do a pretend unmerge for those packages with unacceptable licenses. Change the script to take out the p option of emerge only when you're sure you wont break your system.

Code: Select all

#!/bin/bash
# Program to find licenses of installed packages
                                                                                                                              
# Space separated list of acceptable licenses. See /usr/portage/licenses
oklicenses="GPL-1 GPL-2 as-is"
                                                                                                                              
pkgdir=/var/db/pkg
filelist=`mktemp`
                                                                                                                              
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
                                                                                                                              
sort $filelist >> ${filelist}.sorted
                                                                                                                              
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
                                                                                                                              
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -f2- -d/ >> ${filelist}.bad
                                                                                                                              
# Remove the p at your own peril! Check it with the p first just in case you missed something
for i in `cat ${filelist}.bad`; do
        emerge -pC =${i}
done
                                                                                                                              
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*

Sometimes scripts get errors when cut+pasted from webpages so you can also get this version of the script here:

http://gentoo.kicks-ass.org/scripts/licensegrep-0.1.sh

hbmartin · Post by **hbmartin** » Thu Jan 22, 2004 4:04 am

Thanks a bunch tomk

I'm having fun hacking on it.
Would you mind explaining the diff line to me?

Maybe you should submit it to be part of the gentoolkit

tomk · Post by **tomk** » Thu Jan 22, 2004 1:13 pm

No probs, I had fun hacking it too.

When it gets to the diff you've got two lists, sorted and good. Both are in the format:

License categogy/package-version

The diff then sees what's in sorted but not in good, i.e all the bad stuff. The first grep just gets rid of the name of the file being grepped. The second grep gets the bad lines and the cut removes the license and category. I was going to leave the category, but some packages have more than one license so had to cut it somewhere else.

I'd have to tidy it up a bit before I submitted it. The main problem being that it does each emerge seperately. If I can change it so it does an emerge -pC of everything, so you get a nice list and/or an easier way of removing packages from the list. Maybe if it was reversed so you specify which licenses you don't like rather than the ones you like.

I'll play around with it a bit more

hbmartin · Post by **hbmartin** » Thu Jan 22, 2004 3:59 pm

I prefer being able to specify which licenses I like that having to dig through each license.
I'm having a problem, when I set the oklicenses to

Code: Select all

oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON Sudo W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3"

Things like mod_perl and xscreensaver are showing up in /tmp/tmpXX.bad
Any ideas what might be going wrong?
Besides that I haven't changed the script, except to comment out the emerge loop and the rm tmp files.

tomk · Post by **tomk** » Thu Jan 22, 2004 4:35 pm

Edit: The latest version of the script can be found here:

http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

OK found the problem, the good list needs sorting before it's diffed against the sorted list. I've also cleaned it up a bit so that it does all the emerges at once, gives you a chance to look at the bad list and will do a real unmerge if you tell it to. This version (0.2) should work ok, as before you can download it:

WARNING: If you tell it to this version will unmerge packages, some of which may be vital to your system. Please check the bad list before unmerging anything

http://gentoo.kicks-ass.org/scripts/licensegrep-0.2.sh

Code: Select all

#!/bin/bash
# Program to find and remove packages with unwanted licenses
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
oklicenses="GPL-1 GPL-2 as-is BSD"
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-pC"
 
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        less ${filelist}.bad
fi
 
echo "Pretend unmerge of packages:"
emerge $emergeopts $emergefiles
 
read -n1 -p "Continue with real unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo && exit 0
fi
 
emergeopts="-C"
 
emerge $emergeopts $emergefiles
 
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*

hbmartin · Post by **hbmartin** » Thu Jan 22, 2004 8:09 pm

I just unmerged a bunch of packages with 0.2, it works great!
Nice interface, it's alot easier.
Here's my code for .21

Code: Select all

#!/bin/bash
# Program to find and remove packages with unwanted licenses
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
#OSI Compatible (+ as-is & public-domain)
#I couldn't find some of the OSI licenses in /usr/portage/licenses
#http://www.opensource.org/licenses/
 
oklicenses="as-is public-domain Academic Apache-1.1 Apple Artistic Artistic-2 BSD GPL-1 GPL-2 LGPL-2 LGPL-2.1 PLAN9 IBM MIT MPL-1.0 MPL-1.1 nethack OpenSoftware PHP PYTHON PSF-2.1.1 PSF-2.2 QPL QPL-1.0 Sleepycat SISSL-1.1 SPL ncsa-1.3 W3C wxWinLL-3 ZLIB"
 
#We should probably do OSI compatible + whatever Gentoo needs
 
 
#My system's
#Hopefully I'll narrow this down, since most of these licenses only cover one or two pacakges
#oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3 CRACKLIB DB"
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-C"
 
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
badpackages=`wc -l ${filelist}.bad | cut -f1 -d\ `
if [ $badpackages = '0' ]; then
    echo "No packages found outside of license parameters"; exit 2;
else
    echo "Found ${badpackages} packages outside of license parameters";
fi
 
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        less ${filelist}.bad
fi
 
echo "Pretend unmerge of packages:"
emerge -p $emergeopts $emergefiles
 
read -n1 -p "Continue with real unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo; exit 1
fi
 
 
emerge $emergeopts $emergefiles
 
cp ${filelist}.bad /var/log/unmerged.license
# Comment this line if you want to keep the lists (/tmp/tmpXXXX)
rm -f ${filelist}*
 
exit 0

http://home.earthlink.net/~cocoadev/licensegrep.sh

Changes:
oklicenses is OSI by default (see the comments)
emergeopts only has to be specifed once
checking to see it there are no bad packages
s/echo && exit 0/echo; exit0/
exit codes (0 = sucess, 1 = user canceled unmerge, 2 = no bad packages found)
List of unmerged pacakges copied to /var/log/unmerged.license
Can we get this into CVS somewhere? I love working with CVS (and I think we might need it swoon), so I'd be happy to get a project from sf.net
Actually, nevermind. I want to be able to blame someone's code if this messes up my system

I assume you're going to GPL this?

tomk · Post by **tomk** » Fri Jan 23, 2004 4:27 pm

hbmartin wrote:I assume you're going to GPL this?

Yep GPL, otherwise it would probably delete itself

maxcow · Post by **maxcow** » Sat Jan 24, 2004 1:51 am

maybe Vim's license, bzip2 license, Xfree license and free documentation license should also be included as default too.

tomk · Post by **tomk** » Fri Feb 20, 2004 12:17 pm

Me and hbmartin have been working on this script, making quite a few improvements. It's still a work in progress, but it's usable. You are now given a list of the packages which have incompatable licenses, giving you the option to keep certain packages or licenses.

We've also renamed the script to licenses2kill, you can get it here:

http://gentoo.kicks-ass.org/scripts/licenses2kill.sh

Here's the code:

Code: Select all

#!/bin/bash
# $Id: licenses2kill.sh,v 1.6 2004/01/30 19:57:06 tom Exp $
# Program to find and remove packages with unwanted licenses
# Copyright (c) Tom Knight (tomk@forums.gentoo.org)
#               Harold Martin (cocoadev@earthlink.net)
# This program is distributed under the terms of GPL version 2.
 
# Space separated list of acceptable licenses. See /usr/portage/licenses
#OSI Compatible (+ as-is & public-domain)
#I couldn't find some of the OSI licenses in /usr/portage/licenses
#http://www.opensource.org/licenses/
 
oklicenses="as-is public-domain Academic Apache-1.1 Apple Artistic Artistic-2 BSD GPL-1 GPL-2 LGPL-2 LGPL-2.1 PLAN9 IBM MIT MPL-1.0 MPL-1.1 nethack OpenSoftware PHP PYTHON PSF-2.1.1 PSF-2.2 QPL QPL-1.0 Sleepycat SISSL-1.1 SPL ncsa-1.3 W3C wxWinLL-3 ZLIB"
 
#We should probably do OSI compatible + whatever Gentoo needs
 
 
#My system's
#Hopefully I'll narrow this down, since most of these licenses only cover one or two pacakges
#oklicenses="GPL-1 GPL-2 as-is Apache-1.1 Artistic BSD BZIP2 DIVX EDB LGPL-2.1 FDL-1.1 FLEX FTL Info-ZIP MIT X11 MOTIF MPL-1.1 OPL OpenSoftware PSF-2.2 PYTHON W3C ZLIB fontconfig fmod freedist povlegal-3.5 public-domain wxWinLL-3 CRACKLIB DB"
 
cleanup ()
  {
   rm -f ${filelist}*
  }
 
pkgdir=/var/db/pkg
filelist=`mktemp`
emergeopts="-C"
logfile=/var/log/unmerged.license
 
echo "Searching licenses"
for i in `find $pkgdir -name LICENSE`; do
        license=`cat "$i"`
        echo -n "$license " >> $filelist
        echo $i | cut -f5-6 -d/ >> $filelist
done
 
sort $filelist >> ${filelist}.sorted
 
for i in $oklicenses; do
        grep $i ${filelist}.sorted >> ${filelist}.good
done
 
cat ${filelist}.good | sort -u >> ${filelist}.tmp && mv ${filelist}.{tmp,good}
diff -u ${filelist}.sorted ${filelist}.good | grep -v '^--' | grep '^-' | cut -c2- >> ${filelist}.bad
 
badpackages=`wc -l ${filelist}.bad | awk '{print $1}'`
if [[ $badpackages = '0' ]]; then
    echo "No packages found outside of license parameters"; cleanup; exit 2;
else
    echo "Found ${badpackages} packages outside of license parameters";
fi
 
 
read -n1 -p "Do you want to see the list of bad packages? (y/n) " reply
 
echo
 
if [[ $reply = 'y' ]]; then
        while :; do
                nl -s ") " ${filelist}.bad
                read -p "Enter the number of a package you wish to keep, or d to continue " keep_package_no
                if [[ ${keep_package_no} != 'd' ]]; then
                        if (( ${keep_package_no} > 0 )) && (( ${keep_package_no} <= ${badpackages} )); then
                                keep_package=`head -n ${keep_package_no} ${filelist}.bad | tail -n1 | cut -f2- -d/`
                                keep_license=`head -n ${keep_package_no} ${filelist}.bad | tail -n1 | cut -f1 -d\ `
                                echo "Keeping ${keep_package} package"
 
                                badleft=`wc -l ${filelist}.bad | awk '{print $1}'`
                                declare -i headn
                                declare -i tailn
                                headn=${keep_package_no}-1
                                tailn=${badleft}-${keep_package_no}
                                head -n ${headn} ${filelist}.bad > ${filelist}.tmp && \
                                tail -n ${tailn} ${filelist}.bad >> ${filelist}.tmp && \
                                mv ${filelist}.{tmp,bad}
                                 
                                # Maybe this should add the license to oklicenses for future use?
                                read -n1 -p "Keep all packages with ${keep_license} license? " keep_all
                                echo
                                if [[ $keep_all = 'y' ]]; then
                                        echo "Keeping all ${keep_license} packages"
                                        grep -v ^${keep_license} ${filelist}.bad > ${filelist}.tmp && \
                                        mv ${filelist}.{tmp,bad}
                                fi
                        else
                                echo "Number out of range"
                        fi
                else
                        break
                fi
        done
fi
 
for i in `cat ${filelist}.bad | cut -f2- -d/` ; do
        emergefiles=${emergefiles:-""}"=${i} "
done
 
read -n1 -p "Continue with unmerge? Saying yes will permanently remove packages (y/n) " reply2
 
if [[ $reply2 != 'y' ]]; then
        echo; cleanup; exit 1
fi
 
 
emerge $emergeopts $emergefiles
 
echo "Packages removed by licenses2kill" >> $logfile
echo `date +%c` >> $logfile
echo >> $logfile
cat ${filelist}.bad >> $logfile
echo >> $logfile
 
cleanup
 
exit 0

HolyCoitus · Post by **HolyCoitus** » Tue Jul 20, 2004 6:39 pm

Very nice script so far. I changed a few things to my likings, so I might as well post them back to the forums. Going to do code snippets so that the thread doesn't get too confusing with pasted code.

http://mywebpages.comcast.net/HolyCoitu ... es2kill.sh

Changes:

Added output so you can have flat text files to view afterwards if you so choose.
Added a dump of the licenses that are not on the list, so you can get an idea much easier of what they are
Added a check at the end to see if you are root or in the portage group before attempting to unmerge anything

Code: Select all

# Blank for no logging, otherwise it will output to the file you specify
licenselog=""
packagelog=""

Code: Select all

# separate the licenses from the packages that they associate with, and then save the variable for later use
badlicenses=`cat "${filelist}.bad" | cut -f1 -d\  | sort -u | tee  ${filelist}.liccrop | wc -l`
echo "There are $badlicenses licenses outside of parameters"

read -n1 -p "Do you want to view the list of licenses that are being rejected? (y/n)" licrep

echo

if [[ $licrep = 'y' ]];
then
	if [[ $licenselog != "" ]]
	then
		nl -s ") " ${filelist}.liccrop | tee $licenselog
	else
		nl -s ") " ${filelist}.liccrop
	fi
fi

Code: Select all

		if [[ $packagelog != "" ]]
		then
			nl -s ") " ${filelist}.bad | tee $packagelog
		else
			nl -s ") " ${filelist}.bad
		fi

Code: Select all

# Check to see if the user is root or in the portage group before you try to unmerge things.
if [[ `/usr/bin/whoami` = 'root' || `groups | grep -c "portage"` -ge 1 ]]
then

tomk · Post by **tomk** » Fri Oct 08, 2004 5:56 pm

Seems I missed this when you first posted. I like the changes you've made, especially dumping the list of licenses.

Post by **pjp** » Fri Oct 08, 2004 6:01 pm

Moved from Other Things Gentoo.