findcruft: Yet another script to find obsolete files

[hepta_sean]

Hello!

EDIT: Last version is 1.0.4.

This is a script, which I wrote as a successor to the great cruft script by ecatmur.
Unfortunately ecatmur is not really maintaining this thing anymore, so I wrote a similar script borrowing some parts of cruft but trying to avoid some of the shortcomings of it.
(I think Ed wrote a successor himself - cruft2.py - but he never announced it and I don't know if he's working on it.)

The pretty verbose help message for this beast follows:

findcruft v1.0.4

Usage:
findcruft [ -h | -? ]
findcruft [ -D ] [ -f targetdir ]
findcruft > cruftlist
FINDCRUFT_CONFDIR=/path/to/confdir findcruft

Options:
-h:
Print this help message and exit.
-?:
Print short usage information and exit.
-D:
Switch to debug mode. Gives some diagnostic
information and does not clean the temporary
files after finishing.
-f targetdir:
Switch to force mode. Found cruft is packed
into a tar.bz2 archive in targetdir and
deleted.

This program compares the files on your filesystem to the ones
managed by portage and prints out files which do not belong to
any package.

Additionally it uses the files in /usr/local/lib/findcruft to
exclude false positives, i.e. configuration and similar files
which are not managed by portage, but should nevertheless be on
the system. You can manipulate the location of this directory
by the FINDCRUFT_CONFDIR environment variable.

All of the files in this directory are sourced into findcruft
(and hence processed by bash). Thus, you can use all of bash's
programming features. Additionally the following portage settings
are grabbed from the configuration and can be used in all files:
$PORTDIR, $DISTDIR, $PKGDIR, $PORT_LOGDIR, $PORTDIR_OVERLAY,
$PORTAGE_TMPDIR, $CCACHE_DIR, $ARCH, and $CHOST

The file ignore should output a list of directories which will
be completely ignored, i.e. they are neither searched during the
find phase nor during the building of the false positive list (as
long as you do not do it in your nopackage.local). You can also
add your own additions in an ignore.local file. This is handy if
you have some network file systems or similar which you do not
want to put under too heavy load.

The file symlink should output a list of symlink-target pairs,
which is applied to the contents of packages managed by portage.
This is necessary, because some packages install their files
e.g. into /usr/man, but this is a symlink to /usr/share/man.

You can add your own symlinks to symlink.local. For example if
/opt is a symlink to /usr/opt (because /opt was too big for the
root partition and you wanted it to be on the /usr partition
together with the other large applications) then add a line
echo /opt /usr/opt to symlink.local.

The other files are expected to output a list of false positive
files one file per line onto stdout. So, if you want to ignore
a single file you can just do echo /full/path/to/file. You can
use the predefined functions cruftfile /full/path/file_regexp and
cruftdir /full/path/dir_regexp, which do a grep on the unmanaged
files and ignore all files matching the given regexp or all
directories matching the regexp and their contents, respectively.
Look into the included files for more elaborated examples.

You can also use the functions from the file functions in these
rule files:
* nsplugin plugin.so plugin.a plugin.la adds all files given as
arguments to the false positives in the plugin directory for
Netscape style browsers (Mozilla, Firefox, ...).
* fontdir relative/path /absolute/path adds the generated cache,
directory, and scale files for font directories to the false
positives. If a relative path is given the directory is assumed
to reside under /usr/share/fonts.
* xpiapp relative/path /absolute/path adds the typical false
positives for XPI applications such as Mozilla and Firefox to
the list (i.e. things in the chrome and components directories,
etc.). Relative pathes are assumed under /usr/lib.

The file nopackage contains a list of false positives without
corresponding package (for some of these one might argue, that
they belong to some of the base packages, which are present on
any system, perhaps they will be moved to the corresponding
package sometime). The file nopackage.local can be used for
local settings.

The rest of the files is organized in a category/package
hierarchy. If a package is present on the system (according to
/var/db/pkg) the corresponding file is sourced with the
variables $VERSION set to the version and $SLOT set to the slot
of the package.

Please make additions to ignore, symlink, and nopackage in the
corresponding *.local files, so that they do not get overwritten,
when you unpack a new rule archive into the directory.

It is really easy to install:

Code: Select all

$ cd
$ wget http://user.cs.tu-berlin.de/~sean/findcruft-1.0.4.tar.bz2
$ cd /usr/local
$ su -c "tar -xjf ${HOME}/findcruft-1.0.4.tar.bz2"

Now you can test it:

Code: Select all

$ findcruft > cruftfiles

EDIT: You do not have to be root to run this script anymore. It will however try to do "su -c" if it needs root permissions. At these points you might have to give a password.

The files listed are the ones findcruft couldn't find a reason to exist for.
You can consider to delete them, but beware: They might as well be important application data.

If you think they are false positives and know to which package they belong you can create/edit a rule file in /usr/local/lib/findcruft/<category>/<package>.
Then, we can exchange these rule files in this thread and build new rule archives from time to time.
If it is a false positive due to a personal/local setting on your machine, you can add it into /usr/local/lib/findcruft/nopackage.local.
There it won't interfere with the exchanged rules.

Greetings,
sean

[tlaloctlaloc]

Download is not possible....'Connection refused'.
Bye.

[dzho002]

Same to me

Code: Select all

 wget http://user.cs.tu-berlin.de/~sean/findcruft-1.0.0.tar.bz2
--23:45:44--  http://user.cs.tu-berlin.de/%7Esean/findcruft-1.0.0.tar.bz2
           => `findcruft-1.0.0.tar.bz2'
Resolving user.cs.tu-berlin.de... 130.149.17.156
Connecting to user.cs.tu-berlin.de[130.149.17.156]:80... failed: Connection refused.

[hepta_sean]

Ugh! Sorry for that!

The file server at the university was down. It has some problems this week.

But ATM it's up again. In case this happens again, I have mirrored it on another server:

Code: Select all

$ wget http://tfs.cs.tu-berlin.de/~bbraatz/findcruft-1.0.0.tar.bz2

Greetings,
sean

[soulwarrior]

So far I use this lines in my nopackage.local to accomodate to mysql, nessus & postgresql:

Code: Select all

# for nessus
find /usr/lib/nessus/plugins
find /var/lib/nessus

# for mysql databases
find /var/lib/mysql

# for postgresql databases
find /var/lib/postgresql/data

Maybe this should be moved to the category/package hierarchy?

[Wyckliff]

Nice script, hepta_sean!
Could it be modified to not check network filesystems?
Thanks.

[wescott]

xf86Config came up on mine. I know that one is a false-positive .


Also, I got alot of results in /usr/bin

c++
cc
cpp
g++

...

i686-pc-linux-gnu-c++
i686-pc-linux-gnu-g++
i686-pc-linux-gnu-gcc

...

mtrace
yacc
libdb.a

if all of these (and those I left out) truely are false positives, shouldn't /usr/bin be overlooked?

[teedog]

I wonder if the Gentoo team has plans to officially support/develop some kind of cruft-cleaning script or program like this? Seems like they are very popular with the obsessive-compulsive Gentoo community.

[hepta_sean]

xf86Config came up on mine. I know that one is a false-positive .

Do you mean the program (I think there was one on xfree) or the configuration file /etc/X11/XF86Config?
Do you use xorg.conf? Then it should be either xorgconfig/xorgcfg (the program) or xorg.conf (the configuration file) and the old one is really cruft.

I don't know, which conf file is used if both - XF86Config and xorg.conf - are there (xorg honours the old xf86Configs), so be careful (make some backups in your home dir, before deleting/overwriting).

If you still use xfree, I'll add that one.

Also, I got alot of results in /usr/bin

c++
cc
cpp
g++
...
i686-pc-linux-gnu-c++
i686-pc-linux-gnu-g++
i686-pc-linux-gnu-gcc
...
mtrace
yacc
libdb.a

c++, cpp, g++, i686-pc-linux-gnu-* and co belong to gcc-config on my system:

Code: Select all

sean@lois ~ $ grep /usr/bin/c++ /var/db/pkg/*/*/CONTENTS
/var/db/pkg/sys-devel/gcc-config-1.3.7-r1/CONTENTS:obj /usr/bin/c++ 0bda93acb31d794b3c0d457371f8b306 1098884742
sean@lois ~ $ grep /usr/bin/i686-pc-linux-gnu-c++ /var/db/pkg/*/*/CONTENTS
/var/db/pkg/sys-devel/gcc-config-1.3.7-r1/CONTENTS:obj /usr/bin/i686-pc-linux-gnu-c++ 0bda93acb31d794b3c0d457371f8b306 1098884742

mtrace belongs to glibc:

Code: Select all

sean@lois ~ $ grep /usr/bin/mtrace /var/db/pkg/*/*/CONTENTS
/var/db/pkg/sys-libs/glibc-2.3.4.20041102/CONTENTS:obj /usr/bin/mtrace 5a9cddd7824dae746ecd48c209f5892c 1099945949

yacc belongs to yacc (who would have thought? ):

Code: Select all

sean@lois ~ $ grep /usr/bin/yacc /var/db/pkg/*/*/CONTENTS
/var/db/pkg/dev-util/yacc-1.9.1-r2/CONTENTS:obj /usr/bin/yacc d1788f9a38ae04b5efede90f50390aa1 1088787310

So, all of these shouldn't be reported (the contents of all CONTENTS files are not reported - that's the main functionality of this script).
Something must be really odd on your system.
Do you have those packages?
What's the result of the grep statements above on your system?
Have you compiled/installed them by hand?
Have you unpacked some kind of rescue or stage tarball without (re-)emerging the corresponding packages afterwards?

if all of these (and those I left out) truely are false positives, shouldn't /usr/bin be overlooked?

They shouldn't be reported. /usr/bin should be checked, 'cause it is managed by portage. Only false positives should be things generated in post-installs (rarely) and symlinks (if you use binary packages they are not overwritten on updates and are not considered contents of the package).

To the others: I hope I will get your suggestions in tonight (CET).

Greetings,
sean

[hepta_sean]

O.K., I just uploaded the new version:

Code: Select all

wget http://user.cs.tu-berlin.de/~sean/findcruft-1.0.2.tar.bz2

So far I use this lines in my nopackage.local to accomodate to mysql, nessus & postgresql:

Code: Select all

...

Maybe this should be moved to the category/package hierarchy?

Done!

Nice script, hepta_sean!

Thanks!

Could it be modified to not check network filesystems?

I modified it, so that the ignored directories are not hardcoded anymore, but are echoed to stdout by ignore in /usr/local/lib/findcruft. So there are two possibilities:
1.) You let your network filesystems be echoed by ignore.local.
2.) You write a nice piece of code, which parses fstab or mtab and puts out the filesystems, which are network filesystems.

Up to now, these directories are only ignored in the "Finding files on your filesystem ..." phase. If a find or cruftdir is used in the "Building false positive list ..." phase the filesystems are still accessed.

Hope, that helps.

Greetings,
sean

[Sonderblade]

Would it be possible to make it so it doesn't have to be run as root everytime? It feels unsafe.

[stahlsau]

Would it be possible to make it so it doesn't have to be run as root everytime? It feels unsafe.

Since, as user, you don´t have read-access to all the systems directories, how should that work? don´t think it´s possible.

BTW, nice script, works better for me that the "old" cruft script.

[hepta_sean]

Would it be possible to make it so it doesn't have to be run as root everytime? It feels unsafe.

Since, as user, you don´t have read-access to all the systems directories, how should that work? don´t think it´s possible.

On my system there is only a small number of directories for which root access is really necessary:

Code: Select all

sean@lois ~ $ findcruft
Finding files on the filesystem ...
find: /etc/cups/certs: Permission denied
find: /etc/samba/private: Permission denied
find: /var/lib/xdm: Permission denied
find: /var/lib/samba/private: Permission denied
find: /var/lib/slocate: Permission denied
find: /var/spool/cron: Permission denied
find: /var/spool/cups: Permission denied
Finding files managed by portage ...
Comparing to get unmanaged files ...
Building false positive list ...
find: /etc/samba/private: Permission denied
find: /var/lib/samba/private: Permission denied
find: /etc/cups/certs: Permission denied
find: /var/spool/cups: Permission denied
Comparing to get cruft files ...
findcruft thinks these files are cruft:
---------------------------------------
/etc/.fstab.hal.f
/etc/.fstab.hal.i
/etc/samba/private
/var/spool/cups
--------------------------------------------------
Please check carefully, before removing any files!
Cleaning up ...

I could monitor the find commands to record all "Permission denied" complaints and then just sudo or su -c for these. The rest of the script can run as user. I'll add that to my TODO list.

Until then, the extra paranoid user can run it as user and just ignore the "Permission denied"s and possible cruft reports due to these.

BTW, nice script, works better for me that the "old" cruft script.

Thanks! I hope, I'll maintain it a little longer (too bad, that ecatmur is gone). And I'm planning on also writing a successor to "dep", but that's far more ambitious ...

Greetings,
sean

[stahlsau]

arm, a little OT:
do my posting´s apostrophes "´" look weird? For me they look normal, as they should, but if s/o quotes my posts, they show up as a "´", whatever that may be...
Please give feedback so that i can correct it if it displays wrong.

[soulwarrior]

So far I use this lines in my nopackage.local to accomodate to mysql, nessus & postgresql:

Code: Select all

...

Maybe this should be moved to the category/package hierarchy?

Done!

thanks

I will test your new version.

Maybe it would be nice to add to findcruft a flag, which would put every cruft file in a tar-archiv and delete the cruft files afterwards.
One would have to set the directory, where to store the archiv, in a config file.

Like that one could in the worst case, reboot into a livecd and undo what should not have been done

Of course one could perform this steps manually, but adding this functionality to findcruft would be nice.

What do you think?

[GentooBox]

The script finds a lot of stuff in my /dev/ folder.
and also all my files in /mnt/backup

I have amd64, it also finds all files in /lib64/

Code: Select all

du -h cruft
16M     cruft

thats a lot of "cruft"....

[soulwarrior]

Just did a fresh install of fcron and findcruft wants to delete this files:

Code: Select all

/var/spool/cron/fcrontabs/root
/var/spool/cron/fcrontabs/root.orig

I think fcron generates these files at runtime.

Postfix seems to need some files under /var/spool/postfix to function correctly, which findcruft wants to remove, so I adde the following line to nopacke.local:

Code: Select all

find /var/spool/postfix

[hepta_sean]

Hello!

I did a new version:

Code: Select all

wget http://user.cs.tu-berlin.de/~sean/findcruft-1.0.3.tar.bz2

Would it be possible to make it so it doesn't have to be run as root everytime? It feels unsafe.

It now works for normal users and only does "su -c" when necessary.
If you do not allow users in the wheel group to "su" without password, you'll have to enter the root password for each of these actions - once in finding files and several times in force mode (see below).

Maybe it would be nice to add to findcruft a flag, which would put every cruft file in a tar-archiv and delete the cruft files afterwards.
One would have to set the directory, where to store the archiv, in a config file.

You can now do something like:

Code: Select all

findcruft -f /root

The script will store an archive "cruft-<date>-<number>.tar.bz2" in "/root" (for this example) with all found cruft files in it. It will then delete these files (directories are tried to be removed by "rmdir" - that way you will get an error message if only the directory is suspected to be cruft but not its contents).
It will "su" once for creation of the archive and once for each cruft file. That will be a lot of password typing for those who do not use one of the "su without password" options.
Attention: Please be careful with this option! Only use it if you do not have any false positives left. You can use the archives to restore, but you will perhaps need to boot into a live-cd.

Just did a fresh install of fcron and findcruft wants to delete this files:

Code: Select all

/var/spool/cron/fcrontabs/root
/var/spool/cron/fcrontabs/root.orig

...
Postfix seems to need some files under /var/spool/postfix to function correctly

Added to the corresponding category/package files.

Moreover, I rewrote the "cruftdir" function to not use "find" anymore (we already did a "find" for the whole filesystem in the first phase). Now "cruftfile" and "cruftdir" do "grep" on the unmanaged files. Should be a little faster, cannot get into permission problems and you can use grep's regular expressions.

I also added a debug flag ("-D"), which gives a lot of information on what is happening (variables and sourced configuration files) and prevents the removal of the temporary files in "/tmp/findcruft-*".

The script finds a lot of stuff in my /dev/ folder.
and also all my files in /mnt/backup

These should be ignored! Please try the latest version with the debug option.
It should output (among a lot of other stuff):

Code: Select all

sean@lois ~ $ findcruft -D
...
DEBUG: Sourcing /usr/local/lib/findcruft/ignore.
DEBUG: $IGNORE=/boot /dev /home /media /mnt /proc /root /srv /sys /tmp /usr/local /timestamp.chk /usr/portage /usr/portage/distfiles /usr/portage/packages /var/log/portage /usr/local/portage/manual /var/tmp/portage /var/tmp/portage-pkg /var/tmp/sync /var/tmp/timestamp.chk /var/tmp/ccache /var/cache/edb /var/db/pkg
Finding files on the filesystem ...
...

I have amd64, it also finds all files in /lib64/

I added this to "nopackage" for $ARCH="amd64". Do they really not belong to any package?

arm, a little OT:
do my posting´s apostrophes "´" look weird? For me they look normal, as they should, but if s/o quotes my posts, they show up as a "´", whatever that may be...
Please give feedback so that i can correct it if it displays wrong.

Yeah, your apostrophes ("'") look like single quotes ("´"). I don't know, why ...

Greetings,
sean

[hepta_sean]

Bugfix release:

Code: Select all

wget http://user.cs.tu-berlin.de/~sean/findcruft-1.0.4.tar.bz2

The force mode didn't work with files containing special characters like | or $. Fixed that. Nothing else new.

Greetings,
sean

[soulwarrior]

The script will store an archive "cruft-<date>-<number>.tar.bz2" in "/root" (for this example) with all found cruft files in it. It will then delete these files (directories are tried to be removed by "rmdir" - that way you will get an error message if only the directory is suspected to be cruft but not its contents).

Thanks


I have some more false postives:

Code: Select all

#
# vixie-cron stores the different cron jobs of the users in /var/spool/cron/crontabs/
# only delete the crontabs of users, which don't exist (anymore) on the system
#
for x in /var/spool/cron/crontabs/*
do
    name=`basename $x`
    if [ `grep "^$name:" /etc/passwd` ]
    then
        echo $x
    fi
done

#
# cups-pdf stores the pdf's by default in this directory
# remove directories from non-existant users
echo /var/spool/cups-pdf
echo /var/spool/cups-pdf/SPOOL
for x in /var/spool/cups-pdf/*
do
    name=`basename $x`
    if [ `grep "^$name:" /etc/passwd` ]
    then
        echo $x
    fi
done


#
# mysql stores it's pid & sock in this dir
# only remove it when mysql is not running
#
if [ -n "`/usr/bin/pidof mysqld`" ]
then
    echo /var/run/mysqld/mysqld.pid;
    echo /var/run/mysqld/mysqld.sock;
fi
# don't remove it's log files
cruftdir /var/log/mysql


#
# pid of apache2
# only to be removed, when apache2 is not running
#
if [ -n "`/usr/bin/pidof apache2`" ]
then
    echo /var/run/apache2.pid;
fi
# don't remove it's log files
cruftdir /var/log/apache2


#
# sys-apps/acpid
#
echo /var/log/acpid

Could there be a better way to take into account pid-files or log-files the programs generate at runtime?

[chunderbunny]

Ok, it picked up /etc/portage/profile/virtuals which is a portage config file (user created) which I don't think it should.

[soulwarrior]

I just wondered why findruft did also pickup socket files, so I looked into the findcruft script and saw that find didn't include the "-type f" option.

Maybe it would be an idea to use the "-type f" option for find to only take "real" files into account, or was there an reason to not use this option?

[hepta_sean]

Maybe it would be an idea to use the "-type f" option for find to only take "real" files into account, or was there an reason to not use this option?

I think, I won't do that, since on the one hand "-type f" would also exclude directories (o.k., we could do "-type f -or -type d"), on the other hand, I don't want to exclude sockets, because parts of the most regular cruft on my system are sockets, which are left over e.g. when entrance is not terminated properly.

So, let's just add the sockets to the false positive files. (Most programs create them in /tmp anyway).

I'll look into the other things this weekend.

Greetings,
sean

[soulwarrior]

I think, I won't do that, since on the one hand "-type f" would also exclude directories (o.k., we could do "-type f -or -type d"), on the other hand, I don't want to exclude sockets, because parts of the most regular cruft on my system are sockets, which are left over e.g. when entrance is not terminated properly.

So, let's just add the sockets to the false positive files. (Most programs create them in /tmp anyway).

Okay, I understand. I thought dead sockets wouldn't survive a reboot. So it is in fact better to include the sockets.

Or could one maybe specifically search for dead sockets on a system?

I'll look into the other things this weekend.

thanks

Greetings,
soulwarrior

[djpenguin]

Hey, I've installed your script, but I can't seem to get it to work right...

After getting the latest version:

Code: Select all

bash-2.05b$ cd /usr/local/
bash-2.05b$ su -c "tar -xvjf /tmp/findcruft-1.0.4.tar.bz2"

But when I go to try it out:

Code: Select all

bash-2.05b$ findcruft > cruftfiles
bash: cruftfiles: Permission denied
bash-2.05b$ su root
Password: 
bash-2.05b# findcruft > cruftfiles
bash: findcruft: command not found
bash-2.05b# su djpenguin
bash-2.05b$ findcruft > cruftfiles
bash: cruftfiles: Permission denied
bash-2.05b$ source /etc/profile
djpenguin@chilly-willy local $ findcruft > cruftfiles 
bash: cruftfiles: Permission denied

Any idea what I did wrong? I'd really like to get this going...had to go in a manually remove stuff from /var/tmp/ today so I could do an emerge sync...