Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO - Automated Whitelisting using Dovecot and Procmail
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
reteo
n00b
n00b


Joined: 26 Dec 2004
Posts: 61
Location: Phoenix, Arizona

PostPosted: Fri Sep 29, 2006 9:08 am    Post subject: HOWTO - Automated Whitelisting using Dovecot and Procmail Reply with quote

I run a two-computer system, one server/busybox, and one desktop. I use Dovecot IMAP for reading mail, and I have a server-side set of scripts sorting mail into specific mailboxes.

When new messages come in from legitimate people, I don't want to have to ssh into the server, just to update the procmail recipes with the new email addresses.

So, I decided to create a set of bash scripts that, when combined with cron, will automatically update the recipes list for me. All I need to do is move the new mail into the folder I ultimately want it to continue coming into.

And then, I thought, what the heck, maybe someone else might just like this too.

So, here y'all go, the WhiteDove Recipe Generator.

First script is auto-whitelist, which scans your maildirs for the first instance of "From:" in a message, and places it into a whitelist named after the maildir.

Code:

#!/bin/bash

# The WhiteDove Recipe Generator
# (C) 2006, Lampros Liontos, aka. "Reteo Varala"
#   This program is licensed under the GNU GPL, version 2 or greater.

# A set of scripts to generate an automatically-updating whitelist using
# Dovecot and Procmail.

# auto-whitelist: Generates the whitelist.

###########################################################################
################## Start of configurable data #############################
###########################################################################

# First, you need to fill in the specifics of your system.

# Use the hostname portion only.  Anything else, and you might not get all
# the addresses.
HOST=$HOSTNAME

#This is the location of your maildirs.
ROOT=~/.maildir

# This is the location of your whitelist.
WHITEDIR=~/.whitelists      # Rootdir
TEMP=$WHITEDIR/tmp      # Tempdir
LOCK=$WHITEDIR/lock      # Lockdir

# This is the rootdir of your actual whitelisted mailboxes.
## NOTE: The preceding period is essential, and do not use whitespace.
FOLDERS="$ROOT/.whitebox"

###########################################################################
################## End of configurable data ###############################
###########################################################################

# Now, we need to make certain that the whitelist directory is clean.  This should do nicely.
rm `find $WHITEDIR -type f`

for MAILDIR in $FOLDERS
do
   # First thing's first, make certain a maildir is not being worked on
   # by another process.
   if [ -e $LOCK/`basename $MAILDIR` ];
   then
      # If the lock exists, skip the whole procedure, and move on to
      # the next folder. Let's make certain that a notification is
      # made, however.
      echo "Skipping `basename $FOLDER`, as it is already in progress.";
   else
      # Create a lockfile so we can work on this folder in peace.
      touch $LOCK/`basename $MAILDIR`;
      
      # Because maildirs use three subdirs, we need to make certain
      # that we don't miss anything. Also, since the actual mail
      # messages use the hostname, that is a good way to tell an
      # actual mailfile from a directory or an index file.
      for MAILFILE in `find $MAILDIR/{cur,new,tmp} -type f `;
      do
         # We need to find the "From:" lines in the mailfiles,
         # and pack them into a temporary file for processing.
         # The -m 1 option prevents the reading of From: lines
         # in attached messages.
         grep -h -m 1 "^From:" $MAILFILE >> $TEMP/`basename $MAILDIR`;
      done;
      
      # We have one maildir tempfile prepared, now to process it.
      # To do this, we need to do three things: sort it, eliminate
      # all duplicates, and then write the final result
      if [ -e $TEMP/`basename $MAILDIR` ];
      then
         # Okay, let's process the file.
         cat $TEMP/`basename $MAILDIR` | sort | uniq > $WHITEDIR/`basename $MAILDIR`;
         # Cleanliness is next to godliness.
         rm $TEMP/`basename $MAILDIR`;
      fi
      # The file's made, the tempfile's gone, we're done.  Let's
      # unlock the whitefile, and move onto the next maildir.
      rm $LOCK/`basename $MAILDIR`
   fi
done;


Second script is the other half of the system, which reads all the whitelists generated by auto-whitelist, and then creates a procmail recipe file with every entry as its own recipe, outputting to the mailbox it originally came from.

This also creates a central procmail recipe file which includes all the created whitelist recipe files.

Code:

#!/bin/bash

# The WhiteDove Recipe Generator
# (C) 2006, Lampros Liontos, aka. "Reteo Varala"
#   This program is licensed under the GNU GPL, version 2 or greater.

# A set of scripts to generate an automatically-updating whitelist using
# Dovecot and Procmail.

# procmail-generator: generates the procmail recipes from the whitelists
#                     created by auto-whitelist

###########################################################################
################## Start of configurable data #############################
###########################################################################

# This is the whitelist directory generated by auto-whitelist.
WHITEDIR=~/.whitelists

# This is your procmail directory.
PMDIR=~/.procmail

# The includefile is what you include from procmailrc to use the whitelists.
INCLUDEFILE=rc.auto

# The include directory is where the whitelists will be stored.
INCLUDEDIR=rc.auto-generated

###########################################################################
################## End of configurable data ###############################
###########################################################################

# We need to make certain whitespace is not a separator, or else the "From:"
# lines will be chopped up into one recipe per word.  Not a good idea.
IFS=$'\n'

# This performs two functions.  It erases the original, and creates a heading.
echo "# Automatically generated by procmail-generator" > $PMDIR/$INCLUDEFILE;
echo "" >> $PMDIR/$INCLUDEFILE;
echo "PMDIR=\$HOME/.procmail" >> $PMDIR/$INCLUDEFILE;
echo "" >> $PMDIR/$INCLUDEFILE;

# The following wildcard should flush out all directories.  Since all
# maildirs start with a dot, and should have at least two characters, this
# should do for our needs. 
for WHITEENTRY in $WHITEDIR/.??*;
do
   # Strip the path, so we can process the filename as a whitelist.
   WHITEFILE=`basename $WHITEENTRY`

   # Now, this generates the include line in the include file.
   echo "INCLUDERC=\$PMDIR/$INCLUDEDIR/rc$WHITEFILE" >> $PMDIR/$INCLUDEFILE;
   
   # Now, we clean and re-header the whitelist recipe file.
   echo "# rc$WHITEFILE automatically generated by procmail-generator" > $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   echo "" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;

   # Now to generate the recipes, one per line in the whitelist file.
   for LINE in `grep "From:" $WHITEENTRY`;
   do
      echo ":0" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
      echo "* ^$LINE" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
      echo "$WHITEFILE/" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
      
      # The recipes need to be spaced apart by at least one line.
      echo "" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   done;
done;


Hope someone finds these useful!

What you need to do is place these scripts in your binary directory, and set up cron to run the auto-whitelist first and the procmail-generator next. Finally, in your current .procmailrc, add the following:

Code:
INCLUDERC=${HOME}/${PMDIR}/${INCLUDEFILE}


Where ${HOME} is your home directory, ${PMDIR} is where you store your procmail scripts, and ${INCLUDEFILE} is the name of the file generated by the above script.
_________________
Lampros Liontos (aka. Reteo Varala)

Linux Home Recording - Tips, tricks and techniques for setting up and using a recording studio with Linux.


Last edited by reteo on Thu Dec 14, 2006 9:07 pm; edited 2 times in total
Back to top
View user's profile Send private message
reteo
n00b
n00b


Joined: 26 Dec 2004
Posts: 61
Location: Phoenix, Arizona

PostPosted: Sat Oct 21, 2006 11:59 pm    Post subject: Reply with quote

Looks like I found a bug in the above script.

Whenever a legitimate email address includes characters that have special meaning in regular expressions, it can match addresses other than the exact email address.

I am not certain how to fix this one perfectly, but I added some code to turn all regex-special symbols into the "match-one" symbol, the period.

First, we need to replace the following code in the procmail-generator script:
Code:

# Now to generate the recipes, one per line in the whitelist file.
for LINE in `grep "From:" $WHITEENTRY`;
do
   echo ":0" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   echo "* ^$LINE" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   echo "$WHITEFILE/" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   
   # The recipes need to be spaced apart by at least one line.
   echo "" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
done;


With the following code:
Code:

# Now to generate the recipes, one per line in the whitelist file.
for LINE in `grep "From:" $WHITEENTRY`;
do
   # Let's get rid of regexp-command characters.  The dot
   # should suffice.
   WORK=$LINE;
   for i in '\\' '\[' '\^' '\$' '\|' '\?' '\*' '\+' '\(' '\)';
   do
      LINE2=`echo $WORK | tr $i '.'` ;
      WORK=$LINE2 ;
   done

   echo ":0" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   echo "* ^$LINE2" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   echo "$WHITEFILE/" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
   
   # The recipes need to be spaced apart by at least one line.
   echo "" >> $PMDIR/$INCLUDEDIR/rc$WHITEFILE;
done;

_________________
Lampros Liontos (aka. Reteo Varala)

Linux Home Recording - Tips, tricks and techniques for setting up and using a recording studio with Linux.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum