Seeking advice on a spamfiltering solution

Message

Del Pede · Post by **Del Pede** » Mon Dec 04, 2006 11:10 am

I've decided to add spamfiltering to my postfix server, since the amount of spam is increasing. I run a postfix server, and courier-imap, and it only sorts mail for one domain, and it's all based on users .maildir.

I've been looking at both dspam and spamassasin, but really don't know what to chose, though i'd prefer not to end up with a solution, that isn't dependent of mysql

Thanks in advanced
Del Pede

col · Post by **col** » Mon Dec 04, 2006 11:19 am

1 : greylisting
2 : blacklists = spamhaus, dsbl, spamcop njabl
3 : spamassassin & clamav

should get rid of most spam

Enverex · Post by **Enverex** » Mon Dec 04, 2006 11:23 am

I'm using Spamassassin with Razor, spamhaus, dsbl, spamcop njabl, etc and to be honest, it's not stopping a lot of the spam these days as it's simply not finding anything wrong with them (only minor things like "Message in HTML" or "40k picture with only 80 characters of text" etc which you can't base blocks on). So it's quite worrying how ineffective the best methods have become.

VanDan · Post by **VanDan** » Mon Dec 04, 2006 11:24 am

I just finished setting up a mail system for our home network, and it works so well I'm upgrading the work server as I type

I've got postfix ==> dspam ==> dbmail

It's a real pain in the arse to set up, but once it's set up, it's very, very nice. I have it so there are 2 global addresses for training: spam@<domain> and ham@<domain>. I also have a global access account so I can view / clear the quarantined messages. I can post config files and stuff if you like. I was using sendmail ==> canit ==> courier-imap, but each piece of the system was starting to piss me off in various ways ( sendmail a nightmare to maintain, canit starting to let through more spam, courier-imap not scaling well ).

I swear by this new combo though

Del Pede · Post by **Del Pede** » Mon Dec 04, 2006 11:48 am

VanDan wrote:I just finished setting up a mail system for our home network, and it works so well I'm upgrading the work server as I type

I've got postfix ==> dspam ==> dbmail

It's a real pain in the arse to set up, but once it's set up, it's very, very nice. I have it so there are 2 global addresses for training: spam@<domain> and ham@<domain>. I also have a global access account so I can view / clear the quarantined messages. I can post config files and stuff if you like. I was using sendmail ==> canit ==> courier-imap, but each piece of the system was starting to piss me off in various ways ( sendmail a nightmare to maintain, canit starting to let through more spam, courier-imap not scaling well ).

I swear by this new combo though

I'd love too glance at your config files, that would be much appreciated

I've actually been thinking about migrating from courier-imapd to dovecot, but unfortunately i don't have a test server to crah run on.

Are you setup dependent on a running mysql?

magic919 · Post by **magic919** » Mon Dec 04, 2006 8:41 pm

I run Postfix with DSPAM and serve up mail with Dovecot. I use DSPAM with MySQL but that's not the only storage option. Needs to be trained but quickly adapts to new spam trends. I don't get tons of messages at home and I'm running about 96% accuracy after 7000 messages. It has captured 4 messages in error. That's not bad.

VanDan · Post by **VanDan** » Wed Dec 06, 2006 12:13 am

Here come the config files ... minus comments and whitespace.

/etc/postfix/main.conf:

Code: Select all

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
mail_owner = postfix
mydomain = entropy.homelinux.org
myorigin = $mydomain
inet_interfaces = $myhostname, localhost
mydestination = $myhostname, entropy.homelinux.org
local_recipient_maps = mysql:/etc/postfix/sql-recipients.cf
unknown_local_recipient_reject_code = 550
mynetworks = 192.168.1.0/24 entropy.homelinux.org
relay_domains = *.entropy.homelinux.org
transport_maps = hash:/etc/postfix/transport
mailbox_transport = dbmail-lmtp:127.0.0.1:24
debug_peer_level = 2
debugger_command =
         PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
         xxgdb $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/sbin/sendmail
newaliases_path = /usr/bin/newaliases
mailq_path = /usr/bin/mailq
setgid_group = postdrop
html_directory = /usr/share/doc/postfix-2.3.4/html
manpage_directory = /usr/share/man
sample_directory = /etc/postfix
readme_directory = /usr/share/doc/postfix-2.3.4/readme
home_mailbox = .maildir/
dspam_destination_recipient_limit = 1
dspam-spam_destination_recipient_limit = 1
dspam-ham_destination_recipient_limit = 1

For /etc/postfix/master.cf I added the following stuff down the bottom of the config file, and left everything else as-is:

Code: Select all

# DSpam
dspam           unix    -       n       n       -       -       pipe
  flags=Rhq user=dspam:dspam argv=/usr/bin/dspam --deliver=innocent --user ${recipient}

dspam-spam      unix    -       n       n       -       -       pipe
  flags=Rhq user=dspam:dspam argv=/usr/bin/dspam --user ${recipient} --class=spam --source=error

dspam-ham       unix    -       n       n       -       -       pipe
  flags=Rhq user=dspam:dspam argv=/usr/bin/dspam --user ${recipient} --class=innocent --source=error

# DBMail transport - for delivering to storage
dbmail-lmtp     unix    -       -       n       -       -       lmtp


#SMTP daemon listening on port 10025 for filtered mail from dspam

127.0.0.1:10025 inet    n       -       n       -       -       smtpd
  -o smtpd_authorized_xforward_hosts=127.0.0.0/8
  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o receive_override_options=no_unknown_recipient_checks

/etc/postfix/sql-recipients.cf:

Code: Select all

user = dbmail_user_name_replace_me
password = dbmail_password_replace_me
host = 127.0.0.1
dbname = dbmail
table = dbmail_aliases
select_field = alias
where_field = alias

/etc/postfix/transport:

Code: Select all

spam@entropy.homelinux.org      dspam-spam:
ham@entropy.homelinux.org       dspam-ham:
entropy.homelinux.org           dspam:

I then had to run 'postmap' on the transport file to create the db file that postfix uses.

Important parts ( not all ) of /etc/dbmail/dbmail.conf:

Code: Select all

driver          = mysql
authdriver      = sql
host            = localhost
sqlport         =                
sqlsocket       = /var/run/mysqld/mysqld.sock
user            = dbmail_username_change_me
pass            = dbmail_password_change_me
db              = dbmail               
table_prefix    = dbmail_   
sendmail        = /usr/sbin/sendmail     
[LMTP]
PORT            = 24                 
[POP]
[IMAP]
PORT            = 143                
TIMEOUT         = 4000            
[SIEVE]
PORT            = 2000  # ****** This was NOT the default, but this setting works with smartsieve

/etc/dspam/dspam.conf:

Code: Select all

Home /var/spool/dspam
StorageDriver /usr/lib/dspam/libmysql_drv.so
TrustedDeliveryAgent "/usr/sbin/dbmail-smtp -d %u"
UntrustedDeliveryAgent "/usr/sbin/dbmail-smtp -d %u"
OnFail error
Trust root
Trust dspam
Trust apache
Trust mail
Trust mailnull 
Trust smmsp
Trust daemon
Trust filter
TrainingMode teft
TestConditionalTraining on
Feature chained
Feature whitelist
Feature tb=3
Algorithm graham burton
PValue graham
SupressWebStats off
ImprobabilityDrive on
Preference "spamAction=quarantine"
Preference "signatureLocation=headers"  # 'message' or 'headers'
Preference "showFactors=off"
AllowOverride trainingMode
AllowOverride spamAction spamSubject
AllowOverride statisticalSedation
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride signatureLocation
AllowOverride showFactors
AllowOverride optIn optOut
AllowOverride whitelistThreshold
MySQLServer     /var/run/mysqld/mysqld.sock
MySQLUser               dspam
MySQLPass               dspam_password_change_me
MySQLDb                 dspam
MySQLCompress           true
MySQLCompress           true
MySQLVirtualTable          dbmail.dbmail_aliases
MySQLVirtualUIDField       deliver_to
MySQLVirtualUsernameField  alias
MySQLUIDInSignature    on
HashRecMax              98317
HashAutoExtend          on  
HashMaxExtents          0
HashExtentSize          49157
HashMaxSeek             100
HashConnectionCache     10
Notifications   off
PurgeSignature  off # Specified in purge.sql
PurgeNeutral   90
PurgeUnused    off # Specified in purge.sql
PurgeHapaxes   off # Specified in purge.sql
PurgeHits1S    off # Specified in purge.sql
PurgeHits1I    off # Specified in purge.sql
LocalMX 127.0.0.1
SystemLog on
UserLog   on
TrainPristine off
Opt out
ParseToHeaders off
ChangeModeOnParse off
ChangeUserOnParse off
ClamAVPort      3310
ClamAVHost      127.0.0.1
ClamAVResponse  reject
ServerPID              /var/run/dspam/dspam.pid
ServerMode auto
ServerDomainSocketPath  "/var/run/dspam/dspam.sock"
ProcessorBias on

You have to give the mysql 'dspam' user read access to the dbmail database.

From this point on, you'll have to make your own way

I installed the dspam-web package, which then proceeded to fuck things up considerably. I recommend installing this by hand. I also had to do some dodgy stuff like cd into /var/spool/dspam/data and then make the sym-links:

local -> entropy.homelinux.org

I think I might have had to make other directories as well. I also had to hack the dspam.cgi script to be slightly more descriptive about it's errors when there were path problems, otherwise it just says "Error", without giving any hint as to WTF is wrong.

If you have troubles, post back and I can help you with the finishing touches.

col · Post by **col** » Fri Dec 08, 2006 4:44 am

problem with dspam & spamassassin is that they do not work with the new image based spam....this is where blacklists & greylisting is required.

I find greylisting the most simple & elegant solution to spam. I use a very small 5 minute delay which gets rid of 99% of spam. I whitelist a good server for 1 month.

Enverex · Post by **Enverex** » Fri Dec 08, 2006 11:19 am

How does Greylisting work then?

col · Post by **col** » Fri Dec 08, 2006 2:37 pm

Enverex wrote:How does Greylisting work then?

http://en.wikipedia.org/wiki/Greylisting

it works very well....I have found that almost all spammers do not use RFC compliant delivery agents....or if they do in combination with blacklists they are blacklisted before they get a chance to resend the spam email.

VanDan · Post by **VanDan** » Tue Dec 19, 2006 8:09 am

I just emerged postgrey. It makes a damned nice addition. Thanks for the tip

GNUtritious · Post by **GNUtritious** » Sun Dec 31, 2006 1:37 am

Been using this setup for almost 2 years: http://freespamfilter.org/ It's based on amavisd-new, spamassassin, etc. and works quite well.

steveb · Post by **steveb** » Sun Dec 31, 2006 2:01 am

VanDan wrote:I just emerged postgrey. It makes a damned nice addition. Thanks for the tip

Try SQLGrey. It is more configurable and clever then PostGrey.

cheers

SteveB

Ateo · Post by **Ateo** » Tue Jan 09, 2007 11:07 pm

col wrote:problem with dspam & spamassassin is that they do not work with the new image based spam....

Try FuzzyOcr...

VanDan · Post by **VanDan** » Tue Jan 09, 2007 11:13 pm

Ateo wrote:Try FuzzyOcr...

Are you saying that you've integrated this into a spam filtering system?

steveb · Post by **steveb** » Wed Jan 10, 2007 7:47 am

VanDan wrote:
Ateo wrote:Try FuzzyOcr...
Are you saying that you've integrated this into a spam filtering system?

Yes. SA has interface to FuzzyOCR.

Ateo · Post by **Ateo** » Thu Jan 18, 2007 10:05 pm

VanDan wrote:
Ateo wrote:Try FuzzyOcr...
Are you saying that you've integrated this into a spam filtering system?

Yes. I have FuzzyOcr installed and working. For what it's worth, it works as expected. It has caught all email with images (thus far, it's been about 2 months now) that I would consider spam. I suggest using the tesseract. It's faster than gocr (imo)...

The downside is that it has it's own word database. It doesn't utilize SA filters when scanning.

Here's the ebuild in bugzilla: https://bugs.gentoo.org/show_bug.cgi?id=158445

VanDan · Post by **VanDan** » Thu Jan 18, 2007 10:43 pm

Interesting. dspam is now starting to get a lot more accurate in identifying this image spam, but OCR in a spam filter still sounds intriguing. I'd like to have it work together with dspam though, so I'll devote some time ( later ) to figuring out how to chain the 2 ( dspam & sa ) together.

col · Post by **col** » Fri Jan 19, 2007 10:43 am

dont waste your CPU cycles...use greylisting.

Ateo · Post by **Ateo** » Fri Jan 19, 2007 5:29 pm

col wrote:dont waste your CPU cycles...use greylisting.

greylisting alone doesn't solve everything.

cmaurand · Post by **cmaurand** » Fri Jul 20, 2007 3:01 pm

I've installed fuzzyocr and a thing called scam.sh which uses downloads a database for clam antivirus that allows clam antivirus to deal with pdf content in email. I really need to send that guy some money. I've really lowered the

Curtis