HOWTO - Spam Filtering with Gentoo, Postfix, Amavis & SA

[green sun]

Spam Filtering for Exchange with Gentoo, Postfix, Amavisd-new and SpamAssassin


[Edit 05.11.04 - Added information about validating users on Exchange]
[Edit 04.27.05 - Added information about needing to emerge spamassassin]
[Edit 04.29.05 - Added information about DNS servers]

Step By Step:
1. Install Gentoo
2. Emerge postfix, amavisd-new
3. Configure Postfix
4. Configure Amavisd-new
5. Configure SpamAssassin

To begin, I'm going to assume you have Gentoo up & running on a box. I have been using gentoo-sources with no problems on my spam filter, but I'm sure if you used a different kernel you could squeeze more performance out of it.

For reference, here's the machine I set this all up on:

Code: Select all

Single Processor
Intel Pentium III 450MHz(Katmai)
128MB RAM
RAID 1 on SCSI 8GB Drives
USE_FLAGS = "-X"

For safety's sake, I think you should have some type of RAID on this system. If you need help, this is an excellent 'HOWTO' on software RAID 1 on Gentoo (its what I used) http://forums.gentoo.org/viewtopic.php? ... tware+raid

I'm currently doing spam filtering for 3,700 Exchange mailboxes, and catching between 500 - 1000 spam messages per day. And to be honest, the machine listed above doesn't even break a sweat.

IMPORTANT: If possible, have your DNS servers for the machine on a local network! Using a DNS server a few hops away can considerably slow mail processing. Even using your ISP's DNS can cause major slowdowns and queue backups.

Ok, once your Gentoo box is booting up, emerge the following:

Code: Select all

emerge postfix
emerge amavisd-new

It appears that emerging amavisd-new will no longer emerge SpamAssassin. In this case, you now need to

Code: Select all

emerge spamassassin

as well in order to add SpamAssassin to your system. Thanks to Captain Obvious for pointing this out.

Postfix Configuration

Now you need to configure Postfix to work as a relay to the Exchange server only. You don't want mail being sent from this box. I'm taking the following information directly from Scott Henderson's excellent HOWTO ( http://www.geocities.com/scottlhenderso ... ilter.html ) Please note that I am not running postfix chroot, although you could & Scott gives instructions on doing this. Scott also covers a lot of other things, like sending root mail to another mailbox, improving logging performance, etc. so I recommend you read through his site. I'm just going to cover the basics here.

Edit the file /etc/postfix/master.cf and add this at the bottom

Code: Select all

smtp-amavis	unix	-	-	y	-	2	smtp
	-o smtp_data_done_timeout=1200
	-o disable_dns_lookups=yes

127.0.0.1:10025	inet	n	-	y	-	-	smtpd
	-o content_filter=
	-o local_recipient_maps=
	-o relay_recipient_maps=
	-o smtpd_restriction_classes=
	-o smtpd_helo_restrictions=
	-o smtpd_sender_restrictions=
	-o smtpd_recipient_restrictions=permit_mynetworks,reject
	-o mynetworks=127.0.0.0/8
	-o strict_rfc821_envelopes=yes

Make sure to get tabs between the - y - parts...

This sets up postfix to forward mail to this machine on port 10025, the port Amavisd listens on.

Next, edit the file /etc/postfix/main.cf and add the following at the bottom. Note that I've replaced my specific info with a filler.

Code: Select all

myorigin = my.network.com
myhostname = spamfilter.my.network.com
mydestination = my.network.com
mynetworks = x.x.x.x/y
biff = no
smtpd_banner = SMTPD_BANNER_HERE
message_size_limit = 25000000
local_transport = no local mail delivery
local_recipient_maps =
transport_maps = hash:/etc/postfix/transport
smtpd_helo_required = yes
#smtpd_helo_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
#smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient
content_filter = smtp-amavis:[localhost]:10024

The line mynetworks defines what networks you will trust and relay mail for. Its important to put exactly the machines that you will be sending mail to!! I usually put my local subnet & the IP of the server. BTW, it takes the list in network/subnet style, so for a single machine, it would be 10.0.0.1/32, for a class B it would be 10.1.0.0/16

Of interest are the following lines

Code: Select all

smtpd_helo_required = yes
#smtpd_helo_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
#smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient
content_filter = smtp-amavis:[localhost]:10024

the lines starting with 'smtpd' are some anti-spam settings for postfix. I have had problems with them, so I usually comment them out, but they are there should you want to use them. I've also had problems with postfix doing blackhole lookups (Internet databases of spam senders), so I usually don't do with postfix. All of this is covered in Scott's site listed above.

Secondly, you may want to add these lines:

Code: Select all

header_checks = pcre:/etc/postfix/pcre-header.cf
body_checks = pcre:/etc/postfix/pcre-body.cf

This will allow you to put regular expressions in those files to test against the incomming mail headers and bodys.
IMPORTANT If you add these lines, you need to at least have files (even blank) in the /etc/postfix dir. If anyone wants, send me a PM & I'll send you my copies of the files. I have built up quite a bit of stuff in them.

Next, you may want to edit the file /etc/postfix/access. Here you can put ip ranges, senders and domains to be rejected or accepted by postfix. Personally, I do not use this file, I just use the pcre-header & body files. If you do edit the access file, you need to run

Code: Select all

postmap /etc/postfix/access

after editing it. Otherwise postfix will not recognize your changes.

Ok, that should be the end of configuring postfix. You can start the postfix service now & check it by telneting into localhost on port 25. Got a response? So far, so good.

If you do change any postfix configuration files, esp. if you decide to use the pcre-header & body approach like I do, when you are done editing, just run

Code: Select all

postfix reload

to have the changes recognized.


Configuring Amavisd-new

Ok, now amavisd-new is kinda like this glue that holds together all these different mail processing programs. Its a pretty neat idea, and once you understand that amavisd-new just coordinates these programs, its easy to work with. A stock install of amavisd-new gives clamd. You will need to emerge SpamAssassin yourself. Clamd will handle anti-virus scanning, and SpamAssassin will handle the spam filtering.

First, edit /etc/amavisd.conf & make the following changes

Code: Select all

Locate $mydomain = 'example.com' and change to your domain

Check that $daemon_user and $daemon_group are set to 'amavis' (done by the Gentoo ebuild I believe)

DO NOT remove the # from @bypass_virus_checks_acl!! You want to do virus scanning!

Remove the # from in front of $warnspamsender = 1;

Change $mailfrom_notify_ lines to point to "your.admin.mailbox\@your.domain.com";

Locate $spam_quarantine_to = 'spam-quarantine'; and add a # to the beginning. Go down one line & REMOVE the # from $spam_quarantine_to = "your.admin.mailbox\@your.domain.com";

Ok, that should get amavisd-new passing mail off to the anti-virus & spam filters correctly. Now, here are some important tips for amavisd-new:

Many of the settings for SpamAssassin are taken from amavisd.conf! So, you can edit the SpamAssassin conf file all you want, but the changes won't work... so, here's what you have to set in amavisd-new

Code: Select all

Find the section # SpamAssassin settings

Notice the lines $sa_tag_level_deflt, $sa_tag2_level_deflt, $sa_kill_level_deflt. These are where you set the scores that SpamAssassin gives to a mail to mark it as spam. I have mine set to:

$sa_tag_level_deflt  = 0.0; 
$sa_tag2_level_deflt = 5.0; 
$sa_kill_level_deflt = $sa_tag2_level_deflt;

so, a score of 5 marks it as spam & kills the mail.

Its important to put the scoring information in the amavisd.conf file, as it will be ignored if put in the SpamAssassin conf file.

Virus Scanning

To be honest, this works out of the box. Clamd sets up a cron job to check for updates daily, and I've never had a problem. Clamd logs to /var/log. Great app & it catches a TON of viruses for us.

SpamAssassin Setup

As I mentioned above, the most important setting for SpamAssassin, the scoring of mails, is controlled through /etc/amavisd.conf. All other configuration for SpamAssassin is done in /etc/mail/spamassassin/local.cf Here's a few settings you can play with

Code: Select all

bayes_auto_learn 1
bayes_auto_learn_threshold_nonspam 1
bayes_auto_learn_threshold_spam 14.00

These settings manipulate the bayes learning feature of SpamAssassin. I would recommend setting the threshold to 'learn' high, as otherwise you will get a lot of false positives. A spam score of 14 seems to do a good job for me. Lower & you will see things like many aol.com emails getting marked with a high Bayes score.

Also, local.cf is used to put SpamAssassin rules into. I have many rules that I use. PM me & I'll send them to you.

Final Setup

You should now be ready to start filtering mail! You need to add some programs to your default runlevel

Code: Select all

rc-update add postfix default
rc-update add amavisd default
rc-update add clamd default

Start these services /etc/init.d/<service> start & start testing it out! You can set your SMTP server to the spam filter machine in your local email client & start sending emails. See if they get through...

Final Thoughts

There is lots and lots of additional stuff you can do. You can play around with the headers that are added to the emails, you can quarantene differently, set up different notifications, and on & on. The amavisd.conf file is very well commented, so read through it.

Notice that I didn't touch the Exchange server at all. If you wanted to get advanced, you could do LDAP lookups to verify that the recipient exists. This would be overkill in my case, and probably too much overhead.

[Edit 05.11.04]
Fixer sent me the following link & said he had validation of accounts against Exchange up & working within 2 hours. If you are like his net admin (and like most of us using Exchange) and are quite paranoid about the stability of Exchange, this would help reduce the load on Exchange.

http://www.unixwiz.net/techtips/postfix ... users.html
[End Edit]

Be careful in setting the score limits. Its better for users to get some spam (& tell you about it) than it is to get a bunch of false positives. Users will understand some spam getting through, but will not like valid messages getting bounced.

Personally, I've leaned towards putting spam senders into the pcre files I mentioned above. If I know there is some unique identifier in an email (usually a sender address or web link), I'll add it to the postfix. This has two advantages; it reduces load on the server (amavisd never sees the mail), and doesn't fill up the Exchange mailbox you set up with stuff you know is spam. On the other hand, there is no notification.. just a SMTP reject.

You may want to install logrotate to help manage logs. They get big fast. Also, I use mtail to color my logs when I tail them. A simple grep statement & mtail can give you a good visual of how many mails are getting bounced, passed, etc.

If you want configuration files, PM me.

Resources

Scott Henderson's site - where I learned most of this stuff
http://www.geocities.com/scottlhenderso ... ilter.html

Postfix - difficult documentation tho...
http://www.postfix.org/

Amavisd-new - just read the config file
http://www.ijs.si/software/amavisd/

SpamAssasin
http://www.mirror.ac.uk/sites/spamassas ... index.html

SpamAssassin Config Settings - remember, the score settings need to be in amavisd.conf
http://www.mirror.ac.uk/sites/spamassas ... _Conf.html

[Kirktis]

What would be the appropriate modifications to the postfix config to make this work on a single machine? That is, rather then forwarding mail on to another mail server, I want to do this all on the same machine that handles the mail.

[nevynxxx]

What would be the appropriate modifications to the postfix config to make this work on a single machine? That is, rather then forwarding mail on to another mail server, I want to do this all on the same machine that handles the mail.

Running exchange on a gentoo box? thats new

What you want is covered in many many other spam/virus filtering howto's, this is purely for the situation where exchange cannot be changed, but people want to use gentoo and spamassasin to filter.

[green sun]

What would be the appropriate modifications to the postfix config to make this work on a single machine? That is, rather then forwarding mail on to another mail server, I want to do this all on the same machine that handles the mail.

Actually, this HOWTO should cover any situation where your mailboxes are on a seperate server... since I did nothing to the Exchange server, that could be any type of mail server.

My guess would be that you would change the mynetworks line in main.cf to point to your server. Its probably a little more complex than that tho.. since postfix is sitting on port 25...

[green sun]

Advanced Configuration

Here's a great link I was sent that should help people take spam filtering to the next level. It deals with OpenBSD, but just skip that part & focus on configuring Razor, SA & DCC. I'm going to be implementing some of this stuff over the summer.

http://www.flakshack.com/anti-spam/

[amd]

Ok.. guys... If you want to see SpamAssassin headers you have to define local domains in /etc/amavisd.conf
SpamAssassin writes the X-SPAM headers only if your domain is considered local.

Code: Select all

# domain.com and all it's subdomains.
@local_domains_acl = ( ".domain.com" );

[ckdake]

I'm trying to get this to work. It is very close, however with

content_filter = smtp-amavis:[localhost]:10024

I get emails bounced back saying that postfix cant find the folder localhost,

and when i replace localhost with 127.0.0.1, i get:

Code: Select all

Jun  3 02:18:01 stitch amavis[32315]: (32315-01) FWD via SMTP: [127.0.0.1]:10025 <chris@ithought.org> -> <chris@ithought.org>
Jun  3 02:18:01 stitch postfix/smtpd[32520]: starting TLS engine
Jun  3 02:18:01 stitch postfix/smtpd[32520]: connect from unknown[127.0.0.1]
Jun  3 02:18:01 stitch postfix/smtpd[32520]: BFBE52E888: client=unknown[127.0.0.1]
Jun  3 02:18:01 stitch postfix/smtpd[32520]: warning: connect to mysql server unix:/var/run/mysqld/mysqld.sock: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

any ideas what the problem is?

[ckdake]

turns out I had something chrooted that wasn't supposed to be. Its my fault for blindly copyign and pasting but if you aren't runnign postfix chrooted,

instead of :

Code: Select all

127.0.0.1:10025   inet   n   -   y   -   -   smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes

make sure to use:

Code: Select all

127.0.0.1:10025   inet   n   -   n   -   -   smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes

Thats changing that one little y to an one little n.

[geek]

Thanks for the guide. I've moved from qmail/qmail-scanner/f-prot to postfix/amavisd/clamav. Just a couple notes that help me get things working properly.

I got an error message from postfix about the localhost not having a host A record when trying to connect to amavisd. I changed

Code: Select all

content_filter = smtp-amavis:[localhost]:10024

To This:

Code: Select all

content_filter = smtp-amavis:[127.0.0.1]:10024

in the /etc/postfix/main.cf file and postfix was then able to drop the mail into amavis as expected.

I also had to create /etc/postfix/transport which contains a list of the domains to forward e-mail for and the e-mail server to pass the mail to for that domain:

Code: Select all

mydomain.net smtp:[192.192.192.192]
mydomain2.net smtp:[192.192.192.193]

After creating this file be sure to run the following command to create /etc/postfix/transport.db , the actual file that postfix reads.

Code: Select all

 postmap /etc/postfix/transport

To get clamavd to work correctly you have to edit /etc/amavisd.conf. If you run with the file as is, the virus scanning is done by clamscan. I wanted to use the daemon.

In /etc/amavisd.conf uncomment the following section:

Code: Select all

 ['Clam Antivirus-clamd',

   \&ask_daemon, ["CONTSCAN {}\n", "/tmp/clamd"],
   qr/\bOK$/, qr/\bFOUND$/,
   qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],

and make sure that you change this line to look like this:

Code: Select all

 \&ask_daemon, ["CONTSCAN {}\n", "/tmp/clamd"],

This points amavis to the correct socket for the clam daemon.

I hope this helps anyone else who chooses this excellent solution.
Casey

[hw-tph]

green sun - brilliant post, and thanks to everyone else who provided useful input! This helped me a ton, and I have now officially moved from Exim to postfix.


Håkan

[green sun]

Lots of SA rules here... good resource

http://www.rulesemporium.com/

[TriGuN]

Wait, so does spamassassin still work even if we don't apply any rules to it? or will it just...send all mail regardless?

Also, if I ony have one machine to do all the mail handling, can I just comment out the "transport_map" and be fine?

And lastly, is it supposed to scan outgoing as well as incoming mail? Judging by the headers i'm looking at, spamassassin isn't scanning outgoing mail, however the virus scanner is. Problem?

[green sun]

Wait, so does spamassassin still work even if we don't apply any rules to it? or will it just...send all mail regardless?

Also, if I ony have one machine to do all the mail handling, can I just comment out the "transport_map" and be fine?

And lastly, is it supposed to scan outgoing as well as incoming mail? Judging by the headers i'm looking at, spamassassin isn't scanning outgoing mail, however the virus scanner is. Problem?

1. Yes, spam assassin has built in rules that apply. The rules Ive given are 'extra', to help catch new/different types of mail.

2. Possibly? Im not really sure, as I've only set the scanning machine up as a relay. I would think that you would be all set (since postfix is told to use Amavis for scanning, regardless of the destination), but a little research would probably be in order.

3. In this setup, outgoing mail is sent via Exchange, so it would not be scanned. Are you worried about your users sending spam, or about you being used as an open relay?

[franoculator]

If I have the following in /etc/postfix/master.cf, I have problems.

Code: Select all

smtp-amavis     unix    -       -       y       -       2       smtp
   -o smtp_data_done_timeout=1200
   -o disable_dns_lookups=yes

127.0.0.1:10025 inet    n       -       y       -       -       smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes

If that's in the config, the mail get's bounced from my main mail server.

Code: Select all

Aug  5 21:17:06 mailgate postfix/smtp[7514]: 9EF38CF4: to=<sender@host>, relay=mail.server.com[xxx.xxx.xxx.xxx], delay=1, status=bounced (host mail.server.com[xxx.xxx.xxx.xxx] said: 550 Relaying denied (in reply to RCPT TO command))

Amavis is started, by the way.

Removing the above code from master.cf allows the mail to properly flow to the master mail server, but obviously, the filtering does not take place.

Any suggestions?

[green sun]

If I have the following in /etc/postfix/master.cf, I have problems.

Code: Select all

smtp-amavis     unix    -       -       y       -       2       smtp
   -o smtp_data_done_timeout=1200
   -o disable_dns_lookups=yes

127.0.0.1:10025 inet    n       -       y       -       -       smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes

If that's in the config, the mail get's bounced from my main mail server.

Code: Select all

Aug  5 21:17:06 mailgate postfix/smtp[7514]: 9EF38CF4: to=<sender@host>, relay=mail.server.com[xxx.xxx.xxx.xxx], delay=1, status=bounced (host mail.server.com[xxx.xxx.xxx.xxx] said: 550 Relaying denied (in reply to RCPT TO command))

Amavis is started, by the way.

Removing the above code from master.cf allows the mail to properly flow to the master mail server, but obviously, the filtering does not take place.

Any suggestions?

I'm coming off of a 2 week vacation.. I'll have a look when I get back into the office...

[MaGuS]

Hi,

first I have to say: Thank you for this Posting!

But I have a question: How do I check the learning function. I would love to see amavisd and SpamAssasssian learing span.

Hope someone could help me.

Best regards
Magnus

[green sun]

Hi,

first I have to say: Thank you for this Posting!

But I have a question: How do I check the learning function. I would love to see amavisd and SpamAssasssian learing span.

Hope someone could help me.

Best regards
Magnus

Check out the spamassassin homepage (in the resources of the initial post). Ive not seen anything on 'watching' SA learn about spam, but I know there is a way to feed spam into SA to help it learn.

Covering all the various aspects of SA would be a whole different thread...

[green sun]

If I have the following in /etc/postfix/master.cf, I have problems.

Code: Select all

smtp-amavis     unix    -       -       y       -       2       smtp
   -o smtp_data_done_timeout=1200
   -o disable_dns_lookups=yes

127.0.0.1:10025 inet    n       -       y       -       -       smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes

If that's in the config, the mail get's bounced from my main mail server.

Code: Select all

Aug  5 21:17:06 mailgate postfix/smtp[7514]: 9EF38CF4: to=<sender@host>, relay=mail.server.com[xxx.xxx.xxx.xxx], delay=1, status=bounced (host mail.server.com[xxx.xxx.xxx.xxx] said: 550 Relaying denied (in reply to RCPT TO command))

Amavis is started, by the way.

Removing the above code from master.cf allows the mail to properly flow to the master mail server, but obviously, the filtering does not take place.

Any suggestions?

Please confirm that the following are set in your main.cf correctly

Code: Select all

myhostname = spamfilter.my.network.com
mydestination = my.network.com
mynetworks = x.x.x.x/y 

Not having mynetworks set might freak it out.. its a place to start atleast....

[Robelix]

Is there a way to do not even accept spam/virus-mails?
At the moment my box accepts a mail with an EICAR and then sends back a warning mail.
I'd prefer to scan "in realtime" while receiving the mail - and then rejecting it.
I already tried $final_virus_destiny = D_REJECT; and D_BOUNCE but this does not change anything.

any ideas?
robelix

[MaxMara]

Hi.
I used this tutorial - everything worked, but if he automatically checks the mailq i get the following entry in my maillog:

Code: Select all

Sep 22 08:48:42 webby postfix/qmgr[26080]: warning: connect to transport amavis: No such file or directory

any ideas?

tia
christian

[prodigy7]

Somewhere must be a mistake ... the filter are called smtp-amavis, not amavis

Hi.
I used this tutorial - everything worked, but if he automatically checks the mailq i get the following entry in my maillog:

Code: Select all

Sep 22 08:48:42 webby postfix/qmgr[26080]: warning: connect to transport amavis: No such file or directory

any ideas?

[MaxMara]

i know, but i have

Code: Select all

content_filter = smtp-amavis:[localhost]:10024

in my main.cf

maybe it's because i also have virtual accounts??

[prodigy7]

amavisd is running ?

maybe it's because i also have virtual accounts??

[MaxMara]

yep

Code: Select all

clamav   26483  0.0  2.9 28748 26656 ?       Ss   09:10   0:01 amavisd (master)
clamav   26485  0.0  2.9 29120 27092 ?       S    09:10   0:00 amavisd (child)
clamav   26486  0.0  3.0 29280 27216 ?       S    09:10   0:00 amavisd (child)

it scans all incoming mails for viruses, but he can't access the mailq. that's the problem.

[prodigy7]

How are the permissions and owners are set in /var/spool/postfix/ ?

yep

Code: Select all

clamav   26483  0.0  2.9 28748 26656 ?       Ss   09:10   0:01 amavisd (master)
clamav   26485  0.0  2.9 29120 27092 ?       S    09:10   0:00 amavisd (child)
clamav   26486  0.0  3.0 29280 27216 ?       S    09:10   0:00 amavisd (child)

it scans all incoming mails for viruses, but he can't access the mailq. that's the problem.