How to set up an email server with postfix/cyrus

Message

audiodef · Post by **audiodef** » Wed Apr 20, 2011 7:41 pm

I'm a little confused. I thought I did provide appropriate authentication by entering "webmaster@audiodef.com" and not just "webmaster". I listed my entry for my Thunderbird outgoing mail settings above. Is there something else I need to do?

audiodef · Post by **audiodef** » Wed Apr 20, 2011 8:01 pm

*plays bugle*

Everything works now. I just had to use smtp.audiodef.com instead of audiodef.com.

Let me shake your hand vigourously and buy you a virtual beer.

audiodef · Post by **audiodef** » Wed Apr 20, 2011 8:02 pm

I still want to know what's taking up so much space, especially when I do not store mail on the server.

audiodef · Post by **audiodef** » Wed Apr 20, 2011 8:07 pm

Oh, and I've taken Gosmackyerdaddy out of my MX records. This is awesome. Now watch me fuck it up.

Post by **cach0rr0** » Wed Apr 20, 2011 9:43 pm

audiodef wrote:I'm a little confused.

this is what happens when I try to reply before I've had my daily dose of nicotine - I reply with confusing limericks! Awful habit (the nicotine, not the limericks), but cripes am I incoherent without it.

audiodef wrote:I thought I did provide appropriate authentication by entering "webmaster@audiodef.com" and not just "webmaster". I listed my entry for my Thunderbird outgoing mail settings above. Is there something else I need to do?

you were. are? were/are?

Just that there's two places to authenticate:

-send 'webmaster@audiodef.com' auth data to IMAP (or POP) for reading mail
-send 'webmaster@audiodef.com' auth data to Postfix, so that you can use the Postfix daemon to send mail to external domains

audiodef wrote: Everything works now. I just had to use smtp.audiodef.com instead of audiodef.com

hrm. strange. the test account setup i have on thunderbird, for both incoming and outgoing mail server, i specify simply 'audiodef.com', and as the username I use 'theunmentionedtestaccount@audiodef.com' - meaning, i dont use smtp.audiodef.com anywhere. As far as the username you send to Cyrus or Postfix for authentication, so long as that username exists in the 'aliases' table, it shouldn't matter.

audiodef wrote: I still want to know what's taking up so much space, especially when I do not store mail on the server.

If you want to not store mail on the server, you need to use POP rather than IMAP (that's actually a fairly easy change to make, if you want to go that route - just a quick change to cyrus.conf). The downside with POP of course being, the pitfalls I mentioned earlier - if you don't store mail on the server, if you read mail on one machine, you cant turn around later and try reading those same messages from another machine, or phone, or what have you, unless you specifically tick "leave a copy of messages on server". The main difference between this and courier, is that with the courier HOWTO you have a /home/vmail directory underneath which are 80 zillion subfolders, one for each email address, and inside each subfolder are your messages, one file per message. IMHO this is not only inefficient and slow, nevermind not being particularly scalable nor flexible, but it adds a requirement of an additional SQL lookup to determine which subdirectory to store the mail in under /home/vmail.

Nonetheless, tried the usual method of du / -h --max-depth=1 then walking up and up and up from there?

audiodef wrote: Oh, and I've taken Gosmackyerdaddy out of my MX records. This is awesome. Now watch me fuck it up

Should be safe enough to do at this stage. Give it a day or two of testing, but now that the screwy LMTP issues are sorted out, it should "just work" for a good long while.

audiodef · Post by **audiodef** » Wed Apr 20, 2011 10:34 pm

I would think that even with imap enabled, if I use pop, it would get the messages off the server. So why would db.0005 or whatever it's called weigh 40M?

Hm... some hefty log files. I need to look up how to configure syslog-ng to limit log file sizes. Removing some files (I'm assuming they'll just be recreated anew, hence the need to look up config options for syslog-ng) drastically reduced disk usage. It's still high, though.

I need to ask Mark if my server options are correctly configured. / is 3.9G and df says I'm using 44% but I'm supposed to have a 16G disk size. That does not add up...

Post by **cach0rr0** » Wed Apr 20, 2011 11:57 pm

audiodef wrote: I would think that even with imap enabled, if I use pop, it would get the messages off the server. So why would db.0005 or whatever it's called weigh 40M?

If you use POP, the messages will indeed be removed from the server, unless you tell your mail client not to.
As far as the db files, I wouldn't wager yours will get much bigger than they already are. Mine's been in production for a couple years now, biggest file is 41MB.

audiodef wrote: Hm... some hefty log files. I need to look up how to configure syslog-ng to limit log file sizes. Removing some files (I'm assuming they'll just be recreated anew, hence the need to look up config options for syslog-ng) drastically reduced disk usage. It's still high, though.

emerge logrotate, then set it to rotate the logs daily (it will set up the cron job automatically, assuming youve already merged a cron daemon)

audiodef wrote: I need to ask Mark if my server options are correctly configured. / is 3.9G and df says I'm using 44% but I'm supposed to have a 16G disk size. That does not add up...

Could be inode usage at 44%. Already cleaned out /usr/portage/distfiles and /var/tmp/portage?

I'd also get a bit of spam filtering set up sooner rather than later, for relatively old domains spam is going to make up the vast majority of your mail traffic.
The more spam you drop rather than quarantine, all the better; this is why I have multiple RBL's running.

audiodef · Post by **audiodef** » Thu Apr 21, 2011 12:55 am

I just ran into a fresh problem:

Code: Select all

Apr 20 10:50:03 serverdef sshd[2890]: Server listening on 0.0.0.0 port 22.
Apr 20 10:50:03 serverdef sshd[2890]: Server listening on :: port 22.
Apr 20 10:50:09 serverdef sshd[3022]: SSH: Server;Ltype: Version;Remote: 71.191.169.85-36819;Protocol: 2.0;Client: OpenSSH_5.8p1-hpn13v10
Apr 20 10:50:09 serverdef saslauthd[3092]: detach_tty      : master pid is: 3092
Apr 20 10:50:09 serverdef saslauthd[3092]: ipc_init        : listening on socket: /var/lib/sasl2/mux
Apr 20 10:50:12 serverdef sshd[3022]: Accepted keyboard-interactive/pam for root from 71.191.169.85 port 36819 ssh2
Apr 20 10:50:12 serverdef sshd[3022]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 20 10:51:02 serverdef pop3s[3494]: sql auxprop plugin using mysql engine
Apr 20 10:51:02 serverdef pop3s[3495]: sql auxprop plugin using mysql engine
Apr 20 10:51:02 serverdef pop3s[3496]: sql auxprop plugin using mysql engine
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin Parse the username webmaster
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin try and connect to a host
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin trying to open db 'maildb' on host 'localhost'
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin Parse the username webmaster
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin try and connect to a host
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin trying to open db 'maildb' on host 'localhost'
Apr 20 10:51:02 serverdef pop3s[3494]: begin transaction
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin create statement from userPassword webmaster serverdef
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin doing query SELECT plainpass FROM aliases WHERE email = 'webmaster@serverdef';
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin: no result found
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin create statement from cmusaslsecretPLAIN webmaster serverdef
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin doing query SELECT plainpass FROM aliases WHERE email = 'webmaster@serverdef';
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin: no result found
Apr 20 10:51:02 serverdef pop3s[3494]: commit transaction
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin Parse the username webmaster
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin try and connect to a host
Apr 20 10:51:02 serverdef pop3s[3494]: sql plugin trying to open db 'maildb' on host 'localhost'
Apr 20 10:51:02 serverdef pop3s[3495]: sql plugin Parse the username damien

This happened after I rebooted the server to see if that would clear up df incorrectly reporting disk usage - it worked, but now mail is bjorked somehow. I'm now getting auth failure for all of my mailboxes. I've changed nothing - merely rebooted the server.

It seems like it's going back to checking for "serverdef" instead of "audiodef.com"... but I haven't changed anything since it was finally working.

audiodef · Post by **audiodef** » Thu Apr 21, 2011 2:22 am

Seems like I have to have /etc/conf.d/hostname set to "audiodef.com", not "serverdef". We are now back in action!

audiodef · Post by **audiodef** » Thu Apr 21, 2011 2:48 am

Just looking ahead here... I'll be hosting my gf's web site and her email on my setup. She has her own domain. Is there anything special I need to do in light of discovering that hostname needed to be set to audiodef.com to get her email working with her domain, which is not audiodef.com?

Post by **cach0rr0** » Thu Apr 21, 2011 2:59 am

audiodef wrote:Seems like I have to have /etc/conf.d/hostname set to "audiodef.com", not "serverdef". We are now back in action!

Shouldn't matter. That's what I was getting at - you have to pay special attention to what you put in your 'username' settings inside e.g. thunderbird

If you just put, for example, 'cach0rr0', then it is going to try and append a default domain/realm.
If I put 'cach0rr0@audiodef.com', then it will NOT try to append a default domain/realm

Having said that, you can add this to imapd.conf:

Code: Select all

defaultdomain: audiodef.com

What this does - if a user merely provides 'cach0rr0' as their IMAP username, it will automatically append '@audiodef.com'

Postfix has a similar setting, for people who try to do authenticated mail relay but only provide 'username' instead of 'username@domain.com': http://www.postfix.org/postconf.5.html# ... cal_domain

I omitted this in the guide, largely because this is contrary to the idea of 'virtual hosting' with email. The idea is supposedly that you have more than one domain you host mail for, and as such the *user* needs to specify the domain, instead of your IMAP/SMTP systems just assuming which domain the user is wanting. If you're only going to host mail for 'audiodef.com' and subdomains (e.g. '*.audiodef.com'), you dont even need to do the 'virtual hosting' nonsense. In fact, you don't even need a database (though it does make some things easier). If you host multiple domains, you need some semblance of virtual domains, and you need the user to provide the domain name rather than having one as a default - for example, I have company A, company B, both have a user name 'chris'. In such a case, since they're two different people, mail for 'chris@companya.com' needs to go to a different mailbox from 'chris@companyb.com'. To that same end, those two different people will have different passwords for checking email - enter 'virtual hosting' (i hate this term, but i suppose it seems to fit)

Basically, if i only handle mail for one domain (in my example, whitehathouston.com), I can set a default domain inside both Postfix and Cyrus, and provide only the username 'meat' like so:

http://ompldr.org/vOGN1cQ/imapwhh.png

Because Postfix/Cyrus will append the @whitehathouston.com to that (as dictated by smtpd_sasl_local_domain in main.cf, and defaultdomain in imapd.conf)

Whereas if I handle multiple domains, where 'meat' at one domain is a different person/mailbox from 'meat' at another domain, has to be like so:

http://ompldr.org/vOGN1cw/imapvirtual.png

Hope that makes sense somewhat?

Post by **cach0rr0** » Thu Apr 21, 2011 3:03 am

audiodef wrote:Just looking ahead here... I'll be hosting my gf's web site and her email on my setup. She has her own domain. Is there anything special I need to do in light of discovering that hostname needed to be set to audiodef.com to get her email working with her domain, which is not audiodef.com?

ha! I posted my last reply before I saw this post.

Though, my last reply does explain it. I'm actually glad you're going to be doing another domain, so I didn't have you go through an unnecessary level of complexity this whole time

audiodef · Post by **audiodef** » Thu Apr 21, 2011 4:49 pm

Heh... you must be psychic!

OK, here's the problem. I've been specifying user@domain.com all along in Thunderbird, and yet, mail will not work until I've set hostname="audiodef.com".

So I've missed something somewhere. I did read your latest post carefully, but I'm not seeing what I'm missing...

Also, I don't have an imapd.conf anywhere. Should I?

Post by **cach0rr0** » Thu Apr 21, 2011 7:47 pm

audiodef wrote:Heh... you must be psychic!

OK, here's the problem. I've been specifying user@domain.com all along in Thunderbird, and yet, mail will not work until I've set hostname="audiodef.com".

Thunderbird has a neat habit of truncating things; revisit Server Settings, as well edit the settings under Outbound Servers. Even if you specify 'user@domain' on the initial setup as your email address, Thunderbird assumes you just use 'user' for auth, so it saves it as such.

You can do the same tests via telnet actually

Code: Select all

telnet localhost 143
01 login someuser@audiodef.com theirpassword
#this should log you in successfully
02 logout

I know the server portion is functional, because I can do the tests via both telnet and with a thunderbird instance here on that test account, and it lets me in

To that same end, check your logs and you'll see my logins. Obv my logins aren't successful because of anything I'm doing differently on the server, as I'm not *on* the server

audiodef wrote: Also, I don't have an imapd.conf anywhere. Should I?

you should have an /etc/imapd.conf on the server yeah

audiodef · Post by **audiodef** » Thu Apr 21, 2011 8:33 pm

OK, I see imapd.conf. I must have not have had my morning coffee when I checked earlier. Of course it's there - I put it there.

I may have to consider using something other than Thunderbird if I'm already adding the correct entries and Thunderbird is doing something I did not tell it to do. My server settings are correct. If Thunderbird is sending out something other than what I put in - time to kiss that buggy program good-bye.

Post by **cach0rr0** » Thu Apr 21, 2011 9:06 pm

I'd test with telnet just to be certain
But testing here via telnet all is well, testing here with thunderbird-bin 3.1.9 seems well

might PM me a screencap of your Server Settings (Edit=>Account Settings)
As well a screencap of 'Outgoing Server (SMTP)'

If that all looks kosher, toss my hands up, let's try another client.

audiodef · Post by **audiodef** » Fri Apr 22, 2011 2:07 pm

I tried another client anyway - claws-mail.

Exactly the same thing happened. Despite using user@audiodef.com instead of just user, I got an auth failure if I did not set the hostname to audiodef.com.

I've changed it to serverdef and left it there. Are you able to use the test account I created for you? It's still in there.

audiodef · Post by **audiodef** » Fri Apr 22, 2011 2:20 pm

I just noticed that I can't telnet audiodef.com 25. On the server, I can telnet localhost 25.

I also just noticed that pinging audiodef.com elicits a response from serverdef.audiodef.com regardless of what hostname is set to. I have both audiodef.com and serverdef associated with my IP address in /etc/hosts, but I don't think that's it, as there is no serverdef.audiodef.com in that file.

Anarcho · Post by **Anarcho** » Fri Apr 22, 2011 2:57 pm

audiodef wrote:I just noticed that I can't telnet audiodef.com 25. On the server, I can telnet localhost 25.

I also just noticed that pinging audiodef.com elicits a response from serverdef.audiodef.com regardless of what hostname is set to. I have both audiodef.com and serverdef associated with my IP address in /etc/hosts, but I don't think that's it, as there is no serverdef.audiodef.com in that file.

The hostname comes from the reverse DNS lookup, see:

Code: Select all

T410 ~ $ nslookup 209.177.157.239
Server:		192.168.2.1
Address:	192.168.2.1#53

Non-authoritative answer:
239.157.177.209.in-addr.arpa	name = serverdef.audiodef.com.

audiodef · Post by **audiodef** » Fri Apr 22, 2011 3:27 pm

What's also interesting is that I just noticed I cannot log in to get mail if hostname != audiodef.com AND my IP address in /etc/hosts != serverdef.audiodef.com.

Actually, I have to leave hostname = audiodef.com for now because I need to stay on top of mail to make plans with friends for this weekend, but I would like to set aside a time for you (cach0rr0) to be able to try to login with hostname set to something other than audiodef.com so we can make sure this is working correctly.

Post by **cach0rr0** » Fri Apr 22, 2011 7:10 pm

audiodef wrote: Actually, I have to leave hostname = audiodef.com for now because I need to stay on top of mail to make plans with friends for this weekend, but I would like to set aside a time for you (cach0rr0) to be able to try to login with hostname set to something other than audiodef.com so we can make sure this is working correctly.

soon as you're ready to set it to something seemingly non-functional, give me a shout.

Cyrus will, flat-out, not change the login string you give it, unless you provide it a login string that does not contain a realm (rather, a 'domain', but in auth nomenclature called a realm).

If you give it a realm, it doesn't care what your hostname is set to,what you have in /etc/hosts, it will use the realm/domain you've provided. The hierarchy goes like so:

-if the user provides a domain name in the login string, no further lookups are done, it uses the user-provided domain name
(ex: user@domain => unmodified)

-if the user provides no domain name, it will append the domain name specified in 'defaultdomain' (setting in imapd.conf)
(ex: user => user + @ + $defaultdomain)

-if the user provides no domain name, and 'defaultdomain' is not set, it will append the server's hostname value
(ex: user => user + @ + `hostname`)

If your IMAP client is providing a domain on the login string, those external lookups will not be done at all, full stop.

Postfix has a similar hierarchy:

-if the user provides a domain name, the domain name will be used
ex:

Code: Select all

openssl s_client -connect audiodef.com:25 -starttls smtp
EHLO somehost.somedomain.tld
AUTH LOGIN
base64_encode(user@domain.com) => unmodified
base64_encode(password)

-if the user provides only a username, and no realm, if smtpd_sasl_local_domain is set in main.cf, it will append smtpd_sasl_local_domain
ex:

Code: Select all

openssl s_client -connect audiodef.com:25 -starttls smtp
EHLO somehost.somedomain.tld
AUTH LOGIN
base64_encode(user) => base64_encode(user + @ + $smtpd_sasl_local_domain)
base64_encode(password)

-if the user provides only a username, and smtpd_sasl_local_domain is not set in main.cf, it will append your system's hostname
ex:

Code: Select all

openssl s_client -connect audiodef.com:25 -starttls smtp
EHLO somehost.somedomain.tld
AUTH LOGIN
base64_encode(user) => base64_encode(user + @ + `hostname`)
base64_encode(password)

If you're doing "virtual hosting" type scenario, you don't ever want the lookups to external settings to be done (e.g. you dont want, it to lookup main.cf/imapd.conf settings, you dont want it to lookup `hostname`). And it won't, if your IMAP/SMTP client is providing a domain as part of the login string.

audiodef · Post by **audiodef** » Fri Apr 22, 2011 7:35 pm

That's a really good explanation, thanks.

I'll try to remember to switch hostname before going to bed tonight - usually around 0300 GMT at the latest. I'll reset it Saturday around 1300-1400 GMT to check my mail.

Saturday I'm going to be out for a while starting at 2200 GMT. I'll not be needing mail until Sunday 1300 or 1400 GMT. I'll try to remember to have hostname set to something other than "audiodef.com" before I go out.

Hopefully, you can poke around somewhere in there.

*All times in GMT for ease of translating across time zones, daylight savings, wormholes, temporal anomalies, and (insert demographic category) time.*

audiodef · Post by **audiodef** » Fri Apr 22, 2011 10:31 pm

OK, I'm done with mail for the night. I just changed hostname to serverdef and checked - nope, can't login from Thunderbird or claws-mail with user@domain.com.

Let me know if you're able to use your test account to send and receive.

Post by **cach0rr0** » Fri Apr 22, 2011 11:40 pm

ok, it's Fri Apr 22 18:38:43 CDT 2011 right now, just got back from epic steak dinner

tried a handful of logins:

-SMTP authentication is working fine
-IMAP authentication is not

can you ship me your logs (auth.log and mail.log ) and the contents of imapd.conf (with password and username for SQL nuked obv)?

Post by **cach0rr0** » Sat Apr 23, 2011 12:02 am

actually...
I don't know why it's working this way, but I'm testing here, and I can reproduce the behavior you describe if I have 'defaultdomain' set in imapd.conf

If I remove this setting completely, everything works fine.
See the following:

http://ompldr.org/vOGR1ZQ/nodefaultdom.png

http://ompldr.org/vOGR1Zg/defaultdom.png

EDIT:
Looks like we're not alone.
http://www.mail-archive.com/info-cyrus@ ... 37017.html

That seems very broken. Easy to work around, but very broken.
Remove 'defaultdomain', and set:

Code: Select all

servername: serverdef.audiodef.com

in /etc/imapd.conf