Configure mail server

From Organic Design wiki
Revision as of 14:16, 31 May 2018 by Nad (talk | contribs) (Domain and IP address configuration)
Procedure.svg Configure mail server
Organic Design procedure

Contents

Exim4 (MTA & MDA)

Exim4 is the default mail transfer and delivery agent for Debian, but it's only the light version which is insufficient for running a mail server, so first we need to install the exim4-daemon-heavy package which will replace the light version, and a few other packages we'll be needing.

sudo apt-get install exim4-daemon-heavy dovecot-common dovecot-imapd spamassassin spamc spf-tools-perl


Exim is designed to move messages from one e-mail server to another and to deliver messages to local users mailboxes. It has nothing to do with POP3 or IMAP as those are protocols relating to the retrieval of the mail by a user and their mail client software. The default Exim configuration allows the server to send mail, but if you plan on running an IMAP or POP3 server, then additional configuration is required which is discussed here.

First, run through the standard Exim4 configuration script, say no splitting configuration files, set general mail configuration to internet, use MailDir delivery method, and keep any non-self-explanatory settings as default.

dpkg-reconfigure exim4-config

Mail name, hostname, rDNS and blacklists

It's important that the domain part of the MAIL FROM field in outgoing messages has a proper domain with a valid reverse lookup, otherwise some SMTP servers will reject them. To ensure this, check that the domain name specified in the first line of /etc/mailname is the correct one, and set it if not. Alternatively you can set the primary_host value in the exim configuration to the proper domain, this is best set at the top of the configuration file, and you may need to comment out other occurrences further down.

The hostname should be set to the domain with the hostname -b command and can also be set in /etc/hostname and added as a localhost name in /etc/hosts.

The reverse DNS should be set up in your server host's configuration (e.g. this is done in the Remote Access tab of the Linode configuration), the server host may need to do this for you if their client app doesn't support it.

You may find that your server's IP is listed inn various spam or abuse blacklists, a good service to check all the common blacklists in one go at is MX Toolbox. They also offer a free notification service, or a more advanced paid service.

Maildir delivery method

This should have been set in the package configuration above, but if not it can be set manually as follows. The primary advantage of maildirs is that multiple applications can access the same Maildir simultaneously without requiring any kind of locking whatsoever, but aside from this, Maildir format is more efficient than mbox, for more information see mbox-vs-maildir. Check the /etc/exim4/update-exim4.conf.conf and append dc_localdelivery='maildir_home' if it doesn't already exist. Note that if you change update-exim4.conf.conf, then you'll need to also run update-exim4.conf.

Main configuration for multiple domains

You may want to handle mail for a few domains on one server, in which case some generic usernames like "accounts" will conflict with the local user names, or with the same names used by other domains. This is based on Blair Harrison's method from this Waikato LUG article which adds virtual domain support to Exim4. This allows any incoming email address to be mapped to any other internal user mailbox, or external email address.

The main configuration changes to be made are in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config). Here's a summary of the main settings we use at the top of this file, these are explained in detail below along with their associated specific configurations throughout the file.

primary_hostname = organicdesign.co.nz
message_body_newlines = true
message_body_visible = 500000
MAIN_TLS_ENABLE = true
daemon_smtp_ports = 25 : 2525
system_filter = /var/www/tools/exim-copy-to-sent
system_filter_directory_transport = copy_to_sent
system_filter_pipe_transport = copy_to_sent_pipe
CHECK_RCPT_SPF = true

Next, adjust the local_domains setting to the following:

domainlist local_domains = @ : @[] : localhost : partial-lsearch;/etc/exim4/virtual.domains


Next go to the section starting with "real_local" and add a new section as follows (if in split configuration, this section should be added in a new file called /etc/exim4/conf.d/router/300_exim4-config_virtual)

virtual:
	driver = redirect
	allow_defer
	allow_fail
	data = ${lookup{$local_part@$domain}lsearch*@{/etc/exim4/virtual.users}}
	domains = partial-lsearch;/etc/exim4/virtual.domains
	retry_use_local_part


Now you will need file called /etc/exim4/virtual.domains that is simply a list of all the domains for which you accept mail in the following format:

example.com
example.net


Don't forget to restart the mail server,

/etc/init.d/exim4 restart

Setting up mail users

To add or modify the mail users, edit the /etc/exim4/virtual.users file. Following are some examples to highlight some of the most popular scenarios:

regularuser@example.com     : localuser@localhost
forwardinguser@example.com  : someuser@example.org
foo@example.com             : :fail: Foo no longer lives here.
bar@example.com             : :blackhole:
*@example.com               : catchall1@localhost

regularuser@example.net     : localuser2@localhost
forwardinguser@example.net  : someuser2@example.org
*@example.net               : catchall2@localhost
  • Note: There is no need for Exim to be restarted after changing the virtual.users or virtual.domains files.

Testing delivery

Before moving on to SMTP or IMAP configuration, you should test that email sent to your local users/domains are being delivered properly. The mails will be delivered to the users home directories in ~/Maildir/cur/. You can cat the files to see the textual content of the messages stored there. You can send an email from the command line:

/path/to/exim -v 'user@domain'
message here
^D ( control D )


Another important thing to check before considering the system ready for production use is to send a test message to another mail server that you have access to the logs on. This way you can see exactly what other mail servers are seeing and ensure that everything matches up as it should. Here's an example of a remote server's Exim log entry showing a message received from the Organic Design server.

2013-04-25 11:37:49 1UVPB7-0008Q2-J8 <= aran@organicdesign.co.nz H=organicdesign.co.nz [37.17.226.20]
    P=esmtps X=TLS1.0:RSA_AES_256_CBC_SHA1:32 S=944 id=51795B7E.2020802@organicdesign.co.nz
2013-04-25 11:37:49 1UVPB7-0008Q2-J8 => nad <nad@foo.com> R=local_user T=maildir_home
2013-04-25 11:37:49 1UVPB7-0008Q2-J8 Completed

The most important part to look at here is the H value. The IP in the square brackets is the real IP address that the message has been received from, and the domain directly following the H is the name found after doing a reverse-DNS lookup on the IP. If the name is surrounded by round brackets it means that no reverse-DNS entry was found, and the name that the server is publishing in response to an SMTP HELO or EHLO command (exim's primary_domain value) is being used instead. If there's both a reverse_DNS name and also a bracketed name, then this means the reverse-DNS name is not the same as the name returned from the HELO command. It's very important that there be a reverse-DNS entry and that it exactly match the HELO domain because otherwise many mail servers will reject the message as spam. More information about the Exim log format of received messages can be found here in the Exim documentation.

To test what you server's domain is in the ELHO response connect via SMTP using telnet DOMAIN 25 then type ELHO. If you're connecting from a consumer connection you may need to use the 2525 port since many of them block 25 to prevent spammers.

Setting up a secure SMTP server with authentication

Since many people in the organisation may be working with laptops in the field, it's useful for the server to run it's own SMTP server so that settings don't need to be changed when moving from one ISP to another for internet access. The SMTP server will need to use authentication and encryption so that it can't be used for spamming.

Setting up a secure SMTP server with authentication is quite simple using Exim4's plaintext authenticator with TLS encryption.

SMTP Certificates

First add the following to the first section of /etc/exim4/exim4.conf.template. The second directive adds a second SMTP port since many ISP's restrict outgoing traffic on port 25 to their own SMTP server only. All other TLS options are added under the main/03_exim4-config_tlsoptions section.

MAIN_TLS_ENABLE = true
daemon_smtp_ports = 25 : 2525

Self-signed: Using a self-signed certificate, we prefer to use a wildcard domain for the common name such as *.organicdesign.co.nz so that it's possible for different sub-domains to access the service but get handled differently, then generate the certificate with the /usr/share/doc/exim4-base/examples/exim-gencert command.

LetsEncrypt: A better way is to use your LetsEncrypt certificate instead of a self-signed one by setting the MAIN_TLS_CERTIFICATE to the fullchain.pem file and MAIN_TLS_PRIVATEKEY to the privkey.pem file, and ensuring the files are both accessible by Exim (Note: the certs are symlinks so the target needs to be readable by Exim, and also all directories in the path need to be executable by exim to be allowed to follow symlinks). For example:

MAIN_TLS_PRIVATEKEY = /etc/letsencrypt/live/organicdesign.co.nz-0004/privkey.pem

Cipher and protocol selection

We should also restrict the protocols and ciphers to mitigate POODLE and BEAST attacks etc, the available options are shown in the Exim manual here and here and a good guide here, and we use the following selection:

tls_require_ciphers = SECURE256:+SECURE128:-VERS-TLS-ALL:+VERS-TLS1.2:-RSA:-DHE-DSS:-CAMELLIA-128-CBC:-CAMELLIA-256-CBC
  • Note: tls_require_ciphers replaces gnutls_require_kx, gnutls_require_mac and gnutls_require_protocols which are all obsolete now.

Using a global inline SMTP user and password (legacy)

Next find the section (commented out) containing the string "plain_server" and add a similar entry as follows, but replace the USER and PASSWORD with the global authentication you want all clients to use.

plain_server:
	driver = plaintext
	public_name = PLAIN
	server_condition = "${if and {{eq{$auth2}{USER}}{eq{$auth3}{PASSWORD}}}{1}{0}}"
	server_set_id = $auth2
	server_prompts = :

Using the native Linux user accounts and passwords (current)

To use the native Linux accounts for SMTP authentication, the /etc/shadow file must be made accessible to the Debian-exim group:

chgrp Debian-exim /etc/shadow
chmod g+r /etc/shadow

Next use the same plain_server as shown in the previous section, but change the server_condition as follows:

plain_server:
	driver = plaintext
	public_name = PLAIN
	server_condition = "${if crypteq{$auth3}{${extract{1}{:}{${lookup{$auth2}lsearch{/etc/shadow}{$value}}}}}{1}{0}}"
	server_set_id = $auth2
	server_prompts = :

Test sending mail to an external domain

Restart the exim4 server and the server should be ready to accept SMTP requests!

To set up the clients use the user name and password defined above, smtp.organicdesign.co.nz for the server, 2525 for the port and TLS for security.

To test that the SMTP is working and is not accessible to spammers, testing sending out emails to non-local recipients (as delivery to local domains is unrestricted). Try sending without authentication to ensure that it fails. Also test sending to a local domain from another SMTP server (such as your ISP's one) to ensure that normal delivery is working correctly.

Copying emails into the Sent folder

Many Thunderbird users have been having trouble with the "copying to sent folder" process freezing or being extremely slow. Rather than have every mail be effectively sent twice, once too the recipient and again to the sent folder, it's a lot more efficient to have the server make a copy in the user's Sent folder instead.

Web-mail note: If this is set up, you should also configure your web-mail to not place copies in the Sent folder. Our method of doing this is shown below in the RoundCube web-mail section.

To do this we need to set up an Exim system filter which begins by adding the following lines to the beginning of the configuration where the port and other OD settings are put.

system_filter = /var/www/tools/exim-copy-to-sent
system_filter_directory_transport = copy_to_sent
system_filter_pipe_transport = copy_to_sent_pipe


Next we need to add the two transport that were specified as the system filter transport above. I put this after end transport/30_exim4-config_mail_spool in the configuration and it's defined as follows:

copy_to_sent:
    driver = appendfile
    delivery_date_add
    envelope_to_add
    return_path_add
    group = Debian-exim
    user = ${if match {${local_part:${lookup{$sender_address_local_part@$sender_address_domain}lsearch*@{/etc/exim4/virtual.users}}}}{(.+)}{$1}{Debian-exim}}
    mode = 0660
    maildir_format
    directory = /home/${local_part:${lookup{$sender_address_local_part@$sender_address_domain}lsearch*@{/etc/exim4/virtual.users}}}/Maildir/.Sent/
    create_directory

copy_to_sent_pipe:
    driver = pipe
    user = Debian-exim
    group = Debian-exim


The filter itself is in our tools repo here and so will be available in the file system at /var/www/tools/exim-copy-to-sent, it has the following content:

if
    $sender_address_local_part is not "root"
and 
    "${if def:h_X-Spam-Status {def}{undef}}" is "undef"
and
    "${if match {${lookup{$sender_address}lsearch{/etc/exim4/virtual.users}}}{@localhost}{yes}{}}" is "yes"
then
    unseen save $home/Maildir/.Sent/
    unseen pipe "/var/www/tools/copy-to-sent.pl \"$recipients\""
endif

This filter saves the messages to the user's Sent folder if three conditions apply:

  • The message is not from the server itself
  • The spam headers have not been added yet (the filter will run both before and after they're added)
  • The message is from a local account (i.e. not an incoming message)

One difficulty with using a system filter is that the BCC header is stripped by the time the message arrives at the system filter, but it's quite important for this header to be present in the Sent folder so that users can see who they sent the message to (also it's very useful for sending to a private list and then using the "Edit as new" feature to easily send another message to the same list). The header can be added back again using the $recipients variable which contains the entire list of recipients from all headers including BCC.

Another issue is that the messages are created as unread and Exim doesn't have any commands for changing the unread status as that's really a job for the mail client, not the transfer agent.

To get around these problems I've created the copy-to-sent.pl script that detects which file in the user's Sent/new Maildir folder is the one just added by checking the headers of all new messages and checking which has the new message ID which Exim sets in the MESSAGE_ID environment variable. The script then adds a Bcc header to the new message containing all the recipients that are not present in the To or Cc headers, and marks the message as read.

The system_filter_pipe_transport value in the Exim configuration needs to be set and an appropriate transport created for it before pipe commands are allowed to execute in system filters.

There is still one issue with this method currently which is that the complex expansion of the sender address into a local user has to be done twice because I can't figure out how to cache the result for later use. Perhaps the directory directive is not even needed and something else can be used instead.

LAN based SMTP servers

Often clients wish to run a local mail server on their LAN with the reasoning that they'd like all connection to their mail server to be extremely quick, to be independent of internet connectivity so that they can still access their mail and send mail internally independently of Internet connectivity. One complication with this is that some of the client computers on the LAN are laptops which must still be able to connect from outside the LAN from arbitrary Internet connections without any change of settings.

Here are some limitations and notes:

  • Most ISP's block outgoing traffic on port 25 unless it's to their own SMTP server
  • SSL certificate's domains must match the request even if self-signed
  • Our SMTP connections require authentication using Unix account details

Our solution is to set clients up using smtp.organicdesign.co.nz as their outgoing server using a company-wide user which is set up on the OD server as a zero-privilege user. This works for the external situation, but to allow local mail to be routed directly independently of Internet requires that the local DNS server be set up with an organicdesign.co.nz zone to be authoritative for just the smtp sub-domain but forward everything else on. This will make the local exim instance the first stop for all internal mail sending requests, but will deliver according to the usual rules for all non-local recipients.

DoveCot IMAP server

We use DoveCot for our IMAP. DoveCot responds to mail client requests by retrieving the mails from file and returning them to the client. It does not deal with the receiving the mails or storing them to disk.

We go with a configuration which is as close to default as possible and gives us both POP3 and IMAP services which both work with either TLS or SSL but not plain text. The users are by default the native linux users and passwords on the system.

Our entire /etc/dovecot/dovecot.conf configuration file can be replaced with the following extremely minimal one since nearly everything we use is default and we only use IMAP with local UNIX accounts. SSL is required for incoming connections but unencrypted connections are supported locally so that the webmail and other local applications can access the mail without requiring any SSL configuration. For more detail about the configuration format, see their wiki.

  • Note1: this configuration is for Dovecot 2.x, if you're still using 1.x check the history if this article and look at the revision prior to the comment about changing to Dovecot 2.x
  • Note2: if you have trouble with your client having frequent login trouble, try adding mail_max_userip_connections = 100
  • Note3: You can use your LetsEncrypt certificate instead of a self-signed one using the fullchain.pem and privkey.pem files and ensuring they're accessible by Dovecot
  • Note4: I'm using Artem Sidorenko's recommended SSL cyphers from here, except that we don't disable SSLv2 since it's was removed completely from OpenSSL.
log_path = /var/log/dovecot.log

protocols = imap
protocol imap {
        listen = 127.0.0.1:143
        ssl_listen = *:993
}

mail_location = maildir:~/Maildir
maildir_very_dirty_syncs = yes

userdb {
        driver = passwd
}

passdb {
        driver = pam
}

ssl = required
ssl_prefer_server_ciphers = yes
ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:
                  EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
ssl_protocols = !SSLv3                        # SSLv2 has been removed from OpenSSL so is not even a valid name to disable
ssl_cert = </etc/dovecot/dovecot.pem          # you can also use your LE cert: /etc/letsencrypt/live/..../fullchain.pem
ssl_key = </etc/dovecot/private/dovecot.pem   # you can also use your LE cert: /etc/letsencrypt/live/..../privkey.pem

IMAP Certificate

Note.svg Note: we prefer to use a wildcard domain for the common name such as *.organicdesign.co.nz so that it's possible for different sub-domains to access the service but get handled differently.

The certificate's are build automatically upon installation to match the servers primary domain name. To rebuild them for a different mailserver domain name, run the dovecot-cert.sh script (or dovecot-1.x-cert.sh if still using Dovecot 1.x), enter the organisation details and the mailserver domain in the Common Name field, then restart dovecot.

Spam Assassin

First install the spamassassin and spamc packages, then edit the configuration file /etc/default/spamassassin and set the ENABLED option set to 1. It's also a good idea to set the CRON option to 1 as well so that the rules are kept up to date on a daily basis. Remember to start spamassassin after enabling it;

service spamassassin start

The exim4 configuration needs to be adjusted which is in the /etc/exim4/exim4.conf.template file because our servers use the monolithic configuration method.

Add the following section after the line containing end router/800_exim4-config_maildrop

# 850: Spamcheck router
spamcheck_router:
	no_verify
	check_local_user
	condition = "${if and { {!def:h_X-Spam-Flag:} {!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
	driver = accept
	transport = spamcheck_transport


Add the following section before the line containing end transport/30_exim4-config_remote_smtp_smarthost

Note: be sure to check that the spamc binary is in the location specified, and that it is running.

# 30: Spamcheck transport
spamcheck_transport:
	debug_print = "T: spamassassin_pipe for $local_part@$domain"
	driver = pipe
	command = /usr/sbin/exim4 -oMr spam-scanned -bS
	use_bsmtp
	transport_filter = /usr/bin/spamc
	home_directory = "/tmp"
	current_directory = "/tmp"
	user = Debian-exim
	group = Debian-exim
	return_fail_output
	message_prefix =
	message_suffix =


At this point, after exim4 is restarted, all messages should have extra headers added by spamassassin for example:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.1.7-deb (2006-10-05) on organicdesign.co.nz
X-Spam-Level: ***********************
X-Spam-Status: Yes, score=23.9 required=5.0 tests=NO_DNS_FOR_FROM,
	RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,
	RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RCVD_IN_BL_SPAMCOP_NET,
	RCVD_IN_XBL,UNRESOLVED_TEMPLATE,URIBL_JP_SURBL,URIBL_OB_SURBL,
	URIBL_SBL,URIBL_SC_SURBL,URIBL_WS_SURBL autolearn=spam 
	version=3.1.7-deb

Moving spams to another folder automatically

Mails which have been marked as spam by spamassassin can be automatically moved into a specific mail folder by using an exim filter. First you may have to edit the exim4 configuration again and uncomment the allow_filter option in the 600_exim4-config_userforward section. We haven't had to do this so far because it is currently enabled by default.

To make a new filter, create a file in your home directory called .forward and ensure it is owned by the user in question, then add a rule such as in the following example from our server which forwards the spams straight into the Trash folder.

# Exim filter
if
   $h_X-Spam-Status: CONTAINS "Yes"
	  or
   "${if def:h_X-Spam-Flag {def}{undef}}" is "def"
then
   save $home/Maildir/.Trash/
   finish
endif
  • Important: do not change the comment in the first line!
  • Important: use \40 for spaces in target directory name (escaped octal!)
  • Note: You can use the make-forwards.pl to create them automatically for all users who don't already have them

Teaching spamassassin

Spamassassin comes with a script called sa-learn which is designed to parse mail directories with the information that the messages within them are either spam or "ham" (not spam). We have a Perl script called learn-spam.pl which resides in the /var/www directory and is run each morning. It scans all our user's Inbox/Spam and Inbox/Not Spam mail folders into which users should place their incorrectly assigned spam and non-spam messages. Note: the messages are removed after being processed, so the messages in your Not Spam folder should be copies, whereas those in the Spam folder can just be moved since you won't need them.

The sa-learn man page has some good information about Spamassassin's Bayesian learning mechanism, the following snippet is from there.

Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.

You need to train with both spam and ham mails. One type of mail alone will not have any effect.

Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)

If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run "spamassassin -d" over it in advance.

Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.

To check how many spams and hams the system has processed as well as some other internal data, use sa-learn --dump magic, here's the output of our system on 24 Dec 2008 which clearly shows that we haven't been feeding it enough ham! It shows that if we feed it ham at the same rate as spam, then it would start becoming useful in about another couple of weeks :-)

# sa-learn --dump magic
0.000          0          3          0  non-token data: bayes db version
0.000          0        733          0  non-token data: nspam
0.000          0         22          0  non-token data: nham
0.000          0      29962          0  non-token data: ntokens
0.000          0 1228202529          0  non-token data: oldest atime
0.000          0 1230047836          0  non-token data: newest atime
0.000          0          0          0  non-token data: last journal sync atime
0.000          0          0          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count


Here's a couple of lines of Perl which extract and print the number of spams and hams from the output shown above. I added this to our daily housekeeping script so that it appears in our recent changes each day.

$_ = `sa-learn --dump magic`;
print m/\s([1-9]+\d*).+?am[\x00-\x1f]+.+?([1-9]+\d*).+?am[\x00-\x1f]+.+?([1-9]+\d*).+?ns$/m
      ? "$1 spams and $2 hams have been processed with $3 tokens."
      : "ERROR";


Our training script, /var/www/learn-spam.pl, is run daily from the crontab and contains the following:

#!/usr/bin/perl

# Handle false positives
for (glob "/home/*/Maildir/.INBOX.Not\\ Spam/[cn]??") {
        s/ /\\ /;
        print qx "sa-learn --ham $_";
        qx "rm -fr $_/*";
}

# Handle false negatives
for (glob "/home/*/Maildir/.INBOX.Spam/[cn]??") {
        s/ /\\ /;
        print qx "sa-learn --spam $_";
        qx "rm -fr $_/*";
}


To test if the script is working properly, run it manually from the shell at such a time as you know there are spams to be processed and you should see messages stating that new tokens have been learned such as in the following example output.

archive-iterator: readdir found no mail in '/home/foo/Maildir/.INBOX.Spam/cur' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 16 message(s) (16 message(s) examined)
archive-iterator: readdir found no mail in '/home/bar/Maildir/.INBOX.Spam/cur' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 7 message(s) (7 message(s) examined)
Learned tokens from 6 message(s) (6 message(s) examined)
archive-iterator: readdir found no mail in '/home/baz/Maildir/.INBOX.Spam/new' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 2 message(s) (2 message(s) examined)
archive-iterator: readdir found no mail in '/home/buz/Maildir/.INBOX.Spam/new' directory
Learned tokens from 0 message(s) (0 message(s) examined)

Backing & restoring up the Bayesian database

Since the Bayesian database takes a month or so to build up to a useful state, it's important to be able to back it up in case it gets corrupted or if the server needs to be rebuilt. Such a backup can also be used in the installation process of any new server so that there's no need to go through the initial learning process for each new installation.

To back up the database use sa-learn --backup > mybackup.txt and to restore it again, first clear any existing data if there is any with sa-learn --clear and then use sa-learn --restore mybackup.txt to restore from the backup file.

The actual data is stored in ~/.spamassassin in files called bayes_tok and bayes_seen, so if there's a problem and they dissapear you can force the location sa-learn looks for its data with the --dbpath parameter then do a backup and restore without the parameter.

Updating Spam Assassin's rules

In addition to relying on the Baysean database, it's also useful to run the sa-update script every month or so which downloads refined rules from the Spam Assassin channel. GPG keys are used to ensure that the rules are coming from the correct source, but I haven't got these working yet. Following is the command to run the update without checking the channel for authenticity since it won't replaced the current rules with the downloaded ones if this is not specified. The spamd daemon must be restarted after the rules have been updated.

sa-update --nogpg
service spamassassin restart

Set up a local non-forwarding caching DNS server

DNS block lists (DNSBL) are an important tool for blocking spam because the spammers use a variety of different domains all the time to try and combat spam filters. The DNS block lists maintain a real-time list of spammer's domains which can be queried using DNS. Spam Assassin is configured to use many different DNSBL services by default, but the requests for the data may be blocked if the DNS server you're using is not paying for the DNSBL service and is making too many requests to it. You can easily see if your DNSBL requests are being blocked because you'll see URIBL_BLOCKED in your X-Spam-Status message headers, e.g.

X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,HTML_IMAGE_ONLY_32,
	HTML_MESSAGE,T_DKIM_INVALID,T_RP_MATCHES_RCVD,URIBL_BLOCKED,URI_NOVOWEL
	autolearn=ham version=3.3.2

To get around this problem you simply need to set up your own non-forwarding caching name server. See Configure DNS for installation and configuration details.

Your X-Spam header should be looking more like this:

X-Spam-Status: Yes, score=11.0 required=5.0 tests=ADVANCE_FEE_2_NEW_MONEY,
	BAYES_00,HTML_MESSAGE,LOTS_OF_MONEY,MIME_HTML_ONLY,RCVD_IN_BRBL_LASTEXT,
	RCVD_IN_XBL,RDNS_NONE,URIBL_BLACK,URIBL_DBL_SPAM,URIBL_SBL,URIBL_SBL_A,
	URIBL_WS_SURBL autolearn=no autolearn_force=no version=3.4.0

Filter files

Exim uses a special scripting language that can be placed in user's .forward files to specify more complex forwarding and filtering rules. If the .forward file is using the Exim scripting language it must have # Exim filter on the first line. The default filter file we use is the one shown above in the Spamassassin configuration, but the file can be extended in numerous way to cater for the specific needs of individual users.

Auto-responder / Vacation message

See the extended .forward filter here, and has been tested successfully. This method extends the existing forward filter, but unfortunately it raises an error if the .vacation.msg file isn't present, and it can only be required from within the exim configuration, not from within the filter itself, and there seems to be no condition for testing file existence.

Conditionally delivering messages to specific folders

Sometimes we need to prevent the inbox getting too cluttered by having messages from some very prolific senders or lists get automatically filed. Here's a simple example.

# Exim filter
if $header_from matches "Harry McProlific" then
    save $home/Maildir/.Lists.Unread.Harry/
    finish
endif

Note: don't forget the trailing slash or you'll get the "mailbox has too many links" error on delivery

Setting up mailing lists

To create a simple mailing list, you must have the message_body_visible set to a reasonable number and the message_body_newlines directive set to true in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config).

message_body_newlines = true
message_body_visible = 5000

Note: this number of lines should be very large if you want to be able to send attachments in the mailing list

Next create a normal email account using the adduser command, and add an entry in /etc/exim4/virtual.users for the new address. Then create a .forward file (owned by the user) in their home directory as shown in the following example:

# Exim filter
if $h_subject contains "[Our List]" then
	seen mail
	from $reply_address
	reply_to "Our List<our-list@foo.bar>"
	subject $h_subject
	text $message_body
	to "Our List<our-list@foo.bar>"
	bcc "john@foo.bar,mary@foo.bar"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
else
	seen mail
	from $reply_address
	reply_to "Our List<our-list@foo.bar>"
	subject "[Our List] $h_subject"
	text $message_body
	to "Our List<our-list@foo.bar>"
	bcc "john@foo.bar,mary@foo.bar"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
endif

The name of the list and its email address is added to the reply-to field, and the list of recipients is in the bcc field. Later these directives can be added to the files automatically. The to field contains just a dummy address that should be set up to get black-holed.

We also have a MediaWiki extension called EximMailList that can populate a forward file automatically using the email addresses of the wiki users. It should be called from a cron job on the local host, and requires two settings in the LocalSettings.php file to be set, $wgEximMailListName and $wgEximMailListAddress. The crontab entry should be something like this:

5 * * * * root wget -q -O /home/foo/.forward "http://foo.com/wiki/index.php?title=MediaWiki:Common.css&action=eximfilter"

I've used the article title MediaWiki:Common.css to cater for the case of the wiki being locked down, since the CSS article will always be in the white list.

Sending certain messages to multiple recipients

Sometimes it's useful to be able to have some messages that arrive for a specific user send to a number of different people. This is a little different than simple setting up a number of forwarding addresses in the virtual.users file for the address, because in this case we want the rule to only apply to messages that match certain characteristics such as having a specific subject or From address. One example of where this can come in handy is for a company that's subscribed to a paid mailing list and needs all the staff to receive the information.

If you have mail lists set up in the way described in the previous section, then this can be achieved simply by using a slightly modified version of the mail-list filter rules as follows.

# Exim filter
if $h_from contains "updates@expensive-info.com" then
	seen mail
	from $reply_address
	subject $h_subject
	text $message_body
	to "Organic Design staff<manager@organicdesign.co.nz>"
	bcc "manager@organicdesign.co.nz,sales@organicdesign.co.nz,dev@organicdesign.co.nz"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
endif

Delivering emails directly to web-sites or other local programs

Often sites require the ability to respond to incoming emails, for example the site may have the ability for users to reply to site notifications sent by email from the site. We've previously used the EmailToWiki extension for this, but the extension requires that the emails first be delivered to a mailbox and then a cron-job polls the mailbox and removes/processes messages using the IMAP or POP3 protocol. This is a very unresponsive and resource consuming approach.

A new script has been made called post-email.pl which allows emails to be delivered directly to a site via HTTP instead of being delivered to a mailbox. This script is invoked via an Exim filter in the recipient email-user's .forward file. The filter is a single line consisting of a pipe command which sends the email content to the specified program in the STDIN stream, the URL that the extracted body text and headers will be posted to is specified as a command-line parameter to the program.

The script extracts the useful headers (id, date, from, to, subject) and the email body from the message and formats them appropriately using the same code as the original EmailToWiki extension used and then posts the data to the URL specified in the Exim filter's pipe command.

The script has a feature allowing the $header_to value from the Exim filter to be used in the URL instead of a normal domain so that it can decide what to do with the message based on the recipient address. The domain part of the email address will be used as the site domain. The following example demonstrates this feature.

# Exim filter
pipe "perl $home/post-email.pl \"http://$header_to:/index.php\""
  • The post-email.pl script, or a symlink to it, is expected to be in the same directory as the filter
  • Explicitly specifying the perl interpreter means there's no need to worry about the script's executable permissions
  • The quotes are needed if the URL contains a query-string with ampersands


This example allows some local addresses to deliver normally while all others go into the pipe:

# Exim filter
if
   $header_to does not match "(info|admin)@"
then
   pipe "perl $home/post-email.pl \"http://www.example.com\""
endif

Notes on Exim Filters

  • Exim doesn't need to be restarted for changes to the .forward or virtual.users files.
  • You can also forward messages to other recipients by adding one or more deliver directives to the filter file.
  • To debug filter problems, stop the Exim4 service and restart on the command line with -bd -d+filter options.
  • Remember that vacation messages are only sent to the same user once per week, and deleting the .vacation.log file does not reset this, see manual for details.
  • the foranyaddress $h_to: ( $thisaddress contains "$local_part@" ) rule wasn't working last time I installed this - needs checking

Backing up and synchronising Maildirs

nad@nad-laptop:~$ apt-cache search imapsync
 imapcopy - IMAP backup, copy and migration tool
 imapsync - IMAP synchronization, copy and migration tool


For synchoronising the data from another backed up Maildir, use apt-get install maildirsync and the following example syntax:

maildirsync -rvvv Source/Maildir/ Dest/Maildir results-info.bz2


You can also use rsync to maintain a backup of Maildirs on another server without having to copy the entire structure every time. But beware that Maildir uses colons in its filenames which can cause problems with some filesystems. See the rsync article for a solution to this.

Client setup

Any standard mail client such as Thunderbird or Outlook should connect with no trouble, but our procedure doesn't yet include the generation of a valid SSL certificate, so you'll get a warning initially which you can specify to be ignored for subsequent connections. We don't recommend using any Microsoft products at all, but if you had no choice, then the Set up an IMAP account on Outlook 2007 procedure may be of some help.

Privacy: You should set up PGP so you can communicate privately. This won't work or webmail, but works on desktop clients, or on Android you can use OpenKeyChain.org

Incoming mail settings (IMAP)

We use the IMAP protocol, but it is possible to use both POP3 and IMAP together for the same account because both are simply different protocols for accessing the same mail. There's probably no useful purpose to do that, but if you do then make sure the client settings for the POP setup is configured to leave the messages on the server, otherwise any messages downloaded from the inbox will be removed and will therefore not be available from the IMAP inbox folder either. Note that any messages that have been moved out of the inbox and into another folder using the IMAP protocol will no longer be available for download by the POP3 protocol because it only interacts with the inbox folder which is the root mailbox.

The settings for OrganicDesign IMAP access are: imap.organicdesign.co.nz on port 993 using SSL encryption.

Outgoing mail settings (SMTP)

To set the email client to use the organisations SMTP server, go to Account settings/Outgoing server (SMTP) and add a new one called OrganicDesign with server address smtp.organicdesign.co.nz, port 2525 and STARTTLS security. Use your IMAP login details with normal password authentication method.

You may want to keep your ISP's SMTP server as the default since it will work more quickly being more local and not requiring encryption or authentication, but you can easily make the Organic Design SMTP server the default when working in the field.

Domain and IP address configuration

Following are some important configurations that should be done on the mail-server's domain and IP address. If these aren't done then emails sent by the server are more likely to be routed to the recipient's junk folder, or may even be rejected by the remote server.

Reverse DNS

Any site that sends emails should have reverse DNS correctly configured. Most mail-servers will do a reverse lookup on the sending IP address and ensure it matches the senders specified domain.

This is not done by the domain registrar, it's done by the company hosting the server (the IP address owner), sometimes they include the ability to set it in the server management interface. If not, raise a support ticket asking them if they can set up a PTR record for the server's IP pointing to the domain specified in /etc/mailname or Exim's primary_domain setting.

You can find out more about what reverse DNS is and why it's important here. To check the reverse DNS of any IP address simply do the host command as you would for a domain, e.g.

host 1.2.3.4

Block-lists

You should also check SpamHaus to see if it is listed in any block-lists. If it is then that page will tell you which ones and include links on where to go to get the IP removed from the lists.

SPF records (Sender Policy Framework)

A lot of mail-servers now require senders domains to have an SPF record (official site). I think that the following is a good generic record format to use as the IP4 will fail if the server moves to a new IP address.

v=spf1 a mx ptr -all

This version means that the mail will pass as long as the senders address has a matching A and MX record for the domain it says it's from, and the reverse lookup matches one of the domains A records. The record is simply a TXT record in the domain on the @ prefix for a normal domain, or on the appropriate host if the mailname/rDNS of the server is a sub-domain.

Note: SPF checking for Exim4 has been supported since version 4.52 and in recent version can be enabled simply by setting the CHECK_RCPT_SPF flag and installing the spf-tools-perl package via apt.

RoundCube (IMAP-only webmail)

We need to have access to our IMAP folder structures from a browser, we use the RoundCube webmail application for this purpose.

First ensure that the sub-domains which should have webmail access have a rule in the web-server configuration mapping it to the roundcube code base which can be downloaded from here and should be saved to /var/www/domains/webmail. See set up a new domain name for details on mapping a sub-domain to a web application code-base.

Next, go to your.domain/installer and follow their installation procedure which generates the configuration files for you (don't use real passwords as you can change these after you paste the configuration into the real files on the server). Set the default_host option to localhost to prevent the login from having a host input. Use localhost for the server addresses (you may need to use ssl://localhost if non-ssl is denied by the server). You'll need to manually create the MySQL database:

create database roundcubemail;


The installer tests whether the database exists and is writable, then allows you to initialise it with a button. At this point you can also test the IMAP login and SMTP sending.

Manually populating the database

Under some (currently unknown) circumstances the database tables are not created by the web installer, so you may need to manually populate the database by manually logging in to MySQL and creating a database called roundcubemail, and then importing from the shell with the following command:

mysql -u USER -p PASS roundcubemail < /var/www/domains/webmail/SQL/mysql.initial.sql

Once all these are working, go to the root of the webmail domain and login.

Upgrading

To upgrade the Roundcube code-base, download and unpack the new version into a temporary location, change into that directory then run the update script with the location of the current installation as the parameter.

./bin/installto.sh /var/www/domains/webmail/

Note: This process will rebuild the configuration and the hack shown below to prevent outgoing mail being saves to "Sent" will need to be added again. It would be best to check that the same method still works after the upgrade as well, but checking with Firebug that the save-to option in the compose view still uses the same key and value method in its posted data.

Nginx configuration

The web-mail is best forced to HTTPS, so we have some configuration in both the plain and secure server blocks. In the plain block we simply detect if the request domain starts with "webmail", and if so redirect the request to HTTPS. In the secure block we have the same domain-based rule but instead of redirecting, it rewrites the request to the appropriate file system location of the RoundCube code-base.

server {
	listen 80;
	...
	server_name ~^webmail\.;
	rewrite ^ https://$host:443$uri last;
	...
}
server {
	listen 443;
	ssl on;
	...
	if ( $host ~ ^webmail\. ) {
		rewrite ^/$ /webmail/index.php last;
		rewrite ^ /webmail$uri last;
	}
	...
}

Manual Configuration

In our configuration there is only one setting in db.inc.php which is the mysql database connection line, and there are a few settings in the main.inc.php, the default_host is "ssl://localhost", the des_key, the product_name which is "OrganicDesign Webmail" in our case, and finally the mail_domain should be set to the default domain that user-names should be under to make a valid return address.

Disabling the copying to Sent folder

If your server already handles the copying of sent messages into the "Sent" folder (for example using the procedure shown above), then you'll want to stop RoundCube from also making a copy of the message there. This is easily by un-setting the "Sent" option in RoundCube's "special folders" settings page, but if you do this then the "Sent" folder will be listed alphabetically amongst all the other normal folders. So it's preferable to keep treating "Sent" as a special folder, but disabling the copying functionality.

In RoundCube's "compose message" page there is a drop-down list that allows the message to be saved in a different folder, or no folder at all. If the "Sent" folder is special, then this drop-down list will be set to "Sent" by default. All we need to do to disable the copying-to-sent functionality is to add the following snippet into the beginning config.inc.php configuration file so that a "Sent" target for newly sent messages is changed to no target.

if( array_key_exists( '_store_target', $_POST ) && $_POST['_store_target'] == 'Sent' ) {
	$_POST['_store_target'] = '';
}

Troubleshooting

Stop the exim service and run again from the shell with debugging enabled,

exim4 -bd -d+filter

For more information on the command line options, see chapter 5 of the manual.

suhosin.session.encrypt option value is incorrect

One issue is that on some systems the Roundcube installation procedure gives an error saying that the suhosin.session.encrypt option value is incorrect (it should be disabled) even though it says it is disabled in php.ini. The suhosin settings have their own php.ini configuration file in /etc/php5/apache2/conf.d/suhosin.ini which overrides the settings in the default php.ini. Change the line containing the suhosin.session.encrypt option in this file to disable it (and don't forget to uncomment the line by removing the leading semicolon), then restart the web server and run the Roundcube installer again.

Problems sending to Hotmail/Microsoft

Hotmail have increased their spam security and require a number of steps to be taken for smtp servers to be able to send mail to them without being blocked as spammers. You can de-list blocked IPs at sender.office.com.

.forward files not being processed

The most common reason for this is that the messages that you're expecting to be forwarded do not have a local destination, check that the domain they're under is a local domain, you may need to add it to the dc_other_hostnames setting in /etc/exim4/update-exim4.conf.conf or to your virtual.domains file if you're using the above configuration.

Panic log contains "User 0 set for maildir_home transport is on the fixed_never_users list"

This means that your configuration has required Exim to run as root which it does not allow. The most common reason for this is a local delivery to the root user. Either the domain of the local host should have its email handled by an external server so that the root@yourdomain address does not attempt to be delivered to the local root user, or there should be an alias for the local root user set in /etc/aliases.

Our system has detected that this message does not meet IPv6 sending guidelines regarding PTR records and authentication

This is an error that is often seen in the logs when Exim sends messages to Gmail. Having an SPF record may fix it, but if it continues, you can disable SMTP connections over IPv6 by adding the disable_ipv6 directive into the main/02_exim4-config_options section.

## main/02_exim4-config_options
#################################

disable_ipv6

Spammers clogging up the server

Today I had a problem whereby I couldn't send an email due to there being too many connections to the mail server. I've never had this before and it shouldn't happen since we're a very small operation with only twenty or thirty active email users. Using netstat -5 | grep smtp revealed that indeed there were many concurrent SMTP connections and they were all from a couple of IP addresses. I checked the Exim log to see if they were doing anything, but fortunately Exim was rejecting them due to them not having any valid host name associated with them.

2015-12-01 06:38:49 no host name found for IP address 223.240.93.244
2015-12-01 06:38:51 no host name found for IP address 223.240.93.244
2015-12-01 06:38:53 no host name found for IP address 223.240.93.244
2015-12-01 06:38:59 no host name found for IP address 117.68.193.8
2015-12-01 06:39:01 no host name found for IP address 117.68.193.8
2015-12-01 06:39:03 no host name found for IP address 117.68.193.8
2015-12-01 06:39:10 no host name found for IP address 117.68.193.8
2015-12-01 06:39:16 no host name found for IP address 117.68.193.8
2015-12-01 06:39:21 no host name found for IP address 117.68.193.8
2015-12-01 06:39:28 no host name found for IP address 223.240.78.176
2015-12-01 06:39:30 no host name found for IP address 223.240.78.176

But since they're still clogging up the server I needed to block them at a lower level. The only problem is that the IP address keeps changing. So I wrote ip-block.pl which is called every 5 min from the crontab to check the Exim log of recent activity for items of the format shown above, and any IP addresses that reoccur more than ten times are blocked from accessing port 25 by iptables. Use iptables --list to check what filter rules are currently in place.

# iptables --list | grep smtp
DROP       tcp  --  223.240.84.70        anywhere             tcp dpt:smtp
DROP       tcp  --  175.45.186.150       anywhere             tcp dpt:smtp
DROP       tcp  --  114.96.76.14         anywhere             tcp dpt:smtp
DROP       tcp  --  223.240.75.50        anywhere             tcp dpt:smtp
DROP       tcp  --  114.96.70.100        anywhere             tcp dpt:smtp
DROP       tcp  --  94.249.127.124       anywhere             tcp dpt:smtp
DROP       tcp  --  70.45.94.194         anywhere             tcp dpt:smtp
DROP       tcp  --  179.127.166.29       anywhere             tcp dpt:smtp

Message does not meet IPv6 sending guidelines regarding PTR records and authentication

This is an error that is often seen in the logs when Exim sends messages to Gmail. Having an SPF record may fix it, also try adding an IPv6 rDNS record, but if you can'y do that or it still continues, you can disable SMTP connections over IPv6 by adding the disable_ipv6 directive into the main/02_exim4-config_options section.

See also