Configure mail server

Configure mail server Organic Design procedure

Mail Transfer Agents (MTA) connect to one another on port 25 to route mail to its destination. Other ports such as 465 or 587 are also used, but these are used exclusively for connections from Mail User Agents (MUA) to MTAs. All ports are the same as far as the listening MTA is concerned, any of them can use plain of encrypted connection which is negotiated as part of the SMTP protocol's handshake phase.

In our configuration described here, the MTA listens on ports 25 and 587 (encrypted) for incoming emails and decides whether to relay them somewhere else for delivery, or to deliver them locally. In our configuration we have no relaying at all since all recipient accounts we're handling are accounts on the same host as the MTA. So if an incoming email is destined for a local recipient, we pass it through a spam filter and then deliver it locally without any other restrictions.

But the local clients (MUAs) need to be able to use the MTA for sending mails from their local machines out to other external recipients. To cater for this situation, the MTA is configured so that it will allow delivery of mail to remote recipients as well, but only if the connection is encrypted and uses authentication with credentials that match one of the local UNIX accounts.

If authentication is used, the spam filter is bypassed even if deliver is to a local client, which means that the MTA can also be used as a "smart host" by other MTAs that are unable to send traffic on port 25, which is often then case when hosting on a DSL connection for example. In this case, the firewalled MTA can be configured to connect to an MTA configured as described here on port 587 with authentication.

Exim4 (MTA & MDA)

Exim4 is the default mail transfer and delivery agent for Debian, but it's only the light version which is insufficient for running a mail server, so first we need to install the exim4-daemon-heavy package which will replace the light version, and a few other packages we'll be needing.

sudo apt-get install exim4-daemon-heavy dovecot-common dovecot-imapd spamassassin spamc spf-tools-perl

Exim is designed to move messages from one e-mail server to another and to deliver messages to local users mailboxes. It has nothing to do with POP3 or IMAP as those are protocols relating to the retrieval of the mail by a user and their mail client software. The default Exim configuration allows the server to send mail, but if you plan on running an IMAP or POP3 server, then additional configuration is required which is discussed here.

First, run through the standard Exim4 configuration script, say no splitting configuration files, set general mail configuration to internet, use MailDir delivery method, and keep any non-self-explanatory settings as default.

dpkg-reconfigure exim4-config

When changing configuration in Exim4 there are three different levels:

• Making changes to update-exim4.conf.conf requires running update-exim4.conf and service exim4 restart.
• Making changes to exim4.conf.template requires only service exim4 restart.
• Making changes to virtual.users, virtual.domains or any users .forward file requires no other action.

Mail name, hostname, rDNS and blacklists

It's important that the domain part of the MAIL FROM field in outgoing messages has a proper domain with a valid reverse lookup, otherwise some SMTP servers will reject them. To ensure this, check that the domain name specified in the first line of /etc/mailname is the correct one, and set it if not. Alternatively you can set the primary_host value in the exim configuration to the proper domain, this is best set at the top of the configuration file, and you may need to comment out other occurrences further down.

The hostname should be set to the domain with the hostname -b command and can also be set in /etc/hostname and added as a localhost name in /etc/hosts.

The reverse DNS should be set up in your server host's configuration (e.g. this is done in the Remote Access tab of the Linode configuration), the server host may need to do this for you if their client app doesn't support it. Note that you should have reverse DNS set up for your IPv6 address as well, and if your ISP can't provide this then you'll need to add the disable_ipv6 directive to your configuration (see troubleshooting section below).

You may find that your server's IP is listed in various spam or abuse blacklists, a good service to check all the common blacklists in one go at is MX Toolbox. They also offer a free notification service, or a more advanced paid service.

Maildir delivery method

This should have been set in the package configuration above, but if not it can be set manually as follows. The primary advantage of maildirs is that multiple applications can access the same Maildir simultaneously without requiring any kind of locking whatsoever, but aside from this, Maildir format is more efficient than mbox, for more information see mbox-vs-maildir. Check the /etc/exim4/update-exim4.conf.conf and append dc_localdelivery='maildir_home' if it doesn't already exist.

Main configuration for multiple domains

You may want to handle mail for a few domains on one server, in which case some generic usernames like "accounts" will conflict with the local user names, or with the same names used by other domains. This is based on Blair Harrison's method from this Waikato LUG article which adds virtual domain support to Exim4. This allows any incoming email address to be mapped to any other internal user mailbox, or external email address.

The main configuration changes to be made are in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config). Here's a summary of the main settings we use at the top of this file, these are explained in detail below along with their associated specific configurations throughout the file.

primary_hostname = organicdesign.co.nz
message_body_newlines = true
message_body_visible = 500000
MAIN_TLS_ENABLE = true
daemon_smtp_ports = 25 : 587 : 2525
system_filter = /var/www/tools/exim-copy-to-sent
system_filter_directory_transport = copy_to_sent
system_filter_pipe_transport = copy_to_sent_pipe
CHECK_RCPT_SPF = true
helo_allow_chars = _

Next, adjust the local_domains setting to the following:

domainlist local_domains = @ : @[] : localhost : partial-lsearch;/etc/exim4/virtual.domains

Next go to the section starting with "real_local" and add a new section as follows (if in split configuration, this section should be added in a new file called /etc/exim4/conf.d/router/300_exim4-config_virtual)

virtual:
	driver = redirect
	allow_defer
	allow_fail
	data = ${lookup{$local_part@$domain}lsearch*@{/etc/exim4/virtual.users}}
	domains = partial-lsearch;/etc/exim4/virtual.domains
	retry_use_local_part

Now you will need file called /etc/exim4/virtual.domains that is simply a list of all the domains for which you accept mail in the following format:

example.com
example.net

Don't forget to restart the mail server,

/etc/init.d/exim4 restart

Setting up mail users

To add or modify the mail users, edit the /etc/exim4/virtual.users file. Following are some examples to highlight some of the most popular scenarios:

regularuser@example.com     : localuser@localhost
forwardinguser@example.com  : someuser@example.org
foo@example.com             : :fail: Foo no longer lives here.
bar@example.com             : :blackhole:
*@example.com               : catchall1@localhost

regularuser@example.net     : localuser2@localhost
forwardinguser@example.net  : someuser2@example.org
*@example.net               : catchall2@localhost

• Note: There is no need for Exim to be restarted after changing the virtual.users or virtual.domains files.

Testing delivery

Before moving on to SMTP or IMAP configuration, you should test that email sent to your local users/domains are being delivered properly. The mails will be delivered to the users home directories in ~/Maildir/cur/. You can cat the files to see the textual content of the messages stored there. You can send an email from the command line:

/path/to/exim -v 'user@domain'
message here
^D ( control D )

Another important thing to check before considering the system ready for production use is to send a test message to another mail server that you have access to the logs on. This way you can see exactly what other mail servers are seeing and ensure that everything matches up as it should. Here's an example of a remote server's Exim log entry showing a message received from the Organic Design server.

2013-04-25 11:37:49 1UVPB7-0008Q2-J8 <= aran@organicdesign.co.nz H=organicdesign.co.nz [37.17.226.20]
    P=esmtps X=TLS1.0:RSA_AES_256_CBC_SHA1:32 S=944 id=51795B7E.2020802@organicdesign.co.nz
2013-04-25 11:37:49 1UVPB7-0008Q2-J8 => nad <nad@foo.com> R=local_user T=maildir_home
2013-04-25 11:37:49 1UVPB7-0008Q2-J8 Completed

The most important part to look at here is the H value. The IP in the square brackets is the real IP address that the message has been received from, and the domain directly following the H is the name found after doing a reverse-DNS lookup on the IP. If the name is surrounded by round brackets it means that no reverse-DNS entry was found, and the name that the server is publishing in response to an SMTP HELO or EHLO command (exim's primary_domain value) is being used instead. If there's both a reverse_DNS name and also a bracketed name, then this means the reverse-DNS name is not the same as the name returned from the HELO command. It's very important that there be a reverse-DNS entry and that it exactly match the HELO domain because otherwise many mail servers will reject the message as spam. More information about the Exim log format of received messages can be found here in the Exim documentation.

To test what you server's domain is in the ELHO response connect via SMTP using telnet DOMAIN 25 then type ELHO. If you're connecting from a consumer connection you may need to use the 2525 port since many of them block 25 to prevent spammers.

Setting up a secure SMTP server with authentication

Since many people in the organisation may be working with laptops in the field, it's useful for the server to run it's own SMTP server so that settings don't need to be changed when moving from one ISP to another for internet access. The SMTP server will need to use authentication and encryption so that it can't be used for spamming.

Setting up a secure SMTP server with authentication is quite simple using Exim4's plaintext authenticator with TLS encryption.

SMTP Certificates

First add the following to the first section of /etc/exim4/exim4.conf.template to define the ports we'll listen on. Using port 587 is not only important for accepting encrypted connections, but also because many ISP's restrict outgoing traffic on port 25 to their own SMTP server only. All other TLS options are added under the main/03_exim4-config_tlsoptions section.

MAIN_TLS_ENABLE = true
daemon_smtp_ports = 25 : 587

Self-signed: Using a self-signed certificate, we prefer to use a wildcard domain for the common name such as *.organicdesign.co.nz so that it's possible for different sub-domains to access the service but get handled differently, then generate the certificate with the /usr/share/doc/exim4-base/examples/exim-gencert command.

LetsEncrypt: A better way is to use your LetsEncrypt certificate instead of a self-signed one by setting the MAIN_TLS_CERTIFICATE to the fullchain.pem file and MAIN_TLS_PRIVATEKEY to the privkey.pem file, and ensuring the files are both accessible by Exim (Note: the certs are symlinks so the target needs to be readable by Exim, and also all directories in the path need to be executable by Exim to be allowed to follow symlinks). For example:

MAIN_TLS_CERTIFICATE = /etc/organicdesign/ssl/fullchain.pem
MAIN_TLS_PRIVATEKEY = /etc/organicdesign/ssl/privkey.pem

Note: these certs must be readable by the Debian-exim user, we've found that we need to chgrp the private key to Debian-exim and set its mode to 640, and that symlinks cab cause trouble. We gave a deploy-hook for our LetsEncrypt certificate renewal that puts a copy of the files into /etc/organicdesign instead of using symlinks.

Cipher and protocol selection

We should also restrict the protocols and ciphers to mitigate POODLE and BEAST attacks etc, the available options are shown in the Exim manual here and here and a good guide here, and we use the following selection:

tls_require_ciphers = SECURE256:+SECURE128:-VERS-TLS-ALL:+VERS-TLS1.2:-RSA:-DHE-DSS:-CAMELLIA-128-CBC:-CAMELLIA-256-CBC

• Note: tls_require_ciphers replaces gnutls_require_kx, gnutls_require_mac and gnutls_require_protocols which are all obsolete now.

Using the native Linux user accounts and passwords

To use the native Linux accounts for SMTP authentication, the /etc/shadow file must be made accessible to the Debian-exim group:

chgrp Debian-exim /etc/shadow
chmod g+r /etc/shadow

Next find the authenticators section containing the string "PLAIN" and add a similar entry as follows, removing any other entry with the public name of PLAIN or LOGIN:

plain:
	driver = plaintext
	public_name = PLAIN
	server_condition = "${if crypteq{$auth3}{${extract{1}{:}{${lookup{$auth2}lsearch{/etc/shadow}{$value}}}}}{1}{0}}"
	server_set_id = $auth2
	server_prompts = :
	server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}

If you'd like to use a global username and password for your SMTP authentication instead of the native user accounts, use the following server_condition instead replacing the USER and PASSWORD with the global authentication you want all clients to use.

server_condition = "${if and {{eq{$auth2}{USER}}{eq{$auth3}{PASSWORD}}}{1}{0}}"

Test sending mail to an external domain

Restart the exim4 server and the server should be ready to accept SMTP requests!

To set up the clients use the user name and password defined above, smtp.organicdesign.co.nz for the server, 587 for the port and TLS for security.

To test that the SMTP is working and is not accessible to spammers, testing sending out emails to non-local recipients (as delivery to local domains is unrestricted). Try sending without authentication to ensure that it fails. Also test sending to a local domain from another SMTP server (such as your ISP's one) to ensure that normal delivery is working correctly.

Copying emails into the Sent folder

Many Thunderbird users have been having trouble with the "copying to sent folder" process freezing or being extremely slow. Rather than have every mail be effectively sent twice, once too the recipient and again to the sent folder, it's a lot more efficient to have the server make a copy in the user's Sent folder instead.

Web-mail note: If this is set up, you should also configure your web-mail to not place copies in the Sent folder. Our method of doing this is shown below in the RoundCube web-mail section.

To do this we need to set up an Exim system filter which begins by adding the following lines to the beginning of the configuration where the port and other OD settings are put.

system_filter = /var/www/tools/exim-copy-to-sent
system_filter_directory_transport = copy_to_sent
system_filter_pipe_transport = copy_to_sent_pipe

Next we need to add the two transport that were specified as the system filter transport above. I put this after end transport/30_exim4-config_mail_spool in the configuration and it's defined as follows:

copy_to_sent:
    driver = appendfile
    delivery_date_add
    envelope_to_add
    return_path_add
    group = Debian-exim
    user = ${if match {${local_part:${lookup{$sender_address_local_part@$sender_address_domain}lsearch*@{/etc/exim4/virtual.users}}}}{(.+)}{$1}{Debian-exim}}
    mode = 0660
    maildir_format
    directory = /home/${local_part:${lookup{$sender_address_local_part@$sender_address_domain}lsearch*@{/etc/exim4/virtual.users}}}/Maildir/.Sent/
    create_directory

copy_to_sent_pipe:
    driver = pipe
    user = Debian-exim
    group = Debian-exim

The filter itself is in our tools repo here and so will be available in the file system at /var/www/tools/exim-copy-to-sent, it has the following content:

if
    $sender_address_local_part is not "root"
and 
    "${if def:h_X-Spam-Status {def}{undef}}" is "undef"
and
    "${if match {${lookup{$sender_address}lsearch{/etc/exim4/virtual.users}}}{@localhost}{yes}{}}" is "yes"
then
    unseen save $home/Maildir/.Sent/
    unseen pipe "/var/www/tools/copy-to-sent.pl \"$recipients\""
endif

This filter saves the messages to the user's Sent folder if three conditions apply:

• The message is not from the server itself
• The spam headers have not been added yet (the filter will run both before and after they're added)
• The message is from a local account (i.e. not an incoming message)

One difficulty with using a system filter is that the BCC header is stripped by the time the message arrives at the system filter, but it's quite important for this header to be present in the Sent folder so that users can see who they sent the message to (also it's very useful for sending to a private list and then using the "Edit as new" feature to easily send another message to the same list). The header can be added back again using the $recipients variable which contains the entire list of recipients from all headers including BCC.

Another issue is that the messages are created as unread and Exim doesn't have any commands for changing the unread status as that's really a job for the mail client, not the transfer agent.

To get around these problems I've created the copy-to-sent.pl script that detects which file in the user's Sent/new Maildir folder is the one just added by checking the headers of all new messages and checking which has the new message ID which Exim sets in the MESSAGE_ID environment variable. The script then adds a Bcc header to the new message containing all the recipients that are not present in the To or Cc headers, and marks the message as read.

The system_filter_pipe_transport value in the Exim configuration needs to be set and an appropriate transport created for it before pipe commands are allowed to execute in system filters.

There is still one issue with this method currently which is that the complex expansion of the sender address into a local user has to be done twice because I can't figure out how to cache the result for later use. Perhaps the directory directive is not even needed and something else can be used instead.

LAN based SMTP servers

Often clients wish to run a local mail server on their LAN with the reasoning that they'd like all connection to their mail server to be extremely quick, to be independent of internet connectivity so that they can still access their mail and send mail internally independently of Internet connectivity. One complication with this is that some of the client computers on the LAN are laptops which must still be able to connect from outside the LAN from arbitrary Internet connections without any change of settings.

Here are some limitations and notes:

• Most ISP's block outgoing traffic on port 25 unless it's to their own SMTP server
• SSL certificate's domains must match the request even if self-signed
• Our SMTP connections require authentication using Unix account details

Our solution is to set clients up using smtp.organicdesign.co.nz as their outgoing server using a company-wide user which is set up on the OD server as a zero-privilege user. This works for the external situation, but to allow local mail to be routed directly independently of Internet requires that the local DNS server be set up with an organicdesign.co.nz zone to be authoritative for just the smtp sub-domain but forward everything else on. This will make the local exim instance the first stop for all internal mail sending requests, but will deliver according to the usual rules for all non-local recipients.

DoveCot IMAP server

We use DoveCot for our IMAP. DoveCot responds to mail client requests by retrieving the mails from file and returning them to the client. It does not deal with the receiving the mails or storing them to disk.

We go with a configuration which is as close to default as possible and gives us both POP3 and IMAP services which both work with either TLS or SSL but not plain text. The users are by default the native linux users and passwords on the system.

Our entire /etc/dovecot/dovecot.conf configuration file can be replaced with the following extremely minimal one since nearly everything we use is default and we only use IMAP with local UNIX accounts. SSL is required for incoming connections but unencrypted connections are supported locally so that the webmail and other local applications can access the mail without requiring any SSL configuration. For more detail about the configuration format, see their wiki.

• Note1: this configuration is for Dovecot 2.x, if you're still using 1.x check the history if this article and look at the revision prior to the comment about changing to Dovecot 2.x
• Note2: if you have trouble with your client having frequent login trouble, try adding mail_max_userip_connections = 100
• Note3: You can use your LetsEncrypt certificate instead of a self-signed one using the fullchain.pem and privkey.pem files and ensuring they're accessible by Dovecot
• Note4: I'm using Artem Sidorenko's recommended SSL cyphers from here, except that we don't disable SSLv2 since it's was removed completely from OpenSSL.

log_path = /var/log/dovecot.log

protocols = imap
protocol imap {
        listen = 127.0.0.1:143
        ssl_listen = *,[::]:993
}

mail_location = maildir:~/Maildir
maildir_very_dirty_syncs = yes

userdb {
        driver = passwd
}

passdb {
        driver = pam
}

ssl = required
ssl_prefer_server_ciphers = yes
ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:
                  EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
ssl_protocols = !SSLv3                        # SSLv2 has been removed from OpenSSL so is not even a valid name to disable
ssl_cert = </etc/dovecot/dovecot.pem          # you can also use your LE cert: /etc/letsencrypt/live/..../fullchain.pem
ssl_key = </etc/dovecot/private/dovecot.pem   # you can also use your LE cert: /etc/letsencrypt/live/..../privkey.pem

Spam Assassin

First install the spamassassin and spamc packages, then edit the configuration file /etc/default/spamassassin and set the ENABLED option set to 1. It's also a good idea to set the CRON option to 1 as well so that the rules are kept up to date on a daily basis. Remember to start spamassassin after enabling it; The only configuration you may want to change from the default is the required_scor setting in /etc/spamassassin/local.cf, the default is 5, but this can probably be dropped a bit to filter out more spam without loosing any real messages which are rarely even above zero.

service spamassassin start

Check you have no Perl modules missing with the following command and install them with cpanm if necessary:

spamassassin -D --lint 2>&1 | grep -i failed

The exim4 configuration needs to be adjusted which is in the /etc/exim4/exim4.conf.template file because our servers use the monolithic configuration method.

Add the following block at the end of the router/800_exim4-config_maildrop section:

spamcheck_router:
	no_verify
	check_local_user
	condition = "${if and { {!def:h_X-Spam-Flag:} {!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
	driver = accept
	transport = spamcheck_transport

Add the following block at the end of the transport/30_exim4-config_remote_smtp_smarthost section:

Note: be sure to check that the spamc binary is in the location specified, and that it is running.

# 30: Spamcheck transport
spamcheck_transport:
	debug_print = "T: spamassassin_pipe for $local_part@$domain"
	driver = pipe
	command = /usr/sbin/exim4 -oMr spam-scanned -bS
	use_bsmtp
	transport_filter = /usr/bin/spamc
	home_directory = "/tmp"
	current_directory = "/tmp"
	user = Debian-exim
	group = Debian-exim
	return_fail_output
	message_prefix =
	message_suffix =

At this point, after exim4 is restarted, all messages should have extra headers added by spamassassin for example:

X-Spam-Checker-Version: SpamAssassin 3.1.7-deb (2006-10-05) on organicdesign.co.nz
X-Spam-Level: ***********************
X-Spam-Status: Yes, score=23.9 required=5.0 tests=NO_DNS_FOR_FROM,
	RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,
	RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RCVD_IN_BL_SPAMCOP_NET,
	RCVD_IN_XBL,UNRESOLVED_TEMPLATE,URIBL_JP_SURBL,URIBL_OB_SURBL,
	URIBL_SBL,URIBL_SC_SURBL,URIBL_WS_SURBL autolearn=spam 
	version=3.1.7-deb

Moving spams to another folder automatically

Mails which have been marked as spam by spamassassin can be automatically moved into a specific mail folder by using an exim filter. First you may have to edit the exim4 configuration again and uncomment the allow_filter option in the 600_exim4-config_userforward section. We haven't had to do this so far because it is currently enabled by default.

To make a new filter, create a file in your home directory called .forward and ensure it is owned by the user in question, then add a rule such as in the following example from our server which forwards the spams straight into the Trash folder.

# Exim filter
if
   $h_X-Spam-Status: CONTAINS "Yes"
then
   save $home/Maildir/.Trash/
   finish
endif

• Important: do not change the comment in the first line!
• Important: use \40 for spaces in target directory name (escaped octal!)
• Note: You can use the make-forwards.pl to create them automatically for all users who don't already have them

Teaching spamassassin

Spamassassin comes with a script called sa-learn which is designed to parse mail directories with the information that the messages within them are either spam or "ham" (not spam). We have a Perl script called learn-spam.pl which resides in the /var/www directory and is run each morning. It scans all our user's Inbox/Spam and Inbox/Not Spam mail folders into which users should place their incorrectly assigned spam and non-spam messages. Note: the messages are removed after being processed, so the messages in your Not Spam folder should be copies, whereas those in the Spam folder can just be moved since you won't need them.

The sa-learn man page has some good information about Spamassassin's Bayesian learning mechanism, the following snippet is from there.

Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.

You need to train with both spam and ham mails. One type of mail alone will not have any effect.

Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)

If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run "spamassassin -d" over it in advance.

Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.

To check how many spams and hams the system has processed as well as some other internal data, use sa-learn --dump magic, here's the output of our system on 24 Dec 2008 which clearly shows that we haven't been feeding it enough ham! It shows that if we feed it ham at the same rate as spam, then it would start becoming useful in about another couple of weeks :-)

# sa-learn --dump magic
0.000          0          3          0  non-token data: bayes db version
0.000          0        733          0  non-token data: nspam
0.000          0         22          0  non-token data: nham
0.000          0      29962          0  non-token data: ntokens
0.000          0 1228202529          0  non-token data: oldest atime
0.000          0 1230047836          0  non-token data: newest atime
0.000          0          0          0  non-token data: last journal sync atime
0.000          0          0          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count

Here's a couple of lines of Perl which extract and print the number of spams and hams from the output shown above. I added this to our daily housekeeping script so that it appears in our recent changes each day.

$_ = `sa-learn --dump magic`;
print m/\s([1-9]+\d*).+?am[\x00-\x1f]+.+?([1-9]+\d*).+?am[\x00-\x1f]+.+?([1-9]+\d*).+?ns$/m
      ? "$1 spams and $2 hams have been processed with $3 tokens."
      : "ERROR";

Our training script, /var/www/learn-spam.pl, is run daily from the crontab and contains the following:

#!/usr/bin/perl

# Handle false positives
for (glob "/home/*/Maildir/.INBOX.Not\\ Spam/[cn]??") {
        s/ /\\ /;
        print qx "sa-learn --ham $_";
        qx "rm -fr $_/*";
}

# Handle false negatives
for (glob "/home/*/Maildir/.INBOX.Spam/[cn]??") {
        s/ /\\ /;
        print qx "sa-learn --spam $_";
        qx "rm -fr $_/*";
}

To test if the script is working properly, run it manually from the shell at such a time as you know there are spams to be processed and you should see messages stating that new tokens have been learned such as in the following example output.

archive-iterator: readdir found no mail in '/home/foo/Maildir/.INBOX.Spam/cur' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 16 message(s) (16 message(s) examined)
archive-iterator: readdir found no mail in '/home/bar/Maildir/.INBOX.Spam/cur' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 7 message(s) (7 message(s) examined)
Learned tokens from 6 message(s) (6 message(s) examined)
archive-iterator: readdir found no mail in '/home/baz/Maildir/.INBOX.Spam/new' directory
Learned tokens from 0 message(s) (0 message(s) examined)
Learned tokens from 2 message(s) (2 message(s) examined)
archive-iterator: readdir found no mail in '/home/buz/Maildir/.INBOX.Spam/new' directory
Learned tokens from 0 message(s) (0 message(s) examined)

Backing & restoring up the Bayesian database

Since the Bayesian database takes a month or so to build up to a useful state, it's important to be able to back it up in case it gets corrupted or if the server needs to be rebuilt. Such a backup can also be used in the installation process of any new server so that there's no need to go through the initial learning process for each new installation.

To back up the database use sa-learn --backup > mybackup.txt and to restore it again, first clear any existing data if there is any with sa-learn --clear and then use sa-learn --restore mybackup.txt to restore from the backup file.

The actual data is stored in ~/.spamassassin in files called bayes_tok and bayes_seen, so if there's a problem and they dissapear you can force the location sa-learn looks for its data with the --dbpath parameter then do a backup and restore without the parameter.

Updating Spam Assassin's rules

In addition to relying on the Baysean database, it's also useful to run the sa-update script every month or so which downloads refined rules from the Spam Assassin channel. GPG keys are used to ensure that the rules are coming from the correct source, but I haven't got these working yet. Following is the command to run the update without checking the channel for authenticity since it won't replaced the current rules with the downloaded ones if this is not specified. The spamd daemon must be restarted after the rules have been updated.

sa-update --nogpg
service spamassassin restart

Set up a local non-forwarding caching DNS server

Note: This step has a major impact on SpamAssassin's effectiveness!

DNS block lists (DNSBL) are an important tool for blocking spam because the spammers use a variety of different domains all the time to try and combat spam filters (see using network test in general). The DNS block lists maintain a real-time list of spammer's domains which can be queried using DNS. Spam Assassin is configured to use many different DNSBL services by default, but the requests for the data may be blocked if the DNS server you're using is not paying for the DNSBL service and is making too many requests to it. You can easily see if your DNSBL requests are being blocked because you'll see URIBL_BLOCKED in your X-Spam-Status message headers, e.g.

X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,HTML_IMAGE_ONLY_32,
	HTML_MESSAGE,T_DKIM_INVALID,T_RP_MATCHES_RCVD,URIBL_BLOCKED,URI_NOVOWEL
	autolearn=ham version=3.3.2

To get around this problem you simply need to set up your own local caching non-forwarding name server (see also understanding DNS forwarding). See Configure DNS for installation and configuration details.

Your X-Spam header should be looking more like this:

X-Spam-Status: Yes, score=11.0 required=5.0 tests=ADVANCE_FEE_2_NEW_MONEY,
	BAYES_00,HTML_MESSAGE,LOTS_OF_MONEY,MIME_HTML_ONLY,RCVD_IN_BRBL_LASTEXT,
	RCVD_IN_XBL,RDNS_NONE,URIBL_BLACK,URIBL_DBL_SPAM,URIBL_SBL,URIBL_SBL_A,
	URIBL_WS_SURBL autolearn=no autolearn_force=no version=3.4.0

Domain and IP address configuration

Following are some important configurations that should be done on the mail-server's domain and IP address. If these aren't done then emails sent by the server are more likely to be routed to the recipient's junk folder, or may even be rejected by the remote server.

Reverse DNS

Any site that sends emails should have reverse DNS correctly configured. Most mail-servers will do a reverse lookup on the sending IP address and ensure it matches the senders specified domain. Note that it's best if this domain has a host name part, not just the fully-qualified domain name part, for example mail.mydomain.com.

This is not done by the domain registrar, it's done by the company hosting the server (the IP address owner), sometimes they include the ability to set it in the server management interface. If not, raise a support ticket asking them if they can set up a PTR record for the server's IP pointing to the domain specified in /etc/mailname or Exim's primary_domain setting.

You can find out more about what reverse DNS is and why it's important here. To check the reverse DNS of any IP address simply do the host command as you would for a domain, e.g.

host 1.2.3.4

Block-lists

You should also check MX Toolbox to see if your mail-server IP is listed in any block-lists. If it is then that page will tell you which ones and include links on where to go to get the IP removed from the lists.

TXT records

There are a number of different mechanisms for verifying senders that are based on TXT records in the sending domain. These TXT records should be included in all domains that can be used as from addresses by the MTA.

DKIM (Domain Keys Identified Mail)

Setting up DKIM allows outgoing messages to be signed and the signature verified by receiving servers by looking up the public key in a TXT record associated with the domain.

Create a key pair using openssl as follows:

mkdir /etc/organicdesign/dkim
cd /etc/organicdesign/dkim
openssl genrsa -out dkim-private.pem 2048
openssl rsa -in dkim-private.pem -out dkim.pub -pubout -outform PEM
chown root:Debian-exim dkim-private.pem
chmod 640 dkim-private.pem

We like to keep the selector in a file too, so that our scripts that update DNS records can access all the DKIM information in one place.

echo "selector: YOUR_SELECTOR" > dkim-selector

You need to add the DKIM settings to you Exim4 configuration, they can go right at the top with the other specifics added. Setting DKIM_STRICT means that messages won't be sent and errors will be logged if the outgoing messages cannot be signed for some reason, for example if Exim can't read the private key. Don't forget to check the paniclog as well.

DKIM_CANON = relaxed
DKIM_SELECTOR = ${lookup{selector}lsearch{/etc/organicdesign/dkim/dkim-selector}}
DKIM_DOMAIN = primary_hostname
DKIM_PRIVATE_KEY = /etc/organicdesign/dkim/dkim-private.pem
DKIM_STRICT = true

Then the record itself will be a TXT record with host name of:

YOUR_SELECTOR._domainkey

and the content of your public DKIM key included for example:

v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC30aRx6rlDA7Lkhs/4XJMo
1AtuW8LoBrjo6RZH3yS7nC9EgqV5ntFIzQyCo88hNBz72XwwFAAGKuCVIwcxV06lAHWnUTt+ZyjJlP
9JH76ZJu9vUTaHw753IY3SZR+xEnJuyBr/LZknAEFqHuDP7V3+B6SWuBElSFFnImnP7oeMQQIDAQAB

Note: The key should be a single line in the TXT record. I've broken it here for clarity, and also the .pem file is multi-line as well.

You can test your DKIM signature at dkimvalidator.com.

SPF (Sender Policy Framework)

A lot of mail-servers now require senders domains to have an SPF record (official site). I think that the following is a good clear choice to use, but you'll need to update the record if the server moves to a new IP address.

v=spf1 ip4:SERVER-IP4 ip6:SERVER-IP6 -all

This simply ensures that mail can only be sent from your domains if it comes from your mail server's IP address. Before we had a version that used a PTR rule to make an IP independent rule, but reverse lookups are very resource intensive and cause mail sending to be slow down too. The record is simply a TXT record in the domain on the @ prefix for a normal domain, or on the appropriate host if the mailname/rDNS of the server is a sub-domain.

Note: SPF checking for Exim4 has been supported since version 4.52 and in recent version can be enabled simply by setting the CHECK_RCPT_SPF flag and installing the spf-tools-perl package via apt.

dkimvalidator.com also checks SPF records too.

DMARC (Domain-based Message Authentication, Reporting and Conformance)

DMARC is a protocol for adding TXT record with the host name of _dmarc to the sending domains DNS record that instructs receiving servers on what to do with messages that do not have corresponding SPF records or DKIM signatures. We have the following DMARC record which simple instructs receivers to reject messages that do not have DKIM signatures, or matching SPF rules.

The following advises the receiving MTA to reject any messages which fail either SPF or DKIM because the true server has both enabled. Use none instead of reject until you've confirmed that your SPF and DKIM configurations are functioning properly. For all the parameters and values see here.

v=DMARC1; p=reject

You can test by sending a message to gmail and checking the headers of the message in there. Google adds headers indicating its authentication tests and results. Got to the message and click the menu icon at the top right of the opened message content, then select "show original". You'll see a line like the following:

Authentication-Results: mx.google.com;
   dkim=pass header.i=@organicdesign.co.nz header.s=201809191 header.b=YuECVckW;
   spf=pass (google.com: domain of aran@organicdesign.co.nz designates 213.5.71.227 as permitted sender) smtp.mailfrom=aran@organicdesign.co.nz;
   dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=organicdesign.co.nz

For more detailed results covering everything, you can test your mail sending at mail-tester.com which is a pay service, but gives you three tests per day for free.

Filter files

Exim uses a special scripting language that can be placed in user's .forward files to specify more complex forwarding and filtering rules. If the .forward file is using the Exim scripting language it must have # Exim filter on the first line. The default filter file we use is the one shown above in the Spamassassin configuration, but the file can be extended in numerous way to cater for the specific needs of individual users.

Auto-responder / Vacation message

See the extended .forward filter here, and has been tested successfully. This method extends the existing forward filter, but unfortunately it raises an error if the .vacation.msg file isn't present, and it can only be required from within the exim configuration, not from within the filter itself, and there seems to be no condition for testing file existence.

Conditionally delivering messages to specific folders

Sometimes we need to prevent the inbox getting too cluttered by having messages from some very prolific senders or lists get automatically filed. Here's a simple example.

# Exim filter
if $header_from matches "Harry McProlific" then
    save $home/Maildir/.Lists.Unread.Harry/
    finish
endif

Note: don't forget the trailing slash or you'll get the "mailbox has too many links" error on delivery

Setting up mailing lists

To create a simple mailing list, you must have the message_body_visible set to a reasonable number and the message_body_newlines directive set to true in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config).

message_body_newlines = true
message_body_visible = 5000

Note: this number of lines should be very large if you want to be able to send attachments in the mailing list

Next create a normal email account using the adduser command, and add an entry in /etc/exim4/virtual.users for the new address. Then create a .forward file (owned by the user) in their home directory as shown in the following example:

# Exim filter
if $h_subject contains "[Our List]" then
	seen mail
	from $reply_address
	reply_to "Our List<our-list@foo.bar>"
	subject $h_subject
	text $message_body
	to "Our List<our-list@foo.bar>"
	bcc "john@foo.bar,mary@foo.bar"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
else
	seen mail
	from $reply_address
	reply_to "Our List<our-list@foo.bar>"
	subject "[Our List] $h_subject"
	text $message_body
	to "Our List<our-list@foo.bar>"
	bcc "john@foo.bar,mary@foo.bar"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
endif

The name of the list and its email address is added to the reply-to field, and the list of recipients is in the bcc field. Later these directives can be added to the files automatically. The to field contains just a dummy address that should be set up to get black-holed.

We also have a MediaWiki extension called EximMailList that can populate a forward file automatically using the email addresses of the wiki users. It should be called from a cron job on the local host, and requires two settings in the LocalSettings.php file to be set, $wgEximMailListName and $wgEximMailListAddress. The crontab entry should be something like this:

5 * * * * root wget -q -O /home/foo/.forward "http://foo.com/wiki/index.php?title=MediaWiki:Common.css&action=eximfilter"

I've used the article title MediaWiki:Common.css to cater for the case of the wiki being locked down, since the CSS article will always be in the white list.

See also

• Open source alternatives to MailChimp

Sending certain messages to multiple recipients

Sometimes it's useful to be able to have some messages that arrive for a specific user send to a number of different people. This is a little different than simple setting up a number of forwarding addresses in the virtual.users file for the address, because in this case we want the rule to only apply to messages that match certain characteristics such as having a specific subject or From address. One example of where this can come in handy is for a company that's subscribed to a paid mailing list and needs all the staff to receive the information.

If you have mail lists set up in the way described in the previous section, then this can be achieved simply by using a slightly modified version of the mail-list filter rules as follows.

# Exim filter
if $h_from contains "updates@expensive-info.com" then
	seen mail
	from $reply_address
	subject $h_subject
	text $message_body
	to "Organic Design staff<manager@organicdesign.co.nz>"
	bcc "manager@organicdesign.co.nz,sales@organicdesign.co.nz,dev@organicdesign.co.nz"
	extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding"
endif

Delivering emails directly to web-sites or other local programs

Often sites require the ability to respond to incoming emails, for example the site may have the ability for users to reply to site notifications sent by email from the site. We've previously used the EmailToWiki extension for this, but the extension requires that the emails first be delivered to a mailbox and then a cron-job polls the mailbox and removes/processes messages using the IMAP or POP3 protocol. This is a very unresponsive and resource consuming approach.

A new script has been made called post-email.pl which allows emails to be delivered directly to a site via HTTP instead of being delivered to a mailbox. This script is invoked via an Exim filter in the recipient email-user's .forward file. The filter is a single line consisting of a pipe command which sends the email content to the specified program in the STDIN stream, the URL that the extracted body text and headers will be posted to is specified as a command-line parameter to the program.

The script extracts the useful headers (id, date, from, to, subject) and the email body from the message and formats them appropriately using the same code as the original EmailToWiki extension used and then posts the data to the URL specified in the Exim filter's pipe command.

The script has a feature allowing the $header_to value from the Exim filter to be used in the URL instead of a normal domain so that it can decide what to do with the message based on the recipient address. The domain part of the email address will be used as the site domain. The following example demonstrates this feature.

# Exim filter
pipe "perl $home/post-email.pl \"http://$header_to:/index.php\""

• The post-email.pl script, or a symlink to it, is expected to be in the same directory as the filter
• Explicitly specifying the perl interpreter means there's no need to worry about the script's executable permissions
• The quotes are needed if the URL contains a query-string with ampersands

This example allows some local addresses to deliver normally while all others go into the pipe:

# Exim filter
if
   $header_to does not match "(info|admin)@"
then
   pipe "perl $home/post-email.pl \"http://www.example.com\""
endif

Notes on Exim Filters

• Exim doesn't need to be restarted for changes to the .forward or virtual.users files.
• You can also forward messages to other recipients by adding one or more deliver directives to the filter file.
• To debug filter problems, stop the Exim4 service and restart on the command line with -bd -d+filter options.
• Remember that vacation messages are only sent to the same user once per week, and deleting the .vacation.log file does not reset this, see manual for details.
• the foranyaddress $h_to: ( $thisaddress contains "$local_part@" ) rule wasn't working last time I installed this - needs checking

RoundCube (IMAP-only webmail)

We need to have access to our IMAP folder structures from a browser, we use the RoundCube webmail application for this purpose.

First ensure that the sub-domains which should have webmail access have a rule in the web-server configuration mapping it to the roundcube code base which can be downloaded from here and should be saved to /var/www/domains/webmail. See set up a new domain name for details on mapping a sub-domain to a web application code-base.

Next, go to your.domain/installer and follow their installation procedure which generates the configuration files for you (don't use real passwords as you can change these after you paste the configuration into the real files on the server). Set the default_host option to localhost to prevent the login from having a host input. Use localhost for the server addresses (you may need to use ssl://localhost if non-ssl is denied by the server). You'll need to manually create the MySQL database:

create database roundcubemail;

The installer tests whether the database exists and is writable, then allows you to initialise it with a button. At this point you can also test the IMAP login and SMTP sending.

Manually populating the database

Under some (currently unknown) circumstances the database tables are not created by the web installer, so you may need to manually populate the database by manually logging in to MySQL and creating a database called roundcubemail, and then importing from the shell with the following command:

mysql -u USER -p PASS roundcubemail < /var/www/domains/webmail/SQL/mysql.initial.sql

Once all these are working, go to the root of the webmail domain and login.

Upgrading

To upgrade the Roundcube code-base, download and unpack the new version into a temporary location, change into that directory then run the update script with the location of the current installation as the parameter.

./bin/installto.sh /var/www/domains/webmail/

Note: This process will rebuild the configuration and the hack shown below to prevent outgoing mail being saves to "Sent" will need to be added again. It would be best to check that the same method still works after the upgrade as well, but checking with Firebug that the save-to option in the compose view still uses the same key and value method in its posted data.

Nginx configuration

The web-mail is best forced to HTTPS, so we have some configuration in both the plain and secure server blocks. In the plain block we simply detect if the request domain starts with "webmail", and if so redirect the request to HTTPS. In the secure block we have the same domain-based rule but instead of redirecting, it rewrites the request to the appropriate file system location of the RoundCube code-base.

server {
	listen 80;
	...
	server_name ~^webmail\.;
	rewrite ^ https://$host:443$uri last;
	...
}

server {
	listen 443;
	ssl on;
	...
	if ( $host ~ ^webmail\. ) {
		rewrite ^/$ /webmail/index.php last;
		rewrite ^ /webmail$uri last;
	}
	...
}

Manual Configuration

In our configuration there is only one setting in db.inc.php which is the mysql database connection line, and there are a few settings in the main.inc.php, the default_host is "ssl://localhost", the des_key, the product_name which is "OrganicDesign Webmail" in our case, and finally the mail_domain should be set to the default domain that user-names should be under to make a valid return address.

Disabling the copying to Sent folder

If your server already handles the copying of sent messages into the "Sent" folder (for example using the procedure shown above), then you'll want to stop RoundCube from also making a copy of the message there. This is easily by un-setting the "Sent" option in RoundCube's "special folders" settings page, but if you do this then the "Sent" folder will be listed alphabetically amongst all the other normal folders. So it's preferable to keep treating "Sent" as a special folder, but disabling the copying functionality.

In RoundCube's "compose message" page there is a drop-down list that allows the message to be saved in a different folder, or no folder at all. If the "Sent" folder is special, then this drop-down list will be set to "Sent" by default. All we need to do to disable the copying-to-sent functionality is to add the following snippet into the beginning config.inc.php configuration file so that a "Sent" target for newly sent messages is changed to no target.

if( array_key_exists( '_store_target', $_POST ) && $_POST['_store_target'] == 'Sent' ) {
	$_POST['_store_target'] = '';
}

Client setup

Any standard mail client such as Thunderbird or Outlook should connect with no trouble. We don't recommend using any Microsoft products at all, but if you had no choice, then the Set up an IMAP account on Outlook 2007 procedure may be of some help.

Privacy: You should set up PGP so you can communicate privately. This won't work or webmail, but works on desktop clients, or on Android you can use OpenKeyChain.org

Incoming mail settings (IMAP)

We use the IMAP protocol, but it is possible to use both POP3 and IMAP together for the same account because both are simply different protocols for accessing the same mail. There's probably no useful purpose to do that, but if you do then make sure the client settings for the POP setup is configured to leave the messages on the server, otherwise any messages downloaded from the inbox will be removed and will therefore not be available from the IMAP inbox folder either. Note that any messages that have been moved out of the inbox and into another folder using the IMAP protocol will no longer be available for download by the POP3 protocol because it only interacts with the inbox folder which is the root mailbox.

The settings for OrganicDesign IMAP access are: od.nz on port 993 (default) using SSL encryption.

Outgoing mail settings (SMTP)

To set the email client to use the organisations SMTP server, go to Account settings/Outgoing server (SMTP) and add a new one called OrganicDesign with server address od.nz, port 587 (default) and STARTTLS security. Use your IMAP login details with normal password authentication method.

You may want to keep your ISP's SMTP server as the default since it will work more quickly being more local and not requiring encryption or authentication, but you can easily make the Organic Design SMTP server the default when working in the field.

Using other servers as clients

Mail is very complex to set up reliably these days (hence this huge procedure!). So if you run a number of servers that all need to have mail sending capability, you most likely don't want to set up rDNS, SPF, DKIM and DMARC for all of them. A far simpler solution is to use a simpler package like msmtp to present itself to the system as an MTA that the sendmail command can be symlinked to, but all it actually does is refers all requests to your main mail server.

First make sure that you don't have any other mail servers installed by default such as Exim4 or Postfix and then install MSMTP.

which sendmail
ls -al /usr/sbin/sendmail
dpkg -l |grep exim
apt remove exim4-daemon-light
apt install msmtp

Make sure that your sendmail command is linked to your new msmtp:

ln -s /usr/bin/msmtp /usr/sbin/sendmail

Create a minimal configuration in /etc/msmtprc with the following settings (based on your own SMTP servers configuration):

account default
syslog LOG_MAIL
maildomain your.remote.smtp.server
host your.remote.smtp.server
auto_from on
port 587
tls on
tls_starttls on
auth plain
user your_smtp_server_user
password *******

And test the commonly required forms of mail sending capability:

echo "testing" | mail -s "mail test" foo@bar.baz
echo "Subject: sendmail test" | sendmail -v "foo@bar.baz"
echo '<?php mail("foo@bar.baz","php test","testing");' | php

Backing up and synchronising Maildirs

nad@nad-laptop:~$ apt-cache search imapsync
 imapcopy - IMAP backup, copy and migration tool
 imapsync - IMAP synchronization, copy and migration tool

For synchoronising the data from another backed up Maildir, use apt-get install maildirsync and the following example syntax:

maildirsync -rvvv Source/Maildir/ Dest/Maildir results-info.bz2

You can also use rsync to maintain a backup of Maildirs on another server without having to copy the entire structure every time. But beware that Maildir uses colons in its filenames which can cause problems with some filesystems. See the rsync article for a solution to this.

Troubleshooting

Stop the exim service and run again from the shell with debugging enabled,

exim4 -bd -d+filter

For more information on the command line options, see chapter 5 of the manual.

suhosin.session.encrypt option value is incorrect

One issue is that on some systems the Roundcube installation procedure gives an error saying that the suhosin.session.encrypt option value is incorrect (it should be disabled) even though it says it is disabled in php.ini. The suhosin settings have their own php.ini configuration file in /etc/php5/apache2/conf.d/suhosin.ini which overrides the settings in the default php.ini. Change the line containing the suhosin.session.encrypt option in this file to disable it (and don't forget to uncomment the line by removing the leading semicolon), then restart the web server and run the Roundcube installer again.

Problems sending to Hotmail/Microsoft

Hotmail have increased their spam security and require a number of steps to be taken for smtp servers to be able to send mail to them without being blocked as spammers. You can de-list blocked IPs at sender.office.com.

.forward files not being processed

The most common reason for this is that the messages that you're expecting to be forwarded do not have a local destination, check that the domain they're under is a local domain, you may need to add it to the dc_other_hostnames setting in /etc/exim4/update-exim4.conf.conf or to your virtual.domains file if you're using the above configuration.

Panic log contains "User 0 set for maildir_home transport is on the fixed_never_users list"

This means that your configuration has required Exim to run as root which it does not allow. The most common reason for this is a local delivery to the root user. Either the domain of the local host should have its email handled by an external server so that the root@yourdomain address does not attempt to be delivered to the local root user, or there should be an alias for the local root user set in /etc/aliases.

Our system has detected that this message does not meet IPv6 sending guidelines regarding PTR records and authentication

This is an error that is often seen in the logs when Exim sends messages to Gmail. You'll need the domain to have reverse DNS setting for the IPv6 address, if your ISP can't provide this, then you can disable SMTP connections over IPv6 by adding the disable_ipv6 directive into the main/02_exim4-config_options section.

## main/02_exim4-config_options
#################################

disable_ipv6

Spammers clogging up the server

Today I had a problem whereby I couldn't send an email due to there being too many connections to the mail server. I've never had this before and it shouldn't happen since we're a very small operation with only twenty or thirty active email users. Using netstat -5 | grep smtp revealed that indeed there were many concurrent SMTP connections and they were all from a couple of IP addresses. I checked the Exim log to see if they were doing anything, but fortunately Exim was rejecting them due to them not having any valid host name associated with them.

2015-12-01 06:38:49 no host name found for IP address 223.240.93.244
2015-12-01 06:38:51 no host name found for IP address 223.240.93.244
2015-12-01 06:38:53 no host name found for IP address 223.240.93.244
2015-12-01 06:38:59 no host name found for IP address 117.68.193.8
2015-12-01 06:39:01 no host name found for IP address 117.68.193.8
2015-12-01 06:39:03 no host name found for IP address 117.68.193.8
2015-12-01 06:39:10 no host name found for IP address 117.68.193.8
2015-12-01 06:39:16 no host name found for IP address 117.68.193.8
2015-12-01 06:39:21 no host name found for IP address 117.68.193.8
2015-12-01 06:39:28 no host name found for IP address 223.240.78.176
2015-12-01 06:39:30 no host name found for IP address 223.240.78.176

But since they're still clogging up the server I needed to block them at a lower level. The only problem is that the IP address keeps changing. So I wrote ip-block.pl which is called every 5 min from the crontab to check the Exim log of recent activity for items of the format shown above, and any IP addresses that reoccur more than ten times are blocked from accessing port 25 by iptables. Use iptables --list to check what filter rules are currently in place.

# iptables --list | grep smtp
DROP       tcp  --  223.240.84.70        anywhere             tcp dpt:smtp
DROP       tcp  --  175.45.186.150       anywhere             tcp dpt:smtp
DROP       tcp  --  114.96.76.14         anywhere             tcp dpt:smtp
DROP       tcp  --  223.240.75.50        anywhere             tcp dpt:smtp
DROP       tcp  --  114.96.70.100        anywhere             tcp dpt:smtp
DROP       tcp  --  94.249.127.124       anywhere             tcp dpt:smtp
DROP       tcp  --  70.45.94.194         anywhere             tcp dpt:smtp
DROP       tcp  --  179.127.166.29       anywhere             tcp dpt:smtp

Message does not meet IPv6 sending guidelines regarding PTR records and authentication

This is an error that is often seen in the logs when Exim sends messages to Gmail. Having an SPF record may fix it, also try adding an IPv6 rDNS record, but if you can'y do that or it still continues, you can disable SMTP connections over IPv6 by adding the disable_ipv6 directive into the main/02_exim4-config_options section.

Warning: Unexpected error in SPF check

This most likely means that you don't have the spf-tools-perl package installed which Exim4 now relies on to do SPF checking.

Rejected EHLO, syntactically invalid argument

The most common reason for this is a hostname having an underscore in it, e.g.

2019-02-13 08:23:25 1gtuVZ-0005gN-9n <= foo@bar.baz H=(_) [2a02:908::b419] P=esmtpsa X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no A=plain_server:fred S=15273

It would be best to figure out why there's an underscore, or sometimes ONLY an underscore in the hostname, but you can allow the character in the Exim config with the following directive:

helo_allow_chars = _

R=dnslookup T=remote_smtp defer (port 25 blocked - configure Exim to route all requests to a remote server)

This most likely means that the Exim server is running behind a firewall, or on a service that has port 25 blocked. A mail server cannot operate without being able to send on port 25, because all MTA-MTA traffic occurs on port 25 (other mail ports such as 465 or 587 are only used for MUA-MTA traffic). The only solution is to route mail to another external SMTP server. In the following example, we configure the firewalled Exim to route mail to the Organic Design mail server which uses the PLAIN authentication method.

Note: If the firewalled server is only being used for sending then it may be easiest to use something more light-weight than Exim such as MSMTP, see the using other servers as clients section for details about this.

First add a new router before the current dnslookup router (the one that's failing in the log). We use auth_relay transport which we'll create next instead of the normal remote_smtp, because if we don't authenticate, the remote Exim will send all mail to trash as spam. Also be sure to set the port in the route_list to 587 since 25 is blocked (the double colon is not a typo).

smart_route:
     driver = manualroute
     domains = ! +local_domains
     ignore_target_hosts = 127.0.0.0/8
     transport = auth_relay
     route_list = * smtp.organicdesign.co.nz::587
     no_more

Now we need to create the auth_relay referred to in the previous block:

auth_relay:
    driver = smtp
    port = 587
    hosts_require_auth = $host_address
    hosts_require_tls = $host_address

Note that actually the port doesn't seem to matter here, maybe it's been overridden by being specified in route_list?

Then we need to add a new PLAIN authentication method to replace the current one, search for "PLAIN" to find the current (non-commented-out one) and then either comment it out or change its public_name to PLAINFOO or something to avoid there being two authenticators with the same name.

auth_plain:
    driver = plaintext
    public_name = PLAIN
    hide client_send = ^USER^PASS

Finally restart Exim, send a test message and check the log to see if your new items are being used:

2019-02-16 12:38:18 1guzEV-00058l-Kl => foo@bar.baz R=smart_route T=auth_relay H=smtp.organicdesign.co.nz [213.5.71.227] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes DN="CN=organicdesign.co.nz" A=auth_plain C="250 OK id=1guzEY-000263-0q"
2019-02-16 12:38:18 1guzEV-00058l-Kl Completed

See also

• Configure mail sending - set up minimal sending capability having a functional mail command
• PGP
• SSL - our info on SSL and LetsEncrypt certs
• Routing exim through a remote smtp server
• Mail Server Overview - in DoveCot manual
• Exim cheatsheet
• Exim4 manual
  • Exim4 configuration functions and variables
  • Exim4 configuration directives
  • The PLAIN and LOGIN authenticators
  • Exim4 generic router options
  • The Exim4 queryprogram router - routing messages according to an external program
  • The Exim4 pipe transport
  • Adding a local scan to Exim4
  • Filter files
    • String conditions in filter files
• Arstechnica's email procedure - see also part 2, part 3 and part 4
• Viewing and clearing the mail queue