Configure mail server

Configure mail server Organic Design procedure

Exim4 (MTA & MDA)

Exim4 is the default mail transfer and delivery agent for Debian, but it's only the light version which is insufficient for running a mail server, so first we need to install the exim4-daemon-heavy package which will replace the light version, and a few other packages we'll be needing.

<bash>apt-get install exim4-daemon-heavy, dovecot-common, dovecot-imapd, spamassassin, spamc</bash>

Exim is designed to move messages from one e-mail server to another and to deliver messages to local users mailboxes. It has nothing to do with POP3 or IMAP as those are protocols relating to the retrieval of the mail by a user and their mail client software. The default Exim configuration allows the server to send mail, but if you plan on running an IMAP or POP3 server, then additional configuration is required which is discussed here.

First, run through the standard Exim4 configuration script, say no splitting configuration files, set general mail configuration to internet, use MailDir delivery method, and keep any non-self-explanatory settings as default.

dpkg-reconfigure exim4-config

Mail name

It's important that the domain part of the MAIL FROM field in outgoing messages has a proper domain with a valid reverse lookup, otherwise some SMTP servers will reject them. To ensure this, check that the domain name specified in the first line of /etc/mailname is the correct one, and set it if not. Alternatively you can set the primary_host value in the exim configuration to the proper domain, this is best set at the top of the configuration file, and you may need to comment out other occurrences further down.

Maildir delivery method

This should have been set in the package configuration above, but if not it can be set manually as follows. The primary advantage of maildirs is that multiple applications can access the same Maildir simultaneously without requiring any kind of locking whatsoever, but aside from this, Maildir format is more efficient than mbox, for more information see mbox-vs-maildir. Check the /etc/exim4/update-exim4.conf.conf and append dc_localdelivery='maildir_home' if it doesn't already exist. Note that if you change update-exim4.conf.conf, then you'll need to also run update-exim4.conf.

Main configuration for multiple domains

You may want to handle mail for a few domains on one server, in which case some generic usernames like "accounts" will conflict with the local user names, or with the same names used by other domains. This is based on Blair Harrison's method from this Waikato LUG article which adds virtual domain support to Exim4. This allows any incoming email address to be mapped to any other internal user mailbox, or external email address.

The main configuration changes to be made are in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config).

Next, adjust the local_domains setting to the following:

domainlist local_domains = @ : @[] : localhost : partial-lsearch;/etc/exim4/virtual.domains

Next go to the section starting with "real_local" and add a new section as follows (if in split configuration, this section should be added in a new file called /etc/exim4/conf.d/router/300_exim4-config_virtual)

virtual: driver = redirect allow_defer allow_fail data = ${lookup{$local_part@$domain}lsearch*@{/etc/exim4/virtual.users}} domains = partial-lsearch;/etc/exim4/virtual.domains retry_use_local_part

Now you will need file called /etc/exim4/virtual.domains that is simply a list of all the domains for which you accept mail in the following format:

example.com example.net

Don't forget to restart the mail server,

/etc/init.d/exim4 restart

Setting up mail users

To add or modify the mail users, edit the /etc/exim4/virtual.users file. Following are some examples to highlight some of the most popular scenarios:

regularuser@example.com  : localuser@localhost forwardinguser@example.com  : someuser@example.org foo@example.com  : :fail: Foo no longer lives here. bar@example.com  : :blackhole: *@example.com  : catchall1@localhost regularuser@example.net  : localuser2@localhost forwardinguser@example.net  : someuser2@example.org *@example.net  : catchall2@localhost

• Note: There is no need for Exim to be restarted after changing the virtual.users file.

Testing delivery

Before moving on to SMTP or IMAP configuration, you should test that email sent to your local users/domains are being delivered properly. The mails will be delivered to the users home directories in ~/Maildir/cur/. You can cat the files to see the textual content of the messages stored there. You can send an email from the command line:

/path/to/exim -v 'user@domain' message here ^D ( control D )

Another important thing to check before considering the system ready for production use is to send a test message to another mail server that you have access to the logs on. This way you can see exactly what other mail servers are seeing and ensure that everything matches up as it should. Here's an example of a remote server's Exim log entry showing a message received from the Organic Design server.

2013-04-25 11:37:49 1UVPB7-0008Q2-J8 <= aran@organicdesign.co.nz H=organicdesign.co.nz [37.17.226.20] P=esmtps X=TLS1.0:RSA_AES_256_CBC_SHA1:32 S=944 id=51795B7E.2020802@organicdesign.co.nz 2013-04-25 11:37:49 1UVPB7-0008Q2-J8 => nad <nad@foo.com> R=local_user T=maildir_home 2013-04-25 11:37:49 1UVPB7-0008Q2-J8 Completed

The most important part to look at here is the H value. The IP in the square brackets is the real IP address that the message has been received from, and the domain directly following the H is the name found after doing a reverse-DNS lookup on the IP. If the name is surrounded by round brackets it means that no reverse-DNS entry was found, and the name that the server is publishing in response to an SMTP HELO or EHLO command (exim's primary_domain value) is being used instead. If there's both a reverse_DNS name and also a bracketed name, then this means the reverse-DNS name is not the same as the name returned from the HELO command. It's very important that there be a reverse-DNS entry and that it exactly match the HELO domain because otherwise many mail servers will reject the message as spam. More information about the Exim log format of received messages can be found here in the Exim documentation.

Setting up a secure SMTP server with authentication

Since many people in the organisation may be working with laptops in the field, it's useful for the server to run it's own SMTP server so that settings don't need to be changed when moving from one ISP to another for internet access. The SMTP server will need to use authentication and encryption so that it can't be used for spamming.

Setting up a secure SMTP server with authentication is quite simple using Exim4's plaintext authenticator with TLS encryption. Here we'll enable SMTP for all clients and we'll require authentication in the form of a single user/password used by all the clients (we could upgrade this later to use the client's IMAP login, this, this and this may help with those changes).

SMTP Certificates

Note: we prefer to use a wildcard domain for the common name such as *.organicdesign.co.nz so that it's possible for different sub-domains to access the service but get handled differently.

Generate your certificates as follows:

/usr/share/doc/exim4-base/examples/exim-gencert

Add the following to the first section of /etc/exim4/exim4.conf.template. The second directive adds a second SMTP port since many ISP's restrict outgoing traffic on port 25 to their own SMTP server only.

MAIN_TLS_ENABLE = true daemon_smtp_ports = 25 : 2525

Using a global inline SMTP user and password

Next find the section containing the string "plain_server" and add a similar entry as follows, but replace the USER and PASSWORD with the global authentication you want all clients to use.

plain_server: driver = plaintext public_name = PLAIN server_condition = "${if and {{eq{$auth2}{USER}}{eq{$auth3}{PASSWORD}}}{1}{0}}" server_set_id = $auth2 server_prompts = :

Using the native Linux user accounts and passwords

To use the native Linux accounts for SMTP authentication, the /etc/shadow file must be made accessible to the Debian-exim group:

chgrp Debian-exim /etc/shadow chmod g+r /etc/shadow

Next use the same plain_server as shown above, but change the server_condition to the following:

server_condition = "${if crypteq{$auth3}{${extract{1}{:}{${lookup{$auth2}lsearch{/etc/shadow}{$value}}}}}{1}{0}}"

Test sending mail to an external domain

Restart the exim4 server and the server should be ready to accept SMTP requests!

To set up the clients use the user name and password defined above, smtp.organicdesign.co.nz for the server, 2525 for the port and TLS for security.

To test that the SMTP is working and is not accessible to spammers, testing sending out emails to non-local recipients (as delivery to local domains is unrestricted). Try sending without authentication to ensure that it fails. Also test sending to a local domain from another SMTP server (such as your ISP's one) to ensure that normal delivery is working correctly.

LAN based SMTP servers

Often clients wish to run a local mail server on their LAN with the reasoning that they'd like all connection to their mail server to be extremely quick, to be independent of internet connectivity so that they can still access their mail and send mail internally independently of Internet connectivity. One complication with this is that some of the client computers on the LAN are laptops which must still be able to connect from outside the LAN from arbitrary Internet connections without any change of settings.

Here are some limitations and notes:

• Most ISP's block outgoing traffic on port 25 unless it's to their own SMTP server
• SSL certificate's domains must match the request even if self-signed
• Our SMTP connections require authentication using Unix account details

Our solution is to set clients up using smtp.organicdesign.co.nz as their outgoing server using a company-wide user which is set up on the OD server as a zero-privilege user. This works for the external situation, but to allow local mail to be routed directly independently of Internet requires that the local DNS server be set up with an organicdesign.co.nz zone to be authoritative for just the smtp sub-domain but forward everything else on. This will make the local exim instance the first stop for all internal mail sending requests, but will deliver according to the usual rules for all non-local recipients.

DoveCot (IMAP & POP3)

The organicdesign-server package installs DoveCot, an IMAP and POP3 server. DoveCot responds to mail client requests by retrieving the mails from file and returning them to the client. It does not deal with the receiving the mails or storing them to disk.

We go with a configuration which is as close to default as possible and gives us both POP3 and IMAP services which both work with either TLS or SSL but not plain text. The users are by default the native linux users and passwords on the system.

Our entire /etc/dovecot/dovecot.conf configuration file can be replaced with the following extremely minimal one since nearly everything we use is default (remove the POP3 entries if only IMAP access is preferred):

log_path = /var/log/dovecot.log protocols = imap imaps pop3 pop3s protocol imap { listen = *:143 ssl_listen = *:993 } protocol pop3 { listen = *:110 ssl_listen = *:995 pop3_uidl_format = %08Xu%08Xv } mail_location = maildir:~/Maildir auth default { userdb passwd { } passdb pam { } mechanisms = plain }

IMAP Certificate

Note: we prefer to use a wildcard domain for the common name such as *.organicdesign.co.nz so that it's possible for different sub-domains to access the service but get handled differently.

The certificate's are build automatically upon installation to match the servers primary domain name. To rebuild them for a different mailserver domain name, run the dovecot-cert.sh script, enter the organisation details and the mailserver domain in the Common Name field, then restart dovecot.

Spam Assassin

The spamassassin and spamc packages are required, but have now been included in the organicdesign-server package. The configuration file /etc/default/spamassassin needs to be edited and the ENABLED option set to 1. Remember to start spamassassin from init.d after enabling it.

The exim4 configuration needs to be adjusted which is in the /etc/exim4/exim4.conf.template file because our servers use the monolithic configuration method.

Add the following section after the line containing end router/800_exim4-config_maildrop

# 850: Spamcheck router spamcheck_router: no_verify check_local_user condition = "${if and { {!def:h_X-Spam-Flag:} {!eq {$received_protocol}{spam-scanned}}} {1}{0}}" driver = accept transport = spamcheck_transport

Add the following section before the line containing end transport/30_exim4-config_remote_smtp_smarthost

Note: be sure to check that the spamc binary is in the location specified, and that it is running.

# 30: Spamcheck transport spamcheck_transport: debug_print = "T: spamassassin_pipe for $local_part@$domain" driver = pipe command = /usr/sbin/exim4 -oMr spam-scanned -bS use_bsmtp transport_filter = /usr/bin/spamc home_directory = "/tmp" current_directory = "/tmp" user = Debian-exim group = Debian-exim return_fail_output message_prefix = message_suffix =

At this point, after exim4 is restarted, all messages should have extra headers added by spamassassin for example:

X-Spam-Flag: YES X-Spam-Checker-Version: SpamAssassin 3.1.7-deb (2006-10-05) on organicdesign.co.nz X-Spam-Level: *********************** X-Spam-Status: Yes, score=23.9 required=5.0 tests=NO_DNS_FOR_FROM, RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100, RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL,UNRESOLVED_TEMPLATE,URIBL_JP_SURBL,URIBL_OB_SURBL, URIBL_SBL,URIBL_SC_SURBL,URIBL_WS_SURBL autolearn=spam version=3.1.7-deb

Moving spams to another folder automatically

Mails which have been marked as spam by spamassassin can be automatically moved into a specific mail folder by using an exim filter. First you may have to edit the exim4 configuration again and uncomment the allow_filter option in the 600_exim4-config_userforward section. We haven't had to do this so far because it is currently enabled by default.

To make a new filter, create a file in your home directory called .forward and ensure it is owned by the user in question, then add a rule such as in the following example from our server which forwards the spams straight into the Trash folder.

# Exim filter if $h_X-Spam-Status: CONTAINS "Yes" or "${if def:h_X-Spam-Flag {def}{undef}}" is "def" then save $home/Maildir/.Trash/ finish endif

• Important: do not change the comment in the first line!
• Important: use \40 for spaces in target directory name (escaped octal!)
• Note: You can use the make-forwards.pl to create them automatically for all users who don't already have them

Teaching spamassassin

Spamassassin comes with a script called sa-learn which is designed to parse mail directories with the information that the messages within them are either spam or "ham" (not spam). We have a Perl script called learn-spam.pl which resides in the /var/www directory and is run each morning. It scans all our user's Inbox/Spam and Inbox/Not Spam mail folders into which users should place their incorrectly assigned spam and non-spam messages. Note: the messages are removed after being processed, so the messages in your Not Spam folder should be copies, whereas those in the Spam folder can just be moved since you won't need them.

The sa-learn man page has some good information about Spamassassin's Bayesian learning mechanism, the following snippet is from there.

Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.

You need to train with both spam and ham mails. One type of mail alone will not have any effect.

Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)

If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run "spamassassin -d" over it in advance.

Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.

To check how many spams and hams the system has processed as well as some other internal data, use sa-learn --dump magic, here's the output of our system on 24 Dec 2008 which clearly shows that we haven't been feeding it enough ham! It shows that if we feed it ham at the same rate as spam, then it would start becoming useful in about another couple of weeks :-)

# sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0 733 0 non-token data: nspam 0.000 0 22 0 non-token data: nham 0.000 0 29962 0 non-token data: ntokens 0.000 0 1228202529 0 non-token data: oldest atime 0.000 0 1230047836 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count

Here's a couple of lines of Perl which extract and print the number of spams and hams from the output shown above. I added this to our daily housekeeping script so that it appears in our recent changes each day.

{{{1}}}

Our training script, /var/www/learn-spam.pl, is run daily from the crontab and contains the following:

<perl> !/usr/bin/perl Handle false positives for (glob "/home/*/Maildir/.INBOX.Not\\ Spam/[cn]??") { s/ /\\ /; print qx "sa-learn --ham $_"; qx "rm -fr $_/*"; } Handle false negatives for (glob "/home/*/Maildir/.INBOX.Spam/[cn]??") { s/ /\\ /; print qx "sa-learn --spam $_"; qx "rm -fr $_/*"; } </perl>

To test if the script is working properly, run it manually from the shell at such a time as you know there are spams to be processed and you should see messages stating that new tokens have been learned such as in the following example output.

archive-iterator: readdir found no mail in '/home/foo/Maildir/.INBOX.Spam/cur' directory Learned tokens from 0 message(s) (0 message(s) examined) Learned tokens from 16 message(s) (16 message(s) examined) archive-iterator: readdir found no mail in '/home/bar/Maildir/.INBOX.Spam/cur' directory Learned tokens from 0 message(s) (0 message(s) examined) Learned tokens from 7 message(s) (7 message(s) examined) Learned tokens from 6 message(s) (6 message(s) examined) archive-iterator: readdir found no mail in '/home/baz/Maildir/.INBOX.Spam/new' directory Learned tokens from 0 message(s) (0 message(s) examined) Learned tokens from 2 message(s) (2 message(s) examined) archive-iterator: readdir found no mail in '/home/buz/Maildir/.INBOX.Spam/new' directory Learned tokens from 0 message(s) (0 message(s) examined)

Backing & restoring up the Bayesian database

Since the Bayesian database takes a month or so to build up to a useful state, it's important to be able to back it up in case it gets corrupted or if the server needs to be rebuilt. Such a backup can also be used in the installation process of any new server so that there's no need to go through the initial learning process for each new installation.

To back up the database use sa-learn --backup > mybackup.txt and to restore it again, first clear any existing data if there is any with sa-learn --clear and then use sa-learn --restore mybackup.txt to restore from the backup file.

The actual data is stored in ~/.spamassassin in files called bayes_tok and bayes_seen, so if there's a problem and they dissapear you can forice the location sa-learn looks for its data with the --dbpath parameter then do a backup and restore without the parameter.

Updating Spam Asassin's rules

In addition to relying on the Baysean database, it's also useful to run the sa-update script every month or so which downloads refined rules from the Spam Assassin channel. GPG keys are used to ensure that the rules are coming from the correct source, but I haven't got these working yet. Following is the command to run the update without checking the channel for authenticity since it won't replaced the current rules with the downloaded ones if this is not specified. The spamd daemon must be restarted after the rules have been updated.

sa-update --nogpg /etc/init.d/spamassassin restart

Client setup

Any standard mail client such as Thunderbird or Outlook should connect with no trouble, but our procedure doesn't yet include the generation of a valid SSL certificate, so you'll get a warning initially which you can specify to be ignored for subsequent connections. We don't recommend using any Microsoft products at all, but if you had no choice, then the Set up an IMAP account on Outlook 2007 procedure may be of some help.

Incoming mail settings (IMAP)

We use the IMAP protocol, but it is possible to use both POP3 and IMAP together for the same account because both are simply different protocols for accessing the same mail. There's probably no useful purpose to do that, but if you do then make sure the client settings for the POP setup is configured to leave the messages on the server, otherwise any messages downloaded from the inbox will be removed and will therefore not be available from the IMAP inbox folder either. Note that any messages that have been moved out of the inbox and into another folder using the IMAP protocol will no longer be available for download by the POP3 protocol because it only interacts with the inbox folder which is the root mailbox.

The settings for OrganicDesign IMAP access are: imap.organicdesign.co.nz on port 993 using SSL encryption.

Outgoing mail settings (SMTP)

To set the email client to use the organisations SMTP server, go to Account settings/Outgoing server (SMTP) and add a new one called OrganicDesign with server address smtp.organicdesign.co.nz, port 2525 and STARTTLS security. Use your IMAP login details with normal password authentication method.

You may want to keep your ISP's SMTP server as the default since it will work more quickly being more local and not requiring encryption or authentication, but you can easily make the Organic Design SMTP server the default when working in the field.

Domain setup

Reverse DNS

Any site that sends emails should have reverse DNS correctly configured. Having a reverse DNS correctly set up will help to prevent the site's mails being trashed as spam. Many mail-servers will do a reverse lookup on the sending IP address and ensure it matches the senders specified domain.

This is not done by the domain registrar, it's done by the company hosting the server (the IP address owner), sometimes they include the ability to set it in the server management interface. If not, raise a support ticket asking them if they can set up a PTR record for the server's IP pointing to the domain specified in /etc/mailname or Exim's primary_domain setting.

You can find out more about what reverse DNS is and why it's important here. To check the reverse DNS of any IP address simply do the host command as you would for a domain, e.g.

<bash>host 1.2.3.4</bash>

SPF record

A lot of mail-servers now require senders domains to have an SPF record. I think that the following is a good generic record format to use as the IP4 will fail if the server moves to a new IP address.

v=spf1 a mx ptr -all

This version means that the mail will pass as long as the senders address has a matching A and MX record for the domain it says it's from, and the reverse lookup matches one of the domains A records. The record is simply a TXT record in the domain on the @ prefix.

NZ domains

Our NZ domains are handled through WebDrive. The main setup is done from the associated domain template. Assign a primary and secondary MX record with a subdomain each. Both subdomains must explicitly exist as A records in the template.

Other domains

Our non .co.nz domains are handled through NameCheap, so I'll cover the setup for them, but it should be easy to adjust to any name hosting service. In the all host records page for your domain, go to mail settings at the bottom, and set it to "User Simplified" then click save changes.

Now scroll to the bottom of the page again and fill in the "User Simplified" form, set HOST NAME to "mail" (this setting seems to be superfluous), fill in the MAILSERVER IP and set the MX PREF to 1, then click save changes.

RoundCube (IMAP-only webmail)

Note: there is now a roundcube package available for Debian and Ubuntu via APT, but we need to test where and how exactly it installs before updating this procedure to use it.

We need to have access to our IMAP folder structures from a browser, we use the RoundCube webmail application for this purpose.

First ensure that the sub-domains which should have webmail access have a rule in the web-server configuration mapping it to the roundcube code base which can be downloaded from here and should be saved to /var/www/domains/webmail. See set up a new domain name for details on mapping a sub-domain to a web application code-base.

Next, go to your.domain/installer and follow their installation procedure. Use localhost for the server addresses (you may need to use ssl://localhost if non-ssl is denied by the server). You'll need to manually create the MySQL database:

<mysql>create database roundcubemail;</mysql>

The installer tests whether the database exists and is writable, then allows you to initialise it with a button. At this point you can also test the IMAP login and SMTP sending.

Manually populating the database

Under some (currently unknown) circumstances the database tables are not created by the web installer, so you may need to manually populate the database by manually logging in to MySQL and creating a database called roundcubemail, and then importing from the shell with the following command:

mysql -u USER -p PASS roundcubemail < /var/www/domains/webmail/SQL/mysql.initial.sql

Once all these are working, go to the root of the webmail domain and login.

Upgrading

To upgrade the Roundcube code-base, download and unpack the new version into a temporary location, change into that directory then run the update script with the location of the current installation as the parameter.

<bash>./bin/installto.sh /var/www/domains/webmail/</bash>

Apache configuration

The webmail is best forced to HTTPS, so we have some configuration in both the plain and secure virtual host containers. In the plain container we simple detect if the request domain starts with "webmail", and if so redirect the request to HTTPS. In the secure container we have the same domain-based rule but instead of redirecting, it rewrites the request to the appropriate file system location of the Roundcube code-base.

<VirtualHost *:80> ... RewriteCond %{HTTP_HOST} ^(webmail\..+)$ RewriteRule (.*) https://%1$1 [R,L] ...

<VirtualHost *:443> ... RewriteCond %{HTTP_HOST} ^webmail\. RewriteRule (.*) /webmail$1 [L] ...

Troubleshooting

One issue is that on some systems the installation procedure gives an error saying that the suhosin.session.encrypt option value is incorrect (it should be disabled) even though it says it is disabled in php.ini. The suhosin settings have their own php.ini configuration file in /etc/php5/apache2/conf.d/suhosin.ini which overrides the settings in the default php.ini. Change the line containing the suhosin.session.encrypt option in this file to disable it (and don't forget to uncomment the line by removing the leading semicolon), then restart the web server and run the Roundcube installer again.

Manual Configuration

In our configuration there is only one setting in db.inc.php which is the mysql database connection line, and there are a few settings in the main.inc.php, the default_host is "ssl://localhost", the des_key, the product_name which is "OrganicDesign Webmail" and finally the mail_domain should be set to the default domain that usernames should be under to make a valid return address.

Exim broken after apt-get upgrade

We had a major problem with exim aftyer doing an apt-get upgrade where the configuration files were incompatible with the new format. The error was something like the following:

DEBCONFsomethingDEBCONF found in exim configuration. This is most probably caused by you upgrading to exim4 4.67-3 or later without accepting the suggested conffile changes. Please read /usr/share/doc/exim4-config/NEWS.Debian.gz for 4.67-2 and 4.67-4 2007-07-03 02:23:29 Exim configuration error in line 31 of /var/lib/exim4/config.autogenerated.tmp: malformed macro definition Invalid new configfile /var/lib/exim4/config.autogenerated.tmp, not installing /var/lib/exim4/config.autogenerated.tmp to /var/lib/exim4/config.autogenerated

From the message it would seem that this problem started by deciding to keep my configuration file rather than allow it to be replaced by the new one. I did this because of all the changes shown above that I would have had to do again, but I shouldn't have been so lazy because after it broke it took far longer to figure out how to fix the problem. In the end I just had to completely purge Exim4 and re-install from scratch as follows:

apt-get -y remove --purge exim4-config exim4-daemon-heavy exim4-base mv /etc/exim4 /home/nad/ apt-get install exim4-daemon-heavy dpkg-reconfigure exim4-config

And from there go through all the instructions above to re-apply our configuration from scratch again!

Synchronising Mail Folders

nad@nad-laptop:~$ apt-cache search imapsync imapcopy - IMAP backup, copy and migration tool imapsync - IMAP synchronization, copy and migration tool

For synchoronising the data from another backed up Maildir, use apt-get install maildirsync and the following example syntax:

maildirsync -rvvv Source/Maildir/ Dest/Maildir results-info.bz2

Auto-responder / Vacation message

See the extended .forward filter here, and has been tested successfully. This method extends the existing forward filter, but unfortunately it raises an error if the .vacation.msg file isn't present, and it can only be required from within the exim configuration, not from within the filter itself, and there seems to be no condition for testing file existence.

Notes

• Exim doesn't need to be restarted for changes to the .forward or virtual.users files.
• You can also forward messages to other recipients by adding one or more deliver directives to the filter file.
• To debug filter problems, stop the Exim4 service and restart on the command line with -bd -d+filter options.
• Remember that vacation messages are only sent to the same user once per week, and deleting the .vacation.log file does not reset this, see manual for details.
• the foranyaddress $h_to: ( $thisaddress contains "$local_part@" ) rule wasn't working last time I installed this - needs checking

Setting up mailing lists

To create a simple mailing list, you must have the message_body_visible set to a reasonable number and the message_body_newlines directive set to true in /etc/exim4/exim4.conf.template (or in /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs if you're using split-file config).

message_body_newlines = true message_body_visible = 5000

Note: this number of lines should be very large if you want to be able to send attachments in the mailing list

Next create a normal email account using the adduser command, and add an entry in /etc/exim4/virtual.users for the new address. Then create a .forward file (owned by the user) in their home directory as shown in the following example:

# Exim filter if $h_subject contains "[Our List]" then seen mail from $reply_address reply_to "Our List<our-list@foo.bar>" subject $h_subject text $message_body to "Our List<our-list@foo.bar>" bcc "john@foo.bar,mary@foo.bar" extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding" else seen mail from $reply_address reply_to "Our List<our-list@foo.bar>" subject "[Our List] $h_subject" text $message_body to "Our List<our-list@foo.bar>" bcc "john@foo.bar,mary@foo.bar" extra_headers "Content-type: $h_content-type\nContent-transfer-encoding: $h_Content-transfer-encoding" endif

The name of the list and its email address is added to the reply-to field, and the list of recipients is in the bcc field. Later these directives can be added to the files automatically. The to field contains just a dummy address that should be set up to get black-holed.

We also have a MediaWiki extension called EximMailList that can populate a forward file automatically using the email addresses of the wiki users. It should be called from a cron job on the local host, and requires two settings in the LocalSettings.php file to be set, $wgEximMailListName and $wgEximMailListAddress. The crontab entry should be something like this:

5 * * * * root wget -q -O /home/foo/.forward "http://foo.com/wiki/index.php?title=MediaWiki:Common.css&action=eximfilter"

I've used the article title MediaWiki:Common.css to cater for the case of the wiki being locked down, since the CSS article will always be in the white list.

Delivering emails directly to web-sites

Often sites require the ability to respond to incoming emails, for example the site may have the ability to reply to site notifications of in-site messages. We've previously used the EmailToWiki extension for this, but the extension requires that the emails first be delivered to a mailbox and then a cron-job polls the mailbox and removes/processes messages using the IMAP protocol. This is a very unresponsive and resource consuming approach.

A new script has been made called post-email.pl which allows emails to be delivered directly to a site via HTTP instead of being delivered to a mailbox. This script is invoked via an Exim filter in the recipient email users .forward file. The filter is a single line consisting of a pipe command which sends the email content to the specified program in the STDIN stream, the URL that the extracted body text and headers will be posted to is specified as a command-line parameter to the program.

In this example filter, the post-email.pl script is expected to be in the same directory and has the following content:

# Exim filter pipe "perl $home/post-email.pl \"http://www.example.com\""

• Explicitly specifying the perl interpreter means there's no need to worry about the script's executable permissions
• The quotes are needed if the URL contains a query-string with ampersands

Troubleshooting

Stop the exim service and run again from the shell with debugging enabled,

exim4 -bd -d+filter

For more information on the command line options, see chapter 5 of the manual.

Problems sending to hotmail

Hotmail have increased their spam security and require a number of steps to be taken for smtp servers to be able to send mail to them without being blocked as spammers.

First an SPF (Sender Policy Framework) record must be added to the domains DNS record. I used the MS Sender ID wizard which gave me this following content for the record which is added as a TXT record with the @ prefix:

v=spf1 ptr ip4:69.64.87.188 -all

In addition to this a form must be filled out here to be sent to MS. They got back to me via email quite promptly and got me to fill in this form.

After reading up on Wikipedia about the Sender Policy Framework, I think that the following is probably a better record to use as the PTR is a bit confusing, and IP4 will fail if the server moves to a new IP address.

v=spf1 a mx -all

This version means that the mail will pass as long as the senders address matches at least one A or MX record for the domain.

See also

• Wiki daemon - The wiki daemon can be configured to synchronise distributed maildirs and accounts
• Mail Server Overview - in DoveCot manual
• Exim cheatsheet
• Exim4 manual
  • Exim4 configuration functions and variables
  • Exim4 configuration directives
  • Exim4 generic router options
  • The Exim4 queryprogram router - routing messages according to an external program
  • The Exim4 pipe transport
  • Adding a local scan to Exim4
  • Filter files