Install a new server

Install a new server Organic Design procedure

Choose a hosting provider

First an appropriate hosting provider needs to be found, or if running a server in-house, see the Configure LAN procedure. Some possible points to check out when looking for a server hosting service apart from just the cost are:

• Ease of hardware upgrading - can you upgrade disk/memory/cpu without reinstalling the system?
• Contention rate (how many concurrent clients share the hardware if its a VPS)
• Control panel usefulness (the most important features are rebooting and virtual console access)
• OS choices available (up to date Debian or Ubuntu are most important for us)
• Historical downtime statistics
• What jurisdiction are they hosting in and what laws apply? for example can your run hidden services, i2p/tor routers or torrent daemons?
• Do they accept Bitcoin or Ripple for payment?
• What kind of data backup options do they provide?

Get reverse DNS set up

Any site that sends emails should have reverse DNS correctly configured. Having a reverse DNS correctly set up will help to prevent the site's mails being trashed as spam. Many mail-servers will do a reverse lookup on the sending IP address and ensure it matches the senders specified domain.

This is not done by the domain registrar, it's done by the company hosting the server (the IP address owner), sometimes they include the ability to set it in the server management interface. If not, raise a support ticket asking them if they can set up a PTR record for the server's IP (184.82.117.116) pointing to mathaba.com (naked domain).

You can check the reverse DNS for a domain at DNSstuff in their IP section, and you can find out more about what reverse DNS is and why it's important here.

Download and install Debian or Ubuntu

If the server has no OS then download and install Debian/Ubuntu first. Depending on the kind of access you have to the server and the kind of media it can accept, the following links may be of interest.

• Debian Conversion - Change an existing Linux distro into Debian using only SSH access
• Debian installation from memory stick - This is actually one of the most convenient means of installation even when DVD/CD are available

Setting up the OS environment

First, bring the system up to date.

apt-get update apt-get upgrade

Give the server's root account a friendly name so it looks better in the inbox when it sends mail. Do tis by replacing the name "root" in the full-name field in /etc/passwd as follows:

root:x:0:0:Organic Design server:/root:/bin/bash

Timezone

Getting the following warning plus a bunch of others whenever Perl scripts run?

perl: warning: Falling back to the standard locale ("C").

Configure the locales and tick the time zones you'd like to have available on the system, make sure that en_us_8859_1 is selected because without them it will cause warnings from Perl and some other environments.

<bash>dpkg-reconfigure locales</bash>

Some users may prefer to have their shell language set to something different than the server's default. This can be done by adding the zone to their ~/.bashrc for example,

{{{1}}}

If warnings are still being raised (particularly by Perl scripts), export the LC_ALL variable to one of the existing time zones, for example,

{{{1}}}

Security

By default the server login is the root user with a password, so the first thing I did was to set up another user for myself, add the user to /etc/sudoers with full privileges and no password requirement. Note that you need to use the sudo or visudo utility to modify, not the usual vi or nano utilities. Here we're also adding a line for the svn user so that it can perform some subversion operations as root. Both commands are executed via sudo from some of our repository's post-commit hooks. The first command is used to allow us to have auto-updating repositories on the server, and the second keeps our GitHub repositories synchronised with our local Subversion ones.

fred ALL=NOPASSWD : ALL svn ALL=NOPASSWD : /usr/bin/svn,/etc/sync2git/sync2git

Then we want to disable passwords for SSH access and use RSA keys as typing passwords is insecure. These changes were made to /etc/ssh/sshd_config:

AllowUsers fred bob sam PermitRootLogin no RSAAuthentication yes PubkeyAuthentication yes PasswordAuthentication no

And don't forget to add your public RSA key to ~/.ssh/authorized_keys (you'll probably need to create the directory since the account has just been created).

Note that we're allowing the root user to shell in but only from within the server itself. We do this so that the root user can checkout and update subversion repositories on the server which requires the svn+ssh protocol. The root user's id_rsa.pub file needs to be copied to its authorized_keys file too for this to work.

Restart the SSH server and test that you can login from another terminal window before exiting the current session. You now login as your own user, not the root user, and then use sudo bash to gain a root shell.

<bash>/etc/init.d/ssh restart</bash>

Install The Rootkit Hunter with apt-get install rkhunter and uncomment the following lines as these files are normal on Debian systems and should not be considered as attacks. Also have a look at the Debian README file with zcat /usr/share/doc/rkhunter/README.Debian.gz.

ALLOWHIDDENDIR=/dev/.udev ALLOWHIDDENDIR=/dev/.static ALLOWHIDDENDIR=/dev/.initramfs ALLOWHIDDENDIR=/dev/.mdadm ... RTKT_FILE_WHITELIST="/etc/init.d/hdparm /etc/init.d/.depend.boot" ... USER_FILEPROP_FILES_DIRS="/etc/init.d/.depend.boot"

Then run a properties update on it since we've added some custom files to the whitelist and need notification if they change, and then run a local test to see if there are any warnings.

rkhunter --propupd rkhunter -c

Installing packages

Then begin installing the necessary packages,

apt-get install p7zip-full bzip2 unzip subversion git poppler-utils host

The following if you're going to be using email on the server:

apt-get install exim4-daemon-heavy dovecot-common dovecot-imapd spamassassin spamc maildirsync

The following for servers running wikis:

apt-get install htmldoc librsvg2-bin imagemagick apt-get install php5-mysql php5-gd php5-mcrypt php5-xsl php5-curl php5-sqlite php5-imap apt-get install php5-suhosin php-apc

The following if you're running the Perl robot framework:

apt-get install perlmagick libwww-perl libnet-dns-perl libio-socket-ssl-perl libtimedate-perl apt-get install libnet-scp-expect-perl libcrypt-ssleay-perl libcrypt-cbc-perl libcrypt-blowfish-perl apt-get install libnet-xmpp-perl libnet-imap-simple-ssl-perl libphp-serialization-perl

And the following if you would like math markup support, also install the following, and see Enabling math markup for more details.

apt-get install dvipng tetex-extra cjk-latex ocaml

• Note: on some Debian 7 systems the texlive-latex-extra package is also required.
• If using Nginx, you may have problems with the texvc binary not running from shell_exec, if so see Enabling math markup#Nginx.

You will now have a functioning server and LAMP environment.

MariaDB

Either install the mysql-server package with apt-get, or go through this procedure for installing MariaDB instead which is a truly open source drop-in replacement for MySQL forked from the original by the creators.

One issue that can occur after moving server for both MySQL and MariaDB is the following error produced every day:

/etc/cron.daily/logrotate: error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log ' run-parts: /etc/cron.daily/logrotate exited with return code 1

This is due to the debian-sys-maint user not having permission to access mysqladmin to rotate the logs either due to the MySQL user missing, or having the wrong password (thanks to Lornajane for her solution in this post). Get the password from the /etc/mysql/debian.cnf configuration file and then either update the password if the user exists, or create the user with the correct password if not.

USE mysql UPDATE user SET Password = PASSWORD('**************') WHERE User = 'debian-sys-maint' && Host = 'localhost';

or

GRANT RELOAD, SHUTDOWN, PROCESS, SHOW DATABASES, SUPER, LOCK TABLES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY PASSWORD '**************';

You can check if the maintenance user has its access correctly configured with the following command:

mysqladmin --defaults-file=/etc/mysql/debian.cnf ping

Post install checklist

• /etc/hostname, hostname -F /etc/hostname, /etc/hosts
• DB info for wikia, webmail, crm
• /etc/ssh/sshd_config
• /var/www and /home structures (should be automatically maintained by adding new server as a peer)
• Exim4 (this will need to be configured even for sending mail, see Configure mail server)
• Import spamassassin bayesian rules

Scheduled tasks

We have the following scheduled tasks running in our crontab,

*/1 * * * * root perl /var/www/tools/wikid-monitor.pl > /dev/null 0 3 * * * root perl /var/www/tools/learn-spam.pl > /dev/null 0 4 * * * root perl /var/www/work/daily.pl > /dev/null 0 5 * * * root perl /var/www/work/backup.pl &> /dev/null */5 * * * * root wget -O - -q https://dev.organicdesign.co.nz --no-check-certificate > /dev/null

• The first is to keep the baysean rules up to date for our spam filter, see Configure mail server for more detail.
• The second runs our OD daily backup script which backs up our databases and site structures, users mail, our subversion repositories and server configuration files (this is private and specific to the OD server, a more generic script is backup-host.pl).
• The third is a private script that sends reminder emails about payments and invoices due.
• The forth is to keep categorisation etc up to date in our private work wiki because the site can't run jobs quickly enough from page requests being private.
• The last one runs wikid-monitor.pl to ensure that our robot framework is running and send an alert email if not.

Setting up the Wikia & Bot framework

If you are making a replica of or rebuilding a specific system, then unpack a recent www-yyyy-mm-dd.tgz backup into /var/www and remove specific wiki content.

7za x www-yyyy-mm-dd-tgz tar -xf www.tar /var

If starting a new server from scratch, then the main two things required are /var/www/tools and /var/www/extensions, the other procedures for installing codebases and wikis will add everything else necessary.

Extensions and Tools

You can obtain the scripts and extensions from the OD subversion repository, and then add any additional extensions you need. Note that there are also a number of extensions we use which are in the Wikimedia repository, so it may be easiest to unpack our od-extensions.tgz extensions snapshot instead.

cd /var/www svn co svn+ssh://USER@svn.organicdesign.co.nz/svn/extensions svn co svn+ssh://USER@svn.organicdesign.co.nz/svn/tools

Snapshots are also available as gzipped tar files, od-extensions.tgz and od-tools.tgz. After you have a wiki daemon running, the extensions and tools will be automatically synchronised to Organic Design's current tgz snapshots. The wiki daemon executes update-extensions.sh and update-tools.sh in /var/www/tools, and these can be called manually from root at any time. Any content that exists in the local extensions or tools but not in the OD version will be left alone during updates.

Next create the wikia global configuration in /var/www/tools/wikid.conf used by both the wikia and robot framework. Start with the wikid.conf.sample file.

Starting a bot

Now the the config is in place, try running the bot with the --install directive so that it starts up automatically when the system boots. If not running on a GNU/Linux machine, you may be best installing ActivePERL which comes with all the necessary libraries and is available for most platforms.

/var/www/tools/wikid.pl --install

Testing the bot

Check if the bot is running with pgrep wikid, and check the log in /var/www/tools/wikid.log. If you're running an IRC channel, check that your bot is in there and notifying the channel when articles change properly etc.

PHP

The differences to the default php.ini file in our servers are as follows (the parts after the ... need to be added to the end):

Note: this still needs updating to PHP 5.4.x settings.

max_execution_time = 300 memory_limit = 64M log_errors = On error_log = syslog post_max_size = 100M upload_max_filesize = 100M extension = domxml extension = fileinfo.so ... [suhosin] suhosin.session.encrypt = Off

Note that the suhosin settings have their own php.ini configuration file in /etc/php5/apache2/conf.d/suhosin.ini which overrides the settings in the default php.ini. Change the line containing the suhosin.session.encrypt option in this file to disable it (and don't forget to uncomment the line by removing the leading semicolon), then restart the web server.

Web server

First install Nginx and the fast-cgi module.

apt-get install nginx php5-fpm

Edit the /etc/php5/fpm/php.ini file and set cgi.fix_pathinfo to 0 to avoid this serious security problem allowing arbitrary PHP code to be executed on the server.

If using Apache do the following installation steps instead. Our main server is no longer running Apache, we changed over to Nginx on 1 July 2013.

apt-get install apache2 libapache2-svn libapache2-mod-php5 a2enmod ssl a2enmod rewrite

Our configuration is in our private work subversion repository so that we have an organised history of changes made to the configuration and a good back up of it. We use a symlink from /etc/apache2/sites-available/default to /var/www/work/apache.conf or /etc/nginx/sites-available/default to /var/www/work/nginx.conf which is the configuration file's location on the server in an automatically updating working copy.

The configuration has two main sections for SSL and non-SSL. The non SSL section includes rules for most of the sites and falls back on a set of general rules assuming the sites to be part of the OD wiki farm if no other prior rules have matched. The first two rule sets map all requests having a webmail sub-domain to the Organic Design webmail application, and all requests having an svn sub-domain to the websvn application. There are also two other virtual-host containers at the end which make the tools and extensions subversion repositories publicly readable via HTTP.

Setting up SSL

See SSL

Domain names

Adjust the names of the symlinks in the /var/www/domains directory to local domain names and ensure that those names are added to the /etc/hosts file.

• Note: If you're installing your wikia structure on a local machine, then you must ensure that your domains such as foo.localhost are set in /etc/hosts as aliases for 127.0.0.1
• DNS: if you need to set up a DNS server or Dymamic DNS system, see Configure DNS

Extracting Databases from a Backup

Extract the most recent database backup (this may overwrite existing databases of the same names)

7za x all-yyyy-mm-dd.sql.7z mysql -u root -p < all.sql mysqladmin -u root -p flush-privileges

Setting up SFTP access

The OpenSSH server comes with good SFTP support built in and allows users to be set up that have only SFTP access and can be restricted to specified sub-directories. The configuration is done from /etc/ssh/sshd_config (it must be the OpenSSH server), first enable the SFTP subsystem by un-commenting or adding the following directive:

Subsystem sftp internal-sftp

Next add a section like the following example for each user requiring access,

Match User foo ChrootDirectory /var/www ForceCommand internal-sftp X11Forwarding no AllowTcpForwarding no

You can check for problems in the /var/log/auth.log file. The most common issue will be to do with permissions. The root folder that is given access to the SFTP subsystem must be owned by root and be in the root group. It must be writable only by root, but readable by the SFTP user. The connecting clients use a path relative to the chroot directory given to them in their matching configuration section.

SFTP windows clients

Windows users can use the FileZilla FTP client to connect to the server over SFTP using key-based logins.

First you need to import your private key by going into edit/settings and then SFTP in the treeview and click the Add Key button. This will then allow you to convert the key to the windows ppk format and save it in its list.

You can then set up a new site entry using protocol SFTP, and authentication type Interactive.

Setting up FTP access

Some clients may require standard FTP access which although not very secure, can have some restrictions put on it to make it a little safer such as restricting users to their home directories and using a non standard port. We use the GPL proFTPD server in standalone mode.

apt-get install proftpd

Edit the /etc/proftpd/proftpd.conf file and change the port to something other than 21 and add the following directive to restrict users to their home directories (or set it to a shared FTP directory).

DefaultRoot ~

Following Symlinks

Note that following symlinks is not supported if the DefaultRoot directive is used because the directive creates a "jail" preventing access to any directories outside of it. Some administrators have said that mount --bind can be used to achieve this but it hasn't worked for us as that seems to just create a normal symlink as well.

Next steps

• Configure mail server
• Configure LAN - DHCP server and firewall
• Configure DNS - LAN/Server based DNS and dynamic DNS solutions
• Configure SMB - Samba file shares
• Configure VPN - Remote access to file shares and other LAN resources
• Configure IRC - Run a chat channel, your bot can publish notifications to the channel
• Install a MediaWiki code-base - OD wikia specific procedure
• Configure wiki security
• Tune a MediaWiki using Webpagetest and APC Cache

See also

• Install a new server (CentOS)
• Backup
• Hosting
• Add an IP address to a server
• Installing MySQL 5.1 on Debian Lenny
• Convert a MySQL DB from latin1 to UTF8