Configure SVN

Configure SVN Organic Design procedure

This procedure is used to set up our subversion repositories and configure access to them. We have read-only access for viewing our repositories via WebSVN, and we allow checking out local working copies anonymously via HTTP. For commit access we use the SVN+SSH protocol and use RSA keys to distinguish users without them requiring an account on the server and to avoid the necessity of entering passwords.

Setting up an SVN repositories

The necessary packages should already have been installed as part of the Install a new server package, but if not, you'll need to apt-get install subversion libapache2-svn. Then the first step is to create the repositories as in the following example. We create them in /var/www so that they're included in our content backups.

mkdir /var/www/svn
svnadmin create /var/www/svn/extensions
svnadmin create /var/www/svn/tools

Next create an svn user that all access to the repositories will be done through. This method allows us to have external users commit to our repositories without us having to give each of them their own account on the server. But at the same time it works well for users who do have shell access to the server and who may need to do commits from within the server. The image below shows an example of a user commit that doesn't have a corresponding account on the server (tabatha):

We then set the permissions of the repository structures so that the svn user and the web-server (user www-data on Debian-based systems) can access it, we also set the UMODE to 2 so that new items are created with the correct permissions too. We can also at this point add a number of developer users to the www-data group so that they can access the repository. Note that these developer users are just those that will be accessing the repository from within the server itself, not those that access from remote locations.

adduser svn

chown -R svn:www-data /var/www/svn
chmod -R 2770 /var/www/svn

addgroup earl www-data
addgroup tabatha www-data

The svn user will be used by all remote users to access the repositories securely. We assume here that all users already use RSA key-pairs for their shell access to the server, and that passwords are disabled in /etc/ssh/sshd_config. We also allow only specified users to have shell access to by using the ALLOW USERS directive in /etc/ssh/sshd_config, so the new svn user should be added into this directive too.

Next each of the users public parts of their RSA key-pairs should be appended to /home/svn/.ssh/authorized_keys. This file will have to be created the first time a new key is to be added and its ownership and mode set properly. I'm assuming each user only has one key in the following example:

cd /home/svn
mkdir .ssh
cp /home/earl/.ssh/authorized_keys .ssh/
cat /home/tabatha/.ssh/authorized_keys >> .ssh/authorized_keys

chown -R svn:svn .ssh
chmod -R 600 .ssh

Next, we need to lock down the svn user so that it can't be used for anything else except accessing the Subversion repositories. To do this, set the allowed command and turn off all other services that are usually available to applications using the SSH protocol. We also need to use tunnel to specify the correct name for commits from that key to be logged as instead of all being logged as the "svn" user (note that the names supplied here do not need to have any corresponding user on the server). To do this we prepend the options before each of the RSA keys in the /home/svn/.ssh/authorized_keys file:

command="/usr/bin/svnserve -t -r /var/www/svn/ --tunnel-user=tabatha",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding

Enabling anonymous read-only HTTP access

Note: this process is only for Apache, but we're now running Nginx so anonymous read-only access is only available over SSH now using our guest RSA key.

We need some of the repositories to have public anonymous access over HTTP but only for read operations, not for committing etc. The dav_svn Apache module is used to map URLs to subversion paths, and then we use the LimitExcept directive from the mod_access module to deny all but the read operations.

First we need to install the required components and enable the dav_svn module in Apache:

apt-get install libapache2-svn
a2enmod dav_svn

Then we need to add a virtual host container for the domain or sub-domain that will be used for handling the svn requests:

<VirtualHost *:80>
	ServerAdmin root@organicdesign.co.nz
	ServerName svn.organicdesign.co.nz
	<Location /svn>
		DAV svn
		SVNParentPath /svn
		<LimitExcept GET PROPFIND OPTIONS REPORT>
			Deny from all
		</LimitExcept>
	</Location>
</VirtualHost>

Note: The location used in this container must not be inside the main DOCUMENT_ROOT. In our case the repositories are stored in /var/www/svn, but the DOCUMENT_ROOT is /var/www, but we also have a symlink in /svn which I've used for the path in the DAV container.

Alternatively a container can be set up for each specific repository as follows (we have our tools and extensions repositories available for anonymous read-only access in this way). Note the difference apart from the specific repository path is the use of the SVNPath directive instead of SVNParentPath.

<VirtualHost *:80>
	ServerAdmin root@organicdesign.co.nz
	ServerName tools.organicdesign.co.nz
	<Location /svn/tools>
		DAV svn
		SVNPath /svn/tools
		<LimitExcept GET PROPFIND OPTIONS REPORT>
			Deny from all
		</LimitExcept>
	</Location>
</VirtualHost>

Clients

Now the users can all access the repository using the following example syntax which is the same for both external and in-server repositories. All users checkout the repository with the generic svn user, as their RSA key will determine the identity that their commits should be logged as.

svn co svn+ssh://{{h|svn}}@organicdesign.co.nz/tools

(Sorry this is no longer available - please use the Guest RSA key over SSH now.) For clients who only need read access to the repositories a simple anonymous HTTP request can be made to the sub-domain for the repository as in the following example:

svn co http://tools.organicdesign.co.nz/svn/tools

Note: when setting up working copies on the server, they should be done from root, but then commits from the server done from the appropriate user so that they're not logged as root.

Windows clients

If you need to set up a working copy on a Windows machine, then svn+ssh can be a bit of a problem, but basically it's just a matter of adding the location of your ssh.exe to the Subversion configuration. Tortois works out of the box with PuTTY by using the PuTTY session name in the SVN repo URL instead of the server hostname. A good tutorial covering the set up of PuTTY with RSA keys can be found here.

A couple of other links about svn+ssh for Windows are svn+ssh using the Git SSH utility (Git also comes with a ssh-keygen binary for creating your RSA key-pair) and svn+ssh using Cygwin.

Configuring WebSVN

We use the PHP WebSVN application from Tigris for viewing our repositories over the web. Installation of websvn is a simple matter of downloading into /var/www/domains/websvn and creating a sub-domain and virtual host entry for it as for any other PHP web application. Next make a copy of the default config file:

cd /var/www/domains/websvn/include
cp distconfig.php config.php

And add a single entry in the repositories section pointing at the parent directory of your repositories, for example:

$config->parentPath('/svn');

You may want to add or adjust the file extensions used for syntax highlighting, each language is an array key with an array of file-extensions as its value. For example, we wanted our wikid.conf.sample file to be highlighted as Perl, so we added the following line to add "sample" to the extensions list under the "perl" key:

$extGeshi['perl'][] = 'sample';

You can comment out all the themes in your config file except the one that matches your site best which removes the themes selection dropdown. To change the information, edit the index.tmpl file in the corresponding skins directory which are all found in the templates directory (we use the calm theme for ours).

And that's all there is to it! no simply browse to your sub-domain, ours is http://svn.organicdesign.co.nz

Making some repositories private

A simple method of making a repository private can be done by adding the following to the includes/config.php file. It requires a password to be put into the URL which will then automatically be included in subsequent URL's when clicking on links within the websvn interface. It's not very secure since the password is shown directly in the URL, but is mainly just to prevent bots and general snooping of private work, rather than to protect against high level industrial espionage ;-)

$pass = '*******************';
if( $_GET['pass'] != $pass ) {
    if( preg_match( "|pass=$pass|", $_SERVER['HTTP_REFERER'] ) ) {
        header( "location: " . $_SERVER['SCRIPT_URI'] . "?pass=$pass&" . $_SERVER['QUERY_STRING'] );
        exit;
    }
    $config->addExcludedPath('/var/www/svn/PRIVATE_REPO_1');
    $config->addExcludedPath('/var/www/svn/PRIVATE_REPO_2');
}

Including file permission information in the repository

Executable permission: Some files in our repositories need to be executable, but whenever the repo is updated, they revert to a non-executable state. To store this meta-information with the file use propset to set the executable subversion property on the file.

svn propset svn:executable true FILENAME

File ownership: Subversion doesn't support the maintenance of ownership because the users and groups are not expected to be the same across hosts. But sometimes this can be useful, for example I have my ~/.forward file in a subversion repo because I want simple access to the history and to be able to revert it easily. I link the real file to the one in the server's "live" working copy which automatically updates on commit. The problem is that Exim4 requires that .forward files be owned by the corresponding user. To achieve this you can put a chown command in the repo's post-commit hook, but be sure to also add the command to the sudoers file because the commit process is running as the svn user and won't have permission to change the file's ownership. Use the full command in the sudoers file rather than just the chmod program without any parameters to reduce security risk to a minimum.

Backing up an SVN repository

Rather than just zip up the repository directory, it's best to use the svnadmin dump utility so that it can be easily imported again later regardless of specifics of platform and version.

svnadmin dump /var/www/svn/extensions > ~/extensions-2010-01-01.svn

Later the repository can be imported using the following:

cd /svn/extensions
svnadmin create extensions
svnadmin load extensions < ~/extensions-2010-01-01.svn

Linking external repositories

The following links an external repository (in this case the OD tools repo) into another Subversion repository.

svn propset svn:externals "tools svn+ssh://svn@svn.organicdesign.co.nz/tools" ./

Automatically updating working copies

Sometimes its useful to have working copies that update automatically whenever a commit completes. If the working copy is on the same server as the repository, then it simply a matter of calling svn update from the repo's hooks/post-commit file. The svn user may not have access to the repository since it depends what user created it, so one way around this is to use sudo in the hook to update the repository, and add svn ALL=NOPASSWD: /usr/bin/svn into the sudoers file to give the svn user root access but only for running subversion.

If the working copy being updated is on another server this is more complex. I couldn't figure out a way to do it by executing a command over SSH as it wouldn't work without a pty, so I've made this process using email instead - the remote server will need to be running Exim though for this to work. The following is added to the .forward file of the user on the target server that will be used to make the udpates.

if
   $header_to matches "svnupdate@remote.domain"
then
    pipe "svn update /path/to/working/copy"
endif

This user needs to have an RSA key with an appropriate entry in the svn user's authorized_keys file on the OD server so it has permission to checkout and update working copies. The working copy being updated must be checked out by this user, and the user may not have shell access to the server, so if it doesn't you can check it out as the user using sudo, for exmaple:

And then the repository can be checked out in the dev site. This has to be done by root acting as the znazza user since the znazza user has no shell access.

sudo -u USER svn co svn+ssh://svn@organicdesign.co.nz/repo/foo /path/to/working/copy

This stage can then be tested by making a test commit and then sending an email to svnupdate@remote.domain and ensuring that the working copy updates properly. Then finally the commit hook needs to be added to send the email automatically, so the following line can be added to the work/post-commit hook on the OD server:

mail -s svnupdate svnupdate@remote.domain < /dev/null

Making a mirror of a repository on Github

Eventually we want to move our code repositories over to git but a full migration is too difficult to take on at the moment. Then after the recent change of the web-server from Apache to Nginx we could no longer offer a simple anonymous checking out of our code over HTTP using WebDAV so I thought a good interim solution would be to have a read-only mirror of our public repositories available on Github for easy access.

I found this article by Daniel Pocock which describes my situation exactly and the author has written a script which looks like it'll migrate the data and keep it up to date with a cron job. First I set up an empty repository on at Github and added the server's public key so it has the ability to commit to the repository. Then I went through Daniel's instructions to create the necessary directories and profile file and I pulled his code into /etc/sync2git:

git clone git://github.com/dpocock/sync2git.git

I added an authors.txt with his script and updated the users email addresses, then attempted to run it. The first issue was I needed to install the git-svn package which was fine, but then it gave me further errors which I finally figured out was due to the script not being set up to handle simple repositories with no branches. I was able to get it working by commenting out some lines in the last active block of the script as follows:

cd "${GIT_BARE}"
#  git branch -m trunk master
   git for-each-ref --format='%(refname)' refs/heads/tags | \
      cut -d / -f 4 | \
      while read ref;
      do
         git tag "$ref" "refs/heads/tags/$ref"
         git branch -D "tags/$ref"
      done
   git remote add origin "${GIT_REPO}"
#  git config branch.master.remote origin
#  git config branch.master.merge refs/heads/master
#  git push --tags origin master
   git push --all

I also had to change the SVN_LAYOUT settings from the default to just an empty string which can be done by adding an entry to your project's config file. My one for our extensions repo looks like this:

SVN_REPO = svn+ssh://svn@organicdesign.co.nz/extensions
GIT_REPO = git@github.com:OrganicDesign/extensions.git
SVN_LAYOUT = ""

The script can then be called regularly as a cron job or attached to the post-commit hook of your Subversion repositories. I did the latter using the same mechanism as described above for automatically updating repositories which required the /etc/sync2git/sync2git script to be added to the list of commands available to the svn user in the /etc/sudoers file.

Troubleshooting

Attempt to write a readonly database

For some reason the repo's db/rep-cache.db has got the wrong permissions, it needs to be rwx by the group not just the user.

See also

• Subversion - local article about Subversion
• Subversion access over SSH on Debian - the article I based this new method on