rsync
rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison.
rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.
Some features of rsync include
- Can update whole directory trees and filesystems
- Optionally preserves symbolic links, hard links, file ownership, permissions, devices and times
- Requires no special privileges to install
- Internal pipelining reduces latency for multiple files
- Can use rsh, ssh or direct sockets as the transport
- Supports anonymous rsync which is ideal for mirroring
Using rsync over SSH
Sometimes it's useful to do a one-off backup of a file structure from one host to another, and since all the hosts (in our system) are guaranteed to be able to connect to each other with SSH (after adding appropriate RSA keys), using rsync over SSH is a good way to do this.
The transfer syntax is then done very similarly to SCP, for example to pull new changes from a remote directory to a local one, use:
After the systems are confirmed as being able to connect over SSH you may want to lock them down so that the connection between them can only be used for rsync. The IP and command can be prepended to the key in the remote hosts ~/.ssh/authorized_keys file.
For more security, the command allowed can be restricted to just that specific rsync command. This can be done by manually running the rsync command with the -e'ssh -v' option which will output the exact command sent that can be used in the remote hosts authorized_keys file instead of just "rsync".
Backing up a server with Rsync
Rsync is a very useful tool for backing up and synchronising data, here's a simple way to transfer everything from one server to a location on a remote server.
NOTE: there is a problem when trying to backup Maildirs like this if the target machine is Windows (like the ADrive service) because the file names contain colons. This has been addressed on the form of the transliterate path which adds a --tr=BAD/GOOD option for mapping bad characters to good ones. To install the patch you need to download and unpack the latest source and the patches, then change into the source directory and do the following:
This still doesn't completely solve the problem though because ADrive would need to have implemented this patch as well since it too will be running with the same command-line options. One potential way to deal with this could be CurlFtpFS which allows one to mount a remote FTP resource as a local filesystem, and is available in the standard APT repositories. This means that the Maildir could be synchronised to the locally mounted FTP resource using the --tr option.
