Difference between revisions of "Rsync"

From Organic Design wiki
m (→‎Backing up Maildirs with Rsync)
(→‎Backing up Maildirs with Rsync: two-stage rsync)
Line 25: Line 25:
 
Backing up Maildirs can be a problem with many target systems (even non-Windows ones) because many filesystems don't allow colons in file names. This problem occurs for us using the [http://www.adrive.com ADrive backup service].
 
Backing up Maildirs can be a problem with many target systems (even non-Windows ones) because many filesystems don't allow colons in file names. This problem occurs for us using the [http://www.adrive.com ADrive backup service].
  
−
We've over come this problem using a two step solution. First we install rsync from source with the [https://git.samba.org/?p=rsync-patches.git;a=blob;f=transliterate.diff;h=19d6393537903a1fb7d5581b8216b999fa82a450;hb=135a233d6f4d401c187117ae57fac147f2a863a4 transliterate patch] applied which adds a '''--tr=BAD/GOOD''' option for mapping bad characters to good ones. And then second, we use the [http://curlftpfs.sourceforge.net/ CurlFtpFS] utility which allows one to mount a remote FTP storage resource to a local mount point. This allows one to use a patched local rsync to synchronise with a remote storage facility with the colons replaced with a more suitable character such as a semicolon.
+
We've over come this problem using the [https://git.samba.org/?p=rsync-patches.git;a=blob;f=transliterate.diff;h=19d6393537903a1fb7d5581b8216b999fa82a450;hb=135a233d6f4d401c187117ae57fac147f2a863a4 transliterate patch] which adds a '''--tr=BAD/GOOD''' option for mapping bad characters to good ones.
  
 
To install the patch you need to download and unpack the latest source and the patches, then change into the source directory and do the following:
 
To install the patch you need to download and unpack the latest source and the patches, then change into the source directory and do the following:
Line 35: Line 35:
 
</bash>}}
 
</bash>}}
  
−
Next, the CurlFtpFS utility can be installed via ''apt-get'' on Debian-based systems, then a script run on from ''crontab'' which mounts the remote resource and synchronises the Maildirs. This example script synchronises all the home directories. It assumes that a directory called ''/root/adrive'' already exists.
+
You can't use this option to backup directory to the target server unless the target also has the transliterate patch installed. If it's not installed on the server you'll need to do a two-stage backup. The first to a local directory using the ''--tr'' option, and second synchronising this local directory (that has all the colons replaced) with the remote server without the ''--tr'' option.
−
{{code|<bash>
+
 
−
curlftpfs -o user=foo@bar.baz:foopass ftp://ftp.adrive.com/ /root/adrive/
 
−
rsync -av --inplace --tr=':/;' /home /root/adrive/
 
−
fusermount -u /root/adrive
 
−
</bash>}}
 
−
The first line mounts the ''adrive'' storage resource to the local directory ''/root/adrive'' using the FTP protocol. The second then performs an ''rsync'' synchronisation of the local home directories to the local mount point ensuring that all colons will be replaced semicolons. And the final line unmounts ''adrive'' from the local mount point after the ''rsync'' command has finished running. The ''--inplace'' option on the rsync command is required to prevent the fatal error that would otherwise occur when it tries to execute the ''mkstemp'' commands. It will also issue errors when trying to set the ownership of the target directories, but these are just non-fatal warnings (actually there are patches available to allow ''chown'' and ''chmod'' to function, but I haven't bothered implementing them since their absence isn't preventing it from working).
 
 
[[Category:Software]][[Category:Linux]]
 
[[Category:Software]][[Category:Linux]]

Revision as of 23:07, 16 July 2014

rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison.

rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Some features of rsync include

  • Can update whole directory trees and filesystems
  • Optionally preserves symbolic links, hard links, file ownership, permissions, devices and times
  • Requires no special privileges to install
  • Internal pipelining reduces latency for multiple files
  • Can use rsh, ssh or direct sockets as the transport
  • Supports anonymous rsync which is ideal for mirroring

Using rsync over SSH

Sometimes it's useful to do a one-off backup of a file structure from one host to another, and since all the hosts (in our system) are guaranteed to be able to connect to each other with SSH (after adding appropriate RSA keys), using rsync over SSH is a good way to do this.

The transfer syntax is then done very similarly to SCP, for example to pull new changes from a remote directory to a local one, use:

<bash>rsync -avz -e ssh remoteuser@remotehost:/remote/dir /this/dir/</bash>

After the systems are confirmed as being able to connect over SSH you may want to lock them down so that the connection between them can only be used for rsync. The IP and command can be prepended to the key in the remote hosts ~/.ssh/authorized_keys file. 

from="1.2.3.4",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB...

For more security, the command allowed can be restricted to just that specific rsync command. This can be done by manually running the rsync command with the -e'ssh -v' option which will output the exact command sent that can be used in the remote hosts authorized_keys file instead of just "rsync".

Backing up Maildirs with Rsync

Backing up Maildirs can be a problem with many target systems (even non-Windows ones) because many filesystems don't allow colons in file names. This problem occurs for us using the ADrive backup service.

We've over come this problem using the transliterate patch which adds a --tr=BAD/GOOD option for mapping bad characters to good ones.

To install the patch you need to download and unpack the latest source and the patches, then change into the source directory and do the following:

<bash>

patch -p1 <patches/transliterate.diff ./configure make make install </bash>

You can't use this option to backup directory to the target server unless the target also has the transliterate patch installed. If it's not installed on the server you'll need to do a two-stage backup. The first to a local directory using the --tr option, and second synchronising this local directory (that has all the colons replaced) with the remote server without the --tr option.

Retrieved from "https://organicdesign.nz/wiki/index.php?title=Rsync&oldid=117743"