Difference between revisions of "Backup"

From Organic Design wiki
m (→‎Setting up automated backups over SCP)
(→‎Setting up automated backups over SCP: perms)
Line 7: Line 7:
 
qx( mysqldump -u DB-USER --password='DB-PASS' --default-character-set=latin1 -A >$sql );
 
qx( mysqldump -u DB-USER --password='DB-PASS' --default-character-set=latin1 -A >$sql );
 
qx( 7za a $dir/$s7z $sql );
 
qx( 7za a $dir/$s7z $sql );
−
qx( chmod 644 $dir/$s7z );
+
qx( chmod 640 $dir/$s7z );
  
 
qx( scp $tar.7z scp@REMOTE-SERVER-ADDR );
 
qx( scp $tar.7z scp@REMOTE-SERVER-ADDR );
Line 13: Line 13:
 
<small>'''Note:''' we're using the '''latin1''' character set because of issues with MySQL character-encoding, see [[manually backup a wiki]] for more detail.</small>
 
<small>'''Note:''' we're using the '''latin1''' character set because of issues with MySQL character-encoding, see [[manually backup a wiki]] for more detail.</small>
  
 +
The backups are stored locally in the '''/backup''' directory. If a user needs to read these manually with SCP, then the mode of this directory should be '''770''' and the user should be added to the ''root'' group. The directory and all the backups are owned by ''root'' and in the ''root'' group.
 +
{{code|<bash>addgroup fred root
 +
chmod 770 /backup</bash>}}
  
 
This script connects to the remote server using a user called "scp" which needs to be setup on the remote server and added to the "Allow Users" directive in '''/etc/ssh/sshd_config''' if you use that like we do to restrict only the specified users to have shell access to the server.
 
This script connects to the remote server using a user called "scp" which needs to be setup on the remote server and added to the "Allow Users" directive in '''/etc/ssh/sshd_config''' if you use that like we do to restrict only the specified users to have shell access to the server.

Revision as of 23:53, 8 August 2012

The organic design backups are created daily and compressed to 7zip and distributed over Secure Copy Protocol (SCP) to various other domains.

Setting up automated backups over SCP

We use a general backup script to backup wikis on the various servers we administer which is in our Subversion tools repository here. The script dumps databases and compresses them to 7zip, then sends them over SCP to another domain if format can be done in Perl as follows:

{{{1}}}

Note: we're using the latin1 character set because of issues with MySQL character-encoding, see manually backup a wiki for more detail.

The backups are stored locally in the /backup directory. If a user needs to read these manually with SCP, then the mode of this directory should be 770 and the user should be added to the root group. The directory and all the backups are owned by root and in the root group.

<bash>addgroup fred root

chmod 770 /backup</bash>

This script connects to the remote server using a user called "scp" which needs to be setup on the remote server and added to the "Allow Users" directive in /etc/ssh/sshd_config if you use that like we do to restrict only the specified users to have shell access to the server.

Also, the script can't enter passwords, so an RSA key-pair needs to be created with ssh-keygen -t RSA (it's a good practice to disable password logins completely in /etc/ssh/sshd_config). The private part goes into /root/.ssh/ on the local server running the backup backed up (or the .ssh directory of whatever user will be running the script - we run it from the crontab as root). The public part of the key-pair goes into /home/scp/authorized_keys on the remote server.

One more security precaution is to lock the scp user down so that it can't be used for anything else except transferring files from the backup server into the /home/scp directory. To do this, we restrict the command that the user can execute, and turn off all the other SSH services that are usually available to applications using the SSH protocol. To do this prepend the following before the RSA public key in /home/scp/authorized_keys. 

command="scp -t /home/scp",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding

About LZMA

LZMA is an extremely good compression method which compresses our backups to about one third of the size of the gzip or bzip. I have tested it with the free 7-zip file manager from www.7-zip.org and od-wiki-db-2006-11-20 is 268MB uncompressed, 54.9MB gzipped and only 21.7MB as a 7z! But I'm unable to get the Debian port to work due to dependency issues with low level C libraries that I don't want to mess with.

  • I've found a standalone version at http://sourceforge.net/projects/p7zip and that's compressed it to 24.8MB, not quite as small as the windows one, but still very good.
  • Using switches -t7z -m0=lzma -mx=9 has got it down to 21.1MB - slightly smaller than the windows version :-)

Statistics

7Zip is extremely good at compressing wiki data compared to other algorithms, perhaps due to compressing the history more efficiently, here's a size comparison for compressing a server image which is a standard linux file structure containing no database or web site content.

Compressionserver imagewiki backup
none517MB269MB
7z122MB (76%)21.1MB (92%)
RAR140MB (72%)24.9MB (90%)
Bzip2176MB (66%)38.0MB (86%)
Gzip197MB (62%)54.5MB (80%)
Zip197MB (62%)54.5MB (80%)

See also

Retrieved from "https://organicdesign.nz/wiki/index.php?title=Backup&oldid=110708"