Difference between revisions of "Backup"
(backup plans) |
|||
| Line 1: | Line 1: | ||
| − | + | The organic design backups are created dailyand compressed to [[w:7zip|7zip]] and distributed over SCP to various other domains. We're in the process of re-organising this process to include all the workstations as well. In general the idea is to connect all our workstation and server contents and state in to a distributed file system automatically behind the scenes. This distributed file system would actually mirror all the content into a number of different environments such as: | |
| − | + | *File structures running on the peers and their media | |
| − | + | *Existing P2P spaces using torrents | |
| − | + | *FTP and other cheap online storage services | |
| − | The organic design backups are created | ||
== Simple workstation backup == | == Simple workstation backup == | ||
| Line 13: | Line 12: | ||
{{code|<perl>{{:backup.pl}}</perl>}} | {{code|<perl>{{:backup.pl}}</perl>}} | ||
| − | == | + | == MySQL Replication == |
MySQL 5.0 offers a new thing called [http://dev.mysql.com/doc/refman/5.0/en/replication-implementation.html replication] which allows many ''slave'' databases to be kept syncronised with a single ''master''. Even running the slaves locally is of benefit because regular backup and distribution can be made from the slaves so that the master never needs to be locked. | MySQL 5.0 offers a new thing called [http://dev.mysql.com/doc/refman/5.0/en/replication-implementation.html replication] which allows many ''slave'' databases to be kept syncronised with a single ''master''. Even running the slaves locally is of benefit because regular backup and distribution can be made from the slaves so that the master never needs to be locked. | ||
| Line 19: | Line 18: | ||
*[[20 November 2006]] ''- regarding changeover to LZMA (.t7z)'' | *[[20 November 2006]] ''- regarding changeover to LZMA (.t7z)'' | ||
*[[23 November 2006]] ''- 7z server backups which can be unpacked the same as our template in [[Debian Conversion]]'' | *[[23 November 2006]] ''- 7z server backups which can be unpacked the same as our template in [[Debian Conversion]]'' | ||
| − | *[{{ | + | *[{{localurl:20 December 2006|oldid=44019}} 20 December 2006] ''- backup corruption in transfer'' |
== Backup list == | == Backup list == | ||
| Line 40: | Line 39: | ||
*Using switches <tt>-t7z -m0=lzma -mx=9</tt> has got it down to 21.1MB - slightly smaller than the windows version :-) | *Using switches <tt>-t7z -m0=lzma -mx=9</tt> has got it down to 21.1MB - slightly smaller than the windows version :-) | ||
| − | == Statistics == | + | === Statistics === |
7Zip is extremely good at compressing wiki data compared to other algorthims, perhaps due to compressing the history more efficiently, here's a size comparison for compressing a server image which is a standard linux file structure containing no database or web site content. | 7Zip is extremely good at compressing wiki data compared to other algorthims, perhaps due to compressing the history more efficiently, here's a size comparison for compressing a server image which is a standard linux file structure containing no database or web site content. | ||
<table border cellspacing=0 cellpadding=5> | <table border cellspacing=0 cellpadding=5> | ||
| Line 57: | Line 56: | ||
*[[Templates]] | *[[Templates]] | ||
*[http://download.wikimedia.org/enwiki Wikipedia backups] | *[http://download.wikimedia.org/enwiki Wikipedia backups] | ||
| + | [[Category:IT Support]][[Category:Nad]][[Category:Organic Design]] | ||
Revision as of 01:20, 3 August 2008
The organic design backups are created dailyand compressed to 7zip and distributed over SCP to various other domains. We're in the process of re-organising this process to include all the workstations as well. In general the idea is to connect all our workstation and server contents and state in to a distributed file system automatically behind the scenes. This distributed file system would actually mirror all the content into a number of different environments such as:
- File structures running on the peers and their media
- Existing P2P spaces using torrents
- FTP and other cheap online storage services
Contents
Simple workstation backup
The following backup-workstation.pl Perl script is a simple backup solution for a local workstation which compresses and 7-zip's a list of directory trees.
Simple server backup
The following backup.pl Perl script is a simple backup solution to backup both a directory tree and all databases in a local MySQL server, and save them to a 7-zip compressed file named according to the date. It also logs the backup and it's file size in Server log using SimpleForms. It can be executed periodically from the crontab.
MySQL Replication
MySQL 5.0 offers a new thing called replication which allows many slave databases to be kept syncronised with a single master. Even running the slaves locally is of benefit because regular backup and distribution can be made from the slaves so that the master never needs to be locked.
- 20 November 2006 - regarding changeover to LZMA (.t7z)
- 23 November 2006 - 7z server backups which can be unpacked the same as our template in Debian Conversion
- [/wiki/index.php?title=20_December_2006&oldid=44019 20 December 2006] - backup corruption in transfer
Backup list
The following list of backup files are currently being managed:
- od-wiki-db
- od-server-image
- od-access-log
- peer-logs
- peerd-win32
- nad-org
- nad-kb
- zenia-org
- zenia-docs-settings
- fowin-dunkley
- fowin-luck
About LZMA
LZMA is an extremely good compression method which compresses our backups to about one third of the size of the gzip or bzip. I have tested it with the free 7-zip file manager from www.7-zip.org and od-wiki-db-2006-11-20 is 268MB uncompressed, 54.9MB gzipped and only 21.7MB as a 7z! But I'm unable to get the Debian port to work due to dependency issues with low level C libraries that I don't want to mess with.
- I've found a standalone version at http://sourceforge.net/projects/p7zip and that's compressed it to 24.8MB, not quite as small as the windows one, but still very good.
- Using switches -t7z -m0=lzma -mx=9 has got it down to 21.1MB - slightly smaller than the windows version :-)
Statistics
7Zip is extremely good at compressing wiki data compared to other algorthims, perhaps due to compressing the history more efficiently, here's a size comparison for compressing a server image which is a standard linux file structure containing no database or web site content.
| Compression | server image | wiki backup |
|---|---|---|
| none | 517MB | 269MB |
| 7z | 122MB (76%) | 21.1MB (92%) |
| RAR | 140MB (72%) | 24.9MB (90%) |
| Bzip2 | 176MB (66%) | 38.0MB (86%) |
| Gzip | 197MB (62%) | 54.5MB (80%) |
| Zip | 197MB (62%) | 54.5MB (80%) |
