Difference between revisions of "Backup"

From Organic Design wiki
(→‎Simple backup: add FS backup weekly)
Line 7: Line 7:
 
== Simple backup ==
 
== Simple backup ==
 
The following Perl script is a simple backup solution to backup all databases in a local MySQL server, and save them to a ''7zip'' compressed file named according to the date. It also logs the backup and it's file size in [[Server log]] using [[MW:Extension:Simple Forms|SimpleForms]]. It can be executed periodically from the crontab.
 
The following Perl script is a simple backup solution to backup all databases in a local MySQL server, and save them to a ''7zip'' compressed file named according to the date. It also logs the backup and it's file size in [[Server log]] using [[MW:Extension:Simple Forms|SimpleForms]]. It can be executed periodically from the crontab.
−
{{code|<perl>
+
{{code|<perl>{{:backup.pl}}</perl>}}
−
#!/usr/bin/perl
 
−
use POSIX qw(strftime setsid);
 
−
use HTTP::Request;
 
−
use LWP::UserAgent;
 
−
 
 
−
$dir  = '/home/nad';
 
−
$date = strftime('%Y-%m-%d',localtime);
 
−
$ua  = LWP::UserAgent->new(cookie_jar => {},agent => 'Mozilla/5.0',timeout => 10,max_size => 1024);
 
−
 
 
−
# Return size of passed file in MB
 
−
sub size { return (int([stat shift]->[7]/104857.6+0.5)/10).'MB'; }
 
−
 
 
−
# Post a comment to the wiki's server-log article
 
−
sub comment {
 
−
        $comment = shift;
 
−
        %data = (title => 'Server log',content => "*$comment",summary => $comment,caction => 'append');
 
−
        $post = $ua->post('http://www.organicdesign.co.nz/wiki/index.php',\%data);
 
−
        }
 
−
 
 
−
# Backup and compress databases
 
−
$s7z = "all-$date.sql.7z";
 
−
$sql = "$dir/all.sql";
 
−
qx( mysqldump -u root --password='**********' -A >$sql );
 
−
qx( 7za a $dir/$s7z $sql );
 
−
comment "DB backup: $s7z (".size($sql)."/".size("$dir/$s7z").")";
 
−
 
 
−
# Backup and compress file structure (weekly)
 
−
if ($date =~ /[0-9]+-[0-9]+-(01|09|16|24)/) {
 
−
        $t7z = "www-$date.t7z";
 
−
        $tar = "$dir/www.tar";
 
−
        qx( tar -cf $tar /var/www );
 
−
        qx( 7za a $dir/$t7z $tar );
 
−
        comment "FS backup: $t7z (".size($tar)."/".size("$dir/$t7z").")";
 
−
        }
 
−
</perl>}}
 
  
 
== Up to date backup ==
 
== Up to date backup ==

Revision as of 11:28, 16 September 2007


Organic Design 2 | OD/Wikia | Wikia.php | Templates | Backup | Extensions

Backups in OD2

The organic design backups are created daily by User:Bender, compressed to .t7z and distributed over SSH to various other domains. After the upgrade to MediaWiki1.9.3, we are changing the way that wikid.pl works to integrate with MediaWiki as a proper extension on a per-wiki basis rather than a per-server basis. The backup system of OD2 is designed to stay much more up-to-date and to allow simple restoration of backups into functioning wiki's again. This same mechanism will be used to create template-wiki's.

Simple backup

The following Perl script is a simple backup solution to backup all databases in a local MySQL server, and save them to a 7zip compressed file named according to the date. It also logs the backup and it's file size in Server log using SimpleForms. It can be executed periodically from the crontab.

<perl>Backup.pl</perl>

Up to date backup

MySQL 5.0 offers a new thing called replication which allows many slave databases to be kept syncronised with a single master. Even running the slaves locally is of benefit because regular backup and distribution can be made from the slaves so that the master never needs to be locked.

Backup related news items

Backup list

The following list of backup files are currently being managed:

  • od-wiki-db
  • od-server-image
  • od-access-log
  • peer-logs
  • peerd-win32
  • nad-org
  • nad-kb
  • zenia-org
  • zenia-docs-settings
  • fowin-dunkley
  • fowin-luck

About LZMA

LZMA is an extremely good compression method which compresses our backups to about one third of the size of the gzip or bzip. I have tested it with the free 7-zip file manager from www.7-zip.org and od-wiki-db-2006-11-20 is 268MB uncompressed, 54.9MB gzipped and only 21.7MB as a 7z! But I'm unable to get the Debian port to work due to dependency issues with low level C libraries that I don't want to mess with.

  • I've found a standalone version at http://sourceforge.net/projects/p7zip and that's compressed it to 24.8MB, not quite as small as the windows one, but still very good.
  • Using switches -t7z -m0=lzma -mx=9 has got it down to 21.1MB - slightly smaller than the windows version :-)

Statistics

7Zip is extremely good at compressing wiki data compared to other algorthims, perhaps due to compressing the history more efficiently, here's a size comparison for compressing a server image which is a standard linux file structure containing no database or web site content.

Compressionserver imagewiki backup
none517MB269MB
7z122MB (76%)21.1MB (92%)
RAR140MB (72%)24.9MB (90%)
Bzip2176MB (66%)38.0MB (86%)
Gzip197MB (62%)54.5MB (80%)
Zip197MB (62%)54.5MB (80%)

See also

Retrieved from "https://organicdesign.nz/wiki/index.php?title=Backup&oldid=76484"