Easy backups with lsync. 1

Posted by Samuel Williams Sat, 06 Oct 2007 23:57:00 GMT

Backups are not always the easiest thing to setup and maintain, especially when you have complex requirements. rsync, a tool probably known by most unix geeks, is a convenient way to setup and administrate backups. This tool I have created, called ‘sync’ (but installed as lsync so as not to cause problems), is a simple set of python and ruby scripts to provide an additional layer to rsync, to address some additional requirements/features:

  • Mounting and unmounting local and remote filesystems, and also, mounting local filesystems on a remote machine.
  • Running the backup either on the source system, or destination system, without changing the configuration.
  • Run arbitrary script on source or destination before/after backups.
  • Sending backups to multiple destinations.
  • Using DNS CNAMEs to specify source and destination servers, and have backup clusters reconfigure automatically.

rsync provides a good foundation - it has good error reporting, performs well, widely deployed and tested, integrates with ssh, so this is used as the basis for lsync. On top of this, is a custom configuration parser written in python, a set of methods for using rsync, and Actions, which are sets of scripts which can be executed to perform various tasks.

The configuration file establishes a simple hierarchy. For example, take a look at Examples/site.conf:

[Server]
# We will resolve printserver1a and printserver1b to find which one is equivalent to printserver.
    Hostname: "printserver"
    User: "root"

[PrintA:Server]
    Hostname: "printserver1a"

[PrintB:Server]
    Hostname: "printserver1b"

[SyncConfig:Directory]
    Source: "/etc/sync/"

[PykotaConfig:Directory]
    Source: "/etc/pykota/"

[ApacheConfig:Directory]
    Source: "/etc/apache2/"

[ApacheWWW:Directory]
    Source: "/usr/apache/"

This configuration theoretically synchronises two servers, printserver1a and printeserver1b. One server will be designated a master, while all other servers are designated as destinations. The CNAME “printserver” is used to discover which server is the master (or you can specify it explicitly by using the exact hostname in the [Server] section.

There are also a list of directories. These are directories that will be synchronised on the servers - other directories are ignored.

When this configuration is executed on printserver1a, and printserver resolves to printserver1a, we will use the default method ‘push’, and establish an ssh connection with all other servers, in this case, printserver1b. We will then push out the data for the four directories listed.

The idea behind this is that you can have a cluster of machines, and use this to synchronise them. If the master server falls over, you can designate a new master using DNS CNAME record, and everything will keep on going.

To install this script:

# svn co http://svn.oriontransfer.org/Scripting/Sync
# cd Sync
# setup

This installs files to /usr/libexec/sync/ and a symlink lsync for the main executable. Because I am strange, I wrote part of this program in python, and the other part in ruby, so you’ll need to have both installed.

This script has more features, which I will cover in a future article. Please be aware that this set of scripts is currently in beta - it has been deployed and used successfully in a number of situations, but be aware it is far from complete.

Check the Examples directory for some simple configurations.