March 10, 2009

Content Sync For Load Balancer With Unison

This Unison program is great. It neatly handles syncing between two servers where changes are occurring on both sides. On one client site it takes about 30 seconds to compare/sync a 23GB /var/www/html/ structure with about 100 sites.

1) Install unison (http://www.cis.upenn.edu/~bcpierce/unison/). Requires “ocaml” compiler (http://caml.inria.fr/), and needs the “etags” command from emacs (so “yum install emacs”, whatever)

2) Set up a “/root/.unison/profile-name.prf” file. See example pasted below. Comment out “batch=true” and “silent=true” for initial test runs. Uncomment them for running from a cron job.

3) Run manually to test: unison profile-name

4) If you leave the logging turned on, make an /etc/logrotate.d/unison file, something like this:


/var/log/unison.log {
       notifempty
       daily
       rotate 7
       missingok
       compress
}

/var/log/unison.log {

notifempty

daily

rotate 7

missingok

compress

}

5) Set up cron job(s). Create multiple profiles if specific paths need more frequent syncing. With silent=true, the jobs should produce no output (silent=true automatically sets batch=true).

Sample profile (/root/.unison/profile-name.prf)

# Reasonable defaults
auto=true
confirmbigdeletes=true
contactquietly=true
fastcheck=true
group=true
maxthreads=20
numericids=true
owner=true
times=true

# Dump an archive listing to ~/unison.dump
#dumparchives=true

# Skip confirmation
batch=true

# Suppress output (sets batch=true)
silent=true

# Run in a loop, repeating every X seconds (sort of daemon mode)
#repeat=60

# Log all sync operations
log=true
logfile=/var/log/unison.log

# Backup deleted files
backup=Name *
backuplocation=central
backupdir=/var/www/unison-backups


# Local root
root=/var/www/html/

# Remote root (the double forward-slash between IP and remote path is correct)
root=ssh://1.2.3.4//var/www/html/

# Resolve conflicts in favor of local root
prefer=/var/www/html/


# Limit to specific relative path (can specify multiple path= params)
#path=somepath/

# Don't sync (can specify multiple ignore lines)
ignore=Path */var/cache

# Reasonable defaults

auto=true

confirmbigdeletes=true

contactquietly=true

fastcheck=true

group=true

maxthreads=20

numericids=true

owner=true

times=true

# Dump an archive listing to ~/unison.dump

#dumparchives=true

# Skip confirmation

batch=true

# Suppress output (sets batch=true)

silent=true

# Run in a loop, repeating every X seconds (sort of daemon mode)

#repeat=60

# Log all sync operations

log=true

logfile=/var/log/unison.log

# Backup deleted files

backup=Name *

backuplocation=central

backupdir=/var/www/unison-backups

# Local root

root=/var/www/html/

# Remote root (the double forward-slash between IP and remote path is correct)

root=ssh://1.2.3.4//var/www/html/

# Resolve conflicts in favor of local root

prefer=/var/www/html/

# Limit to specific relative path (can specify multiple path= params)

#path=somepath/

# Don't sync (can specify multiple ignore lines)

ignore=Path */var/cache

Have a project or a problem?

Talk with a senior engineer for practical recommendations—no obligation.

One Comment

One Response

kyle says:

August 28, 2009 at 3:08 pm

also,

if these servers require password authentication you will need to setup a private and public key between the 2 servers for ssh so your cron wont require a password. Heres how i did it…

BOTH SERVERS:

vi /etc/ssh/ssh_config

*** uncomment IdentityFile …. ***

vi /etc/ssh/sshd_config

*** uncomment AuthorizedKeysFile %h/.ssh/authorized_keys ***

/etc/init.d/ssh restart
useradd -s/bin/bash -m unison
passwd unison

SERVER01: (needs identity or private key)
chown unison:unison /home/unison
sudo -u unison ssh-keygen -t rsa

*** enter /home/unison/.ssh/id_rsa passphrase=leave blank***

ssh unison@server02 mkdir -p .ssh
cat /home/unison/.ssh/id_rsa.pub | ssh unison@server02 ‘cat >> .ssh/authorized_keys’
mv /home/unison/.ssh/id_rsa /home/unison/.ssh/identity

SERVER02: (needs authorized_keys or public key)
chmod 0700 /home/unison/.ssh
vi /home/unison/.ssh/authorized_keys

*** append text from id_rsa.pub on server01 to this file and then delete it off server01***

chown unison:unison /home/unison/.ssh/authorized_keys
chmod 0600 /home/unison/.ssh/authorized_key

*** now you can login to 02 from 01 without authentiction via… ***

ssh unison@server02 -i /home/unison/.ssh/identity

*** OR setup cron to run as user unison then you wont need the -i option since it now reads /home/unison/.ssh/identity ***

Reply