Jul 212012
 

The rsync utility has a lot of great uses, such as keeping servers in sync with one another, but sometimes you want it to run unattended. So how do you securely and automatically sync between servers?

Because rsync and ssh work so well together, it’s easy to set up an encrypted, rsync-only link. There are a few different pieces that have to be put in place, so be sure to follow along closely.

First, understand that the method I’m describing is a “pull” rather than a “push”. The backup server connects to the production server to pull the most recent copies of the files. I prefer this method because it reduces the attack surface of the backup servers – they don’t need to accept ssh or rsync connections. They only need to connect to ssh on the production servers. The production servers are also well protected from outside/malicious ssh connections, as you’ll see below.

Production Server Configuration

On the production server, you need to ensure the backup user has permissions to read all the files you wish to sync. This is probably best done by adding the user to a group which has such permissions. To protect the server, we install a script which refuses any SSH connections except for rsync. Log in as the backup user and execute the following:

mkdir -p ~/.ssh
cd ~/.ssh
wget http://www.6by9.net/wp-content/uploads/2012/07/automatic_rsync__validate.txt -O validate-rsync
chmod 700 validate-rsync

Backup Server Configuration

On the backup server, you will use the same username to create the proper host authorization files in that user’s ~/.ssh/ directory:

mkdir -p ~/.ssh
cd ~/.ssh
ssh-keygen -t rsa -N "" -f rsync-key
echo 'from="192.168.1.1",command="'`echo -n ~`'/.ssh/validate-rsync"' `cat rsync-key.pub` | ssh `whoami`@production-server "cat - >> ~/.ssh/authorized_keys"

In the commands above, you need to replace production-server with the name of your production server. The IP address 192.168.1.1 must be replaced by the IP address of the backup server.

To schedule a periodic rsync of the files, cron is most useful. Almost all Linux distributions provide script directories in /etc/cron.* to run scripts hourly, daily, weekly, etc. Log in as root, choose the frequency of backups you prefer and then install the script:

cd /etc/cron.weekly
wget http://www.6by9.net/wp-content/uploads/2012/07/automatic_rsync__production-server.txt -O rsync-production-server
chmod 755 rsync-production-server

Edit the rsync-production-server script and specify the proper values for user, production-server, production-directory and backup-directory. Remember that rsync differentiates between directories that are specified with or without the trailing /.

Conclusion

With these steps complete, try manually running the rsync-production-server script. Remember that it won’t be making any changes on the production server; it will only be syncing the state of the backup server to that of the production server.

If you have any issues, you should add -v flags to the rsync and ssh calls inside the rsync-production-server script. It may also be helpful to examine the sshd logs on the production server to see if the connections are getting through.

If the script runs successfully you will see no output. Cron will automatically run the script from now on.

Multiple production servers may be backed up in a similar fashion. There is no need to generate multiple rsync-key ssh authorization keys, but you will need to create a unique cron backup script for each server.

Download Scripts

rsync backup cron script
rsync ssh login validation script

  2 Responses to “Secure, Automated rsync Between Servers”

Comments (2)
  1. Rsync is a great utility and most linux admins should have abandoned scp a long time ago. However, things get complicated when rsync’ing behind a firewall. For instance, let’s say you have a data center located in Virginia that is managed by a third party organization, i.e. internap and you need a daily dump of a database. For this type of network, most of the time a single ‘admin’ server with port 22 is left open to a narrow range of IPs. The ‘admin’ server has access to the rest of the production network. So now, how do we rsync from your office computer through the ‘admin’ server and onto the production DB server?

    rsync -avz -e "ssh user@remotehost.com ssh" user@firewalledserver:/home/user/rsync-backup/mysqldump.sql /home/backup/
    

    Rsync is neat, using -e you can pass through remote commands to execute.

    • Thanks, Jon, that’s a good tip! I’ve also used -e to pass through non-standard SSH ports:
      rsync -avz -e "ssh -p 1024" user@server:/home/user/ /home/backup/

      Interested people should read through the man page, as there are plenty of other tricks. I’ve used --bwlimit to keep bandwidth usage low.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(to be displayed above your comment)

(for moderation; your e-mail address will not be made public)