Mirror Your Web Site With rsync - Page 2

Submitted by falko (Contact Author) (Forums) on Thu, 2006-04-20 15:08. ::

5 Configure server1.example.com

Now log in through SSH on server1.example.com as someuser (not root!) and do this:

server1:

(Please do this as someuser!)

mkdir ~/.ssh
chmod 700 ~/.ssh
mv ~/mirror-rsync-key.pub ~/.ssh/
cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys
cat mirror-rsync-key.pub >> authorized_keys

By doing this, we have appended the contents of mirror-rsync-key.pub to the file /home/someuser/.ssh/authorized_keys. /home/someuser/.ssh/authorized_keys should look similar to this:

server1:

(Still as someuser!)

vi /home/someuser/.ssh/authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@
mirror

Now we want to allow connections only from mirror.example.com, and the connecting user should be allowed to use only rsync, so we add

command="/home/someuser/rsync/checkrsync",from="mirror.example.com",no-port-forwarding,no-X11-forwarding,no-pty

right at the beginning of /home/someuser/.ssh/authorized_keys:

server1:

(Still as someuser!)

vi /home/someuser/.ssh/authorized_keys

command="/home/someuser/rsync/checkrsync",from="mirror.example.com",no-port-forwarding,no-X11-forwarding,no-pty ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@
mirror

It is important that you use a FQDN like mirror.example.com instead of an IP address after from=, otherwise the automated mirroring will not work!

Now we create the script /home/someuser/rsync/checkrsync that rejects all commands except rsync.

server1:

(We still do this as someuser!)

mkdir ~/rsync
vi ~/rsync/checkrsync

#!/bin/sh

case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac

chmod 700 ~/rsync/checkrsync


6 Test rsync On mirror.example.com

Now we must test on mirror.example.com if we can mirror server1.example.com without being prompted for someuser's password. We do this:

mirror:

(We do this as root!)

rsync -avz --delete --exclude=**/stats --exclude=**/error --exclude=**/files/pictures -e "ssh -i /root/rsync/mirror-rsync-key" someuser@server1.example.com:/var/www/ /var/www/

(The --delete option means that files that have been deleted on server1.example.com should also be deleted on mirror.example.com. The --exclude option means that these files/directories should not be mirrored; e.g. --exclude=**/error means "do not mirror /var/www/error". You can use multiple --exclude options. I have listed these options as examples; you can adjust the command to your needs. Have a look at

man rsync

for more information.)

You should now see that the mirroring takes place:

receiving file list ... done

sent 71 bytes received 643 bytes 476.00 bytes/sec
total size is 64657 speedup is 90.56

without being prompted for a password! This is what we wanted.


7 Create A Cron Job

We want to automate the mirroring, that is why we create a cron job for it on mirror.example.com. Run crontab -e as root:

mirror:

(We do this as root!)

crontab -e

and create a cron job like this:

*/5 * * * * /usr/bin/rsync -azq --delete --exclude=**/stats --exclude=**/error --exclude=**/files/pictures -e "ssh -i /root/rsync/mirror-rsync-key" someuser@server1.example.com:/var/www/ /var/www/ 

This would run rsync every 5 minutes; adjust it to your needs (see

man 5 crontab

). I use the full path to rsync here (/usr/bin/rsync) just to go sure that cron knows where to find rsync. Your rsync location might differ. Run

mirror:

(We do this as root!)

which rsync

to find out where yours is.


8 Links


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Please do not use the comment function to ask for help! If you need help, please use our forum: http://www.howtoforge.com/forums
Comments will be published after administrator approval.
Submitted by vanthevirus (Contact Author) (Forums) on Wed, 2008-04-16 19:14.

i have been spending several hours to make this thing automatic but it keeps asking for password.

but finally i found the reason of my failure. I forgot a COMA!

"... no-port-forwarding,no-X11-forwarding,no-pty, ssh-dss AAAAB3NzaC1kc3MAAAEBALGZJ34a5QwC2 .... " 

 please update your tutorial for the sake of other newbies out there...

anyway, this howto is very helpful. thanks 

Submitted by trigar (Contact Author) (Forums) on Mon, 2008-08-04 17:42.

What finally worked for me was a new-line instead of a comma.  Also, if you use nano - be careful to switch off long-line wrapping (M-L).

Submitted by amcorona (Contact Author) (Forums) on Tue, 2007-08-28 15:24.

I am not sure how important this is but you are using archive mode which preserves file ownership.

You should have the same accounts on both servers if you have some directories on the source server that are owned by different accounts.  I have a file repository that only one specific user has write access to (not apache).  I have to create that owner on the target machine before running this script.  One thing I am not sure about is if the UID has to be identical

 

 

Submitted by Anonymous (Contact Author) (Forums) on Mon, 2006-04-24 00:09.
According to the suggested setup if something goes wrong to the contents of the server1.example.com:/var/www/ the mess would be propagated to the mirror server too. On the other hand, as you say you are offering just a plain rsync over ssh mirroring solution not a bullet proof backup solution..

Submitted by Anonymous (Contact Author) (Forums) on Tue, 2006-06-13 02:32.

If things work up until the last rsync using the public/private key pair, and you're having problems, use the ssh -v switch:


rsync -avz --delete --exclude=**/stats --exclude=**/error --exclude=**/files/pictures -e "ssh -v -i /root/rsync/mirror-rsync-key" someuser@server1.example.com:/var/www/ /var/www/

Submitted by Anonymous (Contact Author) (Forums) on Mon, 2006-05-22 22:31.
I understand Rsync has a windows plugin. Does anyone know how to rsync a /Inetpub/ directory to a linux server /backup/ folder?
Submitted by Anonymous (Contact Author) (Forums) on Thu, 2006-04-27 09:07.
Well, if you run rsync like that then doing incremental backups isn't all that difficult again. This here was the base for my altered script:
http://www.mikerubel.org/computers/rsync_snapshots/
It uses hardlinks. Well, I run the thing as root because I want to keep permissions. Here's my backup.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH


# Make MySQL Backups
#!/bin/bash
# Remove old files
rm -f /mysql_backup/*

#Dump new files
USER=root
PASSWORD=************
HOST=localhost

for i in $(echo 'SHOW DATABASES;' | mysql -u$USER -p$PASSWORD -h$HOST|grep -v '^Database$'); do
mysqldump \
-u$USER -p$PASSWORD -h$HOST \
-Q -c -C --add-drop-table --add-locks --quick --lock-tables \
$i > /mysql_backup/$i.sql;
done;


# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/hourly.3 ] ; then\
$RM -Rf $SNAPSHOT_RW//hourly.3 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/hourly.2 ] ;then\
$MV $SNAPSHOT_RW/hourly.2 $SNAPSHOT_RW/hourly.3 ;\
fi;

if [ -d $SNAPSHOT_RW/hourly.1 ] ; then\
$MV $SNAPSHOT_RW/hourly.1 $SNAPSHOT_RW/hourly.2 ;\
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/hourly.0 ] ; then\
$CP -al $SNAPSHOT_RW/hourly.0 $SNAPSHOT_RW/hourly.1 ;\
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!

$RSYNC\
-avz --delete --delete-excluded \
--exclude-from="$EXCLUDES"\
-e "$SSH -i $KEY" \
root@www.server.com:/ $SNAPSHOT_RW/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/hourly.0 ;


I run this script 4 times daily through cron. Then I have another script which makes daily snapshots for 7 days (backup_daily.sh):

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/daily.6 ] ; then\
$RM -Rf $SNAPSHOT_RW/daily.6 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/daily.5 ] ; then\
$MV $SNAPSHOT_RW/daily.5 $SNAPSHOT_RW/daily.6 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.4 ] ; then\
$MV $SNAPSHOT_RW/daily.4 $SNAPSHOT_RW/daily.5 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.3 ] ; then\
$MV $SNAPSHOT_RW/daily.3 $SNAPSHOT_RW/daily.4 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.2 ] ; then\
$MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.1 ] ; then\
$MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.0 ] ; then\
$MV $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ;\
fi;


# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/hourly.3 ] ; then\
$CP -al $SNAPSHOT_RW/hourly.3 $SNAPSHOT_RW/daily.0 ;\
fi;

# step 4: update the mtime of daily.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/daily.0 ;


And finally I have a weekly script that makes weekly snapshots during a 4-week period (backup_weekly.sh):

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/weekly.3 ] ; then\
$RM -Rf $SNAPSHOT_RW/weekly.3 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/weekly.2 ] ; then\
$MV $SNAPSHOT_RW/weekly.2 $SNAPSHOT_RW/weekly.3 ;\
fi;

if [ -d $SNAPSHOT_RW/weekly.1 ] ; then\
$MV $SNAPSHOT_RW/weekly.1 $SNAPSHOT_RW/weekly.2 ;\
fi;

if [ -d $SNAPSHOT_RW/weekly.0 ] ; then\
$MV $SNAPSHOT_RW/weekly.0 $SNAPSHOT_RW/weekly.1 ;\
fi;


# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/daily.6 ] ; then\
$CP -al $SNAPSHOT_RW/daily.6 $SNAPSHOT_RW/weekly.0 ;\
fi;

# step 4: update the mtime of weekly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/weekly.0 ;



And of course you also need to run crons ^^ # Make Backups
0 0 * * Sun sh /backup/backup_weekly.sh
15 0 * * * sh /backup/backup_daily.sh
45 0,6,12,18 * * * sh /backup/backup.sh



With these scripts there is just a small issue. You need to create first a manual daily.0 and weekly.0 folder :)
Submitted by Anonymous (Contact Author) (Forums) on Fri, 2006-04-21 20:03.
Three things:

1.) rsync does only transfer changed files but not even the whole file... but more than just the changes. rSync has some algorithm that splits up the file in multiple sections and then creates a checksum for it and compares it and only the changed parts will then be transmitted.
For example I had a mysql dump of 270 MB. I deleted it and made a new dump with a few changes. Now rsync noted that the file was changed but it didn't transmit the whole 270 MB again but only 25 MB.

2.) Instead of using --exclude I would rather use --exclude-from="/path/to/file" because I think it's much simpler to add there exclusions. Just add one pattern per line. I have for example this here:

/backup/
/bin/
/dev/
/initrd/
/lib/
/lost+found/
/mnt/
/opt/
/proc/
/sbin/
/sys/
/tmp/
/usr/
/var/log/
/var/cache/
/var/spool/
/var/lib/mysql/

I know, I still need to fine-tune that a bit.
3.) I would also add --delete-excluded for the simple reason that when you exclude something from being backuped then you don't want have older versions on the backup server any longer. This switch takes care of that.
Submitted by Anonymous (Contact Author) (Forums) on Mon, 2006-05-22 18:38.
Nice post and there is another way to do this also. I just implemented the same functionality for our main website by using rdist. The good thing about this is that it checks timestamp automatically and if the files have been modified it will replicate it accordingly. Put it in cron and you don't have to worry about anything else and let it email you stating which files have changed.
Submitted by lazyman (Contact Author) (Forums) on Mon, 2008-04-28 02:46.
If you use the flag -W rsync will copy the entire file not just the blocks it thinks are modified. http://www.oreillynet.com/linux/cmd/cmd.csp?path=r/rsync