Append-only backups with borg to another VPS or dedicated server

1 Configure the backup server
2 Setup the main server and how it connects to the backup server
3 Backup from the main server
1. 3.1 Run backup manually
2. 3.2 Schedule backup with cron
4 Restore files (while logged in at the main server)
1. 4.1 Show a list of backups made
2. 3.4 Verify/restore data
4 Prune old backups (require access both to backup and main server)

This tutorial will show how to backup the data on a server, denominated as the main server, to another host, here named backup server, with the free software backup program Borg. The backup server will be configured in such a way that the main server in normal usage can only append new data, and not delete or alter old backups. This append-only feature, combined with the fact that the backups are stored on a separate server in a separate location, makes the backups protected against data loss from for example natural accidents or a hacker attack against the main server.

Prerequisites to follow this guide is to use Debian Stretch (9) or Debian Buster (10) and have two servers available, one main server from which the backups are taken, and another backup server where the backup archives will be stored. These two servers should be in separate locations for optimum protection.

This guide will start with the configuration on the backup server in the first section. In the second section, we will configure the main server and then perform a backup, a test restore and show how to manually prune of old backup archives.

1 Configure the backup server

1.1 Install borg and create a new borgbackup user

On the backup server (the server where the backups should be stored) install Borg in a root terminal with this command:

apt install borgbackup

Then, choose a password for the borg user that we soon will create. (It is mandatory to have a password for each user, however, this password will almost never be used, as we normally will connect to the backup server with an SSH key.) If you want to generate a random 64 character password use this command:

< /dev/urandom tr -dc A-Za-z0-9 | head -c${1:-64};echo;

Copy the password for the borg user shown to the clipboard, and save it in a secure location somewhere else than on the server in itself.

Now, we will create the borgbackup user:

adduser borgbackup

When you get a question about the password, paste the password you copied earlier, press Enter and then paste the password again followed by Enter. Do not fill in anything else, just press Enter to answer the following questions.

1.2 Generate SSH keys for the borgbackup user

We now need to become the new user with su:

su borgbackup
cd

Then, we generate a new SSH key pair for the user:

ssh-keygen -t ed25519

After pressing Enter, you will get a series of questions. Leave the answers empty, just continue to press Enter several times until the command is finished. Do not set any passphrase for the key, press Enter also on that question.

1.3 Create the backup directory

Now, it is time to create the directory where the backups will be stored, which in borg terminology is defined as the borg repository. We will simply call the directory borgbackup, but you could instead choose to name it after the hostname of your main server.

cd
mkdir borgbackup
chmod go-rwx borgbackup
chmod u+rwx borgbackup

2 Setup the main server and how it connects to the backup server

2.1 Install Borg

Now we move from working on the backup server to instead working on the main server.

On the main server (the server you want to take backups of) install Borg in a root terminal with this command:

apt install borgbackup

Later we will need the external IP address of the main server (the one we are at). Please copy this IP address so you have it readily available. If you do not remember the external IP address of the main server, run this command to display it:

wget -qO- http://ipecho.net/plain | xargs echo

2.2 Allow the main server to access the backup server

Commands that you run on the main server

Now we will prepare the SSH key authentication so that it is possible to connect from the main server to the backup server. First, use this command to show the public SSH key of the root user on the main server:

cat ~/.ssh/id_*.pub

If you have a SSH public key for root you should now see this key displayed. Copy the key to your clipboard, you will need to paste it later in a file on the backup server.

If you do not see the key displayed, you need to generate the keypair. For example, run this command in that case: ssh-keygen -t ed25519 followed by repated presses of Enter.

Now, we need to tell the backup server to allow access from the main server (where we are right now) with the public SSH key. In order to do this, enter this command, but first replace BACKUPHOSTNAME with the hostname of your backup server:

ssh-copy-id -i ~/.ssh/id_*.pub borgbackup@BACKUPHOSTNAME

Connect to the backup server

You will now need to enter the password for the borgbackup user that you created in step 1.1. After that is done, it should be possible to directly access the backup server as the borgbackup user without any password. Now, test that this is possible by typing this command, replacing BACKUPHOSTNAME with the hostname of your backup server:

ssh borgbackup@BACKUPHOSTNAME

Commands that you run at the backup server

We will now tighten the security even further by restricting access only from the IP address of the main server using this key.

Stay at the backup server and run this command:

nano ~/.ssh/authorized_keys

Now, you should see a long line that begins with ssh. Verify that the cursor is in the beginning of the file. Add the following data on the same line, before the ssh key, but please first replace SERVERIPADDRESS with the IP address of your server (that you researched in the beginning of this section):

from="SERVERIPADDRESS",command="borg serve --append-only --restrict-to-path /home/borgbackup/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc

Make sure that there is a space character between the section you have added and the preexisting section starting with ssh that follows afterwards. Then save the file with Ctrl+O and exit with Ctrl+X. We have now limited the access to the backup server in two ways. Firstly, only the IP address of the main server is permitted access. Secondly, through the --append-only option it is only permitted to add data to the borg repository, but deletions and alterations are not permitted.

Disconnect from the backup server and return to the main server

Run this command to exit from the backup server and return back to the main server:

exit

2.3 Choose a borg repository password

A borg repository is the location where the backups are stored. It is protected by a borg repository password, which you know need to choose. The password is needed in order to perform new backups and access to old ones. You could either come up with a password yourself (choose minimum 20 characters for optimum security) or randomly generate a 64 character long password with this command:

< /dev/urandom tr -dc A-Za-z0-9 | head -c${1:-64};echo;

Copy the borg repository password to your clipboard by marking it and pressing Ctrl+C. It is essential to save the borg repository password in a secure place because without this password you cannot access your backups! So save a copy of the borg password right now somewhere else than on your servers, for example in a password manager (such as KeePass) on your own computer.

2.4 Create the borg repository

In Borg terminology, the location where the backups are stored is named the borg repository. The password will now be set as an environmental variable in preparation of creating this repository. Replace PASSWORD with your actual password and execute the command below as the root user of your main server:

export BORG_PASSPHRASE='PASSWORD'

Create the borg repository in this way - but first replace BACKUPHOSTNAME with the hostname of your backup server:

borg init -e repokey-blake2 borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/

Everything has now been prepared to use borg for backups, which will be the topic of the following section.

3 Backup from the main server

3.1 Run backup manually

In the example that follows, the whole system of the main server will be backed up, except certain directories that are not relevant to backup.

Please make sure that you are logged in as root on your main server before continuing. First, we will set the environmental variable for the borg repository password through this command (replace PASSWORD with you actual borg repository password:

export BORG_PASSPHRASE='PASSWORD'

Then we will run the following command to perform a full system backup. Please adjust the directory exclusions, those begin with --exclude, to your needs. For example, you may want to add an exclusion of the /mnt directory by adding --exclude=mnt Please note that you should not enter the first slash in the exclusion pattern, e.g. you write mnt instead of /mnt. You also need to change the BACKUPHOSTNAME to the actual hostname of your backup server. We need to run a cd / command first, as borg normally starts the backup from the directory it is run from.

cd / && borg create --stats --progress --compress lz4 borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/::`hostname`-`date +%Y-%m-%d-%H-%M-%S` ./ --exclude=dev --exclude=mnt/borgbackup -exclude=proc --exclude=run --exclude=root/.cache/ --exclude=sys --exclude=tmp && cd

The first time the backup is run it will take a while (up to several hours if you have a large amount of data to backup). You will see a status line quickly update exactly which file borg is processing in the backup. From the second time onwards backups will bemuch quicker, as only changes since the last backup will be transmitted.

3.2 Schedule backup with cron

It is, of course, best to run an automatic backup at a regular interval, rather than having to remember to run manual backups. In order to accomplish this, we first create a shell script that runs the backup job, which we then edit with the nano editor as the root user on the main server:

touch /usr/local/bin/borgbackup.sh
chmod go-rwx /usr/local/bin/borgbackup.sh
chmod u+rwx /usr/local/bin/borgbackup.sh
nano /usr/local/bin/borgbackup.sh

Insert these lines in the new shell script:

#!/bin/sh
export BORG_PASSPHRASE='PASSWORD'
cd / && borg create --stats --progress --compress lz4 borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/::`hostname`-`date +%Y-%m-%d-%H-%M-%S` ./ --exclude=dev --exclude=proc --exclude=run --exclude=root/.cache/ --exclude=mnt/borgmount --exclude=sys --exclude=tmp && cd

You need to replace PASSWORD with your borg repository password, replace BACKUPHOSTNAME with the hostname of your backup server and finally adjust the exclude patterns to your needs - you may want to add --exclude=mnt to exclude the /mnt directory from backup. Save the file in the nano editor through pressing Ctrl+O, and then exit with Ctrl+X.

Next, test the script through running it from the terminal:

/usr/local/bin/borgbackup.sh

In order to schedule a backup to 02:00 every night, add the backup shell script we just created to /etc/crontab :

nano /etc/crontab

Then add a new line to the crontab with the backup job:

# Backup via Borg to backup server 
00 02           * * *   root    /usr/local/bin/borgbackup.sh

4 Restore files (while logged in at the main server)

4.1 Show a list of backups made

You do not have a functioning backup system until when you have verified that you can restore data from your backups. Therefore our next step is to check the backup we made earlier. This will be done from the root terminal at the main server. We will do this by mounting the backup archive as an extra file system, but first, we will check the list of the backups made through creating a shell script as the root user on your main server:

touch /usr/local/bin/borglist.sh
chmod go-rwx /usr/local/bin/borglist.sh
chmod u+rwx /usr/local/bin/borglist.sh
nano /usr/local/bin/borglist.sh

Insert the following in the new shell script, replacing PASSWORD with your borg repository password, and BACKUPHOSTNAME with the hostname of your backup server:

#!/bin/sh
export BORG_PASSPHRASE='PASSWORD'
borg list -v borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/

Save the file with Ctrl+O followed by Ctrl+X. Then run the shell script in this way:

/usr/local/bin/borglist.sh

You should now see a list of the backups that have been made. If your first backup was made, you should thus see one item in the list. Now mark and copy the archive name to the clipboard. You find the archive name in the first column, it is made up of the hostname of your main server followed by a date and time, on this format: mainhostname-2019-01-31-12-59-59 .

3.4 Verify/restore data

We will now mount the whole borg repository as a FUSE file system mount. This means that the borg repository - including all the files that have been backuped - becomes a filesystem, which we can traverse and inspect through normal commands such as ls and cd.

In order to do be able to easily mount the borg repository, we will create a shortcut of the commands in form of a shell script. Create and edit the shell script in this way:

touch /usr/local/bin/borgmount.sh
chmod go-rwx /usr/local/bin/borgmount.sh
chmod u+rwx /usr/local/bin/borgmount.sh
nano /usr/local/bin/borgmount.sh

Then insert the lines below into the schell script, but first please change PASSWORD below to your borg repository password and change BACKUPHOSTNAME to the hostname of your backup server:

#!/bin/sh
mkdir -p /mnt/borgbackup
export BORG_PASSPHRASE='PASSWORD'
borg mount borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/ /mnt/borgbackup

Save the file with Ctrl+O followed by Ctrl+X. Then run the shell script in this way:

/usr/local/bin/borgmount.sh

The borg repository should now be mounted at /mnt/borgbackup - you can check this by using cd and ls:

cd /mnt/borgbackup
ls

Now you should see a directory in which your backup resides. Go into that directory with cd (replace DIRECTORYNAME with the name of the directory you see displayed as a result of the ls command you just entered) and then use ls to see the contents:

cd DIRECTORYNAME
ls

Now you will be able to see the last archive (the last backup made). You can compare with diff that a file has been backuped correctly, for example for /etc/fstab, with this command:

diff etc/fstab /etc/fstab

If you get no output from diff, the two files are identical and the backup has worked for that file. If you, on the other hand, have changed the file since the last backup you will see which lines in the files that differ.

If you want to you can restore files from the backup, simply by using the cp command to copy files from the subdirectory of /mnt/borgbackup in which you are situated in the terminal right now.

Finally, we will create a script to unmount the borg fuse mount.

Create and edit the shell script in this way:

touch /usr/local/bin/borgumount.sh
chmod go-rwx /usr/local/bin/borgumount.sh
chmod u+rwx /usr/local/bin/borgumount.sh
nano /usr/local/bin/borgumount.sh

Then insert the lines below into the schell script:

#!/bin/sh
cd ~
umount /mnt/borgbackup
rmdir /mnt/borgbackup

Save the file with Ctrl+O followed by Ctrl+X. Then run the shell script in this way:

/usr/local/bin/borgumount.sh

The borg fuse mount is now unmounted and you have been placed in the /root/ directory at the terminal.

4 Prune old backups (require access both to backup and main server)

We now need to switch back and forth between the backup server and the main server. First, we start with commands on the backup server.

After a while, the size of the backups can grow large. For security reasons it is not allowed for the main server to delete old backups automatically. However, we can temporarily allow the removal of old backup archives when needed, in order to save disk space at the backup server.

This removal process is called to prune in borg terminology. The pruning process starts at the backup server, with allowing temporary read-write access. Then we connect to the main server to issue the pruning command. Finally, we access the backup server again to disable read-write access so that old backups are protected from deletion or modification again.

4.1 Allow temporary read+write access (done from the backup server)

We are now going to work at the backup server.

Therefore connect to the backup server and login as root. In order to allow read+write access we need to change the file /home/borgbackup/.ssh/authorized_keys at the backup server. Issue these commands to become the borgbackup user, take a backup of the file and then edit the file with nano:

su borgbackup
cd
cp -a ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak
nano ~/.ssh/authorized_keys

Now, you should in the nano editor see a long line that begins with from= . Cut the line through pressing Ctrl+K and then press Ctrl+U, and once again Ctrl+U so that you have now have two identical lines. Press the Up key on your keyboard two times to set the cursor at the first row. Then type a # character to comment out the first line. Next, press the Down key on your keyboard to get to the second row. Here, you should move to the right with the Right key on your keyboard until you stand at the first - in --append-only. Then press the Del key repeatedly to delete the option --append-only . The file should look approximately like this, note that we have commented away the first line and that --append-only is deleted from the second line:

#from="SERVERIPADDRESS",command="borg serve --append-only --restrict-to-path /home/borgbackup/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh ... (the line continues) ...
from="SERVERIPADDRESS",command="borg serve --restrict-to-path /home/borgbackup/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh ... (the line continues) ..

Save the file with Ctrl+O and exit with Ctrl+X. We have now enabled read+write access to the borg repository.

4.2 Prune old backups (done from the main server)

Now, connect to the main server in a new terminal window as the root user.

We will now create a script at the main server to prune (i.e. remove) old backups. Create and edit the shell script on the main server in this way:

touch /usr/local/bin/borgprune.sh
chmod go-rwx /usr/local/bin/borgprune.sh
chmod u+rwx /usr/local/bin/borgprune.sh
nano /usr/local/bin/borgprune.sh

Then insert the lines below into the shell script. Then replace PASSWORD with your borg repository password and BACKUPHOSTNAME with the hostname of your backup server. You also need to adjust how many old backups you want to keep. In this example, we keep 7 daily backups, 4 weekly backups and 12 monthly backups.

#!/bin/sh
export BORG_PASSPHRASE='PASSWORD'
borg prune --stats --progress borgbackup@BACKUPHOSTNAME:/home/borgbackup/borgbackup/ --prefix `hostname`- --keep-daily=7 --keep-weekly=4 --keep-monthly=12

Press Ctrl+O and Ctrl+X to save and exit from the file. Then, run the prune script you just created on the main server:

/usr/local/bin/borgprune.sh

This command will take a while (can be up to a couple of hours), but you can see the progress it makes along the way, in two passes of getting to 100%.

4.3 Disallow read+write access (done from the backup server)

Now we go back to work at the backup server.

After the prune command has finished on the main server, you now connect to to the backup server as the borgbackup user and edit the /home/borgbackup/.ssh/authorized_keys file again to disallow read+write access.

Do like this on the backup server:

su borgbackup
cd 
nano ~/.ssh/authorized_keys

In the nano editor, take away the # character on the beginning of the first row by pressing the Del key on your keyboard. Then, press the Down key and insert a new # character on the second line. After this, the file should look more or less like this:

from="SERVERIPADDRESS",command="borg serve --append-only --restrict-to-path /home/borgbackup/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh ... (the line continues) ...
#from="SERVERIPADDRESS",command="borg serve --restrict-to-path /home/borgbackup/borgbackup/",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh ... (the line continues) ..

Save the file with Ctrl+O and exit with Ctrl+X. We have now disabled read+write access to the borg repository.

Now backups can continue like normal, in an append-only fashion, and you are protected from deletion attempts from the main server aimed at the backups stored on the backup server.

4.4 Final remarks

You have now created a backup system that is off-site and append-only. This means that the backups are protected in two manners. Firstly, the backups are protected in the sense of being physically separate from the main server. Secondly and finally backups already taken are protected from alteration attempts from the main server (except when you explicitly decide to alter them through manual pruning).