Off-site Backup with Backupninja

Goal: Setup a remote machine that will be used for incremental backup of critical parts of the LURK servers.

Requirements

This document assumes you already have:

• A configure Linux/BSD machine to work as off-site backup
• A lurk user created on the off-site backup that will be used specifically by the backup scripts
• Debian based servers (not sure Backupninja will run otherwise without some slight modifications)
• All the machines (servers and the backup machine) on a working Tinc VPN
• Enough space on the off-site backup machine :)

On the Off-site machine

• As lurk create directories for each server

mkdir /data/lurk/douglas /data/lurk/agnesbaxter

Hardening

Restrict the backup user in /etc/ssh/sshd_config and set longer timeouts for Borg

 Match user lurk
       X11Forwarding no
       PasswordAuthentication no
       AllowTcpForwarding no
       PubkeyAuthentication yes
       ClientAliveInterval 10
       ClientAliveCountMax 30

Copy the public ssh key and add it to lurk's ~/.ssh/authorized_keys. Then restrict the use of that key only to the borg serve command (more info) so that your authorized_keys files looks something like this:

 command="borg serve --restrict-to-path /data/lurk/agnesbaxter" ecdsa-sha2-nistp521 AAAAasdsad[..]asdsad root@agnesbaxter

On each server

Installation

Clone Backupninja from https://0xacab.org/riseuplabs/backupninja

cd /usr/src/ && git clone https://0xacab.org/riseuplabs/backupninja
cd /usr/src/backupninja

./autogen.sh
./configure
make
make install

note that by running the above all configurations will be in /usr/local/etc/

Configuration

/etc/backupninja.conf

Some changes:

reportemail = some@where.nice
when = everyday at 05:55

MySQL local backups

• As root run the command:

ninjahelper

• create a new backup action
• mysql database backup
• path:

/var/backups/mysql # adjust if this location does not have much free space

• all the databases to backup.
• select the debian maintenance user for access
• compress the sql output file
• select the action and test/run/review the config
• leave ninjahelper

PostgreSQL local backups

• As root run the command:

ninjahelper

• create a new backup action
• postgresql database backup
• path:

/var/backups/postgres # adjust if this location does not have much free space

• backup the whole cluster
• compress the backups
• custom
• select the action and test/run/review the config
• leave ninjahelper

Borg Backup

This is the action that will not only allow you to select which part of your local filesystem to remotely send and rotate to the off-site backup machine, but it will also make sure the local backups above are sent as well!

• make sure you have borgbackup installed:

apt install borgbackup

• As root run:

ninjahelper

• create a new backup action, choose borg
• choose file to include & exclude, add paths, wildcard accepted
• configure backup destination:
  • dest_directory /data/lurk/name-of-server-to-backup
  • dest_host 10.0.1.2 Adjust to the Tinc IP of the off-site backup machine.
  • dest_user lurk
  • dest_type remote
• set up ssh keys and test remote connection
• enable encryption by setting encryption = keyfile and choosing a nice long passphrase
• enable pruning, keep 120D (you can adjust if you will run out of space! Keep in mind this is incremental though, so don't panic)
• select the action and test/run/review the config
• check that everything is showing up nicely on the backup server in the destination directory!

Fine tuning

Edit/Change the local filesystem path to include/exclude

By default Babckupninja will backup some folder and exclude some others. This can be changed during the initial configuration of the borg action but can also be done later.

• As root, edit /etc/backup.d/90.borg
• Make changes in the section # files to include in the backup
• Optional: run ninjahelper, select the 90.borg action and run it to make sure it's being sent to the off-site machine. If you're sure of your changes, you can also wait the next backup to happen to see if it worked.