Off-site Backup with Backupninja

From Run Your Own
Jump to navigation Jump to search

Goal: Setup a remote machine that will be used for incremental backup of critical parts of the LURK servers.

Requirements

This document assumes you already have:

  • A configure Linux/BSD machine to work as off-site backup
  • A lurk user created on the off-site backup that will be used specifically by the backup scripts
  • Debian based servers (not sure Backupninja will run otherwise without some slight modifications)
  • All the machines (servers and the backup machine) on a working Tinc VPN
  • Enough space on the off-site backup machine :)

On the Off-site machine

  • As lurk create directories for each server
mkdir /data/lurk/douglas /data/lurk/agnesbaxter

Hardening

Restrict the backup user in /etc/ssh/sshd_config and set longer timeouts for Borg 

 Match user lurk
       X11Forwarding no
       PasswordAuthentication no
       AllowTcpForwarding no
       PubkeyAuthentication yes
       ClientAliveInterval 10
       ClientAliveCountMax 30

Copy the public ssh key and add it to lurk's ~/.ssh/authorized_keys. Then restrict the use of that key only to the borg serve command (more info) so that your authorized_keys files looks something like this: 

 command="borg serve --restrict-to-path /data/lurk/agnesbaxter" ecdsa-sha2-nistp521 AAAAasdsad[..]asdsad root@agnesbaxter


On each server

Installation

Clone Backupninja from https://0xacab.org/riseuplabs/backupninja 

cd /usr/src/ && git clone https://0xacab.org/riseuplabs/backupninja
cd /usr/src/backupninja

./autogen.sh
./configure
make
make install

note that by running the above all configurations will be in /usr/local/etc/

Configuration

/etc/backupninja.conf

Some changes: 

reportemail = some@where.nice
when = everyday at 05:55

MySQL local backups

  • As root run the command:
ninjahelper
  • create a new backup action
  • mysql database backup
  • path:
/var/backups/mysql # adjust if this location does not have much free space
  • all the databases to backup.
  • select the debian maintenance user for access
  • compress the sql output file
  • select the action and test/run/review the config
  • leave ninjahelper

PostgreSQL local backups

  • As root run the command:
ninjahelper
  • create a new backup action
  • postgresql database backup
  • path:
/var/backups/postgres # adjust if this location does not have much free space
  • backup the whole cluster
  • compress the backups
  • custom
  • select the action and test/run/review the config
  • leave ninjahelper

Borg Backup

This is the action that will not only allow you to select which part of your local filesystem to remotely send and rotate to the off-site backup machine, but it will also make sure the local backups above are sent as well!

  • make sure you have borgbackup installed:
apt install borgbackup
  • As root run:
ninjahelper
  • create a new backup action, choose borg
  • choose file to include & exclude, add paths, wildcard accepted
  • configure backup destination:
    • dest_directory /data/lurk/name-of-server-to-backup
    • dest_host 10.0.1.2 Adjust to the Tinc IP of the off-site backup machine.
    • dest_user lurk
    • dest_type remote
  • set up ssh keys and test remote connection
  • enable encryption by setting encryption = keyfile and choosing a nice long passphrase
  • enable pruning, keep 120D (you can adjust if you will run out of space! Keep in mind this is incremental though, so don't panic)
  • select the action and test/run/review the config
  • check that everything is showing up nicely on the backup server in the destination directory!

Fine tuning

Edit/Change the local filesystem path to include/exclude

By default Babckupninja will backup some folder and exclude some others. This can be changed during the initial configuration of the borg action but can also be done later.

  • As root, edit /etc/backup.d/90.borg
  • Make changes in the section # files to include in the backup
  • Optional: run ninjahelper, select the 90.borg action and run it to make sure it's being sent to the off-site machine. If you're sure of your changes, you can also wait the next backup to happen to see if it worked.

Inspecting / Restoring

To inspect or restore the files from the off-site backups you need to know a few things:

  • Where the backup repositories are
  • What key they are encrypted with
  • What the passphrase is for the key

As of writing Feb 2026 we have the following:

  • Agnes has one backup repository (agnes-system in ninjahelper) sent to Manis and Stuff.
  • Larix has two backup repositories (larix-system, mastodon-borg) sent to Manis and Stuff.

Agnes' repo has its own passphrase. Larix has another passphrase but its two repos share the same one.

This passphrase needs to be entered every time you interact with the repo (unless you set the BORG_PASSPHRASE env variable for the duration of the session).

Furthermore, all repos require a key which is not attached to the repo itself. Passphrases and keys are available in the keepass.

Inspecting

n.b. You can find the actual paths and usernames by looking at the appropriate repo config in /usr/local/etc/backup.d/ on either Agnes or Larix.

Get general info on the repository: 

borg info ssh://user@10.0.1.5/path/to/repository

This gives you some details such as whether they key is in the repository (that is on the off-site location) or only on the server the backing up happens from. This becomes relevant when you need to restore a backup from any other location than the server you are backing up from. 

Repository ID: f80fe7d9b22c3425222a979eb607585bd0eb968163cb1e9b
Location: ssh://user@10.0.1.5/path/to/repository
Encrypted: Yes (repokey)
Cache: /root/.cache/borg/f80fe7d9b22c3425222a979eb607585bd0eb968163cb1e9b
Security dir: /root/.config/borg/security/f80fe7d9b22c3425222a979eb607585bd0eb968163cb1e9b
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:              259.11 GB            257.04 GB            104.89 GB 

                       Unique chunks         Total chunks
Chunk index:                  123440               525923


List all the repository snapshots: 

borg list ssh://user@10.0.1.5/path/to/repository

This will output all the different snapshots: 

2026-02-14T11:00:08                  Sat, 2026-02-14 11:00:10 [74easd341e160560585bd0eb968163cb1e9b89a0de405c5daa629c370895]
2026-02-15T11:00:09                  Sun, 2026-02-15 11:00:10 [48d4gdsd4161293e62e286a1e9a0f6b5c2b0082d0ac8fb665a7ea3e5eccc]
2026-02-16T11:00:10                  Mon, 2026-02-16 11:00:11 [dc08d6dqwe6973287f8f932b64474a6e4fcf0fe6950bf7209f1f2efa0068]

Query a specific snapshot from the ones listed above by appending the date (will flood terminal with thousands of file names): 

borg list ssh://user@10.0.1.5/path/to/repository::2026-02-16T11:00:10

Restoring

Restore an important file from a specific snapshot: 

borg extract --list --dry-run ssh://user@10.0.1.5/path/to/repository::2026-02-16T11:00:10 home/mastodon/live/.env.production

n.b. From the docs: This command always extracts to (".") and will recreate the full folder structure (home/mastodon/live/.env.production in the example above). So make sure you cd to the right place before calling borg extract. When parent directories are not extracted (because of using file/directory selection or any other reason), Borg cannot restore parent directories’ metadata, e.g., owner, group, permissions, etc.

Restoring from other location

To restore from a place that does not have the keys, such as from a new server, you first need to import the key for the appropriate repository.

The keys in the keepass are the ones exported by running borg key export ssh://user@10.0.1.5/path/to/repository .

These can be imported on a new machine by writing them to a file on the new server and importing them from there. Note the structure of the key: 

 BORG_KEY f80fe7d9b22c3425222a979eb607585bd0eb968163cb1e9b
hqlhbGdvcml0aG2mc2hhMjU2pGRhdGHaAN4tcXvK6kpfvKnENuufVl3HumRd+ejS+Rw5bp
23gYW8qGcnxPECz9BOWau2jcF12OdOyUJE+JlYIreH90p9PDDkXU7SCJ1R70MtR49+063D
xsmGpOw6zcJLtVFVVv15lhYC9KQzZUd2ChGwOXnu8jTXA+q0OfMzNHXZymEA6eJTKX/qQS
U9z0A+wZqCxwp5w875TITW5/6bDXm4+rmqMDL8P5mgiFsRxZgTHwD3tP4hHuwfG72vqXEx
4/2KApRwYp34gLtOc5Y4P7jLKZq/DRC7Qpu4r6f2IQEcs9ktcrmkaGFzaNoAINJleelRBv
BWLhh/Iqs27Etba8JaIAArppQbzCBFDZUnqml0ZXJhdGlvbnPOAAGGoKRzYWx02gAgipER
XyJdua9isfZGrJ2U8WuHKosRQgC1k/2J9vDcg2CndmVyc2lvbgE=

The header includes the repository id, the key includes linebreaks.

Import the key from a file like so: 

borg key import ssh://user@10.0.1.5/path/to/repository /path/to/keyfile
Retrieved from "https://things.bleu255.com/runyourown/index.php?title=Off-site_Backup_with_Backupninja&oldid=666"