One way fast and lightweight file mirroring

The oneliner

There are countless ways to mirror files. Here is one way to do it with rsync that's quite fast, very lightweight, and will work in many situations. Cherry on top: it also emails you when there is an error.

rsync -avWS --no-compress --progress --delete --timeout 10 -e "ssh -T -o Compression=no -x -p 12345" user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 | tee /dev/tty | ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" | ifne msmtp someone@who.cares

BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder /some/stuff inside /a/local/folder/ with rsync over ssh running as fast as possible, only using timestamps to detect changes. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags:

• -a: archive + preserve most standard file permissions, symlinks, etc
• -v: increasing verbosity of messages
• -W: copy whole files, you trade super slow delta transfers for super fast dumb copies
• -S: handle sparse files
• --no-compress: no compression, will increase speed a bit on modest machines
• --delete: local files will be deleted to really match remote content
• --timeout 10: no patience, remote has 10 seconds to pick up the phone
• -T: disable pseudo terminal for less resources on remote
• -o Compression=no: no compression, will increase speed a bit on modest machines
• -x: disable X11 forwarding
• -p 12345: security through obscurity
• 3>&2 2>&1 1>&3>: swap stderr with stdout so we can build pipes for error messages
• tee /dev/tty: cheeky trick to print to terminal what's being piped
• ifne: if not empty, means that the command after ifne will only run the following command if what arrives from the pipe is non-empty
• sed "0,/^/s/^/Subject: sync failed :\(\n/": prepend all the piped stuff with "Subject: sync failed :(", only once (GNU sed only, sorry)
• msmtp someone@who.cares: msmtmp is a small SMTP client. If you already have a full MTA installed, you can just replace with sendmail

Note: If you have time to waste (both CPU time and your own), or if you need precise accounting of all the bits, you can skip the -W flag. Also, you can probably speed up things further, using for instance the most lightweight allowed ssh cypher working on both the remote and local machine, but this stuff is already pretty speedy like this without many compromises.

2-in-1 cron script and manual sync

With a bit of file locking to ensure that only one instance of the script is running at a time, it is safe to cron the oneliner above with your very own preference for how often it should be called. Not judging. As a bonus that means you can also call the script from any shell to force an instant manual sync, regardless if it's already set to run on a cron, for this very moment where waiting is not an option.

#!/bin/sh

exec 9>/tmp/sync.lock

if ! flock -n 9
then
  echo 'lumpysync is already running, try again later :) \n';
  exit 1
fi

rsync -avWS --no-compress --progress --delete --timeout 10 \
  -e "ssh -T -o Compression=no -x -p 12345" \
  user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 | \
  tee /dev/tty | \
  ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" | \
  ifne msmtp someone@who.cares