One way fast and lightweight file mirroring

The oneliner

There are countless ways to mirror files. Here is one way to do it with rsync that's quite fast, very lightweight, and will work in many situations. Cherry on top: it also emails you when there is an error.

rsync -avWS --no-compress --progress --delete --timeout 10 -e "ssh -T -o Compression=no -x -p 12345" user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 | tee /dev/tty | ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" | ifne msmtp someone@who.cares

BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder /some/stuff *inside* /a/local/folder/ with rsync over ssh running as fast as possible, only using timestamps to detect changes. If you have time to waste (both CPU time and your own), or if you need precise accounting of all the bits, you can skip the -W flag. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags:

• -a: archive + preserve most standard file permissions, symlinks, etc
• -v: increasing verbosity of messages
• -W: copy whole files, you trade super slow delta transfers for super fast dumb copies
• -S: handle sparse files
• --no-compress: no compression, will increase speed a bit on modest machines
• --delete: local files will be deleted to really match remote content
• --timeout 10: no patience, remote has 10 seconds to pick up the phone
• -T: disable pseudo terminal for less resources on remote
• -o Compression=no: no compression, will increase speed a bit on modest machines
• -x: disable X11 forwarding
• -p 12345: security through obscurity
• 3>&2 2>&1 1>&3>: swap stderr with stdout so we can build pipes for error messages
• tee /dev/tty: cheeky trick to print to terminal what's being piped
• ifne: if not empty, means that the command after ifne will only run the following command if what arrives from the pipe is non-empty
• sed "0,/^/s/^/Subject: sync failed :\(\n/": prepend all the piped stuff with "Subject: sync failed :(", only once (GNU sed only, sorry)
• msmtp someone@who.cares: msmtmp is a small SMTP client. If you already have a full MTA installed, you can just replace with sendmail

You can probably speed up things further, using for instance the most lightweight allowed ssh cypher working on both the remote and local machine, but this stuff is already pretty speedy like this without many compromises.

2-in-1 cron script and manual sync

With a bit of file locking to ensure that only one instance of rsync is running at a time, it is safe to cron the following script with your *very own* preference for how often it should be called. As a bonus that means you can also call the script for manual sync, regardless if it's already set to run on a cron.

rsync -q -avWS --no-compress --progress --delete --timeout 10 -e "ssh -T -o Compression=no -x -p 12345" username@server.remote:/distant/content /local/folder/ 2>&1 | ifne cat <(echo "Subject: sync failed\n ") - | ifne sendmail -s "ohoh" someone@who.cares