One way fast and lightweight file mirroring - Revision history

320x200 at 23:39, 21 February 2024

2024-02-21T23:39:32Z

← Older revision		Revision as of 23:39, 21 February 2024
Line 22:		Line 22:
	* <code>3>&2 2>&1 1>&3></code>: swap <code>stderr</code> with <code>stdout</code> so we can build pipes for error messages		* <code>3>&2 2>&1 1>&3></code>: swap <code>stderr</code> with <code>stdout</code> so we can build pipes for error messages
	* <code>tee /dev/tty</code>: cheeky trick to print to terminal what's being piped		* <code>tee /dev/tty</code>: cheeky trick to print to terminal what's being piped
	* <code>ifne</code>: if not empty, means that the command after <code>ifne</code> will only run ~~the following command~~ if what arrives from the pipe is non-empty		* <code>ifne</code>: if not empty, means that the command after <code>ifne</code> will only run if what arrives from the pipe is non-empty
	* <code>sed "0,/^/s/^/Subject: sync failed :\(\n/"</code>: prepend all the piped stuff with "Subject: sync failed :(", only once (GNU sed only, sorry)		* <code>sed "0,/^/s/^/Subject: sync failed :\(\n/"</code>: prepend all the piped stuff with "Subject: sync failed :(", only once (GNU sed only, sorry)
	* <code>msmtp someone@who.cares</code>: <code>msmtmp</code> is a small SMTP client. If you already have a full MTA installed, you can just replace with <code>sendmail</code>		* <code>msmtp someone@who.cares</code>: <code>msmtmp</code> is a small SMTP client. If you already have a full MTA installed, you can just replace with <code>sendmail</code>

320x200: /* The oneliner */

2024-02-21T23:30:44Z

The oneliner

← Older revision		Revision as of 23:30, 21 February 2024
Line 7:		Line 7:
	</code>		</code>

	BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder <code>/some/stuff</code> ''inside'' <code>/a/local/folder/</code> with <code>rsync</code> over <code>ssh</code> running as fast as possible, only using timestamps to detect changes. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags:		BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder <code>/some/stuff</code> ''inside'' <code>/a/local/folder/</code> with <code>rsync</code> over <code>ssh</code> running as fast as possible, only using timestamps to detect changes. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags, and how the pipeline is constructed and activated with the presence of error messages:

	* <code>-a</code>: archive + preserve most standard file permissions, symlinks, etc		* <code>-a</code>: archive + preserve most standard file permissions, symlinks, etc

320x200 at 23:13, 21 February 2024

2024-02-21T23:13:01Z

← Older revision		Revision as of 23:13, 21 February 2024
Line 50:		Line 50:

	[[Category: backup]]		[[Category: backup]]
	~~[[Category: sync]]~~

320x200: /* 2-in-1 cron script and manual sync */

2024-02-21T23:11:32Z

2-in-1 cron script and manual sync

← Older revision		Revision as of 23:11, 21 February 2024
Line 48:		Line 48:
	ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" \| \		ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" \| \
	ifne msmtp someone@who.cares		ifne msmtp someone@who.cares

			[[Category: backup]]
			[[Category: sync]]

320x200: /* 2-in-1 cron script and manual sync */

2024-02-21T23:10:56Z

2-in-1 cron script and manual sync

← Older revision		Revision as of 23:10, 21 February 2024
Line 38:		Line 38:
	if ! flock -n 9		if ! flock -n 9
	then		then
	echo '~~lumpysync~~ is already running, try again later :) \n';		echo 'sync is already running, try again later :) \n';
	exit 1		exit 1
	fi		fi

320x200: /* 2-in-1 cron script and manual sync */

2024-02-21T23:10:39Z

2-in-1 cron script and manual sync

← Older revision		Revision as of 23:10, 21 February 2024
Line 32:		Line 32:
	With a bit of file locking to ensure that only one instance of the script is running at a time, it is safe to cron the oneliner above with your ''very own'' preference for how often it should be called. Not judging. As a bonus that means you can also call the script from any shell to force an instant manual sync, regardless if it's already set to run on a cron, for this very moment where waiting is not an option.		With a bit of file locking to ensure that only one instance of the script is running at a time, it is safe to cron the oneliner above with your ''very own'' preference for how often it should be called. Not judging. As a bonus that means you can also call the script from any shell to force an instant manual sync, regardless if it's already set to run on a cron, for this very moment where waiting is not an option.

	~~<code>~~		#!/bin/sh
	#!/bin/sh
	~~</code>~~
	~~<code>~~
	~~exec 9>/tmp/sync.lock~~

	~~if ! flock -n~~ 9		exec 9>/tmp/sync.lock
	~~then~~
	~~echo 'lumpysync is already running, try again later :) \n';~~
	~~exit 1~~
	fi

	rsync -avWS --no-compress --progress --delete --timeout 10 \		if ! flock -n 9
	-e "ssh -T -o Compression=no -x -p 12345" \		then
	user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 \| \		echo 'lumpysync is already running, try again later :) \n';
	tee /dev/tty \| \		exit 1
	ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" \| \		fi
	ifne msmtp someone@who.cares
	~~</code>~~		rsync -avWS --no-compress --progress --delete --timeout 10 \
			-e "ssh -T -o Compression=no -x -p 12345" \
			user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 \| \
			tee /dev/tty \| \
			ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" \| \
			ifne msmtp someone@who.cares

320x200: /* 2-in-1 cron script and manual sync */

2024-02-21T23:09:45Z

2-in-1 cron script and manual sync

← Older revision		Revision as of 23:09, 21 February 2024
Line 34:		Line 34:
	<code>		<code>
	#!/bin/sh		#!/bin/sh
			</code>
			<code>
	exec 9>/tmp/sync.lock		exec 9>/tmp/sync.lock

320x200 at 23:09, 21 February 2024

2024-02-21T23:09:03Z

← Older revision		Revision as of 23:09, 21 February 2024
Line 34:		Line 34:
	<code>		<code>
	#!/bin/sh		#!/bin/sh

	exec 9>/tmp/sync.lock		exec 9>/tmp/sync.lock

	if ! flock -n 9		if ! flock -n 9
	then		then
Line 42:		Line 42:
	exit 1		exit 1
	fi		fi

	rsync -avWS --no-compress --progress --delete --timeout 10 \		rsync -avWS --no-compress --progress --delete --timeout 10 \
	-e "ssh -T -o Compression=no -x -p 12345" \		-e "ssh -T -o Compression=no -x -p 12345" \

320x200: /* 2-in-1 cron script and manual sync */

2024-02-21T23:08:11Z

2-in-1 cron script and manual sync

← Older revision		Revision as of 23:08, 21 February 2024
Line 30:		Line 30:
	=== 2-in-1 cron script and manual sync ===		=== 2-in-1 cron script and manual sync ===

	With a bit of file locking to ensure that only one instance of ~~rsync~~ is running at a time, it is safe to cron the ~~following script~~ with your very own preference for how often it should be called. As a bonus that means you can also call the script ~~for~~ manual sync, regardless if it's already set to run on a cron.		With a bit of file locking to ensure that only one instance of the script is running at a time, it is safe to cron the oneliner above with your ''very own'' preference for how often it should be called. Not judging. As a bonus that means you can also call the script from any shell to force an instant manual sync, regardless if it's already set to run on a cron, for this very moment where waiting is not an option.

			<code>
			#!/bin/sh

	~~<code~~>		exec 9>/tmp/sync.lock
	rsync -q -avWS --no-compress --progress --delete --timeout 10 -e "ssh -T -o Compression=no -x -p 12345" ~~username~~@~~server~~.remote:/~~distant~~/~~content~~ /local/folder/ 2>&1 \| ifne ~~cat <(echo~~ "Subject: sync failed\n "~~) -~~ \| ifne ~~sendmail -s "ohoh"~~ someone@who.cares
			if ! flock -n 9
			then
			echo 'lumpysync is already running, try again later :) \n';
			exit 1
			fi

			rsync -avWS --no-compress --progress --delete --timeout 10 \
			-e "ssh -T -o Compression=no -x -p 12345" \
			user@somewhere.remote:/some/stuff /a/local/folder/ 3>&2 2>&1 1>&3 \| \
			tee /dev/tty \| \
			ifne sed "0,/^/s/^/Subject: sync failed :\(\n/" \| \
			ifne msmtp someone@who.cares
	</code>		</code>

320x200: /* The oneliner */

2024-02-21T23:00:02Z

The oneliner

← Older revision		Revision as of 23:00, 21 February 2024
Line 7:		Line 7:
	</code>		</code>

	BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder <code>/some/stuff</code> inside <code>/a/local/folder/</code> with <code>rsync</code> over <code>ssh</code> running as fast as possible, only using timestamps to detect changes~~. If you have time to waste (both CPU time and your own), or if you need precise accounting of all the bits, you can skip the <code>-W</code> flag~~. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags:		BUT WHAT DOES IT DO?!?? Basically it tries to mirror the remote folder <code>/some/stuff</code> ''inside'' <code>/a/local/folder/</code> with <code>rsync</code> over <code>ssh</code> running as fast as possible, only using timestamps to detect changes. Normal messages are printed to the terminal, same with error messages, but unlike regular messages, error messages will go through a pipeline to generate an email that will be sent to someone who hopefully cares. Here is a more detailed explanation of the flags:

	* <code>-a</code>: archive + preserve most standard file permissions, symlinks, etc		* <code>-a</code>: archive + preserve most standard file permissions, symlinks, etc
Line 26:		Line 26:
	* <code>msmtp someone@who.cares</code>: <code>msmtmp</code> is a small SMTP client. If you already have a full MTA installed, you can just replace with <code>sendmail</code>		* <code>msmtp someone@who.cares</code>: <code>msmtmp</code> is a small SMTP client. If you already have a full MTA installed, you can just replace with <code>sendmail</code>

	~~You~~ can probably speed up things further, using for instance the most lightweight allowed ssh cypher working on both the remote and local machine, but this stuff is already pretty speedy like this without many compromises.		'''Note:''' If you have time to waste (both CPU time and your own), or if you need precise accounting of all the bits, you can skip the <code>-W</code> flag. Also, you can probably speed up things further, using for instance the most lightweight allowed ssh cypher working on both the remote and local machine, but this stuff is already pretty speedy like this without many compromises.

	=== 2-in-1 cron script and manual sync ===		=== 2-in-1 cron script and manual sync ===