TLDR;

git clone https://github.com/laurent22/rsync-time-backup
cd rsync-time-backup
./rsync_tmbackup.sh -p 2222 -i <sshkey> /home [email protected]:/mnt/backupdrive

Introduction

Backing up your data is crucial to avoid losing important files. Things happen and in cases of hardware failure, software bugs, or other unforeseen events, you will be glad you have a copy of your valuable data. There are generally two kinds of people; those who have lost data and now do backups religiously and those who have not yet felt the pain and live life in blissful ignorance, waiting for disaster to strike. Don't be one of those people!

There are many tools available for backing up data. Rsync is one such tool. It is a popular and reliable choice for creating backups. Rsync is a powerful tool for transferring and synchronizing files between two locations. It can be used to create incremental backups that only transfer changed files so it's very bandwidth efficient.

This brings us to the rsync_tmbackup.sh script. This script provides a convenient wrapper for rsync that makes it easier to create backups with some added advanced features. It is a script created and maintained by Laurent Cozic et al.  In this article, we will explain how to use the rsync_tmbackup.sh script for remote backups.

Assumptions

  • You already have all the ssh configs done for both the source and destination servers.
  • You have enabled and are using certificate authentication.

The Problem Statement

How do I back up data on a remote server when my data storage device sits behind a NAT Firewall I have no control over?

The main issue here has to do with the backup (UnRaid) server located on my local LAN. This server can not be accessed from the WAN side of things as it is blocked by the ISP. So the server cannot initiate the backup. The backup needs to be started from the UnRaid Server's side. The source server is a small Linode instance running a few docker containers. I would like all the mounted storage volumes (which contain the persisted data) to be backed up to the UnRaid server located on my local LAN.

This means that the backup server needs to initiate and manage the backup. So it is a pull-based model.

Enter the rsync_tmbackup.sh script

The rsync_tmbackup.sh script is a shell script wrapping rsync to create "Time Machine-style backups". It is designed to create incremental backups that only transfer the changes between the source and destination. The script uses hard links to save space on the destination, which means that unchanged files are not copied but are instead linked to the previous backup. The result is a set of backups that only take up as much space as the changes made since the previous backup.

The script is easy to use and has many features that make it a powerful backup tool. For example, it can exclude files and directories from the backup, compress the data during transfer, and delete old backups to free up space on the destination. The script also supports remote backups over SSH, which makes it ideal for creating backups of remote servers.

I would encourage you to head over to the GitHub repo and read the instructions which are way more succinctly expressed than I could do it justice.

How to use rsync_tmbackup.sh for remote backups

To use the script for remote backups, you need to have a source and a destination. The source is the directory or directories that you want to back up, and the destination is the directory where you want to store the backups.

Enough talking, to arms!

1. Install rsync_tmbackup.sh

The first step is to install the rsync_tmbackup.sh script. You can download the script from the official GitHub repository. Clone the repository and cd into the directory by running the command:

git clone https://github.com/laurent22/rsync-time-backup.git
cd rsync-tim-backup
chmod +x rsync_tmbackup.sh

You also need to ensure the script is executable

2. Configure the backup

Next, configure the backup by executing the script and passing the required options. For ease of reference here is the output of the scripts documentation:

3. Run the backup

The command I settled on is:

./rsync_tmbackup.sh -i ssh.key -p 22 root@[myservername]:/myfolder /backups

Translated to English: Using public/private key authentication,  the -i option allows specifying the key, use SSH port 22 (most security-minded people change the default SSH port). Copy the myfolder from myservername to the /backups folder on my UnRaid server. And just like that our 'script' is complete.

Important: If you get an error along the lines of the destination is not a backup location, remember to add the file backup.marker that marks the location as a backup location.

4. Automating the backup

So, executing the above command gets as a nice snapshot at the time of execution. I don't want to have to do this manually. UnRaid runs mostly read-only from a flash boot drive. This is probably the source of comments I have read of people having trouble using the built-in crontab. If you are not careful, when the server restarts, you lose your configuration.

For this reason, I opted to use a plugin. Log into the UnRaid web interface then navigate to the apps page. Search for 'user scripts' and install the plugin.

After the installation is complete, head to the plugin tab. You should find the installed plugin in the list of plugin:

Click on the folder icon and it should take you to the user scripts plugin management interface. Create a new job. I called mine NightlyLinodeServerBackup. This will create a folder on the backup machine.

You need to follow the direction in the 'How to add scripts' section at the bottom of the page. For me the default script location was:

/boot/config/plugins/user.scripts/scripts/NightlyLinodeServerBackup

The script and the metadata are managed by adding files to this directory. Just follow the guidelines at the bottom of the page.

I have opted to configure a custom cron job that runs at midnight every night.

5. The backups folder

After 3 days of backups, here are the contents of the directory:

  • latest@ is a symlink to the latest version of the backup
  • <year-month-day-time> folder contains the backup data

I opted for the default data retention settings in which: "Sets are automatically deleted following a simple expiration strategy defined with the --strategy flag. This strategy is a series of time intervals with each item being defined as x:y, which means "after x days, keep one backup every y days". The default strategy is, which means:

  • After 1 day, keep one backup every 1 day (1:1).
  • After 30 days, keep one backup every 7 days (30:7).
  • After 365 days, keep one backup every 30 days (365:30).

Before the first interval (i.e. by default within the first 24h) it is implied that all backup sets are kept. Additionally, if the backup destination directory is full, the oldest backups are deleted until enough space is available."

5. References

[1]: https://github.com/laurent22/rsync-time-backup "Rsync time backup"
[2]: https://rsync.samba.org/ "The rsync web pages"