Automated Incremental Backups

Tired of time consuming and fiddly backing-up of your Raspberry Pi’s SD card? This guide will provide you with care-free, automated incremental backups of your system’s images, to a permanently attached USB flash drive.

In case of irrecoverable events, the images stored on the USB flash drive, may be used to re-flash your SD card, using your favourite image writing software, on any Windows or Linux machine.

Step 1: Add a USB Flash Drive That Automatically Mounts Itself.

Because we will be dealing with files larger than the 4GB limit of the FAT32 file system, and because we will be restoring images on either a Windows or Linux machine, your USB flash drive must be formatted with the NTFS file system. NTFS permits us the storage of file permissions, and is a more fault tolerant journaling file system.

Your USB flash drive must also be large enough to accommodate at least one image the same size as your Raspberry Pi’s SD card, and you will benefit from having extra room to accommodate compressed backup image ‘snapshots’. For my 16GB system SD card, I chose a ludicrously inexpensive 32GB USB 3.0/2.0 flash drive.

Okay, to mount the NTFS drive, you’ll need to install driver support:

sudo apt-get update
sudo apt-get install ntfs-3g

If you insert the USB flash drive, then list block storage devices, you should see the device, along with it’s NTFS volume. The volume of interest in my case is /dev/sda1:

lsblk

Mount the USB flash drive’s NTFS volume under the /media directory:

sudo mount /dev/sda1 /media

Establish the unique universal id (uuid) of the flash drive: (Be aware the uuid will change any time you reformat the USB flash drive, requiring subsequent re-configuration.)

ls -l /dev/disk/by-uuid

Find out the group id (gid) of pi user:

id -g pi

Obtain the user id (uid) of pi user:

id -u pi

Now, to have the system automatically mount the USB flash drive any time you reboot, we will append a line to the /etc/fstab configuration file, filling in the required information ascertained in preceding steps:

sudo nano /etc/fstab
In nano, fill-out the following line, then append it to the end of the file:

UUID=enter_uuid_here /media ntfs-3g ofail,uid=enter_uid_here,gid=enter_gid_here,noatime 0 0

To check if the new configuration works, we must first un-mount the USB flash drive. The gotcha here is that if your current working directory is anywhere in or under /media, then you have to cd elsewhere, or you’ll get a device busy error:

sudo umount /dev/sda1

Now, to automatically mount all the system’s drives mentioned in /etc/fstab:

sudo mount -a

Check that the NTFS volume on the flash drive is mounted, and now has a mount point under /media with:

lsblk

Now reboot your system, and check again that the USB flash drive has been automatically mounted (per above):

sudo reboot

Step 2: Install the Backup Script, and Automate With a Cron Job.

We’ll be installing the shell script bkup_rpimage.sh v1.0, written by the masterful jinx, and made available by other Raspberry Pi forum users on Github.

Firstly, if you don’t already have a bin directory under your home folder, then here’s how to get one…for free! Your .profile configuration will automagically add this directory to your PATH the next time you login, but we won’t rely on that here.

mkdir /home/pi/bin

Next, download the backup script into your bin directory:

cd /home/pi/bin
wget https://raw.githubusercontent.com/lzkelley/bkup_rpimage/master/bkup_rpimage.sh

So it does not become accidentally overwritten, and to make the script executable:

chmod 554 bkup_rpimage.sh

Okay, so now we do our first full manual backup with the new script. Be patient, as this may take anywhere between 20 and 90+ minutes, depending on your file system size, and system’s capabilities:

sudo /home/pi/bin/bkup_rpimage.sh start -c /media/rpi_backup.img

Looking in your /media directory, you should now see an rpi_backup.img file, of about the same size as your SD card. This is the image you will re-flash to your SD card with Etcher, Win32_Disk_Imager, etc., if ever the need arises:

ls -l /media

Subsequently executing the backup command listed above, will do an incremental update to this backed-up image, taking very little time, and only making changes that reflect those made to the SD card since the last full or incremental backup.

Instead of manually invoking the backup script for the update however, we’re going to add a line to our cron configuration, and have the system automatically perform the update for us, each day at midnight:

sudo crontab -e
In the crontab editor, add the following line to the bottom of the file:

0 0 * * * sudo /home/pi/bin/bkup_rpimage.sh start -c /media/rpi_backup.img

See ‘man 5 crontab’ if you’d like to know how to adjust the time or frequency of system image backup updates.

The bkup_rpimage.sh v1.0 shell script provides options for creating compressed backup images of system files, but not in a format useful under Windows. We’ll be installing support for .zip file compression, so compressed image backups may be recovered with either Windows or Linux.

Install zip program for Windows compatible compressed backup images:

sudo apt-get update
sudo apt-get install zip

Compressing backup images places a considerable burden on system resources, especially if the operation is automated and undertaken very regularly. Instead of this, we’ll be compressing backup images manually, and only when the need for ‘snapshots’ arises.

Manually compress a date-stamped snapshot of the backup image, as a background process:

zip /media/rpi_backup.img.$(date +%Y-%m-%d).zip /media/rpi_backup.img &

Compressed ‘snapshots’ of system file backup images should now be available on your USB flash drive, along with the most recent, uncompressed backup image. To gain access to these, simply un-mount the USB drive, then move the USB drive to either a Windows or Linux machine, for SD card re-flashing, using Etcher, Win32_Disk_Imager, etc.:

sudo umount /dev/sda1

Step 4: Retrieving Individual Files.

Say you accidentally overwrite an important file that you know is on your last backup. You don’t want to lose the entire days work since your last backup, so restoring the whole file-system is not an option. How can you access just the lost file?

The backup script bkup_rpimage.sh has options that allow you to easily mount and dismount a backed-up image file, to and from your file-system. Once mounted, you can readily traverse the mounted backup’s tree, seeking any file contained therein:

To mount the backed-up image under /mnt:

sudo /home/pi/bin/bkup_rpimage.sh mount /media/rpi_backup.img /mnt/

Notice that files in the backup image are now attached, and accessible under /mnt:.

cat /mnt/home/pi/.bashrc

…and to un-mount the backup image on the USB drive from the file-system we issue:

sudo /home/pi/bin/bkup_rpimage.sh umount /media/rpi_backup.img /mnt/

Step 5: Some Scripts You May Find Useful.

Rather than trying to remember or lookup the backup system’s incantations, you might find it worthwhile adding these 4 simple scripts to your ~/bin directory:

nano /home/pi/bin/manual_backup.sh

#!/bin/bash
sudo /home/pi/bin/bkup_rpimage.sh start -c /media/rpi_backup.img

nano /home/pi/bin/compress_backup.sh

#!/bin/bash
zip /media/rpi_backup.img.$(date +%Y-%m-%d).zip /media/rpi_backup.img &

nano /home/pi/bin/mount_backup.sh

#!/bin/bash
sudo /home/pi/bin/bkup_rpimage.sh mount /media/rpi_backup.img /mnt/

nano /home/pi/bin/unmount_backup.sh

#!/bin/bash
sudo /home/pi/bin/bkup_rpimage.sh umount /media/rpi_backup.img /mnt/

Please don’t forget to make these scripts read only and executable:

chmod 554 manual_backup.sh compress_backup.sh mount_backup.sh unmount_backup.sh

---