Backing up my BTRFS system

[lyallp]

I have a btrfs filesystem as follows (I have trimmed out irrelevant filesystems) :-

Code: Select all

Filesystem         Size  Used Avail Use%         Mounted on     
/dev/sdb4          434G  113G  318G 26.0 [#....] /
/dev/sdb4          434G  113G  318G 26.0 [#....] /home
/dev/sdb4          434G  113G  318G 26.0 [#....] /var

At this time, I have a USB Dock into which I plug, on a sporadic but less than weekly, schedule, multi-TB SATA drives which auto-mount to my system.

Whilst I really should store one of these 3 SATA drives elsewhere, all 3 currently live on my window sill, next to my computer desk.

Then, I have a script, shown below which I execute, by hand, because it relies on my external drive to be mounted.

Code: Select all

./SystemBacukp.sh /mnt/usb/Data_01

I have used these backups on multiple occasions to restore stuff that I have accidentally deleted or generally messed up.

I was wondering if there are any suggestions as to a better way to do it.

Code: Select all

#!/bin/bash 
backupFilesystem="${1:-/mnt/usb/Backup}"

if [ ! -d "${backupFilesystem}" ]
then
    echo "$0: ${backupFilesystem} is not mounted."
    exit 1
fi

echo "Starting backup at $(date) to ${backupFilesystem}"
backupDir="${backupFilesystem}/Backups"

SNAPSHOT="$(date +'%Y-%m-%d')"
echo "Snapshot = ${SNAPSHOT}"
snapshotsDir="/.snapshots/${SNAPSHOT}"

function CreateSnapshots
{
    if [ -d /.snapshots/* ]
    then
	echo "Removing previous snapshots..."
	# remove any snapshots left over from an aborted previous run
	for s in /.snapshots/*/*
	do
            echo /sbin/btrfs subvolume delete "${s}"
            /sbin/btrfs subvolume delete "${s}"
	done
	echo rm -fr /.snapshots/*
	rm -fr /.snapshots/*
   fi

    # Want to backup boot, which is not normally mounted
    if ! mountpoint -q /boot
    then
        /bin/mount /boot
    fi
    
    mkdir -p "${snapshotsDir}"
    # create  a read only snapshot
    /sbin/btrfs subvolume snapshot -r /		    ${snapshotsDir}/root
    /sbin/btrfs subvolume snapshot -r /home	    ${snapshotsDir}/home
    /sbin/btrfs subvolume snapshot -r /var	    ${snapshotsDir}/var
}

function RemoveSnapshots
{
    /sbin/btrfs subvolume delete ${snapshotsDir}/root
    /sbin/btrfs subvolume delete ${snapshotsDir}/home
    /sbin/btrfs subvolume delete ${snapshotsDir}/var  
    rmdir "${snapshotsDir}"
    if mountpoint -q /boot
    then
	# no longer need /boot mounted
	/bin/umount /boot
    fi
}

CreateSnapshots

# Remove snapshots regardless of exit status
trap "RemoveSnapshots" EXIT

snapshots="/.snapshots/${SNAPSHOT}"

echo "Starting backup at $(date)"

for fileSystemName in root home var boot
do
    
    case "${fileSystemName}" in
	( "root" ) fileSystem="${snapshotsDir}/root" ;;
    	( "home" ) fileSystem="${snapshotsDir}/home" ;;
    	( "var"  ) fileSystem="${snapshotsDir}/var"  ;;
    	( "boot" ) fileSystem="/boot"                ;;
	( * ) echo  "$0:Invalid Filesystem '${fileSystemName}'" >&2
	      exit 1
	      ;;	
    esac

    echo "Starting backup of ${fileSystem} at $(date)"

    backupFile="${backupDir}/${fileSystemName}.tar.bz2"
    previousBackupDir="${backupDir}/${fileSystemName}.prev"
    #
    ## #########################################################################
    ## Backup previous backups in a directory - we don't really know how many
    ## files there are.
    ## #########################################################################
    #
    /bin/mkdir --parents "${previousBackupDir}"
    #
    ## #########################################################################
    ## Remove any previous backups
    ## #########################################################################
    #
    /bin/rm --force "${previousBackupDir}"/*
    #
    ## #########################################################################
    ## Save the current ones away
    ## #########################################################################
    #
    /bin/mv --force "${backupDir}"/${fileSystemName}.tar.bz2* "${previousBackupDir}"
    #
    ## #########################################################################
    ## Backup the filesystem into 1gb chunks
    ## #########################################################################
    #
    /bin/tar --one-file-system  --create --file - "${fileSystem}" | /bin/bzip2 --stdout | /usr/bin/split --bytes=1000m - "${backupFile}."
    echo "Verifying backup of ${fileSystem} at $(date)"
    /bin/cat "${backupFile}".* | /bin/tar jtf - > /dev/null 2>&1 
    rc=$?
    if [ "$rc" -eq 0 ]
    then
        echo "Completed backup of ${fileSystem} at $(date)"
    else
        echo "**FAILED* backup of ${fileSystem} at $(date) - Did not verify!"
    fi
done

# unmount usb disks as they seem to freeze after a period of disuse

/bin/mount -l | /bin/grep '/mnt/usb' | while read device on dir desc ; do /bin/umount -l ${dir} ; done

echo "Completed backup at $(date)"

[pingtoo]

You need to define what is "better" in your mind.

It is not clear what is the concern, is manually execution is concern? is power saving is concern? is data lost is concern?, etc...

backup is big topic, but one that is well understood. So you can list what you have in mind and give priority on importance to you I am sure someone can give you exact answer you want.

[lyallp]

I guess, I have seen talk of backup tools for BTRFS though, as I understand it, they are still in development.

I was wondering if my old-school backup technique could be improved using more modern tools than tar/gzip/split/cat....

My existing system works fine, and I also put it up there just in case anyone else should want to 'leverage' what I have done.

Really, my process must stay manual because I have to manually mount the bare SATA drives in my USB SATA hub first, do the backup, then pull the bare SATA drive from the USB hub and store it.
The fact that I can successfully restore individual files/directories has been proven.
I am lazy and the bare SATA drives are simply sitting on my window sill instead of remote or in a safe.
I also have a weekly backup that uses the same Script (via Cron) to my NAS, thus keeping 2 generations on my NAS, as well as 2 generations on each bare SATA drive (of which there are 3).

I am open to suggestions, however.

Possibly a tool more targeted at BTRFS rather than the snapshot/tar create/tar verify method I use now...

Should I schedule a regular 'tune-up' of my BTRFS system? If so, how?

My system only really reboots when I install a new kernel (which I build from gentoo-sources)

[szatox]

The biggest problem I see with your current setup is that you're doing those backups manually. Manual backup means you will end up without any backup when you need it.
I'd say set up those snapshots as a cron job to let it actually run on schedule, regardless of backup disks being connected or not. This will take care of the vast majority of mistakes you might make, mostly leaving hardware failures to be taken care of. Then, once the backup disk is connected, it's finally time to copy over all the snapshots made since the last backup.
I'd rather copy it over network to NAS instead of rotating spare disks manually though.

And since you asked about btrfs-specific stuff, apparently btrfs can send and receive snapshots, and even send incremental snapshots. This could be more convenient and maybe even faster than tar. Still, if it works, it works; there's nothing wrong with using generic solutions.

[pingtoo]

lyallp,

Taking szatox suggestion in to consideration and together with your script and with your storage (/dev/sdb).

I suggest you separate your current implementation into two parts, One schedule regular snapshot (think TimeMachine), this should help any human mistake should it happen and give quick recovery time., Two, convert existing snapshots into tarball and move it to offline location. This should give you recovery point whenever you have hardware failure or you want to migrate to new hardware with know condition.

Using snapshot with cron in regular interval (hourly) will not take up too much of space and you can move it to your USB storage whenever you find it is convenience. And you can decide if you want all the point in time, or just the latest.

Using a existing "backup tools" is convenient, but if you have full control of how the backup "work" would it be even better?

[lyallp]

Thanks for taking the time to reply.

I will look into the regular snapshots to local disk and copying to my external USB drives on connection, good idea!

[RumpletonBongworth]

The script could be a little better as it contains several bugs.

Firstly, test -d does not prove that the given path is a mountpoint. Consider using mountpoint(1) as you do elsewhere.

Code: Select all

if ! mountpoint -q -- "${backupFilesystem}"

Secondly, the way in which directories are checked for the absence of any subdirectories is incorrect on multiple counts. I won't go into them all, but in particular:

Code: Select all

$ mkdir dir && touch dir/{a,b} && [ -d dir/* ]
bash: [: dir/a: binary operator expected

As regards the CreateSnapshots function, no such test is needed (it would only amount to a TOCTOU). Instead, the function could be written in this fashion:

Code: Select all

#!/bin/bash

# Prevent globs from expanding as their literal selves, should they match no pathnames.
shopt -s nullglob

function CreateSnapshots {
    local i

    # Trailing slash coerces the expansion of directories only.
    # The nullglob option prevents the loop from doing anything if nothing is matched.
    for s in /.snapshots/*/*/
    do
        if (( i++ == 0 ))
        then
             echo "Removing previous snapshots..."
        fi
        btrfs subvolume delete "${s%/}"
    done

    # The following command can be unconditional, owing to -f.
    rm -fr /.snapshots/*/
}

Thirdly, as regards the use of external commands whose arguments are controlled by pathname expansion, there is a risk of exceeding the ARG_MAX limit. You might never encounter it because the limits imposed by Linux are comparatively generous. Still, it is worth keeping in mind.

Code: Select all

# Execution could fail if the cumulative length of the matched paths is great enough.
rm -fr /.snapshots/*/

# Alternative 1 (builtins cannot be stymied by ARG_MAX)
printf '%s\0' /.snapshots/*/ | xargs -r0 rm -fr

# Alternative 2
find /.snapshots -mindepth 1 -maxdepth 1 -type d -exec rm -fr {} +

Fourthly, it's lacking somewhat in the error handling department. Consider which commands are vital for the program to be able to meaningfully continue. For those, appending "|| exit" can make for a reasonable error handling strategy. This can also be done for batches of commands. For example:

Code: Select all

# Should exit be reached, the exit status will be that of the first failing command.
critical_command1 \
&& critical_command2 \
&& critical_command3 \
|| exit

Handling pipelines requires some thought. One must consider which of the commands comprising the pipeline convey an exit status code that is material. If the answer is "not all, but not merely the last", the PIPESTATUS array can be consulted. If the answer is "all of them", temporarily enabling pipefail may be more convenient. An example of the latter case would be:

Code: Select all

# Constrain the scope of pipefail through the use of a subshell.
(
	set -o pipefail
	
	tar --one-file-system  --create --file - "${fileSystem}" \
	| /bin/bzip2 --stdout \
	| /usr/bin/split --bytes=1000m - "${backupFile}."
) || exit

That being said, if split(1) were to fail there, it is all but certain that bzip2(1) would in turn (SIGPIPE being raised upon writing to a defunct pipe). So the above can be simplified as:

Code: Select all

tar --one-file-system -cj "${fileSystem}" | split --bytes=1000m - "${backupFile}."
(( PIPESTATUS[0] == 0 )) || exit # checks the status of bzip2(1)

...

Moving on to matters of style, consider not fully qualifying the paths of external utilities. Not only does it render scripts tedious to read and maintain but it's a fine way of making scripts break and behave unexpectedly across different operating environments. You should be able to trust the value of PATH. If you expect to run the script in an operating environment that does not define PATH sensibly - such as that which is established by cronie - either define PATH at the top of the script or - better yet - define it at the top of the applicable crontab:

Code: Select all

PATH=/sbin:/bin:/usr/sbin:/usr/bin

Given that its containing loop iterates over a static sequence of words, the case command serves no purpose. It can be replaced in its entirety with:

Code: Select all

fileSystem="${snapshotsDir}/${fileSystemName}"

Some of the parameter expansions are missing double quotes. The shellcheck utility makes light work of such matters.

[Asch]

If you ask me, if it ain't broken, don't fix it.

As someone pointed out already, you probably could make it faster with the command btrfs send/receive. However, data compression could become a little bit more complicated. I wouldn't try this if your backup process isn't taking long.

You surely could automate this. However, you would need to keep your backup disks readily connectable to your main system. As I see it, having offline backups is pretty important for data security.

You surely could automate it, but I would recommend still keeping some manual procedure to have some offline backup safely stored somewhere.

Besides, as some put it, having a single backup is like having no backup at all. A combination of snapshots, automated backups and offline copies would be optimal, but could also mean higher storage costs, especially if you are handling terabytes of data.