❄ Backup to cold storage with BTRFS ❄

BTRFS has default mechanisms to send/receive¹ snapshot between different machines. This is an elegant way to perform backups. Unfortunately, we are not there yet. But we can do differently!

Rationale

When you self-host your services at home, it is easy to forget that all your data are co-located with the place you live. If anything happens to your place, a fire, a flood, you name it, your data could be gone for good.
That is why it is better to have an external backup. Several options exists, I chose cold storage² provided with a third party.

Cold storage is pretty cheap and quite good in term of preserving your data. However is not not without its cons:

• You pay by amount of stored data per month. Fairly often, you also pay transferred data too. Better not store duplicate information!
• It is a third party provider. A leak can happen.
• You are dependent on protocols offered by your provider for transfer.

Of course, we have an answer to mitigate the cons, making choosing a third party provider quite a satisfactory option:

• As we pay by amont of data, we should do incremental backup.
• As we shouldn’t trust third party provider completly, we should encrypt our backup before sending it to the wire.
• As for the protocols, if you have luck with your provider, you will be able to use swift API³. This is not send/receive workflow but we have little choice here. This will have to do.

Tooling

As I looked for solutions to answer my narrow use case of backuping my btrfs snapshots to cold storage with incremental and encryption support I wasn’t satisfied with existing tools.

This is why I decided to come up with a tool of my own, which I share here on BTRFS snapshot to swift repository.

It synchronizes subvolume snapshots with swift, defaults to incremental snapshot, optionally support encryption with age⁴, plays well both in interactive and non-interactive scenario
I use my tool in conjunction with systemd capabilities for my backup automation.

A use case with more elaborated examples

Consider you have a system with a btrfs filesystem, subvolumes, and you regularely snapshot these subvolumes, each respectively located in:

• /fs
• /fs/subvol.n
• /fs/snapshots/subvol.n/<date>

⚠ Automating snapshot creation is not detailed here. We consider you already have automated local snapshot creation.

Install btrfs-snapshot-to-swift⁵

The script is not (yet?) available in PyPI, I recommand installing it from repository using pipx for current user and root.

git clone --branch v1.0.2 https://github.com/Nimamoh/btrfs-snapshot-to-swift.git
cd btrfs-snapshot-to-swift
pipx install -f --system-site-packages --editable (pwd)
sudo pipx install -f --system-site-packages --editable (pwd)

⚠ The script relies on libbtrfs python bindings which generally comes along with btrfs. You can check if it is available on your system using pydoc btrfsutil. This is why --system-site-packages is important here, otherwise btrfs python bindings won’t be available.

Provide credentials to your storage

If your provider support access through swift API, you should be able to provide your credentials through environment variables. I’m afraid you will have to see with your third party provider documentation to know how to retrieve it.

For example, for OVH: OVH documentation

You will also need the container name in which you will store your backups.

Once your credentials are corrently loaded and btrs-snapshot-to-swift is correctly installed, you should be able to successfully run a --dry-run command:

sudo btrfs-snapshot-to-swift --dry-run --container-name <container_name> /fs/subvol.n

Encryption with age⁴

Age allows you to encrypt files easily and our script can interact with it, I’ll let you look at the documentation. Briefly, here are the steps to setup age:

• Install age via the method you want.
• Generate an age key at ~/key.age

age-keygen > ~/key.age

Make sure you have permanent access to your encryption key since loosing it will mean not being able to restore your backup

Note the public key (also named recipient) example:

age1vmcfjk7nyu3zk97kxq89hy3x93h5fc0cgpgnwq6rxdajqy2c3s9qs7euqn.

We will need the recipient to enable encrypting files in further steps.

Wrapper script and systemd setup

In order to simplify and streamline usage of btrfs-snapshot-to-swift we can wrap it, for example in /usr/local/bin/archive-subvol:

#!/usr/bin/env fish

set -lx subvol_path $argv[1]
if not set -q subvol_path
    echo >&2 "Define subvolume beforehand!"
    exit 1
end

#
# Not detailed here, but this wrapper script is a good place to load
# your swift credentials.
#

# Since installed with pipx, the tool is in /root/.local/bin
# This script will be called with systemd, that's why we log directly to syslog (optional)
/root/.local/bin/btrfs-snapshot-to-swift \
    --syslog \
    -v \
    --container-name replace_with_your_container_name \
    --age-recipient age1vmcfjk7nyu3zk97kxq89hy3x93h5fc0cgpgnwq6rxdajqy2c3s9qs7euqn \
    "$subvol_path"

which we will be able to call with archive-subvol /fs/subvol.n

We will want to integrate the tool with systemd, for that, we will write a .service and .timer. Both will be taking path to subvolume as parameter.

In /etc/systemd/system/archive-subvol@.service

[Unit]
Description=Archive snapshots of subvolume %f

[Service]
ExecStart=bash -c 'archive-subvol "%f"'
Type=oneshot

And in /etc/systemd/system/archive-subvol@.timer

[Unit]
Description=Monthly archive snapshot of %f

[Timer]
OnCalendar=monthly
AccuracySec=1d
RandomizedDelaySec=1w
Persistent=true

[Install]
WantedBy=timers.target

Don’t forget to sudo systemctl daemon-reload.

Now, you should be able to trigger the backup of /fs/subvol.n snapshots to your distant cold storage via:

sudo systemctl start archive-subvol@fs-subvol.n.sevice

and schedule archiving monthly with:

sudo systemctl enable --now archive-subvol@fs-subvol.n.timer

Restoring snapshots

The backups are stored in incremental fashion, each depending on the previous one until its origin.
Also, each backup is named following this pattern <PARENT_VOL_UUID>\x2f<rel_path>.
To restore backup of a snapshot you will have to run btrfs receive on each snapshot backup in chronological order. Before doing so, you will have to decrypt the file with your age key.

For example, let’s say you have already backup’d three snapshot for a subvolume:

You will have to retrieve the backup from your provider, once on your restoration disk, for each backup file:

• decrypt:

age -d -i ~/key.age backup_filename > backup_filename.decrypted

• restore with receive:

# /fs being your btrfs filesystem which will receive bakup'd data
sudo btrfs receive -f backup_filename.decrypted /fs

Conclusion

If you found this content useful, spotted an error or something else, feel free to let me know with a comment.
For improve request on the script please do so on the github repository⁵