❄ Backup to cold storage with BTRFS ❄
BTRFS has default mechanisms to send/receive1 snapshot between different machines. This is an elegant way to perform backups. Unfortunately, we are not there yet. But we can do differently!
When you self-host your services at home, it is easy to forget that all your data are co-located with the place you live. If anything happens to your place, a fire, a flood, you name it, your data could be gone for good.
That is why it is better to have an external backup. Several options exists, I chose cold storage2 provided with a third party.
Cold storage is pretty cheap and quite good in term of preserving your data. However is not not without its cons:
- You pay by amount of stored data per month. Fairly often, you also pay transferred data too. Better not store duplicate information!
- It is a third party provider. A leak can happen.
- You are dependent on protocols offered by your provider for transfer.
Of course, we have an answer to mitigate the cons, making choosing a third party provider quite a satisfactory option:
- As we pay by amont of data, we should do incremental backup.
- As we shouldn’t trust third party provider completly, we should encrypt our backup before sending it to the wire.
- As for the protocols, if you have luck with your provider, you will be able to use swift API3. This is not send/receive workflow but we have little choice here. This will have to do.
As I looked for solutions to answer my narrow use case of backuping my btrfs snapshots to cold storage with incremental and encryption support I wasn’t satisfied with existing tools.
This is why I decided to come up with a tool of my own, which I share here on BTRFS snapshot to swift repository.
It synchronizes subvolume snapshots with swift, defaults to incremental snapshot, optionally support encryption with age4, plays well both in interactive and non-interactive scenario
I use my tool in conjunction with systemd capabilities for my backup automation.
A use case with more elaborated examples
Consider you have a system with a btrfs filesystem, subvolumes, and you regularely snapshot these subvolumes, each respectively located in:
⚠ Automating snapshot creation is not detailed here. We consider you already have automated local snapshot creation.
The script is not (yet?) available in PyPI, I recommand installing it from repository using pipx for current user and root.
git clone --branch v1.0.2 https://github.com/Nimamoh/btrfs-snapshot-to-swift.git cd btrfs-snapshot-to-swift pipx install -f --system-site-packages --editable (pwd) sudo pipx install -f --system-site-packages --editable (pwd)
⚠ The script relies on libbtrfs python bindings which generally comes along with btrfs. You can check if it is available on your system using
pydoc btrfsutil. This is why
--system-site-packagesis important here, otherwise btrfs python bindings won’t be available.
Provide credentials to your storage
If your provider support access through swift API, you should be able to provide your credentials through environment variables. I’m afraid you will have to see with your third party provider documentation to know how to retrieve it.
For example, for OVH: OVH documentation
You will also need the container name in which you will store your backups.
Once your credentials are corrently loaded and btrs-snapshot-to-swift is correctly installed, you should be able to successfully run a
sudo btrfs-snapshot-to-swift --dry-run --container-name <container_name> /fs/subvol.n
Encryption with age4
Age allows you to encrypt files easily and our script can interact with it, I’ll let you look at the documentation. Briefly, here are the steps to setup age:
- Install age via the method you want.
- Generate an age key at
age-keygen > ~/key.age
Make sure you have permanent access to your encryption key since loosing it will mean not being able to restore your backup
Note the public key (also named recipient) example:
We will need the recipient to enable encrypting files in further steps.
Wrapper script and systemd setup
In order to simplify and streamline usage of btrfs-snapshot-to-swift we can wrap it, for example in
#!/usr/bin/env fish set -lx subvol_path $argv if not set -q subvol_path echo >&2 "Define subvolume beforehand!" exit 1 end # # Not detailed here, but this wrapper script is a good place to load # your swift credentials. # # Since installed with pipx, the tool is in /root/.local/bin # This script will be called with systemd, that's why we log directly to syslog (optional) /root/.local/bin/btrfs-snapshot-to-swift \ --syslog \ -v \ --container-name replace_with_your_container_name \ --age-recipient age1vmcfjk7nyu3zk97kxq89hy3x93h5fc0cgpgnwq6rxdajqy2c3s9qs7euqn \ "$subvol_path"
which we will be able to call with
We will want to integrate the tool with systemd, for that, we will write a
.timer. Both will be taking path to subvolume as parameter.
[Unit] Description=Archive snapshots of subvolume %f [Service] ExecStart=bash -c 'archive-subvol "%f"' Type=oneshot
[Unit] Description=Monthly archive snapshot of %f [Timer] OnCalendar=monthly AccuracySec=1d RandomizedDelaySec=1w Persistent=true [Install] WantedBy=timers.target
Don’t forget to
sudo systemctl daemon-reload.
Now, you should be able to trigger the backup of
/fs/subvol.n snapshots to your distant cold storage via:
sudo systemctl start firstname.lastname@example.org
and schedule archiving monthly with:
sudo systemctl enable --now email@example.com
The backups are stored in incremental fashion, each depending on the previous one until its origin.
Also, each backup is named following this pattern
To restore backup of a snapshot you will have to run
btrfs receive on each snapshot backup in chronological order. Before doing so, you will have to decrypt the file with your age key.
For example, let’s say you have already backup’d three snapshot for a subvolume:
|Original path||Backup filename|
You will have to retrieve the backup from your provider, once on your restoration disk, for each backup file:
age -d -i ~/key.age backup_filename > backup_filename.decrypted
- restore with receive:
# /fs being your btrfs filesystem which will receive bakup'd data sudo btrfs receive -f backup_filename.decrypted /fs
If you found this content useful, spotted an error or something else, feel free to let me know with a comment.
For improve request on the script please do so on the github repository5
Knowing so, if you think you found an error or inexact content, you are more than welcome to notify it through comment below ⏬.
Also, if you found the content useful and it helped you, consider leaving a comment too or, better, give me fuel buying me a coffee with the link on the top of the website. 🙏