How to test that your Borg backups actually restore

Why "the backup ran" is not enough

When borg create exits with code 0, the files it read were stored in the repository. That is all. Between that moment and the day you need a restore, a lot can go wrong:

• The repository gets damaged later — a failing disk, a bad NAS, a sync tool that copied a half-written segment.
• The important data was never in it — a path was renamed, an exclude pattern was too wide, or the database dump it picked up was empty.
• Some files were skipped — Borg exits with code 1 (a warning) when a file could not be read or changed while it was read. Many scripts treat that as success.
• You can't open it — the passphrase, or with keyfile encryption the key file itself, lives only on the server that just died.

The only test that covers the whole chain is the one you'll do on a bad day: take data out of the repository and check that it's right.

Quick manual test: extract one file

Set the repository and passphrase the same way your backup job does, then find the newest archive:

export BORG_REPO=/srv/borg
export BORG_PASSCOMMAND="cat /root/.borg-pass"

borg list --last 1 --short

Extract a single file to stdout and compare it with the original. Borg stores paths without the leading slash, so /etc/hostname is etc/hostname inside the archive:

ARCHIVE=$(borg list --last 1 --short)
borg extract --stdout "::$ARCHIVE" etc/hostname | cmp - /etc/hostname && echo "restore OK"

Nothing is written to disk. If the path is not in the archive, Borg says Include pattern 'etc/hostname' never matched and exits with a warning.

This only works for a file that hasn't changed since the backup. A file that changes all the time will "fail" even when the backup is fine — which is why the next step uses a file you control.

Make it reliable: a canary file

A canary is a small file you write right before each backup, with content you can predict. After the backup, you read it back from the new archive. If it matches, you know three things at once: the backup ran, it contains this run's data, and the repository can give data back.

# before the backup: write the canary
date -u +%Y-%m-%dT%H:%M:%SZ > /var/lib/borg-canary

# the backup itself — make sure the canary's path is included
borg create "::{hostname}-{now}" /etc /home /var/lib/borg-canary

# after the backup: read it back from the newest archive
borg extract --stdout "::$(borg list --last 1 --short)" var/lib/borg-canary | cmp - /var/lib/borg-canary

If several machines share one repository, limit the lookup to this machine's archives: borg list --glob-archives "$(hostname)-*" --last 1 --short.

Automate it

Put the whole sequence in one script and let it shout when something is off. This version mails root; swap the alert function for whatever you use (a chat webhook, a monitoring push).

#!/bin/bash
set -uo pipefail
export BORG_REPO=/srv/borg
export BORG_PASSCOMMAND="cat /root/.borg-pass"
CANARY=/var/lib/borg-canary

alert() { echo "$1" | mail -s "borg on $(hostname): $1" root; }

date -u +%Y-%m-%dT%H:%M:%SZ > "$CANARY"

borg create "::{hostname}-{now}" /etc /home "$CANARY"
rc=$?
if [ $rc -eq 1 ]; then
  alert "backup finished with warnings — some files may be missing"
elif [ $rc -ne 0 ]; then
  alert "backup failed (exit $rc)"; exit 1
fi

LAST=$(borg list --glob-archives "$(hostname)-*" --last 1 --short)
if ! borg extract --stdout "::$LAST" "${CANARY#/}" | cmp -s - "$CANARY"; then
  alert "restore test FAILED: the canary did not come back"
  exit 1
fi

Run it from a systemd timer or cron at the time your backup already runs. Keep a copy of the passphrase somewhere other than this server — and with keyfile encryption, the key too (borg key export). A restore test on the server proves nothing about the day the server is gone.

What this test does not prove — and borg check

Extracting one file reads only the chunks that hold it. It proves the chain works today, not that every byte in the repository is intact. Borg has three levels of checking for the rest:

• borg check — checks the repository's segments and the metadata of every archive. On a remote repository the segment part runs on the storage server, so it costs little bandwidth. Run it after every backup or daily.
• borg check --verify-data — also reads and decrypts every data chunk and verifies it. This is the only check that proves every archive restores. It reads the whole repository, so on a big or remote one it takes hours and downloads everything. Run it weekly or monthly.
• borg check --repository-only --max-duration 3600 — checks the segments for up to an hour and continues where it stopped on the next run. It spreads the segment check over nights, but does not decrypt the data and can't be combined with --verify-data.

Never add --repair just because a check failed — read what it found first. Repair can throw away damaged data for good.

If you use borgmatic

borgmatic runs the checks for you when you list them in its configuration, each with its own frequency:

checks:
  - name: repository
    frequency: always
  - name: archives
    frequency: always
  - name: extract
    frequency: 2 weeks
  - name: data
    frequency: 1 month

extract does a dry-run extract of the newest archive: it reads everything back and writes nothing. data is borg check --verify-data. As with any test, make sure a failure reaches a person — borgmatic's on_error hook or its monitoring integrations do that.