How to monitor restic backups and get alerts when they stop

What breaks without anyone noticing

restic itself has no notifications. It runs from cron or a systemd timer, and errors go to a log or a local mailbox nobody reads. The failures that hurt are the quiet ones:

• The job stops running — a timer disabled during maintenance, a cron file lost in a server move. No run means no error message.
• The job can't open the repository — the password file moved, the mount isn't there after a reboot, credentials for the bucket expired.
• A stale lock — a run was killed and left a lock behind; later runs fail with "repository is already locked".
• A disk fills up — the repository disk or restic's cache directory.
• A backup hangs — a stuck network mount, and the job never finishes.
• Files are skipped — exit code 3: the snapshot was made, but some files weren't readable.
• The backup runs but saves nothing useful — the source path changed and every new snapshot is the same as the last one.

Layer 1: don't swallow exit codes

restic exits 0 on success, 1 on a fatal error and 3 when the snapshot is incomplete because some files couldn't be read. Treat 3 as a warning, not as success:

restic backup /etc /home
rc=$?
[ $rc -eq 3 ] && alert "some files were not read"
[ $rc -ne 0 ] && [ $rc -ne 3 ] && alert "backup failed (exit $rc)"

With systemd, OnFailure= in the backup unit can start a separate notification unit whenever the job fails.

Exit codes only help when the job runs. They say nothing when it never starts or hangs forever.

Layer 2: check that the newest snapshot is fresh

The question that matters is "how old is the newest snapshot?" — and the answer should come from a separate timer, so it still fires when the backup job doesn't. This needs jq:

#!/bin/bash
set -uo pipefail
export RESTIC_REPOSITORY=/srv/restic-repo
export RESTIC_PASSWORD_FILE=/root/.restic-pass
MAX_AGE_HOURS=26   # daily backups, plus some slack

last=$(restic snapshots --json --latest 1 --host "$(hostname)" \
  | jq -r 'map(.time | sub("\\.[0-9]+"; "")) | max // empty') || {
  alert "cannot read the repository"; exit 1; }

if [ -z "$last" ]; then
  alert "no snapshots for this host"; exit 1
fi

age_h=$(( ( $(date +%s) - $(date -d "$last" +%s) ) / 3600 ))
if [ "$age_h" -gt "$MAX_AGE_HOURS" ]; then
  alert "newest snapshot is ${age_h}h old"
fi

This catches a stopped timer, a hanging backup and a broken repository connection. It also tells you when the repository can't be opened at all — which is exactly the day you'd find out otherwise.

Layer 3: a dead man's switch

Both checks above run on the same server as the backup. If that server is down, nothing alerts. A dead man's switch flips it around: the backup pings a URL after each success, and a separate service alerts you when the pings stop. Healthchecks.io works this way, and so do the push monitors in Uptime Kuma, if you'd rather host it yourself.

restic backup /etc /home && curl -fsS -m 10 --retry 3 https://your-monitor.example/ping/backup-web01

Layer 4: prove the data comes back

A fresh snapshot is not the same as a restorable one. Add a restore test after each backup — a canary file read back from the newest snapshot — and re-read the repository's data in nightly slices with restic check --read-data-subset.

Where it gets heavy

On one server, four layers are a couple of short scripts. On twenty, they're twenty sets of scripts, timers and alert settings to keep in step — and a check that silently stopped running looks exactly like a check that's passing.