Skip to content

PR 4: backup Healthchecks.io dead-man's-switch monitoring

jedmund requested to merge jedmund/backup-monitoring into main

Created by: jedmund

Summary

Final PR in the backup series. Adds Healthchecks.io as the dead-man's-switch for nightly borgmatic runs. If 02:00 passes without a successful run plus a grace window, Healthchecks.io alerts via whatever channel you've configured on its side (email, ntfy, Slack, etc.).

This PR is stacked on PR 3 (jedmund/backup-ui). When PRs 2 and 3 merge, GitHub auto-rebases this onto main.

What it does

  • Adds borgmatic's native healthchecks: config block. Default pings on finish and fail states with send_logs: true so the dashboard shows the full run output.
  • borg_healthchecks_url lives in group_vars/backup/vault.yml. Passed through to the container as HEALTHCHECKS_PING_URL, referenced from borgmatic.yaml via {credential env HEALTHCHECKS_PING_URL}.
  • README walks through Healthchecks.io project setup (account, check, schedule, grace, notification channels).

What's not in this PR

  • Netdata Prometheus scrape of borgmatic metrics. Earlier research was ambiguous about whether the borgmatic-collective image actually exposes a /metrics endpoint. Deferred until verified; immediate value of dead-man's-switch alerting is higher than trend graphs anyway.

Setup checklist (after merge)

  1. Create Healthchecks.io account (free tier is plenty for one check)
  2. Create check: name nuc-mini borgmatic, cron schedule 0 2 * * *, grace 1 hour
  3. Copy the ping URL (https://hc-ping.com/<uuid>)
  4. Add to vault: 
    ansible-vault edit group_vars/backup/vault.yml
# add: borg_healthchecks_url: "https://hc-ping.com/<uuid>"
  5. make deploy-backup
  6. Trigger a manual borgmatic run, confirm the check turns green in Healthchecks.io dashboard

Commits

  1. backup: ping Healthchecks.io on every borgmatic run

Test plan

  • After deploy, manual borgmatic run causes the Healthchecks.io check to go green (success ping received)
  • Dashboard shows borgmatic output in the run history (confirms send_logs: true working)
  • Intentionally break a backup (e.g. wrong passphrase) and verify Healthchecks.io receives a fail ping
  • Skip a run (stop the container before 02:00) and verify Healthchecks.io alerts after grace period

Merge request reports