Healthchecks — dead-man switch for cron jobs
Healthchecks inverts the alerting logic: every scheduled job (backup sync, scrub, snapshot) pings it on success, and Healthchecks alerts when a ping fails to arrive — which catches the backup that did not run, a failure mode no log-based alert sees. Installed 2026-08-16, closing the admin wave from the service ideas.
| Guest | srv-healthchecks — unprivileged LXC, CT 115 on rumba, 192.168.88.166, 1 core / 1 GB / 6 GB local-lvm, nesting=1 |
| URL | http://srv-healthchecks.isc3/ — VPN/LAN only, not published through srv-web01 |
| Login | local accounts only, registration closed — superuser pierre-andre.mudry@hevs.ch (password in secretzone/rumba.md). No SSO: the tool does not speak OIDC natively |
| Storage | SQLite in /var/lib/healthchecks/ — a handful of checks, PBS backs it up with the CT |
| Alerting | email through srv-mail and push through ntfy (publisher account) — channels are configured per-project in the UI |
| Managed from | provisioning/healthchecks/ — deploy-healthchecks.sh (idempotent; re-run upgrades to the latest release) |
Checks and how each is pinged
Wired 2026-08-16 by provisioning/healthchecks/wire-jobs.sh (idempotent — it carries the ping
UUIDs and re-asserts every hook):
| Check | Schedule | Pinged by |
|---|---|---|
| rumba vzdump → hdd-backup | daily 02:00, grace 6 h | /usr/local/bin/vzdump-hc-ping on rumba — a vzdump hookscript (script: in /etc/vzdump.conf) that pings on job-end per target storage |
| rumba vzdump → PBS | daily 03:00, grace 6 h | same hookscript, pbs-fs2500 branch |
| PBS pull sync | daily 00:00, grace 8 h | PBS webhook notification endpoint hc-sync + matcher (type=sync, severity info) |
| PBS verify | sat 06:00, grace 12 h | endpoint hc-verify + matcher (type=verify) |
| ZFS scrub hdd | every ≤ 35 d (2nd Sunday) | zed hook /etc/zfs/zed.d/scrub_finish-healthchecks.sh on rumba — pings only if pool hdd reports healthy |
| Oxidized poll | every ≤ 2 h | oxidized hooks: exec on node_success |
Not wired: roster-sync and the manual MikroTik backup (both run manually — a dead-man switch on
an unscheduled job only makes noise; Oxidized covers router-config drift hourly). The NAS-side
tasks are an open item on the ops todo.
Wiring a new job: create the check in the UI with the expected schedule and grace, then append the ping to the job itself:
some-nightly-job && curl -fsS -m 10 --retry 3 http://srv-healthchecks.isc3/ping/<uuid>