Monitoring & alerting

Where to look when something feels wrong — in escalation order, from the room down to a single process. This page is the operator's view; how each tool is built lives on its Infrastructure page.

Where to look, in order

Level	Tool	Where
Room / rack	Rack status page — every machine on/off, PDU temperature & humidity, total power, UPS state & load	http://srv-status.calypso (VPN) or /status / /graph (12 h temperature chart) to @ISC3StatusBot on Telegram
Host / guest	Netdata — real-time CPU / RAM / disk-I/O / network for a PVE host and every VM/CT automatically (cgroup_srv-* sections)	http://rumba:19999, or carnaval0–2:19999 for the playground cluster (VPN) — one click from the machine's name on the status page
Power detail	PDU AP8681 — per-outlet load, per-phase current	SNMP v2c public (read-only) at 192.168.88.10, or the web UI (password in the secretzone)
UPS	UPS monitoring — mains/battery state, load, voltages, temperature	upsc rackups on rumba (NUT), and on the status page + Telegram since 2026-08-03

None of these need a login from the VPN; none is published to the Internet.

Alerts that exist today (Aug 2026)

• Thermal — Telegram. The rack status poller messages the admin chat when the PDU probe crosses 35 °C (warn) or 45 °C (critical), reminds every 30 min while hot, announces recovery, and complains if the probe itself stops answering. This implements the "warn humans" layer of the thermal protection proposal; thresholds and mechanics on the service page.
• Mains loss — Telegram. Since 2026-08-03 the same poller alerts on the UPS ups.status flags: on-battery, overload, replace-battery, and upsd going silent. Now that rumba runs on the UPS this is a real signal — but for the same reason the poller dies with a long outage, so the message marks the start of one.
• Backup failures — mail. The nightly vzdump job on rumba is configured to mail on failure — see Backups.

That is the whole list. Nothing yet alerts on a machine going down, a disk failing, or a full filesystem — the planned NOC (Prometheus/Alertmanager, see the target architecture) would close that gap.

When a thermal alert fires

Lessons of the June 22–23, 2026 heat event (76 °C at the probe, 3.6 kW, no machine shut itself down — full account in the incident log):

1. Look before driving in: the status page / /status shows whether temperature is still rising and how much power the rack is pulling — fans at maximum show up as an abnormal total (~1.1 kW is typical, the June event peaked at 3.6 kW).
2. Warn level (≥ 35 °C): check the room (23N307) and its cooling; this is the stage where a human fixing the AC avoids everything that follows.
3. Approaching critical (≥ 45 °C): start shedding heat manually — the thermal protection proposal defines the order (SLURM drain and playground guests first, storage and calypsomaster last; the MikroTiks and the PDU stay up to keep the remote path alive). The automated layers are not implemented yet — today this is you, over the VPN.
4. After the event: confirm the room is cool on the status page before powering anything back up (deliberately manual), and follow the incident process — write the timeline down while it is fresh.