Thermal protection & emergency shutdown
Date: July 2026 · Status: proposal — layer 1 ("warn humans") implemented Aug 2026 via the rack status service, the other layers are not Scope: the ISC³ rack in 23N307 — automatic reaction to over-temperature events, today (bare-metal) and after the Proxmox migration.
1. Why
During the night of June 22–23, 2026 the room overheated: the PDU probe recorded a maximum of ~76 °C and the rack's power draw peaked at 3.6 kW (all fans at maximum) against ~1.1 kW typical. No machine shut itself down. This is expected behaviour: Dell/Gigabyte thermal protection acts at the component level — fans compensate until a CPU or DIMM exceeds its absolute limit, then the machine hard-trips.
The goal: a layered policy where machines react to ambient temperature early and gracefully, humans are warned first, and the hard component trip remains the last resort.
2. Layered approach
Thresholds are indicative — inlet sensors on the nodes read a few degrees above true room temperature; calibrate against the PDU probe before arming anything destructive.
| Layer | Mechanism | Where it runs | Status (Aug 2026) |
|---|---|---|---|
| Warn | Telegram alerts from the rack status service (35 °C warn / 45 °C crit on the PDU probe) · PDU email alarm still unconfigured, but no longer blocked — the rack has a smarthost since Aug 2026 · later Alertmanager | srv-status · PDU · NOC | ✅ implemented (Telegram) |
| Shed | Stop non-essential load: lab VMs, spare/idle machines via switched PDU outlets | orchestrator | ❌ manual only |
| Self-protect | Watchdog on each node: ipmitool sdr type temperature → inlet ≥ threshold for N minutes → clean shutdown -h | every node (systemd timer, deployed by Ansible) | ❌ not implemented |
| Orchestrate | Watcher on calypsomaster polls the PDU probe over SNMP (public, read-only) and drives a fleet-wide ordered shutdown | calypsomaster | ❌ not implemented |
| Last resort | iDRAC alert action "power off" on the temperature-critical event (hard cut, racadm-scriptable) · component thermal trip | firmware | ⚠️ component trip only (iDRAC action not configured) |
Shutdown order matters: workers first (they generate most of the heat), then rumba/epyc*, then the NAS (verify DSM's own shut down when system overheats option is enabled), and calypsomaster last since it hosts the orchestrator. The two MikroTiks and the PDU stay up — they survive heat better and keep the remote path alive.
Deliberately not automated: turning machines back on. After a thermal event a human should confirm the room is cool (PDU probe readable from the VPN) before powering up via iDRAC/PDU outlets.
3. What Proxmox changes
The future architecture puts Proxmox VE under everything — this mostly makes thermal protection better, with two sharp edges to respect:
Opportunities
- One API for the whole fleet. An orchestrator can cleanly stop every VM and then the hosts (
qm shutdown/pvesh, or node-level shutdown which gracefully stops local VMs first) — no per-OS SSH loops. The shutdown script becomes ~20 lines against one API. - Zones map to shedding priorities. Playground VMs are disposable by design (cloud-init templates) — they can be killed at the 40 °C stage, removing most of the thermal load while Services (GitLab, Moodle) ride it out. Recovery is cheap, so thresholds can be aggressive.
- The NOC closes the loop. Prometheus already planned to scrape everything; adding the PDU's rPDU2 OIDs (power, per-phase current, temperature, humidity) gives dashboards and lets Alertmanager drive the orchestrator via webhook instead of a bespoke poller on
calypsomaster. - Power capping before shutdown. With PVE it's easy to suspend/throttle playground VMs as an intermediate step — less heat generated, services untouched.
Sharp edges
- HA works against a thermal shutdown. If a node shuts down for thermal reasons, the HA manager sees a failed node and restarts its VMs on the surviving nodes, concentrating the heat load on fewer machines. The orchestrator must put nodes in maintenance mode (or disable HA) before the staged shutdown.
- Quorum. Shutting down cluster nodes one by one eventually drops quorum and freezes the remainder. Fine for a full-rack stop, but the sequence must be scripted and tested in advance.
- PBS. A backup running mid-shutdown should be interrupted cleanly; graceful node shutdown handles this, hard PDU cuts don't.
4. Implementation steps
- Now (bare-metal): Ansible role for the per-node IPMI watchdog (layer "self-protect") + PDU email alarm, whose SMTP parameters now exist (Email) — the open question is only whether the NMC firmware does authenticated TLS. Low effort, covers the June scenario.
- With the NOC: PDU + node temps into Prometheus, alert rules replace the standalone poller.
- With Proxmox: rewrite the orchestrator against the PVE API with HA-maintenance handling; per-zone thresholds (playground sheds early, services late).
- Test day: a controlled dry-run (fake threshold) once a semester, to confirm the chain fires end to end.
5. Open questions
- Exact thresholds per layer, calibrated inlet-vs-room (June data point: fans max out well before any shutdown would trigger).
- Whether the school's building management can alarm on the room itself (redundant, independent path).
- UPS interaction: the BlueWalker's USB monitoring could trigger the same orchestrator on power events — same mechanism, second trigger. Half-unblocked since 2026-08-03: the USB cable now physically exists between the UPS and rumba, and the device is confirmed readable (UPS). Two caveats before building on it: the UPS presently carries ~0 load, so a power event on it implies nothing about the machines; and rumba is not on it, so the orchestrator host would itself be unprotected.