Thermal protection & emergency shutdown
Date: July 2026 · Status: proposal — layer 1 ("warn humans") implemented Aug 2026 via the rack status service, the other layers are not Scope: the ISC³ rack in 23N307 — automatic reaction to over-temperature events, today (bare-metal) and after the Proxmox migration.
1. Why
During the night of June 22–23, 2026 the room overheated: the PDU probe recorded a maximum of ~76 °C and the rack's power draw peaked at 3.6 kW (all fans at maximum) against ~1.1 kW typical. No machine shut itself down. This is expected behaviour: Dell/Gigabyte thermal protection acts at the component level — fans compensate until a CPU or DIMM exceeds its absolute limit, then the machine hard-trips. Room-level heat is nobody's problem until it becomes everybody's problem.
The goal: a layered policy where machines react to ambient temperature early and gracefully, humans are warned first, and the hard component trip remains what it should be — a never-reached last resort.
2. Layered approach
Thresholds are indicative — inlet sensors on the nodes read a few degrees above true room temperature; calibrate against the PDU probe before arming anything destructive.
| Layer | Mechanism | Where it runs | Status (Aug 2026) |
|---|---|---|---|
| Warn | Telegram alerts from the rack status service (35 °C warn / 45 °C crit on the PDU probe) · PDU email alarm still unconfigured (needs the institutional SMTP relay) · later Alertmanager | srv-status · PDU · NOC | ✅ implemented (Telegram) |
| Shed | Stop non-essential load: SLURM drain, playground VMs, spare/idle machines via switched PDU outlets | orchestrator | ❌ manual only |
| Self-protect | Watchdog on each node: ipmitool sdr type temperature → inlet ≥ threshold for N minutes → clean shutdown -h | every node (systemd timer, deployed by Ansible) | ❌ not implemented |
| Orchestrate | Watcher on calypsomaster polls the PDU probe over SNMP (public, read-only) and drives a fleet-wide ordered shutdown | calypsomaster | ❌ not implemented |
| Last resort | iDRAC alert action "power off" on the temperature-critical event (hard cut, racadm-scriptable) · component thermal trip | firmware | ⚠️ component trip only (iDRAC action not configured) |
Shutdown order matters: workers first (they generate most of the heat), then rumba/epyc*, then the NAS (verify DSM's own shut down when system overheats option is enabled), and calypsomaster last since it hosts the orchestrator. The two MikroTiks and the PDU stay up — they survive heat far better and keep the remote path alive.
Deliberately not automated: turning machines back on. After a thermal event a human should confirm the room is cool (PDU probe readable from the VPN) before powering up via iDRAC/PDU outlets.
3. What Proxmox changes
The future architecture puts Proxmox VE under everything — this mostly makes thermal protection better, with two sharp edges to respect:
Opportunities
- One API for the whole fleet. An orchestrator can cleanly stop every VM and then the hosts (
qm shutdown/pvesh, or node-level shutdown which gracefully stops local VMs first) — no per-OS SSH loops. The shutdown script becomes ~20 lines against one API. - Zones map to shedding priorities. Playground VMs are disposable by design (cloud-init templates) — they can be killed at the 40 °C stage without regret, removing most of the thermal load while Services (GitLab, Moodle) ride it out. Recovery is cheap, so thresholds can be aggressive.
- The NOC closes the loop. Prometheus already planned to scrape everything; adding the PDU's rPDU2 OIDs (power, per-phase current, temperature, humidity) gives dashboards and lets Alertmanager drive the orchestrator via webhook instead of a bespoke poller on
calypsomaster. - Power capping before shutdown. With PVE it's easy to suspend/throttle playground VMs as an intermediate step — less heat generated, services untouched.
Sharp edges
- HA will fight you. If a node shuts down for thermal reasons, the HA manager sees a failed node and restarts its VMs on the surviving nodes — concentrating the heat load on fewer machines, the exact opposite of the goal. The orchestrator must put nodes in maintenance mode (or disable HA) before the staged shutdown.
- Quorum. Shutting down cluster nodes one by one eventually drops quorum and freezes the remainder. Fine for a full-rack stop, but the sequence must be scripted and tested — not improvised at 45 °C.
- PBS. A backup running mid-shutdown should be interrupted cleanly; graceful node shutdown handles this, hard PDU cuts don't.
4. Implementation steps
- Now (bare-metal): Ansible role for the per-node IPMI watchdog (layer "self-protect") + PDU email alarm once the SMTP relay parameters exist. Low effort, covers the June scenario.
- With the NOC: PDU + node temps into Prometheus, alert rules replace the standalone poller.
- With Proxmox: rewrite the orchestrator against the PVE API with HA-maintenance handling; per-zone thresholds (playground sheds early, services late).
- Test day: a controlled dry-run (fake threshold) once a semester — a shutdown chain that has never fired is documentation, not protection.
5. Open questions
- Exact thresholds per layer, calibrated inlet-vs-room (June data point: fans max out well before any shutdown would trigger).
- Whether the school's building management can alarm on the room itself (redundant, independent path).
- UPS interaction: the BlueWalker's USB monitoring could trigger the same orchestrator on power events — same mechanism, second trigger. Half-unblocked since 2026-08-03: the USB cable now physically exists between the UPS and rumba, and the device is confirmed readable (UPS). Two caveats before building on it: the UPS presently carries ~0 load, so a power event on it implies nothing about the machines; and rumba is not on it, so the orchestrator host would itself be unprotected.