Rack status & thermal alerts

The rack status service (Aug 2026): a self-refreshing web page showing the live state of every machine in the rack plus the PDU environment readings, and a Telegram bot that sends thermal alerts — the first concrete implementation of the "warn humans" layer of the thermal protection proposal, born from the June 2026 heat event.

At a glance

URL	http://srv-status.calypso — VPN / LAN only, not published through the reverse proxy
Bot	@ISC3StatusBot on Telegram — /status summary, /graph 12 h temperature chart, 🔄 Refresh / 📈 Graph buttons; replies only to allowed chats
Guest	srv-status — unprivileged LXC, CT 108 on rumba
Address	192.168.88.158 (static), also srv-status / srv-status.calypso in DNS
Stack	Caddy (static pages + status.json + history CSVs) · Python poller (systemd timer, every 30 s, keeps a 7-day raw + 2-year hourly environment history) · long-polling bot daemon (matplotlib for the graphs)
Content	Version-controlled in provisioning/status/ in this repository; idempotent deploy-status.sh
Resources	1 core, 512 MB RAM, 256 MB swap, 4 GB on local-lvm, starts on boot

Data flow

Everything the service reads is read-only: no credentials that can change anything live in the container.

The bot long-polls Telegram over an outbound HTTPS connection, so there is no webhook, no exposed port and no router change — consistent with the rule that only srv-web01 faces the Internet.

The page

http://srv-status.calypso shows, refreshed every 30 s (Aug 2026 rack inventory):

• Stat tiles — PDU probe temperature (colour-coded against the alert thresholds), relative humidity, total rack power draw, the UPS (load with its mains/battery state, colour-coded), and the machines-up count.
• Machines — one card per rack device (calypso nodes, carnaval0–2, epyc pair, rumba, calypsomaster, NAS, the three routers, PDU, pagode pair, UPS), on/off by ICMP ping of the user/OS address, tinted by role with the ISC petal colours (plus a dedicated orange for carnaval, since all five petal colours are already spoken for). The seven spare R630s show n/a — they are iDRAC-only on 192.168.90.x, which is not routed from the guest LAN. The UPS has no network interface at all, so its card means something narrower: on = upsd answered, with the mains/battery state on the tile rather than in the dot. A machine's name is a link to its own Netdata dashboard at :19999 when that machine runs one — set via "netdata": true in the entry below (today: rumba and the three carnaval nodes).
• Proxmox guests — every CT/VM on rumba with its state, via the API token above.

The machine list lives in provisioning/status/poller/machines.json — when the rack changes, update it (and rack.mdx) and re-run the deploy script. The Proxmox guest list needs no such maintenance: it is read live from the API on every poll, so new or retired guests appear on their own.

Temperature, power & UPS history (/graphs.html, Aug 2026)

http://srv-status.calypso/graphs.html charts the PDU probe temperature, the total rack power draw and the UPS load at seven scales — 12 h, 24 h, 7 d (from the raw 30 s samples) and 30 d, 6 m, 1 y, 2 y (from hourly aggregates). The measures live on different scales (°C, kW, %), so each gets its own stacked chart — never a dual axis — sharing the range selector. Each chart has a mean line plus a min–max band per bucket, min/mean/max tiles for the selected range, and a crosshair tooltip (mouse or keyboard); the temperature chart adds the warn/crit reference lines when they enter the visible range. The y-axes rescale to the data of each range. Rendering is dependency-free inline SVG — nothing is fetched from outside the container.

Two files back it, both kept by the poller in /var/lib/isc-status/ and served read-only under /data/:

File	Contents	Retention
/data/history.csv	one row per 30 s poll: utc-iso,temp_c,power_w,humidity,ups_load,ups_battery_v,ups_input_v	7 days
/data/history-hourly.csv	one row per complete UTC hour: min/avg/max of temperature, power and humidity, the sample count, then min/avg/max of UPS load, battery voltage and input voltage	2 years (731 days)

The hourly file is folded from the raw one on the fly (~2 writes/hour, ≈1.5 MB at full retention), so an outage only loses hours if the poller stays down beyond the raw file's 7-day window. Humidity, battery voltage and input voltage are recorded but not yet charted. History collection started 2026-08-02 ≈09:25 UTC (UPS columns from 2026-08-03 ≈16:54 UTC) — the long scales fill in as the data accumulates.

Only ever append columns

Both files are parsed positionally, and graphs.html reads the hourly sample count at a hard-coded column index 10 — which is why the UPS aggregates sit after the count rather than grouped with the other metrics. The rule and the frozen column ranges are written above HOURLY_COLS in isc-status-poller.py. Note also that Caddy serves only the two files matched explicitly under /data/: a third CSV would need that matcher edited before the browser could see it.

Alerts

On every 30 s run the poller checks two things and messages the configured Telegram chat.

Thermal — the PDU probe against the layer-1 thresholds of the thermal protection proposal:

Event	Behaviour
≥ 35 °C	🌡️ warn message
≥ 45 °C	🔥 critical message
still hot	reminder every 30 min
cooled down	✅ recovery message (2 °C hysteresis, so no flapping at the threshold)
probe unreadable	⚠️ alert after ~5 min of consecutive SNMP failures

UPS (Aug 2026) — the UPS ups.status flags:

Event	Behaviour
OB / LB — mains lost	⚡ critical message
OVER — output overloaded	🔌 critical message
RB — replace battery	🔋 warn message
any other flag set without OL	⚠️ warn message quoting the raw flags
unchanged and not OK	reminder every 30 min
back on mains	✅ recovery message
upsd unreachable	⚠️ alert after ~5 min of silence

Thresholds and the reminder interval are the warn_c / crit_c / renotify_min keys of /etc/isc-status/telegram.json inside the container; alerting is dormant when that file is absent. Remember the June 2026 calibration note: node inlet sensors read a few degrees above the PDU probe.

caution

The UPS alert is keyed on ups.status and nothing else

battery.runtime and battery.charge on this UPS are NUT estimates, not measurements, so the poller does not even publish them — why. Do not add a threshold on either.

Also note what this alert cannot survive: rumba now runs on the UPS, so a long enough outage takes the poller down with it. The message arrives at the start of an outage, not at its end.

The bot

@ISC3StatusBot answers two commands (both in the bot's command menu), and answers only the chat(s) configured in telegram.json (chat_id, plus an optional allowed_chats list); anyone else who finds the bot is silently ignored.

• /status — the current summary: temperature, power, humidity, the UPS state and load, which monitored machines are off, guest states, and the poll timestamp in Swiss local time. Every reply carries 🔄 Refresh (re-edits the message with fresh data) and 📈 Graph inline buttons.
• /graph — a PNG chart of the last 12 h of PDU probe temperature, drawn with matplotlib from the poller's rolling history (/var/lib/isc-status/history.csv, pruned to 7 days). The line breaks visibly where the poller was down or the probe unreadable; the configured warn/crit thresholds are drawn as reference lines when the data gets near them (Aug 2026).

Operations

• Redeploy / rebuild: provisioning/status/deploy-status.sh — idempotent, recreates the container from scratch if CT 108 is gone, refreshes files otherwise.
• Secrets: the Proxmox API token and the Telegram bot credentials exist only inside the container; the secretzone (rumba.md there) documents what they are and how to rotate them.
• Publishing: deliberately internal-only. If it is ever published, do it as a Caddy vhost on srv-web01 — never a router change — and put authentication in front first: the page reveals the power state of every machine in the room.