Rack status & thermal alerts

The rack status service (Aug 2026): a self-refreshing web page showing the live state of every machine in the rack plus the PDU environment readings, and a Telegram bot that sends thermal alerts — the first implementation of the "warn humans" layer of the thermal protection proposal, written after the June 2026 heat event.

At a glance

URL	http://srv-status.isc3 — VPN / LAN only, not published through the reverse proxy
Bot	@ISC3StatusBot on Telegram — /status summary, /graph 12 h temperature chart, 🔄 Refresh / 📈 Graph buttons; replies only to allowed chats
Guest	srv-status — unprivileged LXC, CT 108 on rumba
Address	192.168.88.158 (static), also srv-status / srv-status.isc3 in DNS
Stack	Caddy (static pages + status.json + history CSVs) · Python poller (systemd timer, every 30 s, keeps a 7-day raw + 2-year hourly environment history) · long-polling bot daemon (matplotlib for the graphs)
Content	Version-controlled in provisioning/status/ in this repository; idempotent deploy-status.sh
Resources	1 core, 512 MB RAM, 256 MB swap, 4 GB on local-lvm, starts on boot

Data flow

Everything the service reads is read-only: no credentials that can change anything live in the container.

The bot long-polls Telegram over an outbound HTTPS connection, so there is no webhook, no exposed port and no router change — consistent with the rule that only srv-web01 faces the Internet.

The page

http://srv-status.isc3 shows, refreshed every 30 s (Aug 2026 rack inventory):

• Stat tiles — PDU probe temperature (colour-coded against the alert thresholds), relative humidity (with the computed dew point and its distance to the chilled-water inlet, colour-coded against the condensation thresholds below — Aug 2026), total rack power draw, the UPS (load with its mains/battery state, colour-coded), and the machines-up count.
• Machines — one card per rack device (carnaval0–10, epyc pair, rumba, calypsomaster, NAS, the three routers, PDU, tango pair, UPS), on/off by ICMP ping of the user/OS address, tinted by role with the ISC petal colours (plus a dedicated orange for carnaval, since all five petal colours are already spoken for). The cards are grouped by family — carnaval cluster, compute & hypervisors, storage, network, power, spares — each block headed by its role colour and its own up/monitored count, cards sorted by name inside a block (numeric-aware, so carnaval2 precedes carnaval10); the group order and labels are the GROUPS table in site/index.html, and a role missing from it lands in a trailing Other block. The seven spare R630s show n/a — they are iDRAC-only on 192.168.90.x, which is not routed from the guest LAN. The UPS has no network interface, so its card means something narrower: on = upsd answered, with the mains/battery state on the tile rather than in the dot. A machine's name is a link to its own Netdata dashboard at :19999 when that machine runs one — set via "netdata": true in the entry below (today: rumba and the carnaval nodes).
• Proxmox guests — every CT/VM on rumba with its state, via the API token above.

The machine list lives in provisioning/status/poller/machines.json — when the rack changes, update it (and rack.mdx) and re-run the deploy script. The Proxmox guest list needs no such maintenance: it is read live from the API on every poll, so new or retired guests appear on their own.

Temperature, power & UPS history (/graphs.html, Aug 2026)

http://srv-status.isc3/graphs.html charts the PDU probe temperature, the total rack power draw and the UPS load at seven scales — 12 h, 24 h, 7 d (from the raw 30 s samples) and 30 d, 6 m, 1 y, 2 y (from hourly aggregates). The measures live on different scales (°C, kW, %), so each gets its own stacked chart — never a dual axis — sharing the range selector. Each chart has a mean line plus a min–max band per bucket, min/mean/max tiles for the selected range, and a crosshair tooltip (mouse or keyboard); the temperature chart adds the warn/crit reference lines when they enter the visible range. The y-axes rescale to the data of each range. Rendering is dependency-free inline SVG — nothing is fetched from outside the container.

Two files back it, both kept by the poller in /var/lib/isc-status/ and served read-only under /data/:

File	Contents	Retention
/data/history.csv	one row per 30 s poll: utc-iso,temp_c,power_w,humidity,ups_load,ups_battery_v,ups_input_v	7 days
/data/history-hourly.csv	one row per complete UTC hour: min/avg/max of temperature, power and humidity, the sample count, then min/avg/max of UPS load, battery voltage and input voltage	2 years (731 days)

The hourly file is folded from the raw one on the fly (~2 writes/hour, ≈1.5 MB at full retention), so an outage only loses hours if the poller stays down beyond the raw file's 7-day window. Humidity, battery voltage and input voltage are recorded but not charted (todo). History collection started 2026-08-02, the UPS columns a day later, so the long scales fill in as the data accumulates.

Only ever append columns

Both files are parsed positionally, and graphs.html reads the hourly sample count at a hard-coded column index 10 — which is why the UPS aggregates sit after the count rather than grouped with the other metrics. The rule and the frozen column ranges are written above HOURLY_COLS in isc-status-poller.py. Note also that Caddy serves only the two files matched explicitly under /data/: a third CSV would need that matcher edited before the browser could see it.

Alerts

On every 30 s run the poller checks three things. Every alert goes to the configured Telegram chat and by email to the admin, submitted to the srv-mail relay on plain port 25 (srv-status is on its mynetworks list, so the container still holds no mail credential — Aug 2026). The recipient is the mail_to key of telegram.json, defaulting to the address hard-coded in the poller; an empty string disables the email channel.

Thermal — the PDU probe against the layer-1 thresholds of the thermal protection proposal:

Event	Behaviour
≥ 35 °C	🌡️ warn message
≥ 45 °C	🔥 critical message
still hot	reminder every 30 min
cooled down	✅ recovery message (2 °C hysteresis, so no flapping at the threshold)
probe unreadable	⚠️ alert after ~5 min of consecutive SNMP failures

Condensation (Aug 2026) — the dew point of the room air (Magnus formula from the probe's temperature and humidity, published as dew_point_c in status.json) against the coldest chilled-water surface in the room. The inlet water measured 16–17 °C in Aug 2026 and the alarm first used 16 °C, but the dew point reached 17.3 °C (hourly mean 16.6) without any condensation — the exposed pipe surfaces are warmer than the water — so since 2026-08-28 the reference is 19 °C, 2 °C above the highest value recorded. The alarm level is decided on the 10-minute mean of the dew point, not the instantaneous value: the probe reports integer-% humidity, so a single RH step (plus the A/C cycle) moves the dew point ~1.2 °C at once, across the whole hysteresis. Smoothing cut the first week's message count by roughly a factor of seven, on the same data.

Event	Behaviour
mean dew point ≥ 17 °C (within 2 °C of the reference)	💧 warn message, once per episode
mean dew point ≥ 19 °C (at the reference)	💧 critical message, reminder every 4 h (dew_renotify_min)
mean dew point back below 16 °C	✅ recovery message (1 °C hysteresis)

Warn deliberately does not re-notify: with the earlier 16 °C reference the dew point sat in the warn band ~40 % of summer hours, so a periodic warn reminder would be pure noise. The reference values are water_inlet_c / dew_warn_margin_c in telegram.json (defaults 19 / 2); the page hard-codes the same two constants in index.html (and colours its tile on the instantaneous value, so the tile can change before the alarm does).

UPS (Aug 2026) — the UPS ups.status flags:

Event	Behaviour
OB / LB — mains lost	⚡ critical message
OVER — output overloaded	🔌 critical message
RB — replace battery	🔋 warn message
any other flag set without OL	⚠️ warn message quoting the raw flags
unchanged and not OK	reminder every 30 min
back on mains	✅ recovery message
upsd unreachable	⚠️ alert after ~5 min of silence

Thresholds and the reminder intervals are the warn_c / crit_c / water_inlet_c / dew_warn_margin_c / renotify_min / dew_renotify_min / mail_to keys of /etc/isc-status/telegram.json inside the container; alerting is dormant when that file is absent. Remember the June 2026 calibration note: node inlet sensors read a few degrees above the PDU probe.

caution

The UPS alert is keyed on ups.status and nothing else

battery.runtime and battery.charge on this UPS are NUT estimates, not measurements, so the poller does not publish them — why. Do not add a threshold on either.

rumba now runs on the UPS, so a long enough outage takes the poller down with it. The message arrives at the start of an outage, not at its end.

The bot

@ISC3StatusBot answers two commands (both in the bot's command menu), and answers only the chat(s) configured in telegram.json (chat_id, plus an optional allowed_chats list); anyone else who finds the bot is ignored.

• /status — the current summary: temperature, power, humidity, the UPS state and load, which monitored machines are off, guest states, and the poll timestamp in Swiss local time. Every reply carries 🔄 Refresh (re-edits the message with fresh data) and 📈 Graph inline buttons.
• /graph — a PNG chart of the last 12 h of PDU probe temperature, drawn with matplotlib from the poller's rolling history (/var/lib/isc-status/history.csv, pruned to 7 days). The line breaks visibly where the poller was down or the probe unreadable; the configured warn/crit thresholds are drawn as reference lines when the data gets near them (Aug 2026).

Operations

• Redeploy / rebuild: provisioning/status/deploy-status.sh — idempotent, recreates the container from scratch if CT 108 is gone, refreshes files otherwise.
• Secrets: the Proxmox API token and the Telegram bot credentials exist only inside the container; the secretzone (rumba.md there) documents what they are and how to rotate them.
• Publishing: deliberately internal-only. If it is ever published, do it as a Caddy vhost on srv-web01 — never a router change — and put authentication in front first: the page reveals the power state of every machine in the room.