UPS monitoring (NUT)

The rack's BlueWalker VFI 2000 RMG PF1 has no network interface, so since 2026-08-03 it is read over USB from rumba using NUT (Network UPS Tools). Deployed by provisioning/ups/deploy-ups.sh. Since the same day the UPS also powers rumba (both PSUs) and nothing else, which makes the shutdown policy a live question.

At a glance

Host	rumba — the only machine cabled to the UPS
Version	NUT 2.8.1-5 (Debian 13 main), driver nutdrv_qx / subdriver=cypress
UPS name	rackups — upsc rackups on rumba
upsd	listening on 127.0.0.1:3493 and 192.168.88.51:3493
Consumers	upsmon on rumba (log/notify only) and the rack status page, polling every 30 s from CT 108
Shutdown	none — see Shutdown policy

What it reports

What upsc rackups exposes, with typical values (August 2026). The device's own state — health, battery, what it protects — is on the Power page:

Variable	Value
ups.status	OL	on line (mains), not on battery
ups.load	14 %	≈ 265 W, i.e. all of rumba and nothing else
output.current	1.0–1.1 A
ups.power.nominal	2000	confirms the 2 kVA nameplate
input.voltage / output.voltage	244 V / 229.5 V
battery.voltage	82.0 V	on 72 V nominal, i.e. float
ups.temperature	19 °C	bottom of the rack, cooler than the PDU probe higher up
input.transfer.low / .high	218 V / 242 V	outside this window it switches to battery

Nothing per-outlet is available (outlet.0.switchable: no), nor input current or real/apparent power.

danger

NUT's battery.runtime and battery.charge are unreliable — the real ones are elsewhere

The Q* protocol carries neither runtime nor true charge, so nutdrv_qx estimates both from battery voltage against an uncalibrated runtimecal. The driver computes the charge as (voltage − low) / (high − low), and this unit reports battery.voltage.high: 78.00 while the pack floats at 82.0 V — permanently out of range, so battery.charge is pinned at 100 %.

The firmware also answers the Q* status query with a zero-filled battery voltage every so often — bursty, a few per cent of samples. The protocol has no checksum and nutdrv_qx applies no plausibility filter, so upsd serves the zeros and the derived charge collapses with the UPS steadily on line. Only battery.voltage is affected; ups.status does not flap, so status-keyed alerting is unaffected.

Via NUT, trust only ups.status, ups.load and output.current; treat any single voltage sample as suspect. The status page and the alerting deliberately do not publish charge or runtime, so no threshold can be keyed on them by accident. Netdata's stock UPS alarms needed the same treatment. The genuine values are on the HID interface.

The HID Power Device interface — where the real numbers are

The UPS exposes two USB interfaces. nutdrv_qx uses interface 0, which is just the Cypress serial tunnel. Interface 1 is a proper HID Power Device interface, owned by the kernel's hid-generic at /dev/hidraw1, and it carries genuine battery telemetry — decoded on 2026-08-03:

	Value
RunTimeToEmpty	3300 s — 55 min	a real measurement, and it tracks load: 26 460 s when the UPS was empty
RemainingCapacity	100 %	genuine, unlike NUT's saturated 100 %
RemainingCapacityLimit	10 %	the UPS's own low-battery threshold
PresentStatus	bitfield	real Overload, NeedReplacement, InternalFailure, ShutdownImminent and BelowRemainingCapacityLimit bits

Everything nutdrv_qx reports is there too, and agrees with it — which is what pinned the field mapping.

Read it with ups-read-hid on rumba (--json for machine output), installed by deploy-ups.sh from provisioning/ups/read-hid.py. Read-only, and safe to run while NUT is up since the two drivers are on different interfaces.

Three traps encoded in that script — do not "clean them up"

• The descriptor's unit exponents are unusable (exponent 6 on the voltage reports would render an 82.0 V battery as 8.2 × 10⁸). Every scaled field is really in tenths; the script hard-codes that.
• The UPS returns EPROTO on back-to-back GET_FEATURE requests. A bare loop silently yields nothing for about half the fields, which looks like missing data rather than an error. The script paces reads 50 ms apart and retries.
• usbhid-ups cannot be pointed at interface 1 in NUT 2.8.1: it hard-claims interface 0 and bails. usb_set_altinterface does not help — that sets the alternate setting of interface 0, not the interface number. Using this data from inside NUT needs a driver patch, not a config option.

When the USB link wedges

danger

insufficient permissions on everything is not a permissions problem

upsc says Error: Data stale and nutdrv_qx loops that message a few times a second. It reads like the udev trap the deploy script fixes, and it is not. What it means is that the UPS's USB stack has wedged: the device is still enumerated, its node is still root:nut 664, and the kernel has logged nothing.

Two checks settle it in seconds — both were true on 2026-08-09:

• open the node as the driver's own user: setpriv --reuid=nut --regid=nut --clear-groups \ dd if=/dev/bus/usb/001/006 of=/dev/null bs=1 count=1 succeeds while the driver is failing;
• ups-read-hid --json returns Protocol error for every report. That is the other interface, a different kernel driver — so both interfaces are dead at once, which no permission ever does.

First check the device is still there — on 2026-08-10 it was not

The two checks above assume an enumerated device. On 2026-08-10 the UPS vanished from the bus (usb 1-13: USB disconnect; lsusb -d 0665:5161 empty, /sys/bus/usb/devices/1-13 gone), and then usbreset has nothing to act on and the recipe below is a dead end. Confirm with lsusb -d 0665:5161 before reaching for it; if the device is absent the fix is a physical replug, since its parent is the xHCI root hub and the only remote lever is resetting every USB device on bus 1 (incident).

Recovery is a USB reset, not a udev re-trigger:

systemctl stop nut-driver@rackups.service
usbreset 0665:5161                     # usbutils; resets the link, never the UPS output
systemctl start nut-driver@rackups.service   # probes ~10 s before answering

If that does not take: unbind/rebind the port (echo 1-13 > /sys/bus/usb/drivers/usb/unbind, then bind), then a physical replug. Safe at any time — upsmon runs powervalue 0, so nothing can shut down as a side effect. :::

On the status page

The rack status page polls upsd from CT 108 every 30 s: the UPS tile, a real dot on the UPS machine card, a load chart, load/battery/input columns in both history CSVs, and a Telegram alert keyed on ups.status (mains loss, overload, replace-battery, upsd silent). Details and the CSV column-order rule are on the status-page side.

LIST VAR needs no login — upsd authenticates writes only — so CT 108 holds no UPS credential; the read-only monitor account in upsd.users stays free for a future upsmon secondary.

Netdata alarms

Netdata on the rumba host auto-detects upsd and ships three stock alarms. Two of the three are wrong on this hardware, and both fired for days — as mail, because the node is claimed to Netdata Cloud, which notifies on every transition regardless of the to: role in the agent's health file.

deploy-ups.sh therefore installs provisioning/ups/netdata-upsd-health.conf as /etc/netdata/health.d/upsd.conf — a same-named user file fully replaces the stock one, and netdata's daily auto-update does not touch /etc/netdata/:

• upsd_ups_battery_charge (stock warn < 75 %) keys on the unreliable charge above, so it fires with the UPS on line. It keeps its chart but loses its thresholds.
• upsd_ups_last_collected_secs keeps its purpose — upsd gone silent — but on absolute thresholds (warn > 60 s, crit > 300 s, delay: up 1m) since 2026-08-05. Stock's 5 × $update_every resolved to 5 s, and the driver intermittently lets its data go stale (Data for UPS [rackups] is stale in upsd's log), leaving netdata with collection gaps of 6–14 s — enough for dozens of false alarm pairs. Anything keyed on this UPS must tolerate ten-second gaps; only a gap of minutes means upsd is really gone.
• upsd_10min_ups_load stays stock — ups.load is trustworthy.

Netdata cannot alert on the UPS going away — Telegram is the detector

When collection stops, netdata's whole upsd_local_rackups.* chart tree disappears, and all three alarms above go with it: they have no chart left to bind to. upsd_ups_last_collected_secs is therefore blind to the one failure it was written for, as the 2026-08-09 wedge showed. The status page's Telegram alert is what actually covers that case — it fired after ~5 minutes, as designed. The charts come back on their own once upsd answers again (service discovery re-creates the job within minutes; no restart needed).

caution

Do not add /etc/netdata/go.d/upsd.conf to slow the polling down

Netdata's upsd job is created by service discovery (net_listeners, on port 3493) and is named local, hence the upsd_local_rackups.* charts. A file-based job does not override it: it adds a second job, so upsd gets polled twice under a parallel upsd_rackups.* chart tree, with every alarm duplicated (tried, and reverted). Overriding the discovery rule would mean shipping a copy of the whole stock sd/net_listeners.conf, freezing every other collector's discovery — not worth it, and unnecessary: netdata's 1 s polling never touches the serial link, since upsd answers LIST VAR from the driver's cached copy.

Shutdown policy

upsmon runs with powervalue 0 and MINSUPPLIES 0: it logs and notifies, but nothing is ever shut down. That was correct while the UPS carried nothing, and it is still what is configured.

The shutdown policy is now behind the cabling

rumba no longer has a mains feed independent of this UPS, so in a real outage it runs on battery and is then hard-killed when the battery empties, taking the public sites, the ISC Learn DR mirror and the status page down uncleanly.

The HID interface reports 55 minutes at rumba's load, which is ample for a graceful shutdown and removes any reason to calibrate runtimecal. What remains is a plumbing choice (upsmon decides on nutdrv_qx's LB flag and cannot see the HID interface): trust LB and set powervalue 1, run a small watchdog on ups-read-hid, or bridge the HID data into a dummy-ups device. The trade-offs are weighed in the todo; whether LB is firmware-reported (rather than derived from the bogus charge estimate) is only settled by a controlled discharge.

Either way, rumba shutting itself down also stops the alerting that would report it: the Telegram message marks the start of an outage, never the end.