UPS controlled discharge test

A planned power-fail test of the rack UPS: pull its input, let it run on battery, and record what happens. Not yet performed — this is the runbook.

It answers three things at once, none of which is knowable any other way:

1. Is NUT's LB (low-battery) flag real? That decides whether a graceful shutdown can simply be switched on, or needs a watchdog around it — the three options.
2. What is the battery's actual autonomy today? The UPS advertises 55 min at rumba's load, but that figure does not appear to be corrected for battery ageing (see the warning below).
3. Does the alerting work end to end? The mains-loss Telegram alert should fire within 30 s.

Expect less than 55 minutes, and treat that as the finding

For a healthy pack the arithmetic supports it: 72 V × ~9 Ah ≈ 648 Wh, ~90 % inverter efficiency, and a Peukert derate at this discharge rate leaves ~500 Wh usable → ~100 min at rumba's 265 W. So 55 min is conservative for a new battery.

The risk is age, not size. The UPS was manufactured around 2021 and nothing in it reports a battery install date — not over NUT, not over the HID interface. Five-year-old VRLA cells in a warm rack can be at a third of their rated capacity, and the UPS's runtime figure looks like a load-indexed table lookup rather than a health-corrected measurement, so it would still say 55 min with a tired pack.

If the test comes in far under 55 min, the batteries are the finding — not the instrumentation.

Before

• Do not rely on NetBird for remote access. srv-netbird is VM 109 on rumba, so it dies with the guests. The MikroTik WireGuard runs on the CCR2004 and is unaffected — use that, or be on site.

• Shut the guests down gracefully (srv-web01, srv-gitlab, srv-learn-test, srv-keycloak, srv-runner01, srv-status, srv-test). Note that stopping srv-status also stops the status page and the Telegram alerting, so stop it last if you want to see the mains-loss alert fire.

• Start the logger — this is the step that makes the test worth doing:

  systemd-run --unit=ups-discharge --collect \
      /usr/local/bin/ups-log-discharge /var/log/ups-discharge.csv
  journalctl -fu ups-discharge

  It samples NUT and the HID interface every 10 s and fsyncs each line, so an abrupt power loss costs at most one sample. It prints a banner the moment OB appears and again when LB appears, with the elapsed time.

• While you are at the rack, look for a date sticker on the battery pack and write it down. It is the single best predictor of the result and it is not readable remotely.

During

Pull the UPS's input cord. Then leave it alone and watch journalctl -fu ups-discharge.

• A ⚡ mains-loss message should reach Telegram within 30 s (if srv-status is still up).
• Record roughly when the guests' absence stops mattering — the load should drop from ~14 % to whatever the bare host draws.

Stop at LB — do not run it to a crash

When the logger announces LB, restore mains and then shut rumba down cleanly with shutdown -h now. There is nothing left to learn past that point: LB is the trigger a graceful shutdown would use, so once you have seen it appear (and how much time it left), the question is answered. Letting the battery die instead only tests whether rumba survives having its plug pulled, which is the thing we are trying to prevent.

If LB never appears and the voltage approaches battery.voltage.low (62.4 V — 1.73 V/cell), stop anyway and record that: "LB was never raised" is a complete answer to question 1, and it rules out the cheap option.

After

• Keep /var/log/ups-discharge.csv — it is the discharge curve, and the only record of this battery's real capacity.
• Write the result up in the rumba history and index it in the journal; close or re-scope the todo items.
• If autonomy came in low, the batteries are a purchase, not a configuration change.