Rumba — history & operations

How Rumba got to its current state: the dated operation logs, newest first, then the pre-Proxmox legacy configuration. Current facts live on the main page; finished work across the whole rack is indexed in the ops journal.

Vaultwarden retired (2026-08-16)

CT 111 (srv-vaultwarden, Vaultwarden 1.37.1, deployed 2026-08-04) was destroyed without ever entering service: the secrets plan settled on SOPS + age on 2026-08-13, and no shared secret was ever stored in it. It will not be used. Removed with it: the vault.isc-vs.ch vhost on srv-web01, the vaultwarden Keycloak client (and vault.isc-vs.ch from the oauth2-proxy client's webOrigins), the internal and public DNS records, its address in the mail relay's mynetworks, and provisioning/vaultwarden/ (in git history if ever needed). PBS/vzdump copies of the empty container age out on their own.

Rebooted for the UPS — the USB link stayed dead (2026-08-15)

First reboot since the 2026-07-31 install, to reset the xHCI controller after the UPS left the USB bus — no effect, the UPS did not enumerate. Guest shutdown to guests-back took about six minutes. onboot: 1 also starts guests that were deliberately stopped (it woke the parked CT 102–104); flip the flag when parking one. The reboot also took NetBird down (incident).

Outbound mail enabled (2026-08-10)

SInf granted the egress allow asked for on 2026-08-02 (TCP 465 to Infomaniak only), and the rack can send mail. Two steps the same day: PVE notifications first authenticated to Infomaniak directly, then CT 112 srv-mail was added as a Postfix relay and PVE was repointed at it, so no mail credential remains on the hypervisor — /etc/pve/priv/notifications.cfg now holds only the endpoint name. default-matcher targets rack-relay with match-severity warning,error, and both nightly vzdump jobs run notification-mode notification-system.

Verified end to end: a real backup run reported notified via target rack-relay, and the relay logged status=sent (250 2.0.0 Ok: queued as …). Both allowlists were checked by probe — a non-listed recipient gets 554 Recipient address rejected, and srv-web01, deliberately left out of mynetworks, gets 554 Client host rejected. Design, access-control layers and the remaining unconfigured consumers: Email; reconfigure with provisioning/mail/deploy-relay.sh and provisioning/pve/notify-smtp.sh.

The host's own postfix was then pointed at the relay too (provisioning/mail/node-relay.sh), with a root alias: zfs-zed and smartd were enabled all along and mailing root, so pool degradation and SMART pre-failure warnings had been reaching nobody. upsmon was likewise log-only and now mails ONBATT/LOWBATT/FSD/COMM failures. The 21 stale vzdump reports stuck in the queue since the months without a transport were deleted rather than delivered.

GitLab test environment destroyed (2026-08-06)

The disposable July 2026 GitLab CE evaluation — CT 105 srv-gitlab was destroyed per its teardown checklist: both guests, the five .calypso DNS entries on the CCR2004 (export refreshed), and the docs rows. The test had proven repo hosting, a Bun CI pipeline building this site, GitLab Pages, and runners both in-container and in a dedicated VM. Three deltas to keep if GitLab is ever deployed for real (the plan): isolate the runner's executor in Docker or its own guest (a shell executor runs any pushed code as a local user), publish through srv-web01 with a real isc-vs.ch name instead of HTTP-only .calypso names, and use one wildcard DNS entry (match-subdomain=yes) for Pages instead of one entry per project. The Debian 13 cloud image (rumba:/root/.cache/debian-13-genericcloud-amd64.qcow2, 328 MB) was kept for future VMs.

Both PSUs moved onto the UPS (2026-08-03)

Rumba's two power cords were moved from their PDU outlets to the rack UPS, one at a time so the machine never lost redundancy — ~5 s of lost input per PSU (bounded by the iDRAC event log), no reboot.

What the move changed, measured the same afternoon:

	Before	After
ups.load / output.current	1–2 % / 0.00 A	14 % / 1.0–1.1 A
Rumba's own draw (iDRAC)	—	265 W total
PDU rack total	~1.09 kW	~1.09 kW, unchanged

The UPS load matches rumba's own draw, so rumba is the only thing on it. The unchanged PDU total says the UPS's own input is itself fed from the PDU, so this did not move load off the rack's metering — and a PDU-side failure still reaches rumba, just through the battery.

Consequence worth knowing: rumba no longer has a mains feed independent of this UPS, which makes NUT's monitor-only setting the wrong one for the present cabling — see shutdown policy.

ISC typography in the web UI (2026-08-03)

The web UI uses the isc-hub fonts — Manrope for body and UI text, Inter for panel and window titles — instead of the stock helvetica, arial, verdana. Sixteen self-hosted woff2 faces (latin + latin-ext, weights 400–700) live in /usr/share/pve-manager/css/, so the UI needs no Internet access and no Google Fonts call.

Everything is version-controlled in provisioning/pve/ui-theme/; deploy-ui-theme.sh is idempotent and has a --revert. The standing rules:

• Family only. No font-size, line-height or font-weight is touched — ExtJS lays widgets out against fixed pixel metrics, and a size change clips labels.
• Never trust a font-family grep. The theme sets most widgets through the font: shorthand, so overriding only the font-family rules restyles the containers while every widget inside keeps Helvetica. check-coverage.sh re-derives the full selector list from the live stylesheets and fails if anything is uncovered — run it after every pve-manager or extjs upgrade.
• Icon fonts and frame metrics stay untouched. Font Awesome glyphs are selected by family name (a blanket rule turns icons into tofu), and the dh-*/th-* "families" are ExtJS frame-metric encodings, not fonts. The xterm.js/noVNC consoles and the RRD chart axes are deliberately left alone too.

Survives reboots, not upgrades

A pve-manager upgrade replaces index.html.tpl and drops the <link> — the CSS and font files survive. Re-run deploy-ui-theme.sh after such an upgrade. The pristine template is kept as /usr/share/pve-manager/index.html.tpl.isc-orig.

The hdd ZFS pool creation (2026-08-01)

The eight SATA HDDs had been deliberately excluded from the Proxmox install; on 2026-08-01 they were formed into the RAIDZ2 pool described in the storage layout. At pool creation all eight drives showed ~49,300 power-on hours each with clean SMART (zero reallocated or pending sectors on every disk) — the same model with identical wear, i.e. a correlated-failure profile where surviving any two dead disks matters more than IOPS. That is why RAIDZ2 was chosen over mirrors.

The nightly vzdump job at 02:00 (all guests, snapshot mode, zstd, mail on failure) was set up the same day.

Netdata installation (2026-08-01)

Netdata v2.10.4 was installed on the PVE host from the official repo.netdata.cloud stable channel, with anonymous telemetry disabled at install time (--disable-telemetry). Current dashboard facts are on the main page.

Legacy configuration (before 2026-07-31)

Rumba previously ran Ubuntu with services in LXC containers, on its own isolated subnets — 192.168.93.0/24 (servers, host rumba1 at .93.10) and 192.168.94.0/24 (containers) — behind an SInf-provided gateway. The CCR2004 still carries the old gateway addresses (192.168.92.1, 192.168.93.1, 192.168.94.254) but only the iDRAC subnet is still in use. The services that used to run there, kept for reference:

Service	Old container IP	Actual status (checked 2026-07-31)
ISC Inventory (Snipe-IT)	192.168.94.10	Alive elsewhere — inventory.isc-vs.ch resolves to hannibal and serves Snipe-IT. It had already moved off Rumba; nothing to redeploy.
Reverse proxy (Nginx)	192.168.94.13	Superseded by Caddy in srv-web01, which now holds the inbound 80/443 forwards
ISC Learn platform backup (Moodle) — testing and disaster recovery	192.168.94.14	Gone with the reinstall. Was published as test.isc-vs.ch — see the note below
Monitoring (TPI Rodrigo)	192.168.94.20	Gone with the reinstall — basic host + per-guest monitoring is now covered by Netdata on the host (2026-08-01)

caution

test.isc-vs.ch has been reused

That name was the published URL of the Moodle disaster-recovery copy (see the ISC Learn restore procedure). The DR container disappeared with the reinstall, and the name is now serving a static demo page. Resolved 2026-08-01: the DR copy was rebuilt as VM 107 under the internal-only name learn-test.calypso, so the public name stayed with the demo page.

A copy of the old /home (nine accounts) and the relevant /etc files sits in calypsomaster:~/rumba-preinstall-backup/.

An earlier revision of the Rumba page also listed a Gigabyte R282-Z92 in its hardware section; per the rack layout those are the separate epyc0 / epyc1 servers, also earmarked for Proxmox.