Rumba
The ISC teachers tooling machine, hosted onsite. Since 2026-07-31 it runs Proxmox VE and is the first node of the target ISC³ architecture, whose guiding principle is a single Proxmox base layer across the whole fleet. This page is the current state; the pre-Proxmox legacy configuration and the dated operation logs live in Rumba — history & operations.
At a glance
| Role | Proxmox VE node — teachers tooling & ISC services |
| Machines | 1× Dell R7920 (rumba) |
| Hypervisor | Proxmox VE 9.2.2 |
| Services | Landing page + reverse proxy (srv-web01), four static test sites (CT 101–104), a disposable GitLab CE test instance (CT 105, internal only), the ISC Learn DR mirror (VM 107), the rack status page (CT 108), the NetBird VPN (VM 109) and Netdata monitoring on the host itself |
| Access | VPN, then https://rumba:8006 or SSH |
| Managed from | provisioning/pve/ — the unattended install, then post-install.sh for the node's own configuration: repos, the hdd pool, the storages, both backup jobs, netdata. Idempotent, so re-running it also checks the node still matches this page |
Hardware
- 1× Dell R7920 (
rumba), at U6–U7 in the rack - 2× Intel Xeon Gold 6140 @ 2.30 GHz (72 threads total)
- 187 GB RAM
- NVIDIA T1000 8 GB (plus the onboard Matrox G200eW3)
- Storage: 1× Samsung MZPLJ3T2HBJR 3.2 TB NVMe (system + VM storage), 8× Toshiba MQ01ACF050
500 GB SATA HDDs (7200 rpm, spinning) — since 2026-08-01 the
hddZFS RAIDZ2 pool, see storage layout
Network
Rumba sits directly on the Calypso network LAN. Its eno1 NIC is attached
to the CCR2004 core router on port sfp+5 and is enslaved to the vmbr0 bridge.
| Item | Value |
|---|---|
| FQDN | rumba.calypso |
vmbr0 | 192.168.88.51/24 (static) |
| Gateway / DNS | 192.168.88.1 (CCR2004) |
| iDRAC | 192.168.92.10 (iDRAC9 Enterprise, perpetual licence) |
The address is configured statically rather than by DHCP: the CCR2004 still holds a reservation
for eno1, but its SFP+ link negotiates too slowly for the Proxmox installer's DHCP window — see the
remote install process for the full story.
Outbound IPv4 works, but the network has no usable IPv6 route while DNS does return AAAA records, so
clients that prefer IPv6 (a bare curl https://…) can stall — force IPv4 or adjust /etc/gai.conf
if it becomes a nuisance. ICMP to the Internet is filtered upstream for the whole subnet, so
ping 8.8.8.8 failing is normal and not a symptom.
Guests
All guests are unprivileged Debian 13 LXC containers, except srv-runner01 and srv-learn-test
(VMs).
| ID | Name | Address | Role |
|---|---|---|---|
| 100 | srv-web01 | 192.168.88.150 | Reverse proxy + landing page — the only guest reachable from the Internet; terminates TLS for every public name |
| 101 | srv-test | 192.168.88.151 | Static site backend, test.isc-vs.ch |
| 102 | srv-test1 | 192.168.88.152 | Static site backend, test1.isc-vs.ch |
| 103 | srv-test2 | 192.168.88.153 | Static site backend, test2.isc-vs.ch |
| 104 | srv-test3 | 192.168.88.154 | Static site backend, test3.isc-vs.ch |
| 105 | srv-gitlab | 192.168.88.155 | Disposable GitLab CE test instance (July 2026) — internal only, http://srv-gitlab.calypso over the VPN; 4 cores / 8 GB RAM / 32 GB disk. Destroy when done testing |
| 106 | srv-runner01 | 192.168.88.156 | Disposable CI runner VM for the GitLab test (Aug 2026) — Debian 13 cloud image, 24 vCPU / 64 GB RAM / 40 GB disk, shell-executor gitlab-runner, concurrent = 4. Destroy with srv-gitlab |
| 107 | srv-learn-test | 192.168.88.157 | ISC Learn DR mirror (VM, Aug 2026) — full Moodle restored from hannibal, http://learn-test.calypso/learn over the VPN, mail muzzled by design; 8 vCPU / 16 GB RAM / 40 GB NVMe + 400 GB on hdd-vm. Runbook: fromscratch.md |
| 108 | srv-status | 192.168.88.158 | Rack status page + thermal alerts (Aug 2026) — internal only, http://srv-status.calypso over the VPN; managed from provisioning/status/ |
| 109 | srv-netbird | 192.168.88.159 | NetBird VPN control plane (VM, Aug 2026) — https://vpn.isc-vs.ch through srv-web01, also the routing peer for the rack subnets; 2 vCPU / 4 GB / 20 GB NVMe; managed from provisioning/netbird/ |
| 110 | srv-keycloak | 192.168.88.160 | Keycloak identity broker (Aug 2026) — https://sso.isc-vs.ch through srv-web01, admin console behind the admin gate; federates to SWITCH edu-ID; PostgreSQL in-container; 2 cores / 4 GB / 12 GB; managed from provisioning/keycloak/ |
| 111 | srv-vaultwarden | 192.168.88.162 | Vaultwarden password vault (Aug 2026) — https://vault.isc-vs.ch through srv-web01, /admin behind the admin gate; login through Keycloak, invite-only; official image under systemd (nesting=1,keyctl=1); SQLite + nightly 01:30 dump; 2 cores / 2 GB / 12 GB; managed from provisioning/vaultwarden/ |
Backends serve plain HTTP on the LAN only; they hold no certificates and are not port-forwarded. Adding a site therefore never touches the router.
Monitoring
Netdata runs directly on the PVE host (installed 2026-08-01, v2.10.4). Running on the host — rather than in a guest — lets the cgroups collector see every VM and container automatically: each guest gets its own CPU / RAM / disk-I/O / network charts with no agent inside the guest, and new guests appear without any configuration.
| Item | Value |
|---|---|
| Dashboard | http://rumba:19999 — VPN only, not published through the reverse proxy |
| Per-guest charts | cgroup_srv-* sections on the dashboard |
| Updates | daily auto-update via /etc/cron.daily/netdata-updater |
| Telemetry | anonymous statistics disabled at install (--disable-telemetry) |
| Netdata Cloud | claimed since the install (space room 69456813-…, claim state in /var/lib/netdata/cloud.d/) — so alert transitions are mailed by Netdata Cloud, and that is the only alert mail that leaves rumba: the host's own postfix has no working transport (Email plan). A to: silent in a local health file does not stop those mails; that is a per-room setting in the Cloud UI |
The dashboard needs no login: it is reachable only from the Calypso LAN / VPN, like the Proxmox UI.
If it is ever published publicly, put it behind authentication on srv-web01 first.
Rack status page & thermal alerts
srv-status (CT 108, installed 2026-08-01) serves the rack status page at
http://srv-status.calypso (VPN/LAN only), a 2-year temperature history at
/graphs.html (Aug 2026), and sends Telegram thermal alerts when the PDU probe
runs hot — full description, data flow and operations on the
rack status service page.
Storage layout
| Proxmox storage | Type | Backing | Size |
|---|---|---|---|
local | dir | /var/lib/vz on pve/root | 100 GB |
local-lvm | lvmthin | pve/data thin pool | 2.76 TiB |
hdd-backup | dir | /hdd/backup ZFS dataset | ~2.5 TiB usable (shared with hdd-vm) |
hdd-vm | zfspool | hdd/vm ZFS dataset | ~2.5 TiB usable (shared with hdd-backup) |
pbs-fs2500 | pbs | PBS VM on the FS2500 | 2 TiB datastore |
Plus an 8 GB swap LV.
The hdd ZFS pool (created 2026-08-01)
The eight SATA HDDs, deliberately excluded from the install, form a single RAIDZ2 vdev — pool
hdd, 3.62 TiB raw / ~2.5 TiB usable, mounted at /hdd. RAIDZ2 was chosen over mirrors because all
eight drives are the same model with identical wear — a correlated-failure profile where surviving
any two dead disks matters more than IOPS (details in the
creation log). Pool options: ashift=12, compression=lz4,
atime=off, xattr=sa, acltype=posixacl; vdev members are referenced by /dev/disk/by-id so
device-name reshuffles can't confuse the pool.
hdd-backup(diron datasethdd/backup,is_mountpointguard) holds vzdump backups, ISOs and CT templates. A nightly vzdump job at 02:00 backs up all guests to it (snapshot mode, zstd; mail on failure), pruningkeep-daily=3(trimmed 2026-08-03: one nightly run is ~300 GB — 97 % of it VM 107 — so the pool can never hold a longer schedule). The full 7 daily / 4 weekly / 3 monthly retention lives off-host on the PBS VM on the FS2500, fed by a second job at 03:00.hdd-vm(zfspoolon datasethdd/vm, sparse) is for disk images that tolerate spinning rust — bulk/cold volumes, not OS disks; keep VM system disks onlocal-lvm(NVMe).
Scrubbing runs on the Proxmox default schedule (/etc/cron.d/zfsutils-linux, second Sunday of each
month).
Package repositories
PVE 9 uses the deb822 format in /etc/apt/sources.list.d/, not legacy one-line entries.
Configured 2026-07-31 for a node without a subscription, and reproduced by
provisioning/pve/post-install.sh:
| File | State |
|---|---|
pve-no-subscription.sources | added — download.proxmox.com/debian/pve trixie pve-no-subscription |
pve-enterprise.sources | disabled with Enabled: false (it returns 401 without a subscription) |
ceph.sources | switched from the enterprise component to no-subscription (this node runs no Ceph, but packages stay installable) |
debian.sources | untouched (trixie, updates, security) |
The originals are kept as *.orig in /root/.cache/apt-repos/. apt-get update now completes with
no 401s. Re-enable the enterprise repos if a subscription is ever purchased. Pending updates are
tracked in the ops todo → Rumba.
Access
Everything here is behind the VPN — the node is not published, and the
services it hosts reach the Internet through the srv-web01 reverse
proxy, never by a port opened to this machine.
| Path | Address | Authentication |
|---|---|---|
| Proxmox web UI | https://rumba:8006 | realm ISC SSO (edu-ID, below) or Linux PAM for root@pam |
| SSH | ssh root@rumba | pmudry's key; password in docs/secretzone/rumba.md |
| Guest consoles | web UI → the guest → Console | as above — the way into a container with no SSH |
| Netdata | http://rumba:19999 (monitoring) | none — VPN only, never published |
| iDRAC9 | 192.168.92.10 | secretzone |
The web UI uses the ISC typography (Manrope/Inter), deployed from provisioning/pve/ui-theme/ —
re-run deploy-ui-theme.sh and check-coverage.sh after any pve-manager upgrade; the full
engineering rationale is in the history page.
SSO login (edu-ID via Keycloak)
Since 2026-08-05 this node and the carnaval cluster carry an OpenID
Connect realm named isc, pointed at Keycloak — so a Proxmox login
is an edu-ID login, and Proxmox is a client of the broker rather than of the federation (one AAI
registration for the fleet, and the claim-derived groups come for free). This section documents it
for the whole fleet. root@pam is untouched and stays the preselected realm.
| Realm | isc, type openid, issuer https://sso.isc-vs.ch/realms/isc |
| Keycloak client | proxmox (confidential), created by provisioning/pve/keycloak-client.sh |
| PVE side | provisioning/pve/sso-realm.sh, idempotent — one run per cluster, /etc/pve is shared (so carnaval0 covered all three carnaval nodes) |
| Username claim | email, so the PVE user id is pierre-andre.mudry@hevs.ch@isc |
| Who may log in | only pre-created users — --autocreate 0 |
| What grants access | Administrator on / to the PVE group rack-admins-isc, filled at each login from the Keycloak group rack-admins |
--autocreate 0 is the authorisation model, not a temporary setting. Every member of realm
isc can pass the Keycloak login — that is the whole point of the broker — so autocreation would
turn "has an edu-ID and an HES-SO affiliation" into "has a Proxmox account". Such an account has no
privileges, but it exists, is listed, and is one ACL slip from access. Adding an administrator is
therefore two deliberate acts: membership of the Keycloak group rack-admins — from the roster,
add-only: the sync adds, never removes — and an ADMINS=
entry in provisioning/pve/sso-realm.sh re-run once per cluster (or a manual pveum user add).
Forgetting the second act is a clean "access denied" after a perfectly good SSO login — Martin
Tavernier hit it on 2026-08-06.
Group membership is read from Keycloak on every login (--groups-claim groups --groups-overwrite 1), so revoking rack-admins there removes Proxmox access at the next login
with no action on the node — the same discipline as the claim mappers' syncMode=FORCE. Two
consequences:
--groups-autocreateis 0, and that is what makes the claim safe to trust. It carries every Keycloak group the person is in (hes-so,rack-admins, …); PVE silently ignores the ones it has no group for, so only a group deliberately created on the node can grant anything.- A PVE group membership set by hand is wiped at that user's next login. For a realm-
iscuser, Keycloak is the only place to change groups.
PVE::API2::OpenId maps a claim value X to a PVE group named X-<realm> — so the Keycloak
group rack-admins requires a PVE group rack-admins-isc, and sso-realm.sh derives the name
that way. Create the PVE group under the bare claim name instead and the intersection is empty:
the login succeeds, the user lands on an empty UI, and the only trace is an
openid overwrite groups enabled line in the journal with no matching "added to groups" line
after it. Nothing anywhere says the names failed to match.
Note this is the opposite convention to NetBird, which
matches the claim value verbatim — the same Keycloak group is rack-admins there and
rack-admins-isc here.
Three things that will otherwise cost time:
- The redirect URI is the bare origin, no trailing slash — the web UI sends
redirect-url = location.origin, sohttps://rumba:8006andhttps://192.168.88.51:8006are different registrations and a…/*wildcard matches neither. Every name and address an admin might type is listed inkeycloak-client.sh; a missing one fails at the Keycloak page with "Invalid parameter: redirect_uri", and the admin console that would fix it is itself behind the SSO gate. username-claimcannot be changed on an existing realm — PVE accepts it onrealm addonly, because changing it renames every user. Correcting it means deleting the realm, re-running the script, and cleaning up the orphaned user and ACL lines by hand.- The node resolves the issuer over the LAN, via the CCR2004 static record
sso.isc-vs.ch → 192.168.88.150; the router does not hairpin the public address.sso-realm.shchecks discovery before touching anything, so a DNS regression fails there rather than at a user's login.
Logging out of Proxmox drops its own ticket only — it never calls Keycloak's end-session endpoint, so the SSO session survives and signing back in is immediate. There is no post-logout URI registered on the client for that reason.
user.cfg lives in /etc/pve, which is cluster-local: logging in on rumba fills the group there
and leaves carnaval's copy empty. Until a cluster has seen one real login, sso-realm.sh also
grants each ADMINS= entry Administrator by name (USER_ACL=1), so a login arriving without
a usable claim still lands on a working UI instead of an empty one. Re-run with USER_ACL=0 to
remove those once pveum group list shows the user in rack-admins-isc.
Both clusters have passed that point since 2026-08-05: the group is the only Administrator grant on each, and no user holds one by name.
Next steps
Open items for this node are tracked in the ops todo → Rumba.