Rumba

The first node of the Proxmox production cluster (see Future work), hosted onsite. Since 2026-07-31 it runs Proxmox VE and is the first node of the target ISC³ architecture, whose guiding principle is a single Proxmox base layer across the whole fleet. This page is the current state; the pre-Proxmox legacy configuration and the dated operation logs live in Rumba — history & operations.

At a glance

Role	Proxmox VE node — teachers tooling & ISC services
Machines	1× Dell Precision 7920 Rack (rumba)
Hypervisor	Proxmox VE 9.2.6 (Aug 2026)
Services	Landing page + reverse proxy (srv-web01), four static test sites (CT 101–104), the ISC Learn DR mirror (VM 107), the rack status page (CT 108), the NetBird VPN (VM 109), NetBox (CT 113), Oxidized (CT 114), Healthchecks (CT 115), ntfy (CT 116), the Docker Hub pull-through registry (CT 120) and Netdata monitoring on the host itself
Access	VPN, then https://rumba:8006 or SSH
Managed from	provisioning/pve/ — the unattended install, then post-install.sh for the node's own configuration: repos, the hdd pool, the storages, both backup jobs, netdata. Idempotent, so re-running it also checks the node still matches this page

Hardware

Full inventory read from the host and the iDRAC, August 2026. The machine is a Dell Precision 7920 Rack — a rack workstation, not a PowerEdge R7920; the two use different service manuals and different spare part numbers.

The machine was donated to ISC — it is the one 23N307 server that appears in no ServerShop24 invoice (acquisition register); its T1000 matches the pair bought with calypsomaster in Dec 2023.

Model	Dell Precision 7920 Rack, service tag 5YR15Z2, BIOS 2.25.1
Position	U6–U7 in the rack
CPU	2× Intel Xeon Gold 6140 @ 2.30 GHz — 18 c / 36 t each, 36 c / 72 t total
Memory	192 GB (187 GB usable) — 24× 8 GB DDR4-2666 single-rank Micron, every slot populated
GPU	NVIDIA T1000 8 GB (PCIe slot 3) + onboard Matrox G200eW3
PSU	2× 1600 W Delta, redundant — 262 W drawn at idle-ish load (Aug 2026)

Memory has no free slot

All 24 DIMM slots carry an 8 GB module (2 DIMMs per channel across the 6140's 12 channels), so RAM cannot be added, only replaced. Going to 384 GB means discarding 24 modules for 24× 16 GB; 768 GB means 24× 32 GB. Budget the whole set, not an increment.

Storage

Device	Where	Role
1× Samsung MZPLJ3T2HBJR 3.2 TB NVMe	PCIe slot 6 (PCIeSSD.Slot.6-C)	system + VM storage — pve LVM-thin
8× Toshiba MQ01ACF050 500 GB SATA, 7200 rpm	front bays, on the embedded AHCI controllers	the hdd ZFS RAIDZ2 pool — see storage layout

There is no RAID or HBA card: the eight spinners hang off the chipset SATA controllers, which is why ZFS sees them directly and no controller mode has to be managed.

PCIe slots — five free

Slot	Width	Occupant
1	Gen3 x8	free
2	Gen3 x16	free
3	Gen3 x16	NVIDIA T1000
4	Gen3 x8	free
5	Gen3 x16	free
6	Gen3 x16	Samsung 3.2 TB NVMe
7	Gen3 x16	free
8	Gen3 x8	Mellanox ConnectX-6 (dual port)

Three x16 and two x8 are free, which is where any added NVMe carrier or HBA goes.

A ConnectX-6 is already fitted and not cabled — port ens8f1np1 shows no link. The card is in a Gen3 x8 slot, which caps it near 63 Gb/s; move it to slot 2, 5 or 7 before running anything faster than 25 G through it. §6.1 of the target architecture plans 25 G SFP28 for pve-03 and was written without knowing this card was there.

Network

Rumba sits directly on the ISC³ network LAN. Its eno1 NIC is attached to the CCR2004 core router on port sfp+5 and is enslaved to the vmbr0 bridge.

Item	Value
FQDN	rumba.isc3
vmbr0	192.168.88.51/24 (static)
Gateway / DNS	192.168.88.1 (CCR2004)
iDRAC	192.168.92.10 (iDRAC9 Enterprise, perpetual licence)

Interfaces present (Aug 2026): eno1/eno2 Intel X550 10 G — only eno1 is cabled and up; eno3/eno4 Intel I350 1 G, both unused; ens8f1np1 on the ConnectX-6, unused.

The address is configured statically rather than by DHCP: the CCR2004 still holds a reservation for eno1, but its SFP+ link negotiates too slowly for the Proxmox installer's DHCP window — see the remote install process for the full story.

IPv6 and ICMP

Outbound IPv4 works, but the network has no usable IPv6 route while DNS does return AAAA records, so clients that prefer IPv6 (a bare curl https://…) can stall — force IPv4 or adjust /etc/gai.conf if it becomes a nuisance. ICMP to the Internet is filtered upstream for the whole subnet, so ping 8.8.8.8 failing is normal and not a symptom.

Guests

All guests are unprivileged Debian 13 LXC containers, except isc-adm, srv-docker01, srv-learn-test and srv-netbird (VMs).

ID	Name	Address	Role
100	srv-web01	192.168.88.150	Reverse proxy + landing page — the only guest reachable from the Internet; terminates TLS for every public name
101	srv-test	192.168.88.151	Static site backend, test.isc-vs.ch
102	srv-test1	192.168.88.152	Static site backend, test1.isc-vs.ch
103	srv-test2	192.168.88.153	Static site backend, test2.isc-vs.ch
104	isc-adm	192.168.88.171	ISC registration manager (VM, Sep 2026) — internal only, http://192.168.88.171:8000 over the VPN; runs ISC-HEI/isc-manager from /root/isc-manager as the isc-manager.service systemd unit (rebuilds the frontend at start, FastAPI serves it on 8000); update with just deploy on the VM; SSH as root only, cloud-init keys; 8 vCPU / 8 GB / 40 GB NVMe
105	srv-docker01	192.168.88.154	Docker host (VM, Aug 2026) — a rack administrator's own stacks, published as test3.isc-vs.ch through srv-web01 behind the user gate; the proxy forwards to port 8080 and nothing else; 4 cores / 8 GB / 40 GB NVMe; managed from provisioning/docker01/
106	srv-freeipa	192.168.88.173	FreeIPA (VM, Sep 2026) — found on the node on 2026-09-10 while allocating an address; 2 cores / 4 GB; not yet documented here
107	srv-learn-test	192.168.88.157	ISC Learn DR mirror (VM, Aug 2026) — full Moodle restored from hannibal, refreshed 2026-09-06 (data of the 2026-09-06 06:25 dump, frontpage and content scripts re-applied), http://learn-test.isc3/learn over the VPN, mail muzzled by design; 8 vCPU / 16 GB RAM / 40 GB NVMe + 400 GB on hdd-vm. Runbook: fromscratch.md
108	srv-status	192.168.88.158	Rack status page + thermal alerts (Aug 2026) — internal only, http://srv-status.isc3 over the VPN; managed from provisioning/status/
109	srv-netbird	192.168.88.159	NetBird VPN control plane (VM, Aug 2026) — https://vpn.isc-vs.ch through srv-web01, also the routing peer for the rack subnets; 2 vCPU / 4 GB / 20 GB NVMe; managed from provisioning/netbird/
110	srv-keycloak	192.168.88.160	Keycloak identity broker (Aug 2026) — https://sso.isc-vs.ch through srv-web01, admin console behind the admin gate; federates to SWITCH edu-ID; PostgreSQL in-container; 2 cores / 4 GB / 12 GB; managed from provisioning/keycloak/
112	srv-mail	192.168.88.163	Outbound SMTP relay (Aug 2026) — internal only, holds the sole copy of the Infomaniak credential; rack devices submit on plain 25 with no password of their own, source and recipient allowlists; 1 core / 512 MB / 4 GB; managed from provisioning/mail/
113	srv-netbox	192.168.88.164	NetBox DCIM/IPAM (Aug 2026) — internal only, http://srv-netbox.isc3; Keycloak SSO gated to role-rack-admins; PostgreSQL + Redis in-container, nesting=1; 2 cores / 4 GB / 12 GB; managed from provisioning/netbox/
114	srv-oxidized	192.168.88.165	Oxidized network config backup (Aug 2026) — internal only, http://srv-oxidized.isc3; pulls both MikroTiks hourly into a local git repo; 1 core / 1 GB / 6 GB; managed from provisioning/oxidized/
115	srv-healthchecks	192.168.88.166	Healthchecks dead-man switch (Aug 2026) — internal only, http://srv-healthchecks.isc3; alerts by email + ntfy when a cron job stops pinging; 1 core / 1 GB / 6 GB; managed from provisioning/healthchecks/
116	srv-ntfy	192.168.88.167	ntfy push notifications (Aug 2026) — internal only, http://srv-ntfy.isc3; auth deny-all (publisher/reader); 1 core / 512 MB / 4 GB; managed from provisioning/ntfy/
117	srv-stirling	192.168.88.168	Stirling-PDF toolbox (Aug 2026) — pdf.isc-vs.ch through srv-web01, behind the user gate; also http://srv-stirling.isc3 over the VPN with no authentication; Java 25 + LibreOffice/Tesseract/Ghostscript, no user database; 2 cores / 4 GB / 12 GB; managed from provisioning/tools/
118	srv-it-tools	192.168.88.169	IT-Tools (Aug 2026) — tools.isc-vs.ch through srv-web01, open; static release zip served by Caddy; 1 core / 512 MB / 4 GB; managed from provisioning/tools/
120	srv-registry	192.168.88.172	Docker Hub pull-through cache (Sep 2026) — internal only, http://registry.isc3:5000, used by the carnaval k8s servers; 2 cores / 2 GB / 8 GB + 200 GB cache mount point with backup=0; managed from provisioning/registry/
119	srv-cyberchef	192.168.88.170	CyberChef (Aug 2026) — cyberchef.isc-vs.ch through srv-web01, open; static release zip served by Caddy; 1 core / 512 MB / 6 GB; managed from provisioning/tools/
121	srv-stats	192.168.88.174	Audience statistics (Umami) (VM, Sep 2026) — stats.isc-vs.ch through srv-web01, tracker public, dashboard behind the admin gate; Docker Compose, 2 cores / 2 GB / 16 GB; managed from provisioning/stats/
123	srv-tisc-editor	192.168.88.36	TISC Editor (Typst) (LXC, Sep 2026) — https://tisc.isc-vs.ch through srv-web01, own auth via a Keycloak client, migrated off the legacy VPS marcellus 2026-09-18; Docker (tisc-app-prod/tisc-docs/tisc-nginx/tisc-db), nesting=1,keyctl=1; static IP with a DHCP reservation; 1 core / 2 GB / 10 GB; managed from /root/tisc-editor on the guest itself (ISC-HEI/tisc-editor)

Backends serve plain HTTP on the LAN only; they hold no certificates and are not port-forwarded. Adding a site therefore never touches the router.

Monitoring

Netdata runs directly on the PVE host (installed 2026-08-01, v2.10.4). Running on the host — rather than in a guest — lets the cgroups collector see every VM and container automatically: each guest gets its own CPU / RAM / disk-I/O / network charts with no agent inside the guest, and new guests appear without any configuration.

Item	Value
Dashboard	http://rumba:19999 — VPN only, not published through the reverse proxy
Per-guest charts	cgroup_srv-* sections on the dashboard
Updates	daily auto-update via /etc/cron.daily/netdata-updater
Telemetry	anonymous statistics disabled at install (--disable-telemetry)
Netdata Cloud	claimed since the install (space room 69456813-…, claim state in /var/lib/netdata/cloud.d/) — so alert transitions are mailed by Netdata Cloud, independently of the node's own smarthost. A to: silent in a local health file does not stop those mails; that is a per-room setting in the Cloud UI

The dashboard needs no login: it is reachable only from the rack LAN / VPN, like the Proxmox UI. If it is ever published publicly, put it behind authentication on srv-web01 first.

Rack status page & thermal alerts

srv-status (CT 108, installed 2026-08-01) serves the rack status page at http://srv-status.isc3 (VPN/LAN only), a 2-year temperature history at /graphs.html (Aug 2026), and sends Telegram thermal alerts when the PDU probe runs hot — full description, data flow and operations on the rack status service page.

Storage layout

Proxmox storage	Type	Backing	Size
local	dir	/var/lib/vz on pve/root	100 GB
local-lvm	lvmthin	pve/data thin pool	2.76 TiB
hdd-backup	dir	/hdd/backup ZFS dataset	~2.5 TiB usable (shared with hdd-vm)
hdd-vm	zfspool	hdd/vm ZFS dataset	~2.5 TiB usable (shared with hdd-backup)
pbs-fs2500	pbs	PBS VM on the FS2500	5 TiB datastore

Plus an 8 GB swap LV.

The hdd ZFS pool (created 2026-08-01)

The eight SATA HDDs, deliberately excluded from the install, form a single RAIDZ2 vdev — pool hdd, 3.62 TiB raw / ~2.5 TiB usable, mounted at /hdd. RAIDZ2 was chosen over mirrors because all eight drives are the same model with identical wear — a correlated-failure profile where surviving any two dead disks matters more than IOPS (details in the creation log). Pool options: ashift=12, compression=lz4, atime=off, xattr=sa, acltype=posixacl; vdev members are referenced by /dev/disk/by-id so device-name reshuffles can't confuse the pool.

• hdd-backup (dir on dataset hdd/backup, is_mountpoint guard) holds vzdump backups, ISOs and CT templates. A nightly vzdump job at 02:00 backs up all guests to it (snapshot mode, zstd; mails failures through the smarthost since 2026-08-10), pruning keep-daily=3 (trimmed 2026-08-03: one nightly run is ~300 GB — 97 % of it VM 107 — so the pool can never hold a longer schedule). The full 14 daily / 8 weekly / 6 monthly retention lives off-host on the PBS VM on the FS2500, fed by a second job at 03:00.
• hdd-vm (zfspool on dataset hdd/vm, sparse) is for disk images that tolerate spinning rust — bulk/cold volumes, not OS disks; keep VM system disks on local-lvm (NVMe).
• hdd/hannibal-mirror (dataset, /hdd/hannibal-mirror, since 2026-09-05) holds the on-site copy of ISC Learn, versioned by ZFS snapshots after each pull (backups). It is its own dataset on purpose: a snapshot of hdd/backup would pin every vzdump the nightly rotation deletes — ~300 GB a night — and fill the pool within days. Never snapshot hdd/backup. Anything else that needs versions gets its own dataset the same way; with block cloning active on this pool, cp -a --reflink=auto moves a tree between datasets in minutes without using space (--reflink=always fails across datasets: the kernel rejects it).

Scrubbing runs on the Proxmox default schedule (/etc/cron.d/zfsutils-linux, second Sunday of each month).

Package repositories

PVE 9 uses the deb822 format in /etc/apt/sources.list.d/, not legacy one-line entries. Configured 2026-07-31 for a node without a subscription, and reproduced by provisioning/pve/post-install.sh:

File	State
pve-no-subscription.sources	added — download.proxmox.com/debian/pve trixie pve-no-subscription
pve-enterprise.sources	disabled with Enabled: false (it returns 401 without a subscription)
ceph.sources	switched from the enterprise component to no-subscription (this node runs no Ceph, but packages stay installable)
debian.sources	untouched (trixie, updates, security)

The originals are kept as *.orig in /root/.cache/apt-repos/. apt-get update now completes with no 401s (the enterprise repos would be re-enabled with a subscription). Pending updates are tracked in the ops todo → Rumba.

Access

Everything here is behind the VPN — the node is not published, and the services it hosts reach the Internet through the srv-web01 reverse proxy, never by a port opened to this machine.

Path	Address	Authentication
Proxmox web UI	https://rumba:8006	realm ISC SSO (edu-ID, below) or Linux PAM for root@pam
SSH	ssh root@rumba	the keys in provisioning/pve/root-ssh-keys.sh (below); password in secretzone/rumba.md
Guest consoles	web UI → the guest → Console	as above — the way into a container with no SSH
Node shell	web UI → the node → Shell	only root@pam lands in a root shell; every other user, Administrator included, gets a login: prompt — pve-manager hard-codes it (Nodes.pm: "we do not have a superuser role"), same for the Upgrade button. Use SSH as root instead
Netdata	http://rumba:19999 (monitoring)	none — VPN only, never published
iDRAC9	192.168.92.10	secretzone

The web UI uses the ISC typography (Manrope/Inter), deployed from provisioning/pve/ui-theme/ — re-run deploy-ui-theme.sh and check-coverage.sh after any pve-manager upgrade; the rationale is in the history page.

Root SSH keys

There are no per-person accounts on the node: shell access is root, and who holds it is the list in provisioning/pve/root-ssh-keys.sh — pmudry, Adrien (since 2026-08-14) and Yacine (since 2026-09-01). Running the script on a node asserts that list into /etc/pve/priv/authorized_keys (the file PVE owns; /root/.ssh/authorized_keys is a symlink to it, and on a cluster it replicates to every node, so one run per cluster). It is add-only, so revoking a key means editing the file on the node and the script. The node's own ssh-rsa root@rumba key sits in the same file and is PVE's, not a person's.

SSO login (edu-ID via Keycloak)

Since 2026-08-05 this node and the carnaval cluster carry an OpenID Connect realm named isc, pointed at Keycloak — so a Proxmox login is an edu-ID login, and Proxmox is a client of the broker rather than of the federation (one AAI registration for the fleet, and the claim-derived groups come for free). This section documents it for the whole fleet. root@pam is untouched and stays the preselected realm.

Realm	isc, type openid, issuer https://sso.isc-vs.ch/realms/isc
Keycloak client	proxmox (confidential), created by provisioning/pve/keycloak-client.sh
PVE side	provisioning/pve/sso-realm.sh, idempotent — one run per cluster, /etc/pve is shared (so carnaval0 covered all three carnaval nodes)
Username claim	email, so the PVE user id is pierre-andre.mudry@hevs.ch@isc
Who may log in	only pre-created users — --autocreate 0
What grants rights	Administrator on / to the PVE group role-rack-admins-isc, PVEAuditor on / to role-pve-auditor-isc — each filled at every login from the Keycloak group of the same bare name (role-rack-admins, role-pve-auditor)

--autocreate 0 is the authorisation model, not a temporary setting. Every member of realm isc can pass the Keycloak login — that is the whole point of the broker — so autocreation would turn "has an edu-ID and an HES-SO affiliation" into "has a Proxmox account". Such an account has no privileges, but it exists, is listed, and is one ACL slip from access. Adding an administrator is therefore two deliberate acts: membership of the Keycloak group role-rack-admins — from the roster, add-only: the sync adds, never removes — and an ADMINS= entry in provisioning/pve/sso-realm.sh re-run once per cluster (or a manual pveum user add). Forgetting the second act is a clean "access denied" after a successful SSO login.

Read-only access (since 2026-08-14) is the same pair of acts with role-pve-auditor and AUDITORS= in place of role-rack-admins and ADMINS=, granting PVEAuditor on /. It differs on the Keycloak side: role-pve-auditor is an ordinary two-way roster entitlement, so dropping it from the line revokes the access at the next login, where role-rack-admins has to be taken out in the console. It grants no reach — the person still needs vpn-rack-operators to load the UI at all.

Group membership is read from Keycloak on every login (--groups-claim groups --groups-overwrite 1), so revoking role-rack-admins there removes Proxmox access at the next login with no action on the node — the same discipline as the claim mappers' syncMode=FORCE. Two consequences:

• --groups-autocreate is 0, and that is what makes the claim safe to trust. It carries every Keycloak group the person is in (hes-so, role-rack-admins, …); PVE silently ignores the ones it has no group for, so only a group deliberately created on the node can grant anything.
• A PVE group membership set by hand is wiped at that user's next login. For a realm-isc user, Keycloak is the only place to change groups.

PVE appends the realm name to every group it reads from the claim

PVE::API2::OpenId maps a claim value X to a PVE group named X-<realm> — so the Keycloak group role-rack-admins requires a PVE group role-rack-admins-isc, and sso-realm.sh derives the name that way. Create the PVE group under the bare claim name instead and the intersection is empty: the login succeeds, the user lands on an empty UI, and the only trace is an openid overwrite groups enabled line in the journal with no matching "added to groups" line after it. Nothing anywhere says the names failed to match.

Note this is the opposite convention to NetBird, which matches the claim value verbatim — the same Keycloak group is role-rack-admins there and role-rack-admins-isc here.

Three things that will otherwise cost time:

• The redirect URI is the bare origin, no trailing slash — the web UI sends redirect-url = location.origin, so https://rumba:8006 and https://192.168.88.51:8006 are different registrations and a …/* wildcard matches neither. Every name and address an admin might type is listed in keycloak-client.sh; a missing one fails at the Keycloak page with "Invalid parameter: redirect_uri", and the admin console that would fix it is itself behind the SSO gate.
• username-claim cannot be changed on an existing realm — PVE accepts it on realm add only, because changing it renames every user. Correcting it means deleting the realm, re-running the script, and cleaning up the orphaned user and ACL lines by hand.
• The node resolves the issuer over the LAN, via the CCR2004 static record sso.isc-vs.ch → 192.168.88.150; the router does not hairpin the public address. sso-realm.sh checks discovery before touching anything, so a DNS regression fails there rather than at a user's login.

Logging out of Proxmox drops its own ticket only — it never calls Keycloak's end-session endpoint, so the SSO session survives and signing back in is immediate. There is no post-logout URI registered on the client for that reason.

Group membership is per cluster, so each one needs its own first login

user.cfg lives in /etc/pve, which is cluster-local: logging in on rumba fills the group there and leaves carnaval's copy empty. Until a cluster has seen one real login, sso-realm.sh also grants each ADMINS= entry Administrator by name (USER_ACL=1), so a login arriving without a usable claim still lands on a working UI instead of an empty one. Re-run with USER_ACL=0 to remove those once pveum group list shows the user in role-rack-admins-isc.

Both clusters are past that point: the group is the only Administrator grant on each, and no user holds one by name.

Next steps

Open items for this node are tracked in the ops todo → Rumba.