Carnaval — history & operations
How the playground cluster got to its current state: the dated operation logs, newest first. Current facts live on the main page; finished work across the whole rack is indexed in the ops journal.
SSO login wired to Keycloak (2026-08-05)
The cluster took an edu-ID login the same day rumba did, from one run of
provisioning/pve/sso-realm.sh on carnaval0 — /etc/pve is cluster-wide, so all three nodes were
covered at once. Administrator on / is held by the PVE group rack-admins-isc and by nobody by
name; root@pam is untouched and stays the preselected realm.
Two things were learnt here rather than on rumba, both recorded on the
rumba page: PVE appends the realm name to every group it reads from the claim, and
group membership is per cluster — carnaval needed its own first login before rack-admins-isc
filled, even though rumba's had already succeeded.
GPU passthrough re-verified, reference guest destroyed (2026-08-05)
The A2 cards on carnaval0/1 were confirmed to behave identically to the T4 (their PCI
address is 05:00.0, not 82:00.0). VM 110 cuda-t4, the reference guest from the day before,
was destroyed afterwards, so all three cards are free.
Measured the same day on VM 1150, and the reason the
memory reserve exists in the form it does: PVE reported 62 GB used while
the guest itself used 1.3 GB. That gap is honest — VFIO pins every page and passthrough forces
balloon: 0, so PVE falls back to the qemu process's RSS.
Lab VMs, templates and pools (2026-08-04)
The cluster became able to hand out student guests: ubuntu-2404-base on all three nodes (one
each — fast-vm is node-local), ubuntu-2404-cuda on the T4 node, and the resource pools infra /
isc1-2526 / isc3-2526. All of it from provisioning/pve/carnaval-guests.sh.
Numbers worth comparing against later: clone → SSH-reachable in ~30 s (a ZFS linked clone costs
only what the guest writes), and a clone with the T4 attached ran a 1M-element CUDA kernel with zero
mismatches — Tesla T4, compute 7.5, 15360 MiB, driver 595.71.05, CUDA 13.2 runtime / 12.0 toolkit.
Two findings that became standing rules on the cluster page: a PVE pool is
not a quota, and PVE 9's kernel ships a third NVIDIA module (nova_core) that must be
blacklisted alongside nouveau and nvidiafb. The operator runbook
Running labs on carnaval was written the same day.
Off the cluster but part of the same work: the CRS326 DHCP pool was narrowed from the whole
192.168.91.0/24 to .120–.199 and the duplicate dhcp_test server disabled, so a lab VM asking
for DHCP can no longer be handed a node's address.
carnaval0 wedged by an oversized GPU guest (2026-08-04)
A 112 GB guest on the 128 GB node left the host no headroom and wedged it badly enough to need an iDRAC reset. Corosync kept voting while userspace stopped answering, so the cluster looked healthy from the outside — the failure mode to expect from this, rather than a node dropping out.
The fix is the reserve now enforced by RESERVE_MB=65536 in carnaval-guests.sh, and the reasoning
is a standing rule on the cluster page: a passed-through GPU pins the
guest's whole allocation, so the reserve is the host's only headroom and must cover the ZFS ARC,
qemu's overhead, PVE's services and page cache.
The cluster was built (2026-08-04)
calypso8 / calypso9 / calypso10 were wiped and rebuilt as carnaval0–2, PVE 9.2.2, quorate,
with guest disks on a ZFS pool on each node's NVMe card and IOMMU ready for the A2/T4 passthrough.
This starts phase 3 of the target
architecture. The local /home (74 GB) was rescued to rumba:/hdd/backup/carnaval-preinstall/
first.
Two findings from the install are written up where they belong rather than here: this BIOS cannot boot the NVMe add-in cards — the previous Ubuntu install hid that behind a Clover ESP, one of them on a load-bearing USB stick — and iDRAC8 needs a different install path than rumba's.