Skip to main content

Carnaval — history & operations

How the playground cluster got to its current state: the dated operation logs, newest first. Current facts live on the main page; finished work across the whole rack is indexed in the ops journal.

Slurm and MPI on the GPU VMs (2026-09-18)​

carnaval-slurm.sh gives each GPU VM a one-node Slurm cluster with the card as gres/gpu:1, plus OpenMPI on Slurm's PMIx; in the CUDA recipe (--image, config generated at first boot) — Slurm. The CUDA template was rebuilt the same day on carnaval9 and published to carnaval0–2 and 8 (the other five nodes were off); this meant destroying every linked clone first — gpu0–2, gpu9 and the stopped k8s0–2. gpu0–2 and gpu9 were re-stamped from the new image, and a GPU job plus a 3-rank MPI job verified on each straight out of the first boot.

k8s stopped, gpu0–gpu2 created (2026-09-18)​

k8s0–2 shut down with onboot 0 (kept, not destroyed) to free the A2 cards; gpu0–gpu2 (VMID 1160–1162, .160–.162) stamped from the CUDA templates with the full roster, 33 NAS homes mounted and the card computing on each — the running fleet.

ubuntu keys come from the roster (2026-09-18)​

The ubuntu account of the lab VMs, the k8s servers and the template build VM carried one hardcoded key. The three scripts now read the role-rack-admins keys from the roster (provisioning/pve/admin-keys.sh); the five current keys were pushed to k8s0–2 (carnaval-k8s.sh keys) and gpu9 — account model.

carnaval8's NVMe failed and the GPU lab moved to carnaval9 (2026-09-08)​

The second Swissbit card stopped answering on 2026-09-03, ZFS suspended carnaval8's pool and it was found five days later with gpu8 frozen — the account is in the incident log, the recovery sequence is a standing rule on the cluster page.

The cold power cycle brought the controller back a second time: card at 894.3 GB, pool ONLINE, scrub 0 errors in 11 s, VM 1168 restarted with a clean filesystem and its A2 working. The lab was then rebuilt on carnaval9 as gpu9 (VM 1169, 192.168.91.169, that node's A2, 32 NAS homes mounted) and carnaval8 powered off until its card's fate is decided. Both cards of the pair have now hung.

k3s teaching cluster deployed (2026-08-28)​

carnaval-k8s.sh up created k8s0–2 (VMID 110–112) from the CUDA templates on carnaval0–2, GPU passed through, and installed k3s v1.36.4+k3s1 with kube-vip on 192.168.91.115 (k8s.isc3), the NVIDIA device plugin and Headlamp as web UI — what runs. NetBird got the resource carnaval-infra (192.168.91.96/27) so admins reach API and UI over the VPN. Two traps fixed on the way: only three of the ten voters were powered on, so the cluster had no quorum and no clone could run — carnaval-power.sh now re-quorates after on; and a stale known_hosts entry for a previous VM at .110 stalled the script — VM host keys are no longer checked (disposable VMs, reached through a trusted node).

The whole R630 fleet became carnaval (2026-08-28)​

Calypso was retired and its eight worker nodes wiped and reinstalled as PVE 9.2.2, then the three nodes of the 2026-08-04 cluster were wiped too and rejoined as carnaval8–10 — a fresh cluster carnaval of ten voters, carnavalN = ex-calypsoN with its address kept. carnaval5 stayed out: DIMM_B1 multi-bit errors, node held off. Templates and gpu0–2 of the old cluster were destroyed and recreated; the CUDA template came back from its nas-library dump.

Before the wipe, the local /home of the seven Calypso nodes (67 accounts each), their /etc and the SLURM accounting DB were copied to rumba:/hdd/backup/calypso-preinstall/<node>/ — 99 GB after excluding caches, virtualenvs, course datasets and the homes pmudry named as disposable. No second copy exists by decision. The NAS homes were not involved.

What the day fixed as standing practice, on the process page: one baked ISO per node rather than an answer server (the installer's DHCP was too slow for the fetch on four nodes out of five); perc-raid1.sh over Redfish, with racadm through calypsomaster as the fallback; re-insert the virtual media when an iDRAC drops it after the initrd load; copy corosync.conf from a member when a node that rebooted during other joins refuses to start. Timings worth keeping: RAID-1 recreation ~10 min (two reboots), install ~20 min on the 2013 SAS disks, post-install + join + IOMMU reboot ~10 min — the seven Calypso nodes in about three hours wall-clock, mostly in parallel.

carnaval1's NVMe controller hung and was recovered (2026-08-07)​

The card carrying fast-vm stopped answering the evening before and ZFS suspended the pool; the account is in the incident log and the recovery sequence is a standing rule on the cluster page.

Numbers worth keeping: a cold power cycle brought the controller back (warm resets had already failed twice), the scrub of the whole 17.8 GB took 17 s and found 0 errors, and SMART is clean afterwards — 0 media errors, 0 error-log entries, 0 % used at 6347 power-on hours. So the card is back in service rather than replaced, on watch: carnaval0 runs the identical Swissbit on the same firmware ARR50002.

SSO login wired to Keycloak (2026-08-05)​

The cluster took an edu-ID login the same day rumba did, from one run of provisioning/pve/sso-realm.sh on carnaval0 — /etc/pve is cluster-wide, so all three nodes were covered at once. Administrator on / is held by the PVE group rack-admins-isc and by nobody by name; root@pam is untouched and stays the preselected realm.

Two things were learnt here rather than on rumba, both recorded on the rumba page: PVE appends the realm name to every group it reads from the claim, and group membership is per cluster — carnaval needed its own first login before rack-admins-isc filled, even though rumba's had already succeeded.

GPU passthrough re-verified, reference guest destroyed (2026-08-05)​

The A2 cards on carnaval0/1 were confirmed to behave identically to the T4 (their PCI address is 05:00.0, not 82:00.0). VM 110 cuda-t4, the reference guest from the day before, was destroyed afterwards, so all three cards are free.

Measured the same day on VM 1150, and the reason the memory reserve exists in the form it does: PVE reported 62 GB used while the guest itself used 1.3 GB. That gap is expected — VFIO pins every page and passthrough forces balloon: 0, so PVE falls back to the qemu process's RSS.

Lab VMs, templates and pools (2026-08-04)​

The cluster became able to hand out student guests: ubuntu-2404-base on all three nodes (one each — fast-vm is node-local), ubuntu-2404-cuda on the T4 node, and the resource pools infra / isc1-2526 / isc3-2526. All of it from provisioning/pve/carnaval-guests.sh.

Numbers worth comparing against later: clone → SSH-reachable in ~30 s (a ZFS linked clone costs only what the guest writes), and a clone with the T4 attached ran a 1M-element CUDA kernel with zero mismatches — Tesla T4, compute 7.5, 15360 MiB, driver 595.71.05, CUDA 13.2 runtime / 12.0 toolkit.

Two findings that became standing rules on the cluster page: a PVE pool is not a quota, and PVE 9's kernel ships a third NVIDIA module (nova_core) that must be blacklisted alongside nouveau and nvidiafb. The operator runbook Running labs on carnaval was written the same day.

Off the cluster but part of the same work: the CRS326 DHCP pool was narrowed from the whole 192.168.91.0/24 to .120–.199 and the duplicate dhcp_test server disabled, so a lab VM asking for DHCP can no longer be handed a node's address.

carnaval0 wedged by an oversized GPU guest (2026-08-04)​

A 112 GB guest on the 128 GB node left the host no headroom and wedged it badly enough to need an iDRAC reset. Corosync kept voting while userspace stopped answering, so the cluster looked healthy from the outside — the failure mode to expect from this, rather than a node dropping out.

The fix is the reserve now enforced by RESERVE_MB=65536 in carnaval-guests.sh, and the reasoning is a standing rule on the cluster page: a passed-through GPU pins the guest's whole allocation, so the reserve is the host's only headroom and must cover the ZFS ARC, qemu's overhead, PVE's services and page cache.

The cluster was built (2026-08-04)​

calypso8 / calypso9 / calypso10 were wiped and rebuilt as carnaval0–2, PVE 9.2.2, quorate, with guest disks on a ZFS pool on each node's NVMe card and IOMMU ready for the A2/T4 passthrough. This starts phase 3 of the target architecture. The local /home (74 GB) was rescued to rumba:/hdd/backup/carnaval-preinstall/ first.

Two findings from the install are written up where they belong rather than here: this BIOS cannot boot the NVMe add-in cards — the previous Ubuntu install hid that behind a Clover ESP, one of them on a load-bearing USB stick — and iDRAC8 needs a different install path than rumba's.