Carnaval

The playground cluster — three Proxmox VE nodes for student labs, disposable by design. This is the LAB half of the target architecture; the services half is rumba. Built 2026-08-04 from calypso8 / calypso9 / calypso10, which leave the Calypso sandbox. This page is the current state; the dated operation logs live in Carnaval — history & operations.

At a glance

Role	Student playground: lab VMs, K8s, GPU work — everything here is destroyable
Nodes	carnaval0 – carnaval2 (3× Dell R630), cluster carnaval, 3 votes
Hypervisor	Proxmox VE 9.2.2, same build as rumba
Guest storage	fast-vm — a ZFS pool on each node's NVMe card (888 / 888 / 464 GB)
GPUs	NVIDIA A2 (carnaval0/1) · Tesla T4 (carnaval2) — passthrough working
Network	192.168.91.18 – .20, 1 GbE each (ISC³ network)
Access	VPN, then https://carnaval0.calypso:8006 — edu-ID SSO or root@pam
Backups	None, deliberately — see below

Hardware

Read from the iDRACs and the running nodes, August 2026. All three are Dell R630 with a perpetual iDRAC8 Enterprise licence, an 8-bay backplane with 6 bays free, and a single 750 W PSU in slot 2 — slot 1 is empty on every node, so none of them has power redundancy.

Node	CPU	RAM	GPU	NVMe (guests)	PERC array (root)
carnaval0	2× Xeon E5-2683 v4 (64 threads)	128 GB	NVIDIA A2	894 GB Swissbit SFPC960GM1AR4MT	RAID-1, 279 GB
carnaval1	2× Xeon E5-2683 v4 (64 threads)	128 GB	NVIDIA A2	894 GB Swissbit SFPC960GM1AR4MT	RAID-1, 279 GB
carnaval2	2× Xeon E5-2680 v3 (48 threads)	112 GB (7× 16 GB, slot B3 empty)	Tesla T4	466 GB Samsung 980	RAID-1, 279 GB

The RAID-1 arrays sit on pairs of 300 GB 10k SAS disks manufactured in 2013. They are healthy, but they are the oldest components in the cluster and they carry the OS — which is not a choice:

Proxmox must live on the PERC array, never on the NVMe card

This BIOS cannot boot from the NVMe add-in cards. The card has no FQDD, so it appears only as Unknown.Unknown.1-1 in UefiBootSeq and the firmware fails to resolve it — verified on BIOS 2.17.0 and 2.19.0, the newest available, so a firmware update is not a fix. Installing to nvme0n1 yields a node that installs perfectly and then will not boot.

The pre-rebuild Ubuntu worked around this with a Clover ESP on a bootable disk, whose bundled NvmExpressDxe.efi supplied the missing NVMe driver and chainloaded GRUB on the card. calypso10 did this from a USB stick, which was therefore load-bearing (removed 2026-08-04). If you ever find a mystery 200 MB FAT partition or a virtual disk named clover on one of these machines, that is what it was.

Consequence for the answer files: disk-list = ["sda"], never ["nvme0n1"].

caution

carnaval0 has no PERC cache battery

Its battery reads Failed / Removed, so the controller runs the array Write Through instead of Write Back — host writes are markedly slower than on the other two, and the node's iDRAC health stays Critical until a battery is fitted (an absent BBU is not something clearing the SEL or restarting the iDRAC will clear). Guests are unaffected: they live on the NVMe. carnaval1 and carnaval2 have healthy batteries and run Write Back.

This is fleet-wide rather than carnaval-specific — several Calypso nodes are missing theirs too, and one battery order covers them all: ops todo → R630 hardware. Never force Write-Back to win the speed back: with no BBU that is straight data loss on a power cut, and these nodes have no redundant PSU either.

Storage layout

The split is the opposite of what the hardware suggests, for the boot reason above: the slow old array carries the OS, the fast card carries everything that matters.

Storage	Where	Holds
fast-vm	ZFS pool fast/vm on nvme0n1	All guest disks and container rootfs
local	the PERC array, /var/lib/vz	ISOs, CT templates, dumps
local-lvm	the PERC array	disabled — slow 2013 SAS, must never be a guest's default target

ZFS rather than LVM-thin on the NVMe, for the instant clones a template-driven playground lives on, plus compression and checksumming on a single unredundant device. Scrubs need no setup: zfsutils-linux ships the monthly cron.

note

fast-vm is node-local, not shared

Each node has its own pool named fast, so one storage definition serves all three — but a guest's disk exists only on the node that created it. There is no shared storage and no Ceph here, which means migration copies the disk rather than being instantaneous, and HA is not available. For a cluster whose guests are meant to be rebuilt from templates rather than moved, that is the right trade; it is also why the 1 GbE links are not a problem in practice.

There is no shared ISO store. The only FS2500 export is the student homes share, and putting ISO and template directories inside calypso_homes is the wrong trade; a dedicated export is in the todo. Guests that need the student homes mount them themselves — how, and what a lab VM needs.

Templates, pools and addressing

Everything students get starts from a template and is meant to be thrown away. Built by provisioning/pve/carnaval-guests.sh (idempotent — a re-run reports = for every step).

VMID	Purpose
100–119	Persistent / infrastructure guests. Address = 192.168.91.<VMID>, so VM 110 is .110
1000–1999	Ephemeral lab VMs, addresses from 192.168.91.120–.199
9000+N	ubuntu-2404-base on carnavalN — 9000, 9001, 9002
9010+N	ubuntu-2404-cuda on carnavalN — 9012 exists (on the T4 node)

Templates that exist today:

Template	Node	Sizing	Contents
ubuntu-2404-base	all three	2 GB / 2 cores	Ubuntu 24.04 cloud image, cloud-init, qemu-guest-agent
ubuntu-2404-cuda	all three	63 GB / 56 cores (47 GB / 40 on carnaval2)	the above + NVIDIA 595 server driver + CUDA 12.0 toolkit + build-essential. No GPU attached — add --hostpci0 to the clone

Leave the host at least 64 GB — a GPU guest cannot balloon

A passed-through GPU pins the guest's whole memory allocation, so the reserve is the only headroom the host has, and it must cover the ZFS ARC (12.9 GB here, 10 % of RAM), qemu's per-VM overhead, PVE's own services and page cache. Guest RAM comes out of the host's total, so "host minus a little" is not safe — it once wedged a node hard enough to need an iDRAC reset. RESERVE_MB=65536 in carnaval-guests.sh is what sets this.

Consequence in the web UI: a GPU guest shows its whole allocation as used the moment it starts, while free inside it shows almost nothing. That is honest, not a bug — VFIO pins every page, and passthrough also forces balloon: 0, so PVE has no guest-side figure and falls back to the qemu process's RSS (mem == memhost, a little over maxmem, which includes qemu's own overhead). The PVE number is the host cost — the one the reserve is about; for workload usage read the guest (measured).

Three templates, one per node, on purpose. fast-vm is node-local, so a template on carnaval0 cannot be cloned on carnaval1. A clone is a ZFS linked clone: instant, and it costs only what the guest writes. Measured 2026-08-04: clone to SSH-reachable in ~30 s.

Resource pools infra, isc1-2526 and isc3-2526 exist (named after the cohort groups in ansible-playbooks-conf). Grant a cohort PVEVMUser on its pool so its members can use and power-cycle their guests but not create or resize them.

A PVE pool is not a quota

Proxmox has no per-pool CPU/RAM/disk ceiling — a pool is a grouping and permission boundary only. The sizing budget is enforced by whoever provisions the guests, not by the hypervisor, so do not promise students a hard quota. The pools also have no members yet: the cluster is wired to Keycloak, but only for administrators — granting a cohort PVEVMUser on its pool is an open item.

Give every lab VM an explicit address

Templates deliberately ship with no ipconfig0, so a clone cannot silently pick up a lease. Assign from 192.168.91.120–.199 by hand.

The CRS326's DHCP pool is 192.168.91.120-.199 — it used to span the entire subnet, so a VM asking for DHCP could be handed a node's address (narrowed 2026-08-04). Two things are still true and argue for staying with static addresses: the pool hands out dns-server=8.8.8.8, so a DHCP client cannot resolve .calypso names, and nothing enforces the .100–.119 infrastructure range against it. Remaining fix in the todo.

GPU passthrough

Host-side prerequisites are done on all three nodes: intel_iommu=on iommu=pt on the kernel command line, vfio / vfio_iommu_type1 / vfio_pci loaded, and nouveau, nvidiafb and nova_core blacklisted so nothing on the host claims the card. That third one is easy to miss — it is the Rust NVIDIA driver that arrived with the 7.x kernel PVE 9 ships, and the older passthrough guides only mention nouveau. IOMMU is live: 93 groups on carnaval0/1, 59 on carnaval2, and each GPU sits alone in its own group, which is what makes passthrough clean.

Attaching a card to a clone is two lines (q35 is already set on the template, and these are headless compute cards, so no OVMF or x-vga gymnastics are needed):

qm set <vmid> --hostpci0 0000:82:00.0,pcie=1     # 82:00.0 is carnaval2's T4
qm start <vmid>                                   # the host then binds vfio-pci to the card

Clone ubuntu-2404-cuda and the driver is already there. On a bare ubuntu-2404-base clone, sudo ubuntu-drivers install --gpgpu pulls it (then reboot: it is a kernel module). Note that nvcc also needs build-essential — the cloud image has no C compiler, and without one nvcc fails with the unhelpful Failed to preprocess host compiler properties.

All three cards are verified end to end (template → clone → GPU → compute) and all three are free — nothing is left running. The A2 cards behave identically to the T4; only their PCI address differs (05:00.0 instead of 82:00.0). Measurements and dates: history.

One GPU, one guest

These cards are passed through whole: there is no vGPU licence and no mediated-device split, so a GPU belongs to exactly one running VM at a time. The architecture doc's other option — several students sharing a card through containers on the host — needs the driver installed on the host instead, which is incompatible with passing that same card through. Pick one mode per node.

Backups

No backup job exists on this cluster, and that is deliberate. The target architecture makes playground backups selective — templates and infra guests yes, disposable lab VMs no — so a blanket nightly job would spend hours copying machines whose whole purpose is that they can be destroyed. When the templates phase creates something worth keeping, it gets its own job. Contrast rumba, where every guest is backed up twice nightly.

Provisioning

Everything is reproducible from the repository:

Step	Runnable
Answer files	provisioning/pve/answer.carnaval{0,1,2}.toml
Install over the iDRAC	provisioning/pve/install-via-redfish.sh — not the racadm one, see the process page
Node configuration	provisioning/pve/post-install-carnaval.sh (repos, the fast pool, IOMMU, netdata, storages)
Templates and pools	provisioning/pve/carnaval-guests.sh (cloud image, the three templates, resource pools)

post-install-carnaval.sh is idempotent, so re-running it is also how you check a node still matches what this page claims: on a configured node every step reports =.

Access

Everything here is behind the VPN — no node is published, and nothing on this cluster listens on the public Internet.

Path	Address	Authentication
Web UI (any node)	https://carnaval0.calypso:8006, .19/.20 for the others	realm ISC SSO (edu-ID) or Linux PAM for root@pam
SSH	root@carnaval0.calypso	pmudry's key; passwords in docs/secretzone/carnaval.md
Netdata	:19999 on each host	none — LAN/VPN only, never published
iDRAC8	192.168.90.18–.20 (network page)	secretzone

SSO login

Since 2026-08-05 the cluster authenticates administrators against Keycloak, so a Proxmox login is an edu-ID login. The mechanism, the provisioning scripts and the traps are documented once for the whole fleet on the rumba page; what is specific to this cluster:

• Administrator on / is held by the PVE group rack-admins-isc and by nobody by name — membership is refilled from the Keycloak group rack-admins at every login, so revoking there revokes here at the next attempt.
• Only pre-created users may log in (--autocreate 0). Passing the Keycloak login is not enough to obtain an account on this cluster; adding an administrator is a pveum user add plus the Keycloak group.
• Students are not in this yet. The cohort pools have no members — see the caution above and the todo.
• root@pam is untouched and stays the preselected realm: it is the break-glass path for the case where Keycloak itself is what broke.

Wired by provisioning/pve/sso-realm.sh, one run on carnaval0 for all three nodes (how it went).

Day-to-day operation — handing out lab VMs, attaching a GPU, end-of-semester cleanup — is the runbook Running labs on carnaval.

Open items: ops todo → Carnaval.