Carnaval
The playground cluster — ten Proxmox VE nodes for student labs, disposable by design. This is the LAB half of the target architecture; the services half is rumba. Built 2026-08-04 from three R630 and extended to the whole fleet on 2026-08-28, when the previous Ubuntu/SLURM cluster on the same chassis was retired (archive). This page is the current state; the dated operation logs live in Carnaval — history & operations.
At a glance
| Role | Student playground: lab VMs, K8s, GPU work — everything here is destroyable |
| Nodes | carnaval0 – carnaval10 (11× Dell R630), cluster carnaval; carnaval5 is powered off with a dead DIMM, so 10 nodes vote. carnaval8 was off from 2026-09-08 and is on again on 2026-09-18 (found online; the decision on its NVMe card is still open, below); carnaval3, 4, 6, 7, 10 were off that day |
| Hypervisor | Proxmox VE 9.2.2, same build as rumba |
| Guest storage | fast-vm — a ZFS pool on each node's NVMe card (466 GB, 894 GB on carnaval7–9) |
| GPUs | one per node: NVIDIA A2 on carnaval0–9, Tesla T4 on carnaval10 — passthrough working |
| Network | 192.168.91.10 – .20, 1 GbE each (ISC³ network) |
| Access | VPN, then https://carnaval1.isc3:8006 — edu-ID SSO or root@pam |
| Backups | None, deliberately — see below |
| Power budget | At most 10 nodes powered at once in 23N307 (decision 2026-08-28); 1.75 kW / 29.7 °C at the PDU probe with the ten idle |
Hardware
Read from the iDRACs and the running nodes, August 2026. All are Dell R630 with a perpetual
iDRAC8 Enterprise licence (firmware 2.86), an 8-bay backplane with 6 bays free, a single 750 W PSU
in slot 2 — slot 1 is empty on every node, so none has power redundancy — and one NVMe add-in card.
carnavalN is the ex-calypsoN chassis and kept its address: node 192.168.91.(10+N), iDRAC
192.168.90.(10+N); the three nodes of the 2026-08-04 cluster became carnaval8–10.
| Node | CPU | RAM | GPU (PCI) | NVMe (guests) | PERC | BIOS |
|---|---|---|---|---|---|---|
carnaval0 | 2× E5-2690 v3 (48 threads) | 128 GB | A2 82:00.0 | 466 GB Samsung 980 | H730P | 2.16 |
carnaval1 | 2× E5-2697 v4 (72 threads) | 128 GB | A2 83:00.0 | 466 GB Samsung 980 | H730P | 2.19 |
carnaval2 | 2× E5-2697 v4 (72 threads) | 128 GB | A2 83:00.0 | 466 GB Samsung 980 | H730P | 2.19 |
carnaval3 | 2× E5-2697 v4 (72 threads) | 128 GB | A2 83:00.0 | 466 GB Samsung 980 | H730P | 2.19 |
carnaval4 | 2× E5-2690 v3 (48 threads) | 128 GB | A2 83:00.0 | 466 GB Samsung 980 | H730P | 2.16 |
carnaval5 | 2× E5-2690 v3 (48 threads) | 128 GB | A2 | Samsung PM983-class | H730P | 2.16 |
carnaval6 | 2× E5-2690 v3 (48 threads) | 128 GB | A2 83:00.0 | 466 GB Samsung 980 | H730P | 2.16 |
carnaval7 | 2× E5-2683 v4 (64 threads) | 128 GB | A2 05:00.0 | 894 GB Swissbit | H730 | 2.17 |
carnaval8 | 2× E5-2683 v4 (64 threads) | 128 GB | A2 05:00.0 | 894 GB Swissbit | H730 | 2.17 |
carnaval9 | 2× E5-2683 v4 (64 threads) | 128 GB | A2 05:00.0 | 894 GB Swissbit | H730 | 2.17 |
carnaval10 | 2× E5-2680 v3 (48 threads) | 112 GB (slot B3 empty) | Tesla T4 82:00.0 | 466 GB Samsung 980 | H730 | 2.19 |
Provenance per chassis is in the acquisition register.
carnaval5 is powered off — DIMM_B1Its POST logs multi-bit errors on DIMM_B1 and the node does not boot (2026-08-28). It keeps its
name and address and is still Ubuntu inside; it joins the cluster once the module is replaced or
pulled (todo). Nothing else in the fleet is held back.
The PVE root sits on a RAID-1 over each node's two 300 GB 10k SAS disks (2013 vintage). Slow and old, but the only choice — see below. Five of the ex-Calypso nodes had a single-disk RAID-0 there; it was rebuilt as RAID-1 during the 2026-08-28 reinstall.
This BIOS cannot boot from the NVMe add-in cards. The card has no FQDD, so it appears only as
Unknown.Unknown.1-1 in UefiBootSeq and the firmware fails to resolve it — verified on BIOS
2.17.0 and 2.19.0, the newest available, so a firmware update is not a fix. Installing to
nvme0n1 yields a node that installs successfully and then will not boot.
The pre-rebuild Ubuntu worked around this with a Clover ESP on a bootable disk, whose bundled
NvmExpressDxe.efi supplied the missing NVMe driver and chainloaded GRUB on the card. If you ever
find a mystery 200 MB FAT partition or a virtual disk named clover on one of these machines, that
is what it was.
Consequence for the answer files: disk-list = ["sda"], never ["nvme0n1"].
Several controllers have no BBU (carnaval7, 8 and two more from the 2026-08-04 fleet sweep;
carnaval5 unconfirmed) and run the array Write Through — host writes are slower, the iDRAC health
stays Critical until a battery is fitted. Guests are unaffected: they live on the NVMe. One battery
order covers the fleet: ops todo → R630 hardware. Never force
Write-Back on a node with no BBU: that is data loss on a power cut, and these nodes have no
redundant PSU either.
A PVE cluster needs more than half its nodes online to change anything (/etc/pve turns read-only
otherwise; VMs keep running). With ten voters that is six, and nodes are routinely powered off by
budget — so carnaval-power.sh on (and its quorum action) sets pvecm expected to the members
online. Safe because the absent nodes are off, not partitioned: no other half can decide on its own;
corosync raises the figure itself when nodes come back. No HA is configured and none is planned: the
pools are node-local, so a failed node's guests are rebuilt from templates, not moved.
Storage layout
The split is the opposite of what the hardware suggests, for the boot reason above: the slow old array carries the OS, the fast card carries the guests.
| Storage | Where | Holds |
|---|---|---|
fast-vm | ZFS pool fast/vm on nvme0n1 (466 GB; 894 GB on carnaval7–9) | All guest disks and container rootfs |
local | the PERC array, /var/lib/vz | ISOs, CT templates, dumps |
local-lvm | the PERC array | disabled — slow 2013 SAS, must never be a guest's default target |
nas-library | NFS, /volume2/pve-library on the FS2500 (added 2026-08-16) | Shared ISOs/CT templates/snippets, and the CUDA template dumps — what it is for, and what it must never hold |
ZFS rather than LVM-thin on the NVMe, for the instant clones a template-driven playground lives on,
plus compression and checksumming on a single unredundant device. Scrubs need no setup:
zfsutils-linux ships the monthly cron.
fast-vm is node-local, not sharedEach node has its own pool named fast, so one storage definition serves all ten — but a guest's
disk exists only on the node that created it. There is no shared storage and no Ceph here, which
means migration copies the disk rather than being instantaneous, and HA is not available. For a
cluster whose guests are meant to be rebuilt from templates rather than moved, that is the right
trade; it is also why the 1 GbE links are not a problem in practice.
If HA or fast migration is ever wanted here, the answer is ZFS replication (pvesr), not shared
storage: the pool is named fast on every node, which is the precondition, and periodic
incremental zfs send gives both — at the cost of losing up to one replication interval on a node
failure. Ceph is out (1 GbE), NFS from the FS2500 would put every guest's I/O across that same
link. Note the nodes have no redundant PSU, so HA would be papering over single points of failure
it cannot prevent.
Guests that need the student homes mount them themselves — how, and what a lab VM needs.
Recovering a suspended pool
fast pool needs a cold power cycle, not a rebootIf zpool status reads SUSPENDED and lsblk shows nvme0n1 at 0 B, the card's controller has
stopped answering. Both Swissbit N-30m2 cards have done it, five weeks apart: on carnaval1
2026-08-06 and on carnaval8
2026-09-03.
The path that worked both times: kill -9 the kvm processes, then sync -f / — plain sync
would hang on the pool — then from the iDRAC ForceOff, ~50 s with power removed, On (iDRAC8
offers no PowerCycle). kill -9 leaves the kvm process in D state; carry on regardless. Scrub
before trusting the contents: 0 errors both times, 17 s over 17.8 GB and 11 s over 9.8 GB.
Do not reach for zpool clear — zpool status advises it, but there is no block device left to
clear — nor for a normal reboot, qm stop or a warm reset: txg_sync sits in D state on dirty
transactions, and the controller has already ignored resets of its own.
Nothing will alert you: the iDRAC cannot see an add-in NVMe, so the SEL stays empty — 17 h
unnoticed in August, five days in September. A guest on the suspended pool answers ping and its
QEMU agent from RAM while every write blocks, so it reads as alive; the symptom on gpu8 was an
SSH session that authenticated and then hung, with 1026 processes in D state and a load average
of 1027.
SMART reads clean before and after each hang — Critical Warning 0x00, 0 media errors, 0
error-log entries, 0 % used — so a clean report says nothing about whether the controller will
lock up again. Identify a card by its EUI rather than by the node it sits in: …03550000 hung in
carnaval8 on 2026-09-03 at 7398 power-on hours; …05590000, in carnaval9 at 6555 hours, is
the card that hung on carnaval1 (6347 hours at the August recovery, and too few hours have
passed since for it to be the other one). carnaval7 carries a third card of the same model on
the same firmware ARR50002. Their fate is an open item.
Templates, pools and addressing
Everything students get starts from a template and is meant to be thrown away. Built by
provisioning/pve/carnaval-guests.sh (idempotent — a re-run reports = for every step).
| VMID | Purpose |
|---|---|
100–119 | Persistent / infrastructure guests. Address = 192.168.91.<VMID>, so VM 110 is .110. Reserved: 110–112 = k8s0–2 (destroyed 2026-09-18, recreated by carnaval-k8s.sh up), .115 = the k8s API VIP (Kubernetes) |
1000–1999 | Ephemeral lab VMs, addresses from 192.168.91.128–.191 (the NetBird lab range) |
9000+N | ubuntu-2404-base on carnavalN — 9000 … 9010 |
9100+N | ubuntu-2404-cuda on carnavalN — 9100 … 9110 (was 9010+N with three nodes) |
Templates that exist today:
| Template | Node | Sizing | Contents |
|---|---|---|---|
ubuntu-2404-base | every node | 2 GB / 2 cores | Ubuntu 24.04 cloud image, cloud-init, qemu-guest-agent |
ubuntu-2404-cuda | every node | host RAM − 64 GB / host threads − 8 (63 GB / 56 cores on a 128 GB, 64-thread node) | the above + NVIDIA 595 server driver + CUDA 12.0 toolkit + build-essential, clang, cmake, nvtop + Slurm 23.11 and OpenMPI 4.1 (one-node cluster per clone, generated at first boot). No GPU attached — add --hostpci0 to the clone. Rebuilt 2026-09-18 on carnaval9, published to 0, 1, 2, 8; 3, 4, 6, 7, 10 were off and still carry the August image (todo) |
A passed-through GPU pins the guest's whole memory allocation, so the reserve is the only headroom
the host has, and it must cover the ZFS ARC (12.9 GB here, 10 % of RAM), qemu's per-VM overhead,
PVE's own services and page cache. Guest RAM comes out of the host's total, so "host minus a
little" is not safe — it once wedged a node hard enough to need an iDRAC
reset. RESERVE_MB=65536 in carnaval-guests.sh is what
sets this.
Consequence in the web UI: a GPU guest shows its whole allocation as used the moment it starts,
while free inside it shows almost nothing. That is not a bug — VFIO pins every page, and
passthrough also forces balloon: 0, so PVE has no guest-side figure and falls back to the qemu
process's RSS (mem == memhost, a little over maxmem, which includes qemu's own overhead).
The PVE number is the host cost — the one the reserve is about; for workload usage read the guest
(measured).
One template per node, on purpose: fast-vm is node-local, so a template on carnaval0
cannot be cloned on carnaval1. A clone is a ZFS linked clone: instant, and it costs only what
the guest writes. Measured 2026-08-04: clone to SSH-reachable in ~30 s. The CUDA copies are
one build published through the NAS — the
image pipeline (since 2026-08-16).
Resource pools infra, isc1-2526 and isc3-2526 exist (named after the cohort groups in
ansible-playbooks-conf). Grant a cohort PVEVMUser on its pool so its members can use and
power-cycle their guests but not create or resize them.
Proxmox has no per-pool CPU/RAM/disk ceiling — a pool is a grouping and permission boundary
only. The sizing budget is enforced by whoever provisions the guests, not by the hypervisor, so do
not promise students a hard quota. The pools also have no members yet: the cluster is wired to
Keycloak, but only for administrators — granting a cohort PVEVMUser on
its pool is an open item.
Templates deliberately ship with no ipconfig0, so a clone cannot pick up a lease — and there is
nothing to lease anyway: the CRS326's DHCP servers are disabled since 2026-08-05 (the pool used to
span the entire subnet and could hand out node addresses,
narrowed 2026-08-04). carnaval-lab-vm.sh allocates from
192.168.91.128–.191; assign by hand only inside that range.
GPU passthrough
Host-side prerequisites are done on every node: intel_iommu=on iommu=pt on the kernel
command line, vfio / vfio_iommu_type1 / vfio_pci loaded, and nouveau, nvidiafb and
nova_core blacklisted so nothing on the host claims the card. That third one is easy to miss —
it is the Rust NVIDIA driver that arrived with the 7.x kernel PVE 9 ships, and the older
passthrough guides only mention nouveau. IOMMU is live on every node (59–102 groups depending on
the chassis), and each GPU sits alone in its own group, which is what makes passthrough clean.
Attaching a card to a clone is two lines (q35 is already set on the template, and these are
headless compute cards, so no OVMF or x-vga gymnastics are needed):
qm set <vmid> --hostpci0 0000:82:00.0,pcie=1 # the address is per chassis — see the hardware table
qm start <vmid> # the host then binds vfio-pci to the card
Clone ubuntu-2404-cuda and the driver is already there. On a bare ubuntu-2404-base clone,
sudo ubuntu-drivers install --gpgpu pulls it (then reboot: it is a kernel module). Note that
nvcc also needs build-essential — the cloud image has no C compiler, and without one nvcc
fails with the unhelpful Failed to preprocess host compiler properties.
The A2 cards behave identically to the T4; only the PCI address differs per chassis (82:00.0,
83:00.0 or 05:00.0 — the hardware table has them, and carnaval-lab-vm.sh reads
it from the node rather than from a table). Measurements and dates:
history.
These cards are passed through whole: there is no vGPU licence and no mediated-device split, so a GPU belongs to exactly one running VM at a time. The architecture doc's other option — several students sharing a card through containers on the host — needs the driver installed on the host instead, which is incompatible with passing that same card through. Pick one mode per node.
Backups
No backup job exists on this cluster, and that is deliberate. The target architecture makes playground backups selective — templates and infra guests yes, disposable lab VMs no — so a blanket nightly job would spend hours copying machines whose whole purpose is that they can be destroyed. When the templates phase creates something worth keeping, it gets its own job. Contrast rumba, where every guest is backed up twice nightly.
The one exception since 2026-08-16: publishing a CUDA template leaves its vzdump on
nas-library — the cold copy that survives a cluster
reinstall. Nothing is scheduled; the dump is refreshed by the publish itself.
Provisioning
Everything is reproducible from the repository:
| Step | Runnable |
|---|---|
| Answer files | provisioning/pve/answer.carnaval{0..10}.toml — static address, disk-list = ["sda"], root hash redacted (real one in secretzone/carnaval.md) |
| Boot array | provisioning/pve/perc-raid1.sh <idrac> — recreates the PERC volume as RAID-1 over Redfish, which also destroys the old bootloader so the virtual CD wins; --show is read-only |
| racadm | provisioning/pve/idrac-racadm.sh <idrac> <args> — racadm over SSH through calypsomaster (a current macOS OpenSSH cannot talk to iDRAC8's SSH) |
| Install over the iDRAC | provisioning/pve/build-auto-iso.sh (one baked ISO per node) + install-via-redfish.sh --no-bootonce — why baked, why Redfish |
| Kubernetes | `provisioning/pve/carnaval-k8s.sh up |
| Node configuration | provisioning/pve/post-install-carnaval.sh (repos, the fast pool, IOMMU, netdata, storages, node identity — front LCD + web-UI notes) |
| Names | provisioning/network/carnaval-dns.sh — carnavalN and carnavalN.isc3 on the CCR2004, add-or-set |
| Templates and pools | provisioning/pve/carnaval-guests.sh (cloud image, the templates on every node, resource pools) |
| Power cycle | provisioning/pve/carnaval-power.sh off|on|status [N] — clean guest + node shutdown, power-on over the iDRACs, restarts the guests that were running |
post-install-carnaval.sh is idempotent, so re-running it is also how you check a node still
matches what this page claims: on a configured node every step reports =.
Adding a node — the seven spares
The seven spare R630 in U26–33 become carnaval11–17 when wanted (addresses .91.21–.27,
iDRACs .90.21–.27). Their iDRACs have never answered on the management LAN, so the first step
is in the room: cable the iDRAC port, note its MAC, reserve the address on the CRS326. Then, per
node: Redfish inventory (CPU, RAM, NVMe present or not — without a card the node has no fast
guest storage), perc-raid1.sh, an answer.carnavalN.toml from the template plus a line in
answer-map.csv, a baked ISO, install-via-redfish.sh --no-bootonce, post-install-carnaval.sh --create-pool, pvecm add, carnaval-dns.sh N, a reboot for the IOMMU, and the status-page and
NetBox lists. Mind the 10-nodes power ceiling and the quorum arithmetic above
before powering one on for good.
Access
Everything here is behind the VPN — no node is published, and nothing on this cluster listens on the public Internet.
| Path | Address | Authentication |
|---|---|---|
| Web UI (any node) | https://carnavalN.isc3:8006 | realm ISC SSO (edu-ID) or Linux PAM for root@pam |
| SSH | root@carnavalN.isc3 | pmudry's key; one root password for the cluster in secretzone/carnaval.md |
| Netdata | :19999 on each host | none — LAN/VPN only, never published |
| iDRAC8 | 192.168.90.(10+N) (network page) | secretzone/isc3.md; VNC console on :5901 where enabled (carnaval0, password in the same file) |
Node-to-node SSH — what qm migrate and the API proxy use — reads each peer's host key from
/etc/pve/nodes/<node>/ssh_known_hosts and connects with HostKeyAlias=<node>, one file per node,
written at boot. The cluster-wide /etc/pve/priv/known_hosts holds a single stale entry from the
2026-08-28 rebuild: it is the pre-8.2 merged scheme, PVE 9 no longer consults it, and
pvecm updatecerts would restore that older scheme rather than fix anything. So a near-empty file
there is not a broken cluster — test the real path instead, with both the alias and the per-node
file:
ssh -o HostKeyAlias=carnaval9 -o UserKnownHostsFile=/etc/pve/nodes/carnaval9/ssh_known_hosts \
-o GlobalKnownHostsFile=none root@192.168.91.19 hostname
An interactive ssh root@carnavalN from another node does fail, since root's own known_hosts is
no longer cluster-managed. Nothing depends on it.
SSO login
Since 2026-08-05 (re-run on the rebuilt cluster 2026-08-28) the cluster authenticates administrators against Keycloak, so a Proxmox login is an edu-ID login. The mechanism, the provisioning scripts and the traps are documented once for the whole fleet on the rumba page; what is specific to this cluster:
Administratoron/is held by the PVE grouprole-rack-admins-iscand by nobody by name — membership is refilled from the Keycloak grouprole-rack-adminsat every login, so revoking there revokes here at the next attempt.- Only pre-created users may log in (
--autocreate 0). Passing the Keycloak login is not enough to obtain an account on this cluster; adding an administrator is apveum user addplus the Keycloak group. - Students are not in this yet. The cohort pools have no members — see the caution above and the todo.
root@pamis untouched and stays the preselected realm: it is the break-glass path for the case where Keycloak itself is what broke.
Wired by provisioning/pve/sso-realm.sh, one run on any node for the whole cluster
(how it went).
Day-to-day operation — handing out lab VMs, attaching a GPU, end-of-semester cleanup — is the runbook Running labs on carnaval.
Open items: ops todo → Carnaval.