Carnaval software stack
How software reaches the carnaval cluster: golden images (templates) built once, published to every node through the NAS, then lab VMs stamped out of them for the week's scenario. The hardware page describes the machines and their template inventory; the labs process page is the day-to-day VM handling; this page is the image lifecycle and the account model.
The big picture
The ubuntu-2404-base templates (9000–9002) are simpler and stay out of the diagram: each
node builds its own directly from the Ubuntu cloud image (carnaval-guests.sh --templates), nothing to distribute. The CUDA image goes through the pipeline above
because it is expensive to build (~10 min) and may carry manual tuning that a re-run of a
recipe cannot reproduce.
Build and publish an image
Fully scripted change (a new package, a version bump): put it in the recipe —
carnaval-guests.sh, GUEST block — never in a template by hand, or the next rebuild
silently loses it. Then:
# destroy the current CUDA templates first (the script refuses to overwrite;
# qm destroy itself refuses while linked clones exist — destroy the week's VMs first,
# and the k8s VMs too: they are clones of the same template)
ssh root@carnaval0 'qm destroy 9100 --purge 1' # idem 9101/carnaval1, 9102/carnaval2 …
./provisioning/pve/carnaval-guests.sh --cuda # build once on the first node of NODES, publish to the rest
With a manual-tuning window in the middle — the reason this pipeline exists:
./provisioning/pve/carnaval-guests.sh --cuda-prepare # scripted install, build VM left running
ssh -J root@carnaval0 ubuntu@192.168.91.200 # tune by hand, as long as you like (.200+N)
NOTE='what was tuned' ./provisioning/pve/carnaval-guests.sh --cuda-seal
NODES="carnaval9 carnaval8" picks the build node and the publish targets (2026-09-18: built on
carnaval9, carnaval0–2 and 8 online). A node that is off at publish time gets the image later:
destroy its old 910N, then NODES="<build node> carnavalN" carnaval-guests.sh --cuda-seal.
--cuda-seal does, in order: guest hygiene (shell histories, apt caches, machine-id,
cloud-init clean last, so nothing done by hand leaks into the clones' identity),
seal on carnaval0, vzdump to nas-library, qmrestore onto the other two
nodes, and re-sizing per host — carnaval2 has less RAM and fewer cores, and a GPU guest
sized for carnaval0 would not fit it. The three
templates end up bit-identical; NOTE lands in their description so the manual part stays
visible.
Re-running --cuda-seal on an already-sealed template skips straight to dump + restore — that
is also how a template gets re-published, or its cold copy refreshed, without a rebuild.
The build addresses .200–.210 sit outside the NetBird resource carnaval-labs
(192.168.91.128/26), so the operator's machine cannot reach them — polling from it looks
like a VM that never booted (cost a failed build, 2026-08-15). The script polls and
installs via ssh -J root@<node>; do the same for any manual step.
Deploy VMs from the image
One command per VM; the address is allocated (or given as the octet) inside
192.168.91.128–.191, and the DNS names are registered on the CCR before the VM is even
reachable:
provisioning/pve/carnaval-lab-vm.sh carnaval0 1160 160 --gpu --dns=gpu0
Every VM answers to lab-<vmid> / lab-<vmid>.isc3; --dns=<alias> adds the name
students actually type for the week's scenario (gpu0, slurm0, k8s-master…), and the
alias simply moves to whichever VM carries it the following week. A week's fleet is a loop:
n=0; for node in carnaval0 carnaval1 carnaval2; do
provisioning/pve/carnaval-lab-vm.sh $node $((1160+n)) $((160+n)) --gpu --dns=gpu$n
n=$((n+1))
done
Tear-down goes through the same script — carnaval-lab-vm.sh <node> <vmid> --destroy —
which also purges every DNS record pointing at the VM's address, alias included. The
current standing fleet (gpu0–gpu2 and gpu9, VMIDs 1160–1162 and 1169, re-stamped 2026-09-18) and the rest
of the day-to-day handling live on the labs process page.
Who gets in — the account model
Accounts are not per-VM: every lab VM carries the whole cohort, so students find their account on any machine of the week. The sources, all in this repository:
| Question | Source |
|---|---|
| Who gets an account | provisioning/keycloak/roster.csv, lines carrying vpn-carnaval — the same file that drives Keycloak and NetBird, so access is granted in one place and the git diff is the authorization change |
| Who gets root | the roster's role-rack-admins lines (passwordless sudo). Students never do: root on any box in 192.168.91.0/24 can become any student over NFS |
| How they log in | provisioning/keycloak/ssh-keys — one <name> <key> line per key, blank line between people. Deliberately not the Ansible tree, which is shared with calc |
| Their numeric UID | stat'ed live from their NAS home's ownership at VM-creation time — the one source NFS actually enforces (why) |
Every guest also has an ubuntu account (passwordless sudo) whose keys are those of the roster's
role-rack-admins, read by provisioning/pve/admin-keys.sh at creation — the break-glass login on
a lab VM whose roster block failed, and the only account on the k8s servers. Since 2026-09-18 that
list is not a constant in the scripts any more.
A roster member without a NAS home is skipped with a warning; one without a key gets their account but no way in. As of August 2026 the 25/26 cohort has no keys collected yet — ops todo → Carnaval.
The cohort is baked in at VM creation, so someone added to the roster mid-week only reaches
the next stamped VM. For the fleet already running,
provisioning/pve/carnaval-refresh-users.sh --commit regenerates the account block from the
current roster and pushes it into every running lab VM over qm guest exec — add-only:
removing someone still happens at the weekly re-stamp.
Slurm and MPI — one scheduler per GPU VM
Each GPU lab VM is its own one-node Slurm cluster (slurmctld, slurmd and munge on the VM,
the passed-through card as gres/gpu:1): one card and the whole roster, so jobs queue for it
instead of colliding. OpenMPI 4.1 comes with it, on the same PMIx as Slurm's mpi/pmix plugin.
Installed by provisioning/pve/carnaval-slurm.sh --image in the CUDA
recipe (templates rebuilt 2026-09-18); the same script without the flag installs it on a running VM
stamped from an older image (ssh ubuntu@gpuN.isc3 'sudo -n bash -s' < provisioning/pve/carnaval-slurm.sh).
| Config | generated at the clone's first boot by carnaval-slurm-setup.service from the hostname, CPUs, memory and PCI GPU count — nothing per-VM is in the image. Regenerate: rm /etc/slurm/slurm.conf, restart the unit |
| Partition | gpu, default, all cores, RealMemory = RAM − 1 GB, DefMemPerCPU = that / cores, 24 h max, 1 h default |
| Default allocation | a GPU job that asks for nothing else gets 8 cores and about 12 GB (DefCpuPerGPU=8); the memory cgroup is enforced, so ask for more with --cpus-per-task / --mem rather than let a job be killed |
| GPU | srun --gres=gpu:1 … / #SBATCH --gres=gpu:1. cgroup.conf constrains devices: a job that does not ask for the card does not see it (No devices found); an interactive shell outside Slurm still does |
| MPI | mpicc prog.c && srun -n 4 ./prog — MpiDefault=pmix, so ranks are launched by srun directly; mpirun inside an sbatch script works too. Ranks stay on the VM: one node per cluster |
| What works | sbatch, srun, squeue, sinfo, scontrol. No slurmdbd, so no sacct |
| Version | Slurm 23.11.4, OpenMPI 4.1.6, PMIx 5.0.1, all from noble; cgroup v2 (Sept 2026) |
Kubernetes — the teaching cluster
A k3s cluster in three VMs, one per node on carnaval0–2, each with the node's GPU passed
through. Deployed 2026-08-28 by provisioning/pve/carnaval-k8s.sh up; the design and its
reasons are on the future page (phase 2, student access, is
still open there).
The three VMs were destroyed on 2026-09-18: gpu0–gpu2 hold the cards
(lab VMs), and the VMs were linked clones of the CUDA
templates being rebuilt. carnaval-k8s.sh up recreates the cluster from the current template in
about ten minutes, once the lab VM on each node is stopped — one GPU, one running VM. The table
below describes the cluster as built.
| VMs | k8s0–k8s2 = VMID 110–112 = 192.168.91.110–.112, 16 vCPU / 32 GB pinned, 120 GB root on fast-vm, onboot |
| API | https://k8s.isc3:6443 — kube-vip VIP 192.168.91.115, ARP mode, floats between the three servers |
| SSH | ubuntu@k8sN through its node (ssh -J root@carnavalN), keys = the roster's role-rack-admins; carnaval-k8s.sh keys pushes the current list to the running servers |
| Versions | k3s v1.36.4+k3s1 (three servers, embedded etcd, all schedulable), kube-vip v1.2.3, NVIDIA device plugin v0.20.0 — pinned in the script |
| GPUs | one nvidia.com/gpu per node; the nvidia RuntimeClass is created by k3s itself once nvidia-container-toolkit is on the VM (cloud-init) |
| Storage | k3s local-path-provisioner: PVs live on the VM disk. No Longhorn, no NFS PVs |
| Ingress / LB | k3s's bundled traefik and servicelb, untouched |
| Images | docker.io pulls go through the pull-through cache registry.isc3:5000 on rumba (since 2026-09-09; provisioning/k8s/registries.yaml), other registries direct |
| Manifests | provisioning/k8s/ — kube-vip, device plugin, Headlamp; cohort namespaces come with phase 2 |
| Web UI | Headlamp v0.45.0 at http://k8s.isc3 (traefik ingress on the VIP); login with a token from carnaval-k8s.sh token <name> [720h], which also creates a named cluster-admin ServiceAccount — revoke with kubectl -n headlamp delete sa <name> |
| Kubeconfig | carnaval-k8s.sh kubeconfig → ~/.kube/carnaval.yaml (root admin), or a per-person one built on a token |
| Backups | none — destroy + up rebuilds it in ~10 min; anything worth keeping is a manifest in git |
Reaching it: the NetBird resource carnaval-infra (192.168.91.96/27) is in net-carnaval-guests
(since 2026-09-08; admin-only in net-carnaval-hosts from 2026-08-28 until then), so API and web UI
answer for every vpn-carnaval member — the same reach as the lab VMs. Reach is not access: the
API takes a token, and until phase 2 the only tokens are the cluster-admin ones carnaval-k8s.sh token issues. The deployment script itself goes through the node (ssh -J), so it works before
the route does.
While the cluster runs, the three cards on carnaval0–2 are taken: GPU lab VMs go on
carnaval3–10.
The library — nas-library
An NFS storage on all three nodes (/volume2/pve-library on the
FS2500, 6.7 TB volume, added 2026-08-16), content types
iso,vztmpl,snippets,backup. It exists for exactly two things: shared ISOs/CT templates
uploaded once instead of per node, and the CUDA template dumps — the transport of the
pipeline above and the cluster's only cold copy (first dump 2026-08-16; the 2026-09-18 rebuild left
two identical ~4.9 GB dumps of 9109, 14:04 from the build and 14:17 from the republish — one can go).
Guest disks stay on each node's local fast-vm — that is what makes clones instant; a
template served from NFS would pull 15–20 GB through the node's 1 GbE at every clone. And
carnaval's no-backups rule stands: nothing schedules
vzdumps here, and lab VMs don't get manual ones either. Template dumps via --cuda-seal
are the one sanctioned use of backup.
Open items: ops todo → Carnaval.