Running labs on carnaval

Day-to-day operation of the carnaval playground cluster: handing a VM to a student, giving one a GPU, and cleaning up afterwards. The cluster page describes what the machines are; this page is what you type. One script does the whole job — what it gets right on your behalf, and the qm sequences for the cases it does not cover, are on mechanics & manual steps.

Two places you work from:

• provisioning/pve/carnaval-lab-vm.sh runs on your own machine, from a checkout of this repository — it reads the roster, the keys and the UIDs, then drives the node over SSH. It needs the VPN up and your SSH key loaded in the agent.
• Everything else is root on any node (ssh root@carnaval0) or the web UI at https://carnaval0.isc3:8006 (realm Linux PAM). Any node shows the whole cluster.

What exists today

Nodes	carnaval0 – carnaval10 (carnaval5 off; carnaval8 on again since 2026-09-18, card on watch; 3, 4, 6, 7, 10 off on 2026-09-18) — 192.168.91.(10+N), cluster carnaval, quorate (since 2026-08-28)
Templates	ubuntu-2404-base on every node (9000+N) · ubuntu-2404-cuda on every node (9100+N)
Guest storage	fast-vm, a ZFS pool on each node's NVMe — 466 GB, 894 GB on carnaval7–9
Pools	infra, isc1-2526, isc3-2526
GPUs	A2 on carnaval0–9, Tesla T4 on carnaval10 — PCI address per chassis on the cluster page; the script reads it from the node
Accounts	admins log in with edu-ID SSO (or root@pam); no student PVE accounts yet — students reach their lab VM by SSH, not the web UI
Standing guests	none — gpuN (VMID 1160+N, .160+N, one per node, that node's GPU attached) are recreated on demand. Running: gpu0–gpu2 since 2026-09-18 (the k8s VMs are gone, why), gpu9 since 2026-09-08
Backups	none, deliberately — why; the templates have a cold copy on the NAS

Create a lab VM

Node and VMID; the address is allocated for you — pass an octet only if you want a particular one. There is no list of names: every lab VM carries the whole roster (the account model — vpn-carnaval get an account, role-rack-admins get root, students never do):

provisioning/pve/carnaval-lab-vm.sh carnaval1 1121
provisioning/pve/carnaval-lab-vm.sh carnaval2 1142 --gpu --dns=gpu2
provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 121      # octet chosen, not allocated

The address is the lowest free one in 192.168.91.128–.191, read from the guests the cluster actually runs. That range is the NetBird resource carnaval-labs: a VM outside it is unreachable over the VPN, which is why the script refuses an octet outside it rather than trusting you.

Each VM is registered on the CCR2004 as lab-<vmid> / lab-<vmid>.isc3 beside every other .isc3 record — Proxmox cannot write it, its SDN speaks PowerDNS and nothing else. --dns=<alias> adds the name students actually type for the week's scenario (gpu2, slurm0…); the alias moves to whichever VM carries it next week, and the script refuses an alias that would capture a name resolving outside the lab range.

A lab that changes node gets that node's name, not the old one

An alias pointed at a freshly built VM carries a fresh SSH host key, so everyone who used the old name is refused with REMOTE HOST IDENTIFICATION HAS CHANGED until they run ssh-keygen -R. Move the lab and the name together — carnaval9 gets --dns=gpu9 on octet .169 — and let --destroy drop the old name. The script will not stop you: an alias whose current address is inside the lab range passes its guard, which is exactly what makes handing the same name to next week's VM work.

Everyone in the roster gets an account with their real UID, their SSH key, and their own NAS home mounted at /exports/<name> with ~/nas_home linked. Students log in as themselves over the VPN: ssh prenom.nom@gpu2.isc3. The VM answers in about 30 seconds; cloud-init needs a couple of minutes more to create the accounts and mount the shares, and the script prints the command that watches for it.

What you pick, and what is picked for you:

VMID	1000–1999
Last octet	128–191 — the script refuses anything outside the lab range
Node	any of the three; the script clones that node's template and attaches that node's card
Size	4 GB / 4 cores for a base VM, the CUDA template's own sizing for --gpu/--cuda. Change it afterwards with qm set <vmid> --memory 8192 --cores 8
Name	lab-<vmid>, and no pool — add one with qm set <vmid> --pool isc3-2526 if the end-of-semester sweep below is to find it

Root goes to the roster's role-rack-admins only — sudo is not a per-VM decision, because root on any machine in 192.168.91.0/24 can become any student over NFS (why, and what students can reach without it). Admins are in the roster like everyone else: real UID, own ~/nas_home, no shared ubuntu account needed.

Onboarding someone new takes two things: a line in provisioning/keycloak/roster.csv with the vpn-carnaval group, and their key in provisioning/keycloak/ssh-keys (one <name> <key> line per key) — add both and commit. Their NAS home must exist, because its ownership is where their UID comes from (why); without one the script skips them with a warning, without a key the account is created but nobody can log into it. Existing VMs don't pick up roster changes — recreate them (they are disposable) or re-run the provisioning by hand.

The week's fleet

The script takes one VM at a time; loop it over the three nodes and give the week's scenario its names — a node's NVMe and its single GPU are shared by its own guests only:

n=0; for node in carnaval0 carnaval1 carnaval2; do
  provisioning/pve/carnaval-lab-vm.sh $node $((1160+n)) $((160+n)) --gpu --dns=gpu$n
  n=$((n+1))
done

Building a new image for the week (a package added to the recipe, or hand-tuning before it is cloned everywhere) is the image pipeline.

Give a VM a GPU

--gpu is the whole job — it implies --cuda, so it clones that node's CUDA template (driver and toolkit already inside) and attaches that node's card:

provisioning/pve/carnaval-lab-vm.sh carnaval2 1142 --gpu --dns=gpu2

Students share the card through Slurm: srun --gres=gpu:1 nvidia-smi, or #SBATCH --gres=gpu:1 in a batch script; MPI programs run with srun -n 4 ./prog.

Check inside the guest that the card computes rather than merely enumerates: nvidia-smi, then compile something.

One GPU, one running VM

The card is passed through whole — no vGPU licence, no mediated-device split — so a second VM configured with the same hostpci0 refuses to start while the first is running. Check the node before promising someone a card, or use a different one (reasoning).

A GPU guest also shows its entire memory allocation as used in the web UI while free inside it shows almost nothing — why.

Clean up

Disposability is the point: destroy rather than tidy.

provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 --destroy

Use that rather than qm destroy by hand: it also drops every DNS record pointing at the VM's address — lab-<vmid>, .isc3 form and weekly alias alike — and a record that outlives its VM points at the address the next VM will be allocated.

At the end of a semester, everything in a cohort pool can go:

pvesh get /pools/isc3-2526 --output-format json | grep -o '"vmid":[0-9]*' | cut -d: -f2 \
  | while read id; do qm stop $id 2>/dev/null; qm destroy $id --purge 1; done

Nothing here is backed up, so a destroy is final. That is deliberate — but it means anything a student wants to keep must leave the VM, and ~/nas_home is where it should go.

Going further

• Mechanics & manual steps — why the UID has to come from the NAS, what sudo on a lab VM really grants, the qm sequences to build a guest without the script, and rebuilding the cluster from the repository.
• Live items are tracked in one place — ops todo → Carnaval. In short: a k8s-node template and the Kubernetes-on-VMs work, cohort accounts (there are none, so no student can log into the web UI yet — federating PVE to Keycloak would avoid hand-creating ~40 a year), collecting the 25/26 cohort's SSH keys, a carnaval0 PERC battery, and deciding which guests eventually deserve a selective backup job.