Running labs on carnaval

Day-to-day operation of the carnaval playground cluster: handing a VM to a student, giving one a GPU, and cleaning up afterwards. The cluster page describes what the machines are; this page is what you type. One script does the whole job — what it gets right on your behalf, and the qm sequences for the cases it does not cover, are on mechanics & manual steps.

Two places you work from:

• provisioning/pve/carnaval-lab-vm.sh runs on your own machine, from a checkout of this repository — it reads the students' keys and their UIDs, then drives the node over SSH. It needs the VPN up and your SSH key loaded in the agent.
• Everything else is root on any node (ssh root@carnaval0) or the web UI at https://carnaval0.calypso:8006 (realm Linux PAM). All three nodes show the whole cluster.

What exists today

Nodes	carnaval0 / carnaval1 / carnaval2 — 192.168.91.18 – .20, cluster carnaval, quorate
Templates	ubuntu-2404-base on every node (9000/9001/9002) · ubuntu-2404-cuda on every node (9010/9011/9012)
Guest storage	fast-vm, a ZFS pool on each node's NVMe — 888 / 888 / 464 GB
Pools	infra, isc1-2526, isc3-2526
GPUs	A2 on carnaval0/1 (05:00.0) · Tesla T4 on carnaval2 (82:00.0) — passthrough working
Accounts	admins log in with edu-ID SSO (or root@pam); no student PVE accounts yet — students reach their lab VM by SSH, not the web UI
Backups	none, deliberately — why

No persistent guest: as of 2026-08-05 the cluster holds the six templates and nothing else, so all three GPUs are free.

Hand a lab VM to a student

Node, VMID, then everyone who gets an account on it. The address is allocated for you — pass an octet only if you want a particular one:

provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 louis.heredero
provisioning/pve/carnaval-lab-vm.sh carnaval2 1140 kevin.voisin jeremy.duc --gpu
provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 121 louis.heredero      # chosen, not allocated

The address is the lowest free one in 192.168.91.128–.191, read from the guests the cluster actually runs. That range is the NetBird resource carnaval-labs: a VM outside it is unreachable over the VPN, which is why the script refuses an octet outside it rather than trusting you.

Each VM also gets a DNS name, lab-<vmid>.calypso, on the CCR2004 beside every other .calypso record — so labs can use a name. Proxmox cannot write it: its SDN speaks PowerDNS and nothing else.

Every name gets an account with their real UID, their SSH key, and their own NAS home mounted at /exports/<name> with ~/nas_home linked — the same convention as on Calypso. Students then log in as themselves over the VPN: ssh louis.heredero@192.168.91.121. The VM answers in about 30 seconds; cloud-init needs a minute more to create the accounts and mount the share, and the script prints the command that watches for it.

What you pick, and what is picked for you:

VMID	1000–1999
Last octet	120–199 — the script refuses anything outside the lab range
Node	any of the three; the script clones that node's template and attaches that node's card
Size	4 GB / 4 cores for a base VM, the CUDA template's own sizing for --gpu/--cuda. Change it afterwards with qm set <vmid> --memory 8192 --cores 8
Name	lab-<vmid>, and no pool — add one with qm set <vmid> --pool isc3-2526 if the end-of-semester sweep below is to find it

Put yourself on the VM rather than sharing ubuntu. Name yourself among the students and you get your real UID and your own ~/nas_home, which the shared ubuntu account has neither of; --sudo=pmudry keeps root to you while the students on the same VM stay confined:

provisioning/pve/carnaval-lab-vm.sh carnaval2 1140 140 kevin.voisin pmudry --gpu --sudo=pmudry

caution

--sudo hands over the whole cohort's work, not just this VM

Root on any machine in 192.168.91.0/24 can become any student over NFS, so bare --sudo (root for every student on the VM) is deliberately not the default — reserve it for a group already trusted with the cohort's files, and prefer --sudo=<names>. Why, and what the students can reach without it.

Onboarding someone new takes two things: their key at provisioning/ansible/conf/authorized_keys/<name>.pub, one file per person — add it and commit — and an existing home on the NAS, because that home's ownership is where their UID comes from (why). Without a key the account is still created, and the script warns that nobody can log into it.

A whole cohort

The script takes one VM at a time; loop it, and spread the class over the three nodes — a node's NVMe and its single GPU are shared by its own guests only.

i=0; for who in kevin.voisin marco.pfeiffer jeremy.duc louis.heredero; do
  provisioning/pve/carnaval-lab-vm.sh carnaval1 $((1130+i)) $((130+i)) "$who"
  i=$((i+1))
done

Give a VM a GPU

--gpu is the whole job — it implies --cuda, so it clones that node's CUDA template (driver and toolkit already inside) and attaches that node's card:

provisioning/pve/carnaval-lab-vm.sh carnaval2 1140 140 kevin.voisin --gpu

Check inside the guest that the card computes rather than merely enumerates: nvidia-smi, then compile something.

One GPU, one running VM

The card is passed through whole — no vGPU licence, no mediated-device split — so a second VM configured with the same hostpci0 refuses to start while the first is running. Check the node before promising someone a card, or use a different one (reasoning).

A GPU guest also shows its entire memory allocation as used in the web UI while free inside it shows almost nothing — why.

Clean up

Disposability is the point: destroy rather than tidy.

provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 --destroy

Use that rather than qm destroy by hand: it also drops the lab-1121.calypso record, and a record that outlives its VM points at the address the next VM will be allocated.

At the end of a semester, everything in a cohort pool can go:

pvesh get /pools/isc3-2526 --output-format json | grep -o '"vmid":[0-9]*' | cut -d: -f2 \
  | while read id; do qm stop $id 2>/dev/null; qm destroy $id --purge 1; done

Nothing here is backed up, so a destroy is final. That is deliberate — but it means anything a student wants to keep must leave the VM, and ~/nas_home is where it should go.

Going further

• Mechanics & manual steps — why the UID has to come from the NAS, what sudo on a lab VM really grants, the qm sequences to build a guest without the script, and rebuilding the cluster from the repository.
• Live items are tracked in one place — ops todo → Carnaval. In short: a k8s-node template and the Kubernetes-on-VMs work, cohort accounts (there are none, so no student can log into the web UI yet — federating PVE to Keycloak would avoid hand-creating ~40 a year), DNS names for the lab VMs, a carnaval0 PERC battery, a dedicated NFS export for shared ISOs, and deciding which guests eventually deserve a selective backup job.