Carnaval labs — mechanics & manual steps

What carnaval-lab-vm.sh gets right on your behalf and why it matters, plus the qm sequences for the cases it does not cover. The everyday commands are on Running labs on carnaval.

Why the UID must come from the NAS

The student homes on the FS2500 are mode 750 owned by numeric UIDs, and NFS authorises by number. A default cloud-init user (uid 1000) mounts the share and can read nothing, so each account has to be created with that student's real UID. The script reads it from the NAS itself (since 2026-08-05) by stat-ing the home directory over SSH: that ownership is what NFS enforces, so it cannot be stale by construction.

Only the NAS ownership is authoritative — every derived list has drifted

Not the Ansible conf/users/uid-*.yml, and not a Calypso node's getent passwd either, which this runbook claimed until 2026-08-05. Measured that day: five accounts disagree — kevin.voisin is 10032 on the NAS and 25301 on every node, and louis.heredero is right on calypso0/2 and wrong on calypso1, so the nodes do not even agree with each other. A wrong number is silent: the account is created, the mount succeeds, and the student cannot read their own home.

The script therefore refuses to build a VM when a home is owned by uid 0, or when it does not exist at all. The permanent fix is the UID register, built 2026-08-05 — stat-ing the NAS stays correct meanwhile, since that is also what seeds the register.

The homes also require NFSv4.1: the share carries a Synology ACL that v3 cannot see, and over v3 the POSIX bits read 000 and every student is denied (mounting rules). The mount is non-fatal on purpose, so a lab VM stays usable when the NAS is unreachable — the accounts are still created, ~/nas_home is just empty.

What sudo on a lab VM actually grants

Root on a lab VM is root over the whole cohort's files

The NAS trusts the client (sec=sys), so root on any machine in 192.168.91.0/24 can mount the entire homes export and become any UID — root_squash stops uid 0, not sudo -u '#10020'. Grant it only to someone you would trust with everyone's files.

That is why --sudo takes names. --sudo=pmudry keeps root to you while the students on the same VM stay confined; bare --sudo means every student on that VM, so it is reserved for a group already trusted with the cohort's work.

Without it, a student gets one NFSv4.1 mount of their own home at /exports/<name> and no way to widen it from inside the VM — another student's path simply does not exist there. The same exposure does remain on the Calypso nodes, where students are root-equivalent through docker, so until those are rebuilt 192.168.91.0/24 should be read as "trusted as much as its most privileged student" (todo → Carnaval).

Clone a lab VM by hand

Only worth it when the script does not fit — no student accounts, no NAS homes, and everyone shares the ubuntu account the admin key lands on. fast-vm is node-local, so the template must be the one belonging to that node (9000 → carnaval0, 9001 → carnaval1, 9002 → carnaval2). A clone is a ZFS linked clone: instant, and it costs only what the guest writes.

ssh root@carnaval1                                   # the node that will host it
qm clone 9001 1121 --name lab-kevin --pool isc3-2526
qm set 1121 --ipconfig0 ip=192.168.91.121/24,gw=192.168.91.1 \
            --nameserver 192.168.88.1 --searchdomain calypso
qm set 1121 --sshkeys /root/keys/kevin.pub           # replaces the admin key
qm set 1121 --memory 4096 --cores 4                  # size it for the lab
qm start 1121

It then answers as ssh ubuntu@192.168.91.121, in roughly 30 seconds.

danger

Never set ip=dhcp

Pick an address from 192.168.91.120–.199 and set it explicitly — templates ship with no ipconfig0 precisely so a clone cannot silently pick one up. The subnet's DHCP pool was narrowed to that same range on 2026-08-04 (it used to cover the whole /24, node addresses included), but it still hands out dns-server=8.8.8.8, so a DHCP client would not resolve .calypso names.

With a GPU

Clone ubuntu-2404-cuda instead and attach the card, on the node that physically holds it:

ssh root@carnaval2
qm clone 9012 1140 --name cuda-lab --pool isc3-2526
qm set 1140 --ipconfig0 ip=192.168.91.140/24,gw=192.168.91.1 \
            --nameserver 192.168.88.1 --searchdomain calypso
qm set 1140 --hostpci0 0000:82:00.0,pcie=1 --memory 8192 --cores 8
qm start 1140

nvcc needs build-essential, which the base cloud image lacks — the CUDA template already has it, along with the driver and the toolkit.

Rebuilding from scratch

The cluster is reproducible from the repository, in this order:

Step	Runnable
Install a node	provisioning/pve/install-via-redfish.sh — not the racadm one, why
Configure a node	provisioning/pve/post-install-carnaval.sh (repos, fast pool, IOMMU, netdata)
Form the cluster	pvecm create carnaval / pvecm add, then post-install-carnaval.sh --storages
Templates and pools	provisioning/pve/carnaval-guests.sh (--pools, --templates, --cuda)
One lab VM with students on it	provisioning/pve/carnaval-lab-vm.sh

post-install-carnaval.sh and carnaval-guests.sh are idempotent: on a healthy cluster every step prints =, which makes a re-run the cheapest way to check nothing has drifted. --cuda takes about ten minutes per node, because each one fetches the driver and the CUDA toolkit itself.

Why these are shell scripts and not Ansible

The fleet does have Ansible (ansible-playbooks + -conf), and for the guest-facing half that would be the better home — the student roster and the SSH keys already live there, so carnaval-lab-vm.sh reaching into that repo for authorized_keys/*.pub is duplication waiting to bite. What is here is deliberate only for the substrate: qm / pvesh / pvecm one-shot orchestration against three hosts, matching how the rest of provisioning/ already works (post-install.sh set that pattern for rumba) and needing no control node. Migrating the lab-VM and cohort side to a playbook with community.general.proxmox_kvm is a fair next step — it is in the todo.