Carnaval operator manual
How the carnaval playground cluster works and how to run it, written for a system administrator who has never seen it. It explains the model first and the commands second; the cheat sheet is the same material reduced to one line per task, and the reference pages linked throughout hold the numbers, the measurements and the history.
1. What carnaval is
Carnaval is the student playground of the ISC³ datacenter (rack in room 23N307): eleven Dell R630 running Proxmox VE, one NVIDIA card each, on which teachers hand out virtual machines for a course or a week. Everything on it is meant to be destroyed and recreated from this repository.
What follows from that, and shapes every procedure below:
- No backups, no high availability. A VM that matters is rebuilt from a template, not restored. The only cold copy is the CUDA template dump on the NAS (why).
- Nothing is installed by hand. Software goes into the image recipe; accounts come from a roster file; names are registered by the script that creates the VM. A change made by hand inside a VM is lost at the next re-stamp, which is weekly in a normal semester.
- Nothing is public. Every node, VM and console sits behind the NetBird VPN. There is no port forwarding to carnaval and there must never be one.
- Students never get root on a VM, because root on any machine of the lab subnet can read every student's files on the NAS (why).
The services half of the rack (rumba: web, Keycloak, GitLab, mail, backups) is a separate
hypervisor with the opposite policies, documented on its own page.
2. The pieces and how they fit
| Piece | Role in carnaval | Reference |
|---|---|---|
carnaval0–carnaval10 | Proxmox VE 9 nodes, one cluster. Each has a ZFS pool fast-vm on its own NVMe card (node-local: a guest disk exists on one node only) and one GPU passed through whole to one VM at a time | hardware, storage |
| FS2500 NAS | The student homes, one directory per person, exported over NFSv4.1 to the lab subnet and mounted inside every lab VM. Also nas-library, the NFS share the CUDA template travels through | NAS |
| CCR2004 router | DNS for the .isc3 zone: node names, lab-<vmid> and the weekly aliases (gpu0…). No DHCP on the lab subnet, on purpose | network |
| NetBird | The VPN. Who may connect, and which subnets they see, is decided by Keycloak groups carried in the login token | VPN service |
| Keycloak | Identity: edu-ID login, groups synchronised from the roster. Also the SSO of the Proxmox web UI for administrators | Keycloak |
| This repository | The source of truth: roster, keys, UID register, image recipe, every script. A change to carnaval is a commit here first | provisioning/pve/, provisioning/keycloak/, provisioning/uid/ |
Addresses to keep in mind (all 192.168.x.y, reachable over the VPN only):
| Range | What |
|---|---|
91.10–91.20 | the nodes, carnavalN = 91.(10+N) |
90.10–90.20 | their iDRACs, same rule |
91.100–91.119 | infrastructure VMs (the k8s servers, VMID = last octet). Not routed to VPN clients: reach them through a node, ssh -J root@carnavalN |
91.128–91.191 | lab VMs. This is exactly the NetBird resource students get, so a VM outside it is unreachable for them |
91.200–91.210 | the template build VM, one address per node, transient |
91.250 / 88.250 | the NAS as the VMs mount it / as operators query it |
3. Setting yourself up
- VPN. Install the NetBird client, point it at
https://vpn.isc-vs.ch, log in with edu-ID (Connecting). Your account must be on the roster with thevpn-rack-operatorsgroup to see the nodes, andvpn-carnavalto see the lab VMs. - A checkout of this repository. All scripts run from your own machine and read the roster, the keys and the register from the checkout. Commit what you change: the git history is the authorisation record.
- Your SSH key, loaded in an agent (
ssh-add -lmust list it). Root on the nodes is key-based; the list of keys isprovisioning/pve/root-ssh-keys.sh, run once per cluster to add one. A rack command that hangs with no output usually means the agent is empty, not that the host is slow. - Passwords. The cluster's root password, the iDRAC credentials and the router API password
are in
secretzone/(a private directory of this repository, never linked from the docs). - Web UI:
https://carnaval0.isc3:8006(any node shows the whole cluster). Log in with the ISC SSO realm (edu-ID, needsrole-rack-adminsin the roster) or asroot@pam. - Tools. Everything is bash over
ssh,qm,pveshandpvecm, pluspython3for the router API. Nothing to install beyond an OpenSSH client.
4. Who gets in: the account model
Three files in the repository decide everything, and no VM holds an account that is not derived from them.
provisioning/keycloak/roster.csv is the list of people: one line
email,groups[,nasname]. The groups are of three kinds, told apart by their prefix:
| Kind | Examples | Effect |
|---|---|---|
| identity | students, staff | admission to the VPN at all. Never drop it from a line |
vpn-* | vpn-carnaval, vpn-rack-operators, vpn-rack-mgmt | one NetBird route each: the lab VMs, the nodes and appliances, the iDRACs. A route is not a login |
role-* | role-rack-admins, role-pve-auditor | rights inside applications: Administrator in Proxmox, superuser in NetBox, and on the lab VMs passwordless sudo plus a key on the ubuntu break-glass account |
For a lab VM the rule is simple: every line carrying vpn-carnaval becomes an account on every
lab VM, named after the email's local part (or the nasname column when the NAS home is named
differently), and role-rack-admins adds root. Nobody else.
provisioning/keycloak/ssh-keys is how they log in: one <name> <public key> line per key,
several lines for several keys. A roster line without a key produces an account nobody can enter.
The NAS home is where the numeric UID comes from. Homes are 750 and owned by a number, and
NFS authorises by number, so the account inside the VM must carry the same UID as the home; the
script reads it by stat-ing the home over SSH at creation time. The register
provisioning/uid/uid-map.csv issues new numbers (UIDs). A roster line
with no NAS home is skipped with a warning.
The same roster drives Keycloak (roster-sync.sh writes the groups) and, through the login token,
NetBird. One file, three consumers: the git diff of roster.csv is the authorisation change.
Adding a person
# 1. one line in provisioning/keycloak/roster.csv, e.g.
# prenom.nom@hevs.ch,staff;vpn-carnaval (a teacher)
# prenom.nom@students.hevs.ch,students;vpn-carnaval (a student)
# 2. their key in provisioning/keycloak/ssh-keys: prenom.nom ssh-ed25519 AAAA…
# 3. the rest is one command: UID, NAS home, Keycloak groups
provisioning/uid/onboard.sh prenom.nom [their.pub] --commit
# 4. the VMs already running do not read the roster again — push the newcomer into them
provisioning/pve/carnaval-refresh-users.sh --commit
Without --commit, both commands print what they would do. The refresh is add-only and idempotent.
The email must be the address edu-ID releases for the person, or the symptom is "logs in fine, no
access". Taking someone out is the reverse: delete the line, run roster-sync.sh (a full run,
minutes), and re-stamp the VMs — the refresh never removes.
What each kind of person can do
| VPN | Lab VMs | Proxmox web UI | Nodes | |
|---|---|---|---|---|
Student (students;vpn-carnaval) | lab range only | own account, own NAS home, no root | no account | no |
Teacher (staff;vpn-carnaval) | lab range only | same as a student | no account | no |
Rack admin (role-rack-admins + vpn-rack-operators) | nodes, appliances, lab range | own account + sudo, and the ubuntu account | Administrator via edu-ID | root@carnavalN by key |
5. Images: where software lives
Every VM is a ZFS linked clone of a template: instant to create, and it costs disk only for what
the guest writes. Templates are per node because the pool is node-local; the same VMID formula on
every node keeps that invisible (9000+N base, 9100+N CUDA on carnavalN).
| Template | Contents |
|---|---|
ubuntu-2404-base | Ubuntu 24.04 cloud image, cloud-init, guest agent. Each node builds its own from the cloud image |
ubuntu-2404-cuda | the above plus the NVIDIA server driver, the CUDA toolkit, a C/C++ toolchain, nvtop, and Slurm + OpenMPI. Built once, on one node, and published to the others through the NAS |
The CUDA image goes through a pipeline (provisioning/pve/carnaval-guests.sh): clone the base
template into a build VM, run the scripted recipe over SSH through the node, optionally hand-tune,
then seal (clean shell histories, machine-id, cloud-init clean), turn it into a template,
vzdump it to nas-library, qmrestore it on every other node with sizing adapted to that host.
The dump on the NAS doubles as the cold copy. The whole thing takes about ten minutes plus a few
per receiving node (details).
Where to put software. In the recipe: the GUEST block of carnaval-guests.sh for packages
and tools, carnaval-slurm.sh for the scheduler side. Then rebuild. Installing on the running VMs
as well is fine for the current week (ssh ubuntu@gpuN.isc3 'sudo -n apt-get install -y …' in a
loop), but only the recipe survives.
Rebuilding the image. Proxmox refuses to destroy a template that still has linked clones, so the order is fixed:
# 1. every VM stamped from the CUDA template goes first — the gpuN lab VMs, and the k8s VMs
provisioning/pve/carnaval-lab-vm.sh carnaval0 1160 --destroy # … for each
provisioning/pve/carnaval-k8s.sh destroy # if the k8s VMs exist
# 2. the old templates on the online nodes
for n in 0 1 2 8 9; do ssh root@carnaval$n "qm destroy 910$n --purge 1"; done
# 3. build on the first node of NODES, publish to the rest
NODES="carnaval0 carnaval1 carnaval2 carnaval8 carnaval9" provisioning/pve/carnaval-guests.sh --cuda
# 4. stamp the fleet again (section 6)
A node that was powered off at that time still carries the old image. When it is back:
qm destroy 910N --purge 1 on it, then
NODES="<build node> carnavalN" provisioning/pve/carnaval-guests.sh --cuda-seal, which
re-dumps the sealed template and restores it there.
6. Lab VMs: the weekly cycle
One script, provisioning/pve/carnaval-lab-vm.sh, does the whole job from your machine. Given a
node, a VMID and options, it:
- reads the roster and the keys, and
stats every home on the NAS for the UIDs; - allocates an address in the lab range (or takes the octet you give, refusing one outside the range);
- writes a cloud-init user-data snippet on the node: the
ubuntuaccount with the admins' keys, a provisioning script that creates every roster account with its UID and key, mounts each person's NAS home at/exports/<name>(~/nas_homeinside), and grantssudoto the admins; - clones the node's template (CUDA when
--gpuor--cuda), sizes it, attaches the node's GPU when asked (it reads the PCI address from the node, it differs per chassis); - registers
lab-<vmid>and, with--dns=<alias>, the name students will type, on the router; - starts the VM and waits for SSH.
The VM answers in about 30 seconds; cloud-init needs two to three minutes more to create the
accounts and mount the shares. ssh ubuntu@<name> 'cloud-init status --wait && ls /exports' tells
you when it is done.
Conventions that make the fleet predictable: the GPU VM on carnavalN is gpuN, VMID 1160+N,
address .160+N. Only one VM at a time can hold a node's card, so before promising a GPU check that
nothing else on that node has hostpci0 and is running.
provisioning/pve/carnaval-lab-vm.sh carnaval2 1162 162 --gpu --dns=gpu2 # one GPU VM
provisioning/pve/carnaval-lab-vm.sh carnaval1 1121 # one plain VM, address allocated
for n in 0 1 2 9; do # the fleet
provisioning/pve/carnaval-lab-vm.sh carnaval$n $((1160+n)) $((160+n)) --gpu --dns=gpu$n
done
ssh root@carnaval2 'qm set 1162 --memory 16384 --cores 8' # resize (stop/start)
provisioning/pve/carnaval-lab-vm.sh carnaval2 1162 --destroy # VM and its DNS names
Always destroy through the script: a qm destroy by hand leaves DNS records pointing at the
address the next VM will get. When a lab moves to another node, give it that node's name (gpu9 on
carnaval9): reusing the old name means a new host key behind a known name, and every student
gets REMOTE HOST IDENTIFICATION HAS CHANGED. The end-of-semester sweep and the qm sequences for
unusual cases are on the labs page and its mechanics.
7. Slurm and MPI inside a GPU VM
The CUDA image carries Slurm and OpenMPI, and every GPU VM boots as its own one-node cluster:
controller, compute daemon and munge on the VM, the card declared as gres/gpu:1. With one card
and a whole cohort on the VM, the scheduler is what turns collisions into a queue. Nothing per VM is
in the image: at first boot a small unit writes slurm.conf, gres.conf and cgroup.conf from the
VM's hostname, CPUs, memory and PCI GPU count, and creates the munge key.
srun --gres=gpu:1 nvidia-smi # the card, through the scheduler
sbatch job.sh # with #SBATCH --gres=gpu:1 in the script
mpicc hello.c -o hello && srun -n 4 ./hello # MPI ranks launched by srun (PMIx)
squeue ; sinfo # there is no accounting database, so no sacct
Defaults worth knowing: a job that does not ask for the card does not see it (devices are
constrained by cgroup; an interactive shell outside Slurm still sees it); a GPU job that asks for
nothing else gets 8 cores and about 12 GB, enforced, so bigger jobs say --cpus-per-task and
--mem; 24 h maximum, 1 h default. To regenerate the configuration on a VM:
rm /etc/slurm/slurm.conf && systemctl restart carnaval-slurm-setup slurmctld slurmd. All of it is
provisioning/pve/carnaval-slurm.sh, which also installs the same thing on a running VM stamped
from an older image (reference).
8. Kubernetes
provisioning/pve/carnaval-k8s.sh up builds a three-server k3s cluster in VMs 110–112 on
carnaval0–2, each with the node's GPU, a floating API address k8s.isc3 and a web UI. It takes
about ten minutes and is idempotent. destroy removes the VMs and their names, token <name> makes
a personal cluster-admin token, keys pushes the admins' current keys to the ubuntu account of the
running servers. The servers are the only account holders: students do not SSH to them (their access,
when it comes, is kubectl through edu-ID).
Two things to keep in mind: the VMs hold the GPUs, so gpu0–2 must be stopped before up, and
they are linked clones of the CUDA template, so a template rebuild destroys them. There is nothing
to lose: manifests are in provisioning/k8s/, and up recreates everything
(what runs, design).
9. Operating the nodes
Power. At most ten nodes may be powered at once (room budget). carnaval-power.sh off [N]
shuts down the guests on every node, then the nodes; on [N] powers them over the iDRACs, waits,
restores quorum and restarts the guests that were running. Lab VMs deliberately have no onboot.
A Proxmox cluster needs more than half its voters to change anything, and nodes are routinely off
here, so on (and the quorum action) sets the expected votes to the members online. That is safe
because absent nodes are off, not partitioned.
Storage. Proxmox itself lives on the slow PERC RAID-1, never on the NVMe: these BIOSes cannot
boot from the add-in cards. Guests live on the NVMe pool fast-vm. local-lvm is disabled and
must never become a guest's target. Several controllers have no cache battery and run Write Through;
do not force Write Back, these nodes have a single PSU.
Memory. A passed-through GPU pins the whole guest allocation, so the host keeps a 64 GB reserve
and the CUDA template is sized host RAM − 64 GB. The web UI shows a GPU guest at full memory from
the moment it starts; that is the host cost, not the workload.
Monitoring. Netdata on :19999 of each node, VPN-only. Nothing alerts on an NVMe failure: the
iDRAC cannot see the card. Two of the Swissbit cards have already stopped answering, weeks apart,
and were found by hand (open item).
A suspended pool. If zpool status says SUSPENDED and the NVMe device reads 0 B, the card's
controller has hung. Do not reboot, do not zpool clear: kill the kvm processes, sync -f /, then
from the iDRAC ForceOff, wait ~50 s, On, and scrub before trusting the pool. The full sequence and
why each step is what it is: recovering a suspended pool.
Adding a node. Seven spare R630 exist. The procedure (Redfish inventory, PERC RAID-1, an answer
file, a baked ISO installed over the iDRAC, post-install, pvecm add, DNS) is on the
hardware page; every step is a script in
provisioning/pve/.
10. When something is wrong
| Symptom | Likely cause | What to do |
|---|---|---|
A new VM answers SSH as ubuntu but nobody from the roster can log in | cloud-init still running, or the NAS was unreachable at boot | ssh ubuntu@<vm> 'cloud-init status; ls /exports'; if the NAS was down, re-stamp |
| One person cannot log in, the others can | no key, or no NAS home | carnaval-refresh-users.sh without --commit prints key=MISSING or no home for them |
| Someone logs in to NetBird fine but reaches nothing | their token has no vpn-* group: wrong email in the roster, or roster-sync.sh not run | fix the roster line, roster-sync.sh --only <email>, reconnect the client |
REMOTE HOST IDENTIFICATION HAS CHANGED | the name now points at a fresh VM | ssh-keygen -R <name> on the client; next time, move the name with the node |
| A GPU VM refuses to start | another VM on that node holds the card | stop the other one (qm list on the node) |
| A VM pings and accepts SSH but every command hangs | its node's NVMe pool is suspended | section 9, suspended pool |
/etc/pve is read-only, qm refuses everything | no quorum: too few nodes online | carnaval-power.sh quorum, or power nodes on |
qm destroy 91xx says the base volume is in use | linked clones still exist | destroy the VMs first (lab VMs through the script, k8s through carnaval-k8s.sh destroy) |
| A rack command hangs with no output | the SSH agent has no key | ssh-add |
nvcc fails to preprocess | a base-image clone: no C compiler | use the CUDA template, or apt install build-essential |
11. Where everything lives
| Scripts | provisioning/pve/ — carnaval-lab-vm.sh, carnaval-refresh-users.sh, carnaval-guests.sh, carnaval-slurm.sh, carnaval-k8s.sh, carnaval-power.sh, admin-keys.sh, root-ssh-keys.sh, the install and post-install scripts |
| People | provisioning/keycloak/roster.csv, provisioning/keycloak/ssh-keys, provisioning/keycloak/roster-sync.sh; provisioning/uid/ for the register, onboard.sh and create-home.sh |
| Secrets | secretzone/carnaval.md, secretzone/isc3.md |
| Current state | hardware, software stack, network, NAS |
| Procedures | labs, mechanics, cheat sheet, Proxmox install over the iDRAC |
| What happened | history, incidents, journal |
| What is left to do | todo → Carnaval |