Skip to main content

Carnaval cheat sheet

One page, one command per task, for the carnaval playground cluster. The detail behind each line is on the linked page; this one is meant to be open in a terminal. Everything runs from a checkout of this repository, on your own machine, with the VPN up and your SSH key loaded (ssh-add -l shows it — a rack command that hangs with no output means the agent is empty).

All scripts are in provisioning/pve/. Paths below are relative to the repository root.

The model in four lines​

  • Who gets in is provisioning/keycloak/roster.csv: a vpn-carnaval line is an account on every GPU/lab VM, role-rack-admins adds root. Students never get root.
  • How they log in is provisioning/keycloak/ssh-keys, one <name> <key> line per key.
  • VMs are disposable: stamped from a template with the whole roster baked in, destroyed rather than tidied. Software belongs in the image recipe, not in a VM.
  • One GPU, one running VM per node: the card is passed through whole.

People​

TaskCommand
Add someoneone line in roster.csv (email,groups[,nasname], e.g. prenom.nom@hevs.ch,staff;vpn-carnaval), their key in ssh-keys, then provisioning/uid/onboard.sh <name> [key.pub] --commit — UID, NAS home, Keycloak groups
Push them into the VMs already runningprovisioning/pve/carnaval-refresh-users.sh --commit (add-only; without --commit it shows the plan)
Add or replace a keyedit ssh-keys, commit, then the same carnaval-refresh-users.sh --commit
Give someone rootadd role-rack-admins to their roster line; takes effect at the next stamp or refresh. Also puts their key on the ubuntu break-glass account of every guest
Remove someonedelete the roster line, ssh root@srv-keycloak 'bash -s' < provisioning/keycloak/roster-sync.sh (minutes), then re-stamp the VMs — the refresh never removes
"Logs in fine, no access"their edu-ID address is not the one in the roster — account model. No home on the NAS → skipped with a warning at stamp time; no key → account with no way in

Full recipe with the traps: Adding a person and the roster header itself.

VMs​

TaskCommand
A GPU VM for the weekprovisioning/pve/carnaval-lab-vm.sh carnaval2 1162 162 --gpu --dns=gpu2 — node, VMID (1160+N by convention), last octet (.128–.191, allocated if omitted), the card, the name students type
A plain VMprovisioning/pve/carnaval-lab-vm.sh carnaval1 1121
The whole fleetfor n in 0 1 2 9; do provisioning/pve/carnaval-lab-vm.sh carnaval$n $((1160+n)) $((160+n)) --gpu --dns=gpu$n; done
Is it ready?about 30 s to answer, 2–3 min more for cloud-init: ssh ubuntu@gpu2.isc3 'cloud-init status --wait && sinfo && ls /exports'
Resizessh root@carnaval2 'qm set 1162 --memory 16384 --cores 8' (needs a stop/start)
Destroyprovisioning/pve/carnaval-lab-vm.sh carnaval2 1162 --destroy — also drops its DNS names. Never qm destroy by hand: the stale record points at the next VM
Move a lab to another nodedestroy on the old node, stamp on the new one with the new node's name (gpu9 on carnaval9): a reused name means a changed host key for everyone
Kubernetesprovisioning/pve/carnaval-k8s.sh up (three VMs on carnaval0–2, ~10 min) · status · token <name> · keys (admin keys) · destroy. Needs the card, so the gpu0–2 VMs must be stopped first

Details, qm sequences and the why: Running labs, mechanics.

Software on the VMs​

WantDo
Something permanent (a package, a driver, a tool for a course)add it to the recipe — carnaval-guests.sh, the GUEST block, or carnaval-slurm.sh for the scheduler side — then rebuild the image (below). Anything installed by hand in a VM is gone at the next stamp
Something now, on the running VMsfor h in gpu0 gpu1 gpu2 gpu9; do ssh ubuntu@$h.isc3 'sudo -n apt-get install -y <pkg>'; done — and put it in the recipe too
Rebuild the image1. destroy every VM stamped from it (gpuN, and carnaval-k8s.sh destroy if the k8s VMs exist — they are linked clones too) · 2. ssh root@carnavalN 'qm destroy 910N --purge 1' on each online node · 3. NODES="carnaval0 carnaval1 carnaval2 carnaval8 carnaval9" provisioning/pve/carnaval-guests.sh --cuda (first node builds, the rest receive; ~10 min) · 4. stamp the fleet again
A node that was off at rebuild timeqm destroy 910N --purge 1 on it, then NODES="<build node> carnavalN" provisioning/pve/carnaval-guests.sh --cuda-seal
Hand-tune before sealing--cuda-prepare, ssh into the build VM through the node, then NOTE='what' … --cuda-seal

What the image carries today and the pipeline: software stack.

Slurm and MPI on a GPU VM​

Each GPU VM is its own one-node cluster. A job that does not ask for the card does not see it.

srun --gres=gpu:1 nvidia-smi                 # the card, through the scheduler
sbatch job.sh                                # with  #SBATCH --gres=gpu:1  inside
mpicc hello.c -o hello && srun -n 4 ./hello  # MPI ranks, launched by srun (PMIx)
squeue ; sinfo                               # no sacct: there is no accounting DB

Defaults and knobs: Slurm section.

Nodes​

TaskCommand
Power the fleet down / up cleanlyprovisioning/pve/carnaval-power.sh off [N] · on [N] · status — on restarts the guests that were running
Web UIhttps://carnaval0.isc3:8006, edu-ID login or root@pam; any node shows the whole cluster
A fast pool that stopped answeringthe NVMe recovery sequence on the cluster page — do not reboot blindly

When something is off​

  • Nobody can log in to a new VM — cloud-init still running, or the NAS was unreachable at boot (accounts need it for the UID): ssh ubuntu@<vm> 'cloud-init status; ls /exports'.
  • One person cannot — carnaval-refresh-users.sh (plan only) prints their line: key=MISSING or no home.
  • REMOTE HOST IDENTIFICATION HAS CHANGED — the name now points at a fresh VM: ssh-keygen -R gpu2.isc3.
  • A second GPU VM refuses to start — the card is held by the running one on that node.

Open items: ops todo → Carnaval.