Kubernetes on carnaval
Date: August 2026 · Status: phase 1 built 2026-08-28 (what runs) — in bash (carnaval-k8s.sh) rather than the playbook below, since the whole carnaval tooling is bash; phase 2 open
Scope: a k3s cluster in VMs on the three carnaval nodes, first as
reproducible automation, then kept running as the shared teaching cluster.
Decisions taken
- VMs on Proxmox, one per node — never bare metal (the nodes are the PVE cluster) and not LXC (k3s in a container needs cgroup/apparmor workarounds, and GPU passthrough is only proven here VM-style). ZFS linked clones make a broken cluster rebuildable in minutes, which replaces backups.
- k3s over kubeadm/Talos — single binary, embedded etcd, bundled CNI/ingress/local-path storage; fastest to automate and lightest on RAM.
- Three server nodes, all schedulable — embedded etcd needs 3 for quorum, and the cluster then survives one node loss (carnaval1's NVMe has already hung once).
- All three GPUs passed through to the worker VMs; the NVIDIA device plugin exposes them as
schedulable resources, so the scheduler arbitrates instead of "one GPU, one VM". Trade-off:
while this cluster runs, no card is free for
ubuntu-2404-cudalab VMs. - Storage: k3s's bundled local-path-provisioner (PVs land on the VM disk, on the NVMe pool). Longhorn rejected: replicated storage over 1 GbE on a disposable cluster. NFS PVs possible later if the dedicated FS2500 export happens.
- No backup job, matching the cluster's posture: re-running the playbook is the recovery path; anything worth keeping is a manifest in git.
Topology
| VM | Node | Address | GPU | Sizing |
|---|---|---|---|---|
k8s0 (VMID 110) | carnaval0 | 192.168.91.110 | A2 | 16 vCPU / 32 GB |
k8s1 (VMID 111) | carnaval1 | 192.168.91.111 | A2 | 16 vCPU / 32 GB |
k8s2 (VMID 112) | carnaval2 | 192.168.91.112 | T4 | 16 vCPU / 32 GB |
Infrastructure-range VMIDs (address = VMID), static IPs via cloud-init per the standing no-DHCP
rule. A passed-through GPU pins the VM's whole allocation (balloon: 0), so 32 GB is pinned per
node — inside the hosts' 64 GB reserve, leaving ~30 GB for other lab VMs on carnaval0/1 (~15 GB on
carnaval2). Growing later is easy; shrinking a live etcd member is not.
API endpoint: a kube-vip VIP on 192.168.91.115 plus k8s.isc3 on the CRS326, so
kubeconfigs don't die with one node.
Deliverables
One script, one playbook, one directory of manifests — each layer in the tool that fits it:
| Artifact | Tool | Does |
|---|---|---|
carnaval-guests.sh --k8s | bash (extends the existing script) | k8s-node template at VMID 9020+N per node: base + NVIDIA 595 driver + nvidia-container-toolkit (no CUDA toolkit). k3s is not baked in — its version pin lives in the playbook, so the template survives k3s upgrades |
provisioning/ansible/k8s-cluster.yml | Ansible (the guest-facing half, per the standing direction) | community.general.proxmox_kvm clones 9020+N → 110/111/112 with cloud-init + hostpci0; installs k3s (--cluster-init on k8s0, joins the rest — base it on upstream k3s-io/k3s-ansible); applies the manifests; drops the admin kubeconfig, rewritten to the VIP, into the secretzone. Teardown variant destroys the three VMs |
provisioning/k8s/ | plain YAML | kube-vip, NVIDIA device plugin, per-cohort namespace + RBAC + ResourceQuota templates. kubectl apply is already declarative and idempotent — wrapping it in Ansible adds nothing |
Plus: k8s{0-2}.isc3 + k8s.isc3 DNS records on the CRS326 (then re-export the config —
the stale-mirror trap), and the K8s section becomes the first real content in the
carnaval software stack page.
Phase 2 — student access
Staff get admin kubeconfigs first. Then k3s takes apiserver OIDC arguments and
Keycloak is already there: edu-ID logins, namespace-per-cohort RBAC and
ResourceQuota — enforcement that PVE pools do not provide.
Blocked on the cohort accounts existing at all
(todo → carnaval).