Skip to main content

Ops journal

Completed actions, newest first. This is where finished items from the todo land, so that page only ever contains live work. Each entry links to the page holding the details — the journal is the index, not the archive.

2026-08-06​

  • The identity model collapsed to students / staff — identity now grants admission only; the old teachers / admins reach became the entitlements appliances-users (appliance subnet
    • carnaval PVE hosts) and mgmt-users (iDRACs), so every grant is per-resource. NetBird policies repointed by provisioning/netbird/migrate-role-groups.py (access model, role groups). First staff member: Marta Rende.
  • Enrolling is now a documented two-step — roster line + sync, with the un-enroll path and its deprovisioning tail: enrolling people.
  • The ISC wiki is retired — DokuWiki archived off hannibal (rumba:/hdd/backup/wiki-archive/, where it is) and wiki.isc-vs.ch turned into a 301 vhost on srv-web01 that maps the old doku.php?id=… links onto their new pages (the vhost).
  • The documentation has its own name: docs.isc-vs.ch — GitHub Pages custom domain (static/CNAME), so the site is served at the root instead of under /dc-isc/. GitHub redirects isc-hei.github.io/dc-isc/<path> there by itself, so old links keep working.
  • A non-enrolled edu-ID login to the VPN now gets told why — Keycloak denies NetBird logins lacking the roster-derived vpn-access role with a themed EN/FR "not enrolled — contact the staff" page, instead of NetBird's bare error after a successful login (the VPN gate, provisioning/keycloak/vpn-access-gate.sh). The first binding broke every edu-ID login for ~20 min — the trap and its fix are a standing rule on the Keycloak page.
  • Pre-created roster accounts now link silently on the first edu-ID login — Keycloak's default linking demanded an email round-trip or a password, both impossible here; fixed with the eduid-autolink first-broker flow, and edu-ID made authoritative for the stored profile (provider syncMode=FORCE, VERIFY_PROFILE required action off — which removed the type-your-name form new students hit). Validated end to end with two students, VPN gate included (details, provisioning/keycloak/first-broker-autolink.sh).

2026-08-05​

  • DHCP is off on the node subnet — the CRS326 pool overlapped the carnaval lab-VM range while the VMs are assigned statically, and nothing had used the server since the MAAS client that held its only lease disappeared. Pool moved to 192.168.91.200-.240 so a careless re-enable cannot land in .128–.191, both servers disabled (todo → network follow-ups).
  • Lab VMs get their address and their name by themselves — carnaval-lab-vm.sh allocates the lowest free octet in 192.168.91.128–.191 from the guests the cluster actually runs, registers lab-<vmid>.calypso on the CCR2004, and --destroy drops both. The range is the NetBird resource, so a VM can no longer land where no VPN client reaches it (how to run it).
  • Numeric UIDs are issued from a register, and Ansible reads it — provisioning/uid/uid-map.csv seeded from the ownership NFS actually compares, with uid-alloc.py to allocate and verify and create-home.sh to make a home with the number it holds. Applying it will renumber 13 rack accounts, uid-alloc.py --check-ansible lists them (UIDs).
  • elia.pacioni renumbered 1027 → 1100 — 1027 is IscAdmin in the NAS's own /etc/passwd, so DSM rendered that student home as owned by the admin account. Moved on the NAS and the three nodes together, so nothing drifts apart.
  • Seven departed accounts removed from the Calypso nodes, their SSH keys and the register, and the last two /home trees dropped from the carnaval pre-install rescue on rumba. Their numbers are free again; nothing else in the 203 rescued homes was touched.
  • The roster now drives VPN access per resource, not per role. roster.csv became one line per person (email,groups[,nasname]), 18 third-years and 3 teachers are in, and the flat 192.168.91.0/24 grant was split so calypso-users and carnaval-users mean different things (role groups, access model).
  • UID bands widened and given a hole for DSM — the old plan ran out after 500 students and collided with the Synology's own accounts (UID allocation).
  • The SSO login page now names the service you are signing in to and hides the local username/password form behind a disclosure — edu-ID is the only thing the card offers at first glance, which matches reality: realm isc holds no password at all. Theme in provisioning/keycloak/theme/ (Keycloak page).
  • New index: service access paths — the URL or address of every service and what it asks you for, in one table set, linking to the page that owns each one.
  • Proxmox now takes an edu-ID login through Keycloak, on rumba and on the carnaval cluster — realm isc (client proxmox), root@pam untouched and still preselected. Only pre-created users are admitted (--autocreate 0), and the sole Administrator grant on each cluster is the PVE group rack-admins-isc, filled from the Keycloak groups claim at every login — so revoking there revokes here. PVE appends the realm name to claim groups, one of the traps recorded on the node page.
  • The recurring netdata UPS mails were a false alarm, now silenced at the source — upsd_ups_last_collected_secs flapped 3–4×/hour (40 pairs in two days) on stock's 5 s threshold, which the driver's stale-data spells cross routinely; it now warns at an absolute 60 s (why, and why the collector must not be reconfigured). The mail path was Netdata Cloud — rumba is claimed, and its own postfix has delivered nothing in days.
  • Carnaval nodes added to the rack status page, with a netdata deep-link — provisioning/status/poller/machines.json still listed calypso8/9/10 at 192.168.91.18–.20, the exact addresses carnaval0–2 have carried since the 2026-08-04 rebuild; replaced with the real names and role. Netdata itself needed no work — the agents have been running unsupervised on all three since the build — only surfacing them: any machine card flagged netdata: true now links straight to its :19999 dashboard (how).
  • No lab VM mounts the whole homes tree any more — the pre-2026-08-05 VMs are gone (the cluster holds only the six templates), so that item closes; the same exposure on the Calypso nodes, where docker makes students root-equivalent, stays open until the rebuild.
  • --sudo now actually grants root — useradd leaves the account with no password at all, so group membership alone left sudo prompting for a credential that cannot exist. carnaval-lab-vm.sh writes a NOPASSWD drop-in per granted name, the way cloud-init already does for ubuntu (how to run it).
  • A GPU guest reads as 100 % memory used, and that is honest — VFIO pins every page and passthrough forces balloon: 0, so PVE reports the qemu process RSS rather than guest usage: 62 GB against 1.3 GB actually in use (why). Same pass corrected two stale claims — ubuntu-2404-cuda exists on all three nodes, and the reference guest VM 110 is gone, so all three GPUs are free.
  • Lab-VM UIDs now come from the NAS, not a login node — carnaval-lab-vm.sh stats the home directory, because the ownership NFS enforces is the only source that cannot be stale: a Calypso node's passwd disagreed for 4 of 67 accounts, and a wrong number fails silently. It also refuses a home owned by uid 0. The permanent fix — issuing UIDs once instead of rediscovering them — is designed in UID allocation; the four drifted accounts and the unused groupe1–groupe4 dirs are open items.
  • --sudo takes names — --sudo=pmudry gives root to one person on a lab VM instead of the whole class, so a teacher can hold a privileged account of their own with their real UID and NAS home (how to run it).
  • Two home-ownership repairs on the FS2500 — olivier.amacker's home was owned root:root, restored to his 10011:10011; the four unused groupe1–groupe4 project directories and their groups on calypso0/1/2 are gone (open items).
  • A NetBird restart destroyed its own store — encryptionKey was empty, so the server had been minting a throwaway key at every start; ~90 min VPN outage, carried by the legacy WireGuard, and the store rebuilt from config-snapshot.json with the new replay-config.py (incident, what the account looks like now).
  • The VPN logs in through Keycloak, and access follows the roster — netbird up goes straight to ISC SSO then edu-ID, groups come from a groups claim, and admission is membership of a role group rather than a manual approval nobody could see (how it works, what it took). Role groups exist twice on purpose — jwt for federated users, *-manual for local ones — because sync ignores a claim naming a hand-made group. Remaining: fill the roster for the cohort, and re-verify with a second identity (open items).
  • edu-ID logout no longer ends on a French error page — the provider's logout URL is now empty, so logout ends at Keycloak and returns to the client (why).
  • NetBird lazy connections off — they were stranding a connected peer with correct routes; relay-only with three peers gains nothing from them (detail).
  • History sub-pages made uniform — one name for all of them: <page>-history.md, titled "history & operations", newest entry first. nas-operations.md was renamed and the rumba page's sections flipped to newest-first; the convention is now in the document-work skill.
  • Ansible lives in this repository — provisioning/ansible/, playbooks and their configuration together, laid out the way the setup script used to assemble them so no playbook changed. The two GitHub repos are archived. Why: the roster and SSH keys sat in one repo while everything consuming them sat elsewhere (how it works now).
  • Lab VMs no longer expose the whole cohort's homes — one NFSv4.1 subdirectory mount per student instead of the homes tree, and sudo is now opt-in (--sudo), since root on any box in the node subnet can impersonate any UID over sec=sys NFS. Verified: a student reads their own home, another student's path does not exist (how to run it).
  • The MikroTiks are driven over their API, not SSH — provisioning/network/routeros-api.py. RouterOS echoes log lines into interactive sessions (stock action=echo topics=critical), which races every scripted pattern-match against the ~10 s login timeout; the API has no terminal and answers in under a second. Used it to disable the Dude server, the source of the constant login failure … via winbox entries (it polls with credentials that no longer match).
  • Student-home NFS mounts work again — the 2026-08-02 hardening had made every new NFSv4.1 mount of calypso_homes/homes fail while the live ones kept working, so a node reboot would have lost its homes. Fixed with one non-inheriting r-x ACL entry for guest on the share root, keeping root_squash: what it was and why that fix.
  • NAS writes went from 65 to 102–109 MB/s — both exports switched to async, which was the entire bottleneck; the client-side sync everyone assumed was the problem costs nothing on top (measurements and the durability trade-off). Writes now match reads at 1 GbE wire speed. Also corrected on the way: the claim that the FS2500 refuses NFSv4 was wrong.

2026-08-04​

  • Carnaval has an operator runbook — Running labs on carnaval: handing a VM to a student, attaching a GPU, cohort loops, end-of-semester cleanup, and rebuilding the cluster from the repo. Also on 2026-08-04, the CRS326 DHCP pool was narrowed from the whole 192.168.91.0/24 to .120-.199 and the duplicate dhcp_test server disabled, so a VM can no longer be handed a node's address; dns-server is still 8.8.8.8 (todo).

  • Carnaval can hand out lab VMs, GPU included — ubuntu-2404-base templates on all three nodes (one each: fast-vm is node-local), an ubuntu-2404-cuda template on the T4 node, and the pools infra / isc1-2526 / isc3-2526. Clone → SSH in ~30 s, and a clone with the T4 passed through ran a CUDA kernel with zero mismatches. Runnable: provisioning/pve/carnaval-guests.sh; conventions, VMID/address plan and the passthrough recipe on the cluster page. Two things worth knowing: a PVE pool is not a quota, and PVE 9's kernel ships a third NVIDIA module (nova_core) that also has to be blacklisted on the host. Remaining: todo → Carnaval.

  • The playground cluster exists — calypso8/9/10 wiped and rebuilt as the 3-node Proxmox cluster carnaval (carnaval0–2, PVE 9.2.2, quorate), guest disks on a ZFS pool on each node's NVMe, IOMMU ready for the A2/T4 passthrough. Starts phase 3. Local /home (74 GB) rescued to rumba:/hdd/backup/carnaval-preinstall/ first. Two findings are written up where they belong: this BIOS cannot boot the NVMe cards — the previous install hid that behind a Clover ESP, one of them on a USB stick — and iDRAC8 needs a different install path than rumba's. Remaining: todo → Carnaval.

  • PSU criticals cleared on calypso0–calypso3, and the R630 fleet swept for PERC batteries — an iDRAC GracefulRestart re-inventories the chassis and drops a deliberately absent PSU (Critical → OK); clearing the SEL alone does not, and it never clears an absent BBU. Three nodes (calypso1, calypso2, carnaval0) are missing theirs and stay Critical until the kits arrive — table, order and traps in ops todo → R630 hardware.

  • Admin panels are behind a login instead of an IP range, and the split-horizon DNS layer is gone — oauth2-proxy on srv-web01 (Caddy forward_auth, Keycloak group rack-admins) now guards sso.isc-vs.ch/admin* and vault.isc-vs.ch/admin*, so they work from anywhere with no VPN and no client-side DNS trickery: the NetBird sso-split-horizon nameserver group and the vault.isc-vs.ch static record are deleted. Source-IP protection was what forced an internal answer for a public name, and that layer broke repeatedly on its own (RouterOS cache, macOS AAAA/CNAME). The sso.isc-vs.ch record stays, for a different reason: rack services resolve the issuer for OIDC discovery and the CCR does not hairpin. A LAN/VPN branch stays as break-glass, since the gate authenticates against Keycloak itself: how it works, runnables in provisioning/oauth2-proxy/.

  • Password vault live, and it is Keycloak's first consumer — Vaultwarden 1.37.1 as CT 111 (srv-vaultwarden), published at vault.isc-vs.ch, login through the realm isc (SSO_ONLY, invite-only). Keycloak login does not replace the per-user master password — service page. Needed trustEmail=true on the eduid provider, otherwise brokered users arrive unverified and Vaultwarden refuses the signup.

  • Rumba's own configuration is now code — provisioning/pve/post-install.sh covers everything the installer leaves out: deb822 repos, the hdd RAIDZ2 pool, the five storages, both nightly backup jobs, netdata. Idempotent, so it doubles as a check that the node still matches its page. Closes the first of the IaC audit gaps; items 4–7 there remain.

  • Keycloak realm and NetBird access model are snapshotted into git — export-realm.sh → realm-isc.json and export-config.sh → config-snapshot.json, secrets stripped, to be re-run after any console/dashboard change and committed. Both are diff tools, not restore paths: neither API can import what it exports — Keycloak, NetBird.

  • edu-ID federated login works and feeds three institution groups — RRA approved hes-so_isc3_vs_oidc_sso, a real login brokers a user with PKCE S256, and hes-so / hevs / isc are filled from the claims by provisioning/keycloak/groups-and-mappers.sh (how). The org unit took an RR amendment: HES-SO leaves LDAP ou (claim swissEduPersonOu) empty and populates eduPersonOrgUnitDN, where cn=RACA-TICO-ISCO,ou=acad is what marks an ISC member — affiliation alone never can. Measured values, the four traps and the claim-dump recipe in the secretzone (eduid-oidc.md).

  • SSO login page now wears the ISC brand — Keycloakify theme (petal colors, logo, dark mode) built in provisioning/keycloak/theme/, deployed with deploy-theme.sh, loginTheme=isc on the realm. Build/deploy notes on the Keycloak page; provider jars now survive in-place upgrades.

  • Acronym glossary extracted to its own page — moved out of the target architecture into infra/glossary, linked from the three section landing pages.

  • Keycloak admin console reachable for NetBird admins — sso-split-horizon nameserver group (admins only) + the public sso record flattened CNAME → A: macOS resolves AAAA via the public resolver, and a public CNAME there poisons split-horizon answers. DNSSEC on the domain was toggled off/on while chasing this and verified harmless. Standing rule and details on the Keycloak page.

  • Rumba's guests now back up off-host — PBS 4.2 deployed as a VMM VM on the FS2500 (srv-pbs.calypso, PVE storage pbs-fs2500), nightly 03:00 job for all guests, chain verified with a live CT backup. Same day: Volume 2 declared PBS-only, datastore grown to 5 TiB, retention widened to 14/8/6 (≈ half a year). Local 02:00 job trimmed to keep-daily=3 — the hdd pool would have overflowed on ~Aug 8 (14 slots × ~300 GB never fit in ~1.9 TB). Details, the PBS-vs-NFS rationale and the VMM/token traps on the service page.

  • Site footer cleaned up — "Documentation interne" line removed, neutral ⚙️ instead of the heart, build stamp now carries the build time as well (docusaurus.config.ts).

  • srv-web01 page reframed as "Landing page & reverse proxy" — the container terminates TLS for every public service, and the page (plus a new Role row) now says so up front instead of reading as a mere web page.

2026-08-03​

  • Netdata's stock UPS battery-charge alarm silenced — it keys on nutdrv_qx's junk charge estimate, and the UPS firmware intermittently zero-fills battery.voltage in its Q* replies, so the alarm fired with the UPS on line. Override now installed by deploy-ups.sh; details.
  • Rumba moved onto the rack UPS (both PSUs, ~5 s per swap, no reboot). UPS load 14 % ≈ 265 W — rumba is the only thing on it, and the UPS input is itself a PDU outlet. Measurements in the rumba history. Still open: NUT is monitor-only, so a long outage hard-kills rumba — todo.
  • UPS status on the status page — NUT polled every 30 s, UPS tile + load chart + history columns, Telegram alerts keyed on ups.status only (never on the estimated runtime).
  • UPS real battery telemetry decoded — the second USB interface (HID Power Device) reports a measured autonomy of 55 min at rumba's load; NUT's own charge/runtime estimates are junk on this unit. Read with ups-read-hid; details and traps in the HID section.
  • NAS given one dedicated 10 GbE port per subnet — second cable, eth3 static 192.168.91.250/24; rumba ↔ NAS measures 1.1 GB/s. Bonding rejected on purpose. NAS history; rebuilt nodes must follow how a node must mount it.
  • UPS monitored over USB (NUT) — the BlueWalker has no network interface, so USB from rumba is the only management path. Service page; runnables in provisioning/ups/.
  • Rumba's update backlog cleared — 0 pending, pve-manager 9.2.6 / kernel 7.0.14-8-pve.
  • Docs reorganized into present / past / future tiers — machine/service pages hold current state only; dated logs moved to history sub-pages; incidents in the incident log; open items consolidated in the todo.
  • NetBird masquerade turned off — appliances now see each user's own overlay address (100.79.x.x), so logs and per-IP protections act per user; the July 2026 lockout pattern is gone on this path. Service page.
  • Keycloak admin console needed split-horizon DNS — the VPNs are split tunnels, so VPN clients resolved sso.isc-vs.ch to its public address and got the Internet-facing 403. Fixed with a static entry on the CCR2004; traps on the service page.
  • Keycloak identity broker deployed — CT 110, https://sso.isc-vs.ch, realm isc, admin console LAN/VPN-only. Service page; runnables in provisioning/keycloak/.
  • SWITCH edu-ID OIDC client registered — hes-so_isc3_vs_oidc_sso, awaiting RRA approval. One client for the broker; GitLab, Grafana and Moodle become Keycloak clients with no further AAI paperwork. Record and constraints in the secretzone (eduid-oidc.md).
  • NetBird: edu-ID needs no licence — external OIDC login and JWT group sync are Community features; only SCIM directory sync is Enterprise. Do not buy it. Details.
  • Proxmox web UI given the ISC typography — self-hosted Manrope/Inter, idempotent deploy with --revert. Run check-coverage.sh after every pve-manager/extjs upgrade. Rumba history.
  • NetBird group profiles validated client-side — admin and student profiles match the access model exactly, split tunnel confirmed. NetBird history.
  • Connecting docs made NetBird-first — using/connect.md is the NetBird path; WireGuard moved to a legacy subpage.
  • NetBird deployed in production — vpn.isc-vs.ch, VM 109, everything on the existing 443; relay-only is the permanent mode (the uplink drops inbound UDP 3478). The MikroTik WireGuard stays as the emergency/admin path. Service page.

2026-08-02​

  • Network performance audit run and fixes applied — node↔node is wire-speed 1 GbE; the 91↔88 path was software-routed at 38.7 MB/s, fixed with on-link routes (61.5 MB/s NFS writes), a fasttrack-connection rule and the layer-3-and-4 bond hash policy. No new hardware needed. Network history; resulting config in the storage fast path.
  • Calypsomaster audited (read-only): nothing load-bearing runs on it. Findings and the conversion-to-second-PVE-node plan in the todo.
  • Ansible repos clarified: ansible-playbooks (+ private -conf) are the current fleet management; the copies on calypsomaster are stale duplicates of the archived repo.
  • FS2500 given its own NAS page with the first disk-level readout — Volume 2's Toshibas are 7.4 years old, older than the NAS itself.
  • FS2500 NFS exports hardened — three real holes closed (a /24 mask that admitted every VPN user as root, no_root_squash on the node rule, delete rights on the homes ACL). Details.
  • FS2500 data scrubbing run for the first time ever (Volume 1 had never been scrubbed in 1.8 years) — 0 errors, mismatch_cnt=0; quarterly schedule enabled on both pools. Details.
  • FS2500 Volume 2 rebuilt from 7-disk RAID 5 to 6-disk RAID 6 + a hot spare covering both pools — survives two failures, and Volume 1 gets automatic rebuilds for the first time. Capacity 11 → 6.7 TB, at no cost (the volume was empty). Details.
  • FS2500 rumba_backup stub removed — 6.7 TB of free RAID 6 available for the DS923↔rack replication goal (still in the todo).

Earlier in 2026​

  • Patch panel for room 23N307 searched and bought (server-room list). Placement constraint to respect at install time: outside the rack, not inside — rack space will be premium.