What still needs to be fixed

Finished items move to the Ops journal so this page only holds live work.

Various

• Rename What still needs to be fixed → Open points / TODOs, and move it to the top of the Operations section.

• move the Incident log below the ops journal

• Vaultwarden might / shall be removed -> sops + age is a better solution

• NFS configuration issue if the students are root (which would allow them to install things) -> they would be able to mount any NAS mount from other students (can we protect this)

• Create a page for upcoming scheduled events (maintenance, outages, etc.) and link it from the status page — see the status.eduid.ch example.

• Create a VM for services uptime, like on https://status.claude.com/ for instance or the status.eduid.ch page for instanc

• Create a special zone for ISC admin tools from git, for Eliane and me only (impacts netbird and proxmox)

• Move isc-inventory to a VM on ISC3 (back from hannibal), put its documentation from Processes -> Intory to Services (on-site)t

• Move everything from legacy marcellus vps and discard it. First inventory what actually runs on it (vhosts, containers, cron jobs) — the list on vps.md was never filled in.

• Rename pagode, not following the naming convention.

• Check if calypso4->calypso2 still has users on it. If no, stop them.

• UPS and power items — including giving rumba a graceful shutdown on mains loss — have their own UPS and power section below.

• Remove reference to hasdrubal (don't know what it is)

• Review Change management and Incident management pages, the information is odd — and write the notification templates they link to (marked with <!-- TODO --> comments in both pages). Nothing was lost in the wiki migration: those were red links there too, so the templates have to be written from scratch.

• Delete /srv/www/wiki.isc-vs.ch on hannibal after 2026-09-06 — kept a month as the rollback path for the retired wiki; the archive is what survives.

• Before 2026-09-21, re-issue hannibal's isc.hevs.ch certificate without wiki.isc-vs.ch. That name is one of 11 SANs on the certificate shared by all the other vhosts (files, marks, inventory, ingegamez, in-egalite, isc.hevs.ch), it no longer resolves to hannibal, and certbot fails a whole lineage on one failed authorization — so the automatic renewal (certbot.timer, standalone on :80, which Apache leaves free) would take every one of them down with it. Expiry is 2026-10-21, renewal starts a month before. Re-issue with the same list minus the wiki name, and confirm the domain-list change when certbot asks:

  sudo certbot certonly --cert-name isc.hevs.ch --standalone \
    -d isc.hevs.ch -d files.isc-vs.ch -d hannibal.isc-vs.ch \
    -d in-egalite.ch -d www.in-egalite.ch -d ingegamez.isc-vs.ch \
    -d inventory.isc-vs.ch -d isc-inventory.isc-vs.ch \
    -d marks.isc-vs.ch -d marks.isc-vs.dev

• Format correctly the offline docs of the ISC Learn DRP page.

• Document the SLURM configuration section of calypso-stack.md — left empty since the wiki import.

• Network follow-ups from the 2026-08-02 audit (applied fixes in the journal, details):

  1. Fast-path route + nas-fastpath.service on calypso5–10 when they are powered back on — rerun the idempotent provisioning/network/followups.exp. Long-term: fold the unit into the Ansible playbooks.
  2. CRS326 hygiene — mostly done 2026-08-04: the DHCP pool is now 192.168.91.120-.199 (was the whole /24), dns-server is 192.168.88.1 (was 8.8.8.8) and the duplicate dhcp_test server is disabled. Done 2026-08-05: the pool moved to 192.168.91.200-.240 and both DHCP servers are disabled — nothing had used them since the MAAS client that held the only lease went away, and the pool overlapped the carnaval lab-VM range. The pool was moved rather than deleted so that re-enabling the server cannot land back in .128–.191. Still open: device DNS points at the retired 172.30.7.1, and it is still on RouterOS 6.
  3. CCR2004 leftovers: Dude disabled 2026-08-05 (it was polling with stale credentials, filling the log with login failure … via winbox — and RouterOS echoes those into every SSH session, which is what made scripted router access unreliable; drive the routers with provisioning/network/routeros-api.py instead). Still to review: the L2TP server, and an L2TP client dialing out to wifx.ch (185.43.244.107) — why?

• For the production cluster, does it make sense to use HDR200 instead or HDR100 is enough ? Considering that we will have epyc0 and epyc1 on it

• Open points specific to the research infrastructure (Chacha, Disco, Tango, "Dance New") live on the CALC@HEI todo page.

Rumba

Open items for the Proxmox node; done work is in the journal and the history page.

• Destroy the disposable GitLab test pair (CT 105 srv-gitlab + VM 106 srv-runner01) when the testing phase ends — teardown process.
• Verify the first full 03:00 run to PBS (scheduled for the night of 2026-08-03→04; the ~300 GB first pass of VM 107 is the long pole) and check the dedup factor after a few nights.
• Cluster the node with epyc0 / epyc1 per the target architecture — see also the calypsomaster conversion (two nodes need a QDevice third vote).

NAS (FS2500)

Ranked actions from the August 2026 disk-health readout. The 2026-08-02 hardening, first-ever data scrubbing and Volume 2 rebuild are done — history page.

1. Protect calypso_homes — it is the only copy of the student homes: daily immutable Btrfs snapshots plus an off-box backup. Deferred on 2026-08-02 until the Proxmox side settles, since the intended target is Rumba's hdd pool; the constraints established that day (Snapshot Replication not installed, push-not-pull because IscAdmin cannot read the 750 homes) are recorded under Backups below.

2. Re-point the node mounts to 192.168.91.250. The NAS now has one dedicated 10 GbE port per subnet — eth2 = 192.168.88.250 (DSM, rumba/PBS backups), eth3 = 192.168.91.250 static, in the Calypso node subnet (done 2026-08-03; bonding was considered and rejected — redundancy is not the goal, throughput and simplicity are).

   What remains is the node side, and since the Calypso machines will be rebuilt as Proxmox nodes it belongs in the image/playbook, not a per-node script. The rules and both the fstab and pvesm add nfs forms are written up once on the NAS page — how a node must mount it. No export change is needed; the NAS already exports to 192.168.91.0/24 and matches on the client address.

   Same edit drops the leftover sync and the duplicate /exports/ line from the nodes' fstab. Neither is urgent: with the exports now async, a client-side sync was measured to cost almost nothing (2026-08-05).

   Afterwards, delete /usr/local/etc/rc.d/nas-fastpath.sh, the manual 192.168.91.0/24 dev eth2 route, and nas-fastpath.service from any surviving node.

3. (done 2026-08-05) Write throughput — the exports were sync, holding writes to 65 MB/s. Switching both rules to async took them to 102–109 MB/s, i.e. 1 GbE wire speed, and small files from 175 to ~300/s; the client-side sync turned out to be irrelevant on top of it (measurements). The durability trade-off this accepts is a standing caution on the NAS page and is an argument for item 1 above.

4. Until the nodes are rebuilt, the fast-path route can vanish. Nodes still mounting 192.168.88.250 depend on it, and /usr/local/etc/rc.d/nas-fastpath.sh re-adds it only at DSM boot, so a link flap costs ~40 % of read throughput silently. Re-check ip route after physical work behind the NAS; if the rebuild is far off, a nas-fastpath.timer (OnUnitActiveSec=5min) would bound it.

5. Tidy eth0/eth1 on the NAS. Both are still BOOTPROTO=dhcp with no cable, so plugging either one in re-opens the DSM gateway-election hole described on the NAS page.

6. Buy one cold-spare S+RJ10 — all three 10 G RJ45 modules are now in service, so a module failure has no spare.

7. Enable DSM notifications and SSD-lifespan alerts so a degraded array or wearing disk is not discovered by accident.

8. When the 16 GB DIMM arrives (~Aug 10, senetic): install in slot B (→ 24 GB total), then raise the PBS VM to 4 vCPU / 8 GB in VMM and eject its install ISO (Edit → Others — the API can't do it).

UPS and power

The rack UPS has powered rumba and nothing else since 2026-08-03 (what changed). It is monitored with NUT and shown on the status page, which alerts on mains loss.

1. Shut rumba down cleanly before the battery runs out

The problem in plain terms. If the building loses power, the UPS carries rumba for about 55 minutes. When the battery empties, rumba loses power instantly — the equivalent of pulling the plug on a running hypervisor, with its guests and ZFS pools live. Nothing currently prevents that: NUT's upsmon is deliberately configured never to shut anything down (powervalue 0).

What we want is for rumba to notice "I am on battery and the battery is nearly empty" and shut itself down gracefully with time to spare. upsmon is the component that would do it. The only real question is where it gets a trustworthy battery level from, because the driver it currently listens to (nutdrv_qx) reports a battery percentage that is provably wrong — permanently stuck at 100 %. The UPS does expose correct data, but on a second USB channel NUT cannot read (details).

Three ways to close that gap:

• A. Trust the one flag NUT already has. Besides the broken percentage, the protocol carries a single yes/no "battery low" flag, and upsmon can act on that alone — so this is a one-line change (powervalue 1, MINSUPPLIES 1). The catch: we do not know whether the UPS's firmware sets that flag itself, or whether NUT synthesises it from the broken percentage. If it is synthesised, the shutdown could fire far too early (a needless outage) or never (no protection at all). Cheapest, but currently unverified — see item 2 below, which would settle it.
• B. Write our own small shutdown watchdog. A short script on a timer reads the good channel with ups-read-hid and runs shutdown when the real remaining time falls under a margin (say 15 min), or when the UPS's own genuine "shutdown imminent" bit sets. We have already confirmed this data is correct, and 55 minutes of runtime leaves a comfortable margin. Roughly 50 lines plus a systemd timer. The downside is that it is ours to maintain, doing by hand a job NUT exists to do. Recommended — it is the only option that acts on numbers we have actually verified.
• C. Feed the good data into NUT. Run a second, fake NUT device (dummy-ups) that a script keeps updated from the good channel, and point upsmon at that instead. upsmon then does its normal job with correct numbers, using only stock NUT. Most moving parts, and it needs a staleness guard — otherwise a dead updater leaves upsmon reading a frozen "battery full" forever, which is worse than no protection because it looks like protection.

2. Run one controlled power-fail test

Decided 2026-08-03 — this is the chosen path. Runbook: UPS controlled discharge test; the logger it depends on is provisioning/ups/log-discharge.py, installed as ups-log-discharge.

Pull the UPS's input at a chosen moment, with rumba's guests shut down beforehand, and watch what NUT reports as it discharges. This is the only thing that answers three questions at once: whether the "battery low" flag of option A is real, what the battery's actual autonomy is today (the 55 min is the UPS's own estimate, and batteries age), and whether the shutdown path works end to end. It costs a planned outage — but so does discovering the answers during an unplanned one.

3. Decide what else the UPS should protect

rumba uses only ~14 % of the 2 kVA, so there is headroom. The obvious candidate is the NAS, which holds the only copy of the student homes, but a 2 kVA unit cannot cover the whole rack (PDU peak 3.63 kW). Needs a physical cord trace at the rack, and rename the PDU's 24 factory-default outlet names while doing it so this is answerable remotely next time. Also worth checking whether the chassis has an intelligent-card slot for a real SNMP interface.

R630 hardware — PERC batteries and PSUs

Fleet sweep of 2026-08-04 over the eleven reachable node iDRACs (192.168.90.10–.20). Two parts have been harvested from these chassis over time: the redundant PSU (all eleven now run on PS2 alone) and the PERC H730P Mini battery.

Action — order 5 × battery kit (pack with holder, 070K80 / 0H132V / 037CT1 / 0HD8WG, €33.61 net, in stock): three confirmed missing plus the two unconfirmed below. Buy the kit, not the €29.40 bare pack — whether the holder was pulled with the battery cannot be read remotely, and the bracket is a separate €5.03 part.

Node	iDRAC	Host	PERC battery	Reading
calypso0	.10	on	Ok / Ready	live
calypso1	.11	on	Failed / Removed	live
calypso2	.12	on	Failed / Removed	live
calypso3	.13	off	Ok / Ready	stale
calypso4	.14	off	Unknown / Ready	stale
calypso5	.15	off	no battery object enumerated	stale
calypso6	.16	off	Unknown / Ready	stale
calypso7	.17	off	Unknown / Removed	stale
carnaval0	.18	on	Failed / Removed	live
carnaval1	.19	on	Ok / Ready	live
carnaval2	.20	on	Ok / Ready	live

Read with racadm storage get batteries -o. A powered-off node returns last-known inventory, not a live reading — racadm says so itself (STOR0102 … Check if the server has power), which is why five rows are marked stale: calypso5 and calypso7 need confirming once powered.

The seven spare R630s (.21–.27) answer on no protocol — their iDRACs were never put on the management LAN, so their BBUs are unknown. Don't assume a spare is complete.

Never force Write-Back on a node with no BBU

With the battery absent the H730P runs Write-Through, which is safe. Forcing Write-Back to win back write speed is straight data loss on a power cut — and these nodes no longer have a redundant PSU either.

Trap when clearing these: an iDRAC GracefulRestart re-inventories the chassis and drops a genuinely absent PSU (done on .10–.13 on 2026-08-04, health Critical → OK; clearing the SEL alone does not work, and the iDRAC needs 80–160 s to return). It does not clear an absent BBU — that re-asserts within a minute, because it is a live fact and not a stale log entry. No software fix exists short of the controller-side batterywarning=off knob, which needs perccli installed on the host and may not satisfy iDRAC's own health rollup anyway.

VPN and identity — NetBird, Keycloak, edu-ID

The remote-access and single-sign-on stack: the NetBird VPN, the Keycloak broker federating to SWITCH edu-ID, and the legacy MikroTik WireGuard being retired behind them. Architecture: §3bis / §8.

• NetBird follow-ups (deployed 2026-08-03, validated from outside the same day, and both group profiles validated client-side — admin full sweep + student negative test passed, details in the journal; see the service page):
  1. Onboard users: teachers/admins now; students at the autumn intake — coordinate with the September WireGuard purge item below (new students go straight to NetBird).
  2. edu-ID (Phase 2) — client registered and Keycloak deployed on 2026-08-03 (see the journal). Remaining, in order:
     • (done 2026-08-03) identity provider eduid created in realm isc, callback verified against the registered URI, scopes and user-info set;
     • (done 2026-08-04) RRA approved hes-so_isc3_vs_oidc_sso;
     • (done 2026-08-04) first federated login proven end to end, claim set measured, PKCE S256 enabled, eduPersonScopedAffiliation imported — values and the four traps found doing it are in the secretzone (eduid-oidc.md);
     • (done 2026-08-04) RR entry amended (eduPersonOrgUnitDN in, dead swissEduPersonOu out) and the institution groups hes-so / hevs / isc wired to the claims by provisioning/keycloak/groups-and-mappers.sh;
     • eduPersonOrgUnitDN is not being released yet, measured on a login 2026-08-05 — so hevs and isc stay empty and only hes-so is granted. Chase SWITCH if it has not appeared within a few days of the RR amendment, then tighten every consumer from hes-so to isc and confirm a student carries RACA-TICO-ISCO too (the DNs were measured on a staff record only);
     • (done 2026-08-05) the groups claim is emitted — netbird, oauth2-proxy and vaultwarden each carry a Group Membership mapper named groups with full.path=false;
     • (done 2026-08-05) role groups students / teachers / admins exist and are filled from provisioning/keycloak/roster.csv by roster-sync.sh — see role groups. Fill the roster for the teachers/admins now and the students at the autumn intake (item 1 above);
     • (done 2026-08-06) idp-auto-link first-broker-login flow (eduid-autolink) — pre-created roster accounts now link silently on first login; hit by the first student before the intake, validated by the same student — how and why it is safe;
     • verify replay-config.py --jwt-twins against the post-collapse group model — it pairs <name>/<name>-manual from the pre-2026-08-06 identities; the policies now match entitlement jwt groups (appliances-users, mgmt-users) that have no manual twin, so a store rebuild may need it extended before the twins step works;
     • (done 2026-08-06) group-migration cleanup: Keycloak and NetBird groups teachers / admins deleted (the NetBird pair resurrected once — live sessions re-assert cached claims, the trap — and stayed gone after the sessions were killed; glance at the group list once more after 2026-08-07);
     • optional NetBird group slimming (21 → 16 as of 2026-08-06): with a single local account, one break-glass group suffices — drop teachers-manual / students-manual from the five policies and delete them, keeping admins-manual; rack-router tags the routing peer but nothing references it — confirm and drop. hes-so / rack-admins stay: the claim carries them, JWT sync would just re-create them;
     • require MFA for the Keycloak admin account. /admin* no longer depends on the client's address — it is behind the admin gate and the group rack-admins — but that group currently has one member, and the gate authenticates against Keycloak itself: add a second admin before relying on it. Since 2026-08-05 the same group is also the sole Administrator grant on both Proxmox clusters, so it now carries two systems; root@pam remains the break-glass for that half. MFA on it would be requested per-flow via acr-values (PVE's realm takes one), which is why REFEDS MFA was deliberately left off at the edu-ID resource level;
     • (done 2026-08-05) Keycloak added as the Generic OIDC login provider in NetBird — see the service page.
  3. (done 2026-08-05) admission is jwt_allow_groups = admins/teachers/students, i.e. roster membership, and user_approval_required is off; the local login is hidden (localAuthDisabled: true) so ISC SSO is the only option and no longer sits behind "Choose another login method". Role groups reach federated users through the *-manual / jwt pairing — how and why.
  4. Fill roster.csv for the autumn cohort — the mechanism is done and proven with one admin, but the file holds a single line. Needs the class list and the address edu-ID actually releases for students; a wrong domain means "logs in fine, no access" for everyone. Verify with the first student login that they also carry RACA-TICO-ISCO (item above).
  5. Set the peer session expiration deliberately (Settings → Authentication, currently 86400 s = 24 h after the rebuild) before the first federated cohort — it is the whole deprovisioning tail, and it is also why an enrolled device sees no provider change for that long.
  6. Re-verify the whole chain with a second identity. Everything so far was proven with one edu-ID account, which cannot exercise: two people with different roles at once, a user who is in no role group being refused by jwt_allow_groups (only reasoned, never observed), and the idp-auto-link path for a roster account that has never logged in. A colleague's account or the first student is the test.
• Early September 2026 — purge the legacy WireGuard generation info (waiting for the student turnover: current students leave, the autumn intake goes straight to NetBird — wg-gen.sh stays for the admin/emergency peers only; using/connect.md was already cleaned 2026-08-02 and restructured 2026-08-03). Remove what describes the pre-August-2026 full-tunnel generation in docs/secretzone/wireguard.md: the "Legacy batch generation" section and the full-tunnel note above the attribution table — and prune using/connect-wireguard.md (the legacy subpage created 2026-08-03), whose "configs still in circulation" framing stops being true at the turnover. Good moment to also disable/remove the old student peers (wg20–wg52) on the CCR2004, prune the attribution table accordingly, and decide what to do with Marc's OneDrive key archive (…/CALYPSO/calypso (de Marc)/wireguard_conf_gen/, pregenerated keys ≤ 100).
• Hide the NetBird admin console from the Internet — https://vpn.isc-vs.ch/admin is currently reachable from outside (the login fails only because nobody is enrolled yet). It is needed by the owner account alone. Use the admin gate, not a remote_ip matcher: import admin_gate in front of the console path costs one line, works from anywhere and needs no DNS entry (the earlier source-IP recipe required split-horizon DNS and was retired on 2026-08-04). Two things specific to this vhost: NetBird's dashboard, API, embedded IdP, gRPC and relay all share it, so the matcher must cover the console without touching /api/*, /relay* or the gRPC content type — and /oauth2/* is already NetBird's own, so the gate needs a different prefix here (proxy_prefix in the oauth2-proxy config) instead of admin_gate_endpoint.

UID allocation — wire the register up

The register exists since 2026-08-05: provisioning/uid/uid-map.csv, seeded from home ownership on the NAS, issued by uid-alloc.py and made into a home by create-home.sh. Ansible reads it on the rack; what is left is applying it and deciding about the machines that share the same lists.

1. Do not apply the 13 renumberings on Calypso. Ansible would usermod 13 rack accounts (uid-alloc.py --check-ansible lists them), but every Calypso node is rebuilt as a carnaval node once the current cohort finishes, which wipes the /etc/passwd carrying the drift. Reinstalling beats renumbering. Make the rebuild draw its accounts from the register — that is the item that matters.
2. The research machines are frozen — Gregory has two PRs open on dance.yml. Do not touch that inventory or uid_others until they land. Afterwards: the conf/users id: columns survive only because those lists serve both fleets with different numbering, so deleting them means unifying the two namespaces first — renumber and chown one side.
3. Point carnaval-lab-vm.sh at the register rather than stating the NAS. Not urgent: the NAS is the terrain, so it cannot be stale — but it means two code paths for one fact. This one outlives Calypso, so it is the item that matters.
4. Re-run uid-alloc.py --verify whenever the NAS homes change, and redo the burned-number sweep if calypso3–7 come back before they are retired — they were down when the register was seeded, so accounts of theirs could hold numbers the allocator believes are free.

Worth doing before the autumn intake: that is the next batch of numbers to hand out, and the point at which pre-created roster accounts start needing them.

Carnaval (playground cluster)

Built 2026-08-04 — the cluster page holds the current state. What remains:

1. The homes tree is still exposed on the Calypso nodes, where students are in docker and so root-equivalent: they can mount the whole tree themselves, and NFS authorises by numeric UID (root_squash only stops uid 0). Only the node rebuild closes it, and until then 192.168.91.0/24 should be read as "trusted as much as its most privileged student". Found while repairing the mounts.

2. carnaval0's missing PERC battery is part of the fleet-wide sweep and its battery order — see R630 hardware. Nothing carnaval-specific to do beyond fitting it; the node stays Critical and runs Write-Through until then.

3. A dedicated NFS export for shared ISOs/templates. The only FS2500 export is the student homes share, so the cluster has no shared ISO store and each node keeps its own copies on local. Mount whatever is created with vers=4.1 (mounting rules) — the claim that the FS2500 refuses v4 was wrong, and is fixed since 2026-08-05. A new export needs its own squash decision: if PVE is to write guest disks or dumps as root, it needs no_root_squash, which is another reason not to reuse the homes share.

4. Templates still to build: k8s-node, plus the Kubernetes-on-VMs work. ubuntu-2404-base, ubuntu-2404-cuda (all three nodes) and the pools are done.

5. Move the guest-facing half to Ansible. Now that the playbooks live in this repository (provisioning/ansible/), carnaval-lab-vm.sh reads the keys from conf/authorized_keys/ with no cross-repo path — and since 2026-08-05 it reads UIDs from the NAS home ownership rather than a Calypso node, so it no longer depends on a login node at all. community.general.proxmox_kvm does what the script does with proper idempotence instead of hand-built cloud-init YAML. The substrate scripts (install, pool, cluster) can stay as they are. The UID columns in conf/users/*.yml should be deleted, not repaired, as part of the same job — see UID allocation.

6. Grant the cohorts PVEVMUser on their pools once PVE accounts exist — there are none yet, so there are no student accounts yet. The federation half is done since 2026-08-05 — the cluster takes an edu-ID login through Keycloak, but only for pre-created administrators. What is left is the part that avoids hand-creating ~40 accounts a year: --groups-claim groups plus --autocreate 1, then the pool ACLs on the claim-derived students group rather than on individuals.

7. Then decide which guests deserve a selective backup job — there is deliberately none today.

8. (done 2026-08-05) Re-export the MikroTik config — the mirrors were three changes stale, not one: the carnaval DNS records, Dude disabled, and the CRS326 pool still showed the whole 192.168.91.0/24 with dns-server=8.8.8.8 (process).

9. Realign five students' UIDs with their GIDs — decision needed. Found 2026-08-05: on the Calypso nodes kevin.voisin, jeremy.duc, marco.caporizzi and olivier.amacker have a 25 3xx uid against a 10 0xx gid, while their NAS homes are owned uid == gid in the 10 0xx band. Only the uid was renumbered at some point; the gid still carries the original number. Consequence today: each can only read their own ~/nas_home through the group bits of 750 — they cannot write to it.

   louis.heredero is the fifth and the odd one: 25304/10020 on calypso1 only, correct on calypso0 and calypso2. So the nodes disagree with each other as well as with the NAS.

   The low-risk direction is uid ← gid (kevin.voisin 25301 → 10032, jeremy.duc 25303 → 10017, marco.caporizzi 25302 → 10003, olivier.amacker 25305 → 10011, louis.heredero 25304 → 10020 on calypso1): it restores uid == gid and matches the NAS and the student band. It needs /etc/passwd edits on a live login node plus a recursive chown of each local /home tree, so it wants a quiet slot. The permanent fix is the UID register.

   olivier.amacker was the worst case — his (empty) home was owned root:root, so he had no access at all; it was restored to 10011:10011 on 2026-08-05, leaving him level with the other three.

Calypsomaster → second Proxmox node

Live audit of 2026-08-02 (read-only, nothing removed). Conclusion: nothing load-bearing runs on it; convert it to the second PVE node of the admin zone.

What it currently does (all effectively dead)

• Kubernetes control plane (kubeadm v1.35, built ~Jan 2026): only calypso9/calypso10 enrolled and both NotReady; zero user workloads — only kube-system pods. Leftovers of a past course (rallyrobopilot manifests, containers exited 21 months ago).
• MAAS 3.6.2 + PostgreSQL 16: its inventory contains exactly one machine — itself; DHCP off on every VLAN. The "squid on port 8000" mentioned elsewhere is MAAS's internal proxy, not a service.
• Docker registry (192.168.88.248:5000 in calypso-stack.md): container up but started without published ports — unreachable, so already dead in practice (5.2 GB of old course images in /var/lib/registry).
• Zabbix Agent 2 pointing at a Zabbix server on epyc0 that no longer exists.
• No external dependencies (verified from calypso0, the router config and the Ansible repos): workers use router DNS, mount the NAS directly, SLURM controller is calypso0. The only soft one: the incident log documents the NAS-lockout recovery tunnel via calypsomaster — rumba serves that role identically.

Hardware & network facts for the conversion

• Dell R740XD, 2× Xeon Gold 6154 @ 3.0 GHz (72 threads — faster clocks than rumba's 6140), 188 GB RAM, 3.5 TB behind a PERC H730 (SSD vs HDD unknown from the OS — check via iDRAC).
• PCIe slots 1 and 3 (x8) and 4 (x16) are free → up to three Samsung PM1735-style 3.2 TB NVMe add-in cards fit (same model as rumba's system disk). U.2 drives would not fit (SAS backplane).
• No recabling needed: its 10 G DAC lands on the CCR2004 sfp-sfpplus4, same bridge-LAN as rumba (sfp-sfpplus5) — already in the admin segment; the target-architecture zones are logical, not physical. iDRAC stays where it is (it is the reinstall path). The 1 GbE ports (eno3/eno4, down) are a free option for a dedicated corosync link later.
• DHCP quirk: the .248 lease is bound to MAC …:4E:54 but the active port is …:4E:56 — use a static IP in the PVE answer file anyway (SFP+ negotiates too slowly for installer DHCP, same trap as on rumba). Also: the router has a lease "iDRAC Calypso - R740" at 192.168.88.249 while the documented iDRAC is 192.168.90.248 — verify which answers before the remote install.

Plan

1. Gate: confirm with the K8s course owners (artifacts belong to pim/remi) that the cluster is not needed for the autumn 2026 semester.
2. Rescue (total well under 100 MB of unique data):
   • /home/pmudry/rumba-preinstall-backup (7 MB) — the only irreplaceable data on the box (old rumba /home + /etc referenced in the rumba history page) → copy to rumba /hdd/backup or the NAS.
   • Tarball of /etc (etckeeper git repo with daily autocommits = full config history).
   • Skippable: pim's 46-line uncommitted playbook diff (concepts already in ansible-playbooks-conf), registry blobs, the 2024 /backup_calypso-master_conf, etcd backups, the five playbook copies (all preserved in ansible-playbooks-archive on GitHub).
3. Reinstall Proxmox VE over the iDRAC with the proven remote-install process — build the ISO elsewhere (the process notes suggest calypsomaster itself, which won't exist mid-reinstall).
4. Fit the PM1735 NVMe cards (up to 3) — e.g. ZFS mirror for VM storage.
5. Cluster with rumba — two nodes need a QDevice third vote until epyc0/epyc1 join; consider a dedicated corosync link over the unused 1 GbE ports.
6. Cleanup: router DNS/lease rename (calypsomaster, calypso-master, .calypso entries), move the host out of the mgmt group in ansible-playbooks-conf's calypso_rack.yml, purge kubelet remnants on calypso9/calypso10 when those are next reinstalled.
7. Docs: calypso-stack.md (registry section is wrong today regardless), incidents.md (recovery tunnel → rumba), thermal-protection.md (orchestrator role → srv-status), rack.mdx (role/name), tooling/ansible.md (pre-split layout, see the Various item above).

Backups (from the 2026-08-01 ISC Learn DR exercise)

1. DS923↔rack replication (goal decided 2026-08-01) so ISC Learn backups get a second, reachable copy and the restore-from-backup path can actually be tested. Landing spot needs a new decision (2026-08-04): FS2500 Volume 2 was the candidate but is now dedicated to the PBS datastore — remaining options are rumba's hdd pool (~1.6 TB free, already holds the staging mirror) or a share on Volume 1. See backups and the uplink restrictions page for the connectivity constraints (the DS923 is unreachable from Calypso — the replication must run from the school-intranet side or via WireGuard).
2. Fix hannibal's fail2ban sshd jail: it watches port 2002 while sshd listens on 20002 — SSH brute-force protection is currently a no-op (details).
3. Protect the 428 GB of student homes on the FS2500 — calypso_homes is still the only copy in existence: no snapshots, no off-box backup. Deferred (decided 2026-08-02) until the Proxmox datacenter is in a better state, since the natural target is rumba's hdd pool and the storage design there is still moving. Groundwork already established:
   • Snapshot Replication is not installed on the NAS; calypso_homes is a real Btrfs subvolume, so daily + immutable snapshots (DSM 7.2) are available once it is.
   • Rumba's hdd pool has ≈ 1.6 TB free and rsync present — ample for 428 GB.
   • Push, not pull: the student homes are 750 owner-only, and IscAdmin cannot read their contents, so an rsync pulled from rumba would silently back up nothing. Hyper Backup on the NAS runs as root and is the workable direction; a rumba-side ZFS snapshot of the target dataset then gives the tamper resistance that a push target otherwise lacks.

Infrastructure as code — gaps found 2026-08-04

Audit of provisioning/ against the live rack, asking whether it alone could recreate the Proxmox services. It could not: it covered the services well but not the substrate, and two services' real configuration existed only inside a database. Items 1–3 were closed on 2026-08-04 (see the journal); 4–7 below are what remains.

What is already sound, and needs no work: guests 100–104 and 108–110 are each created and configured by their own idempotent deploy script, secrets are generated in-container rather than committed, and srv-web01's live Caddyfile is byte-identical to the repo copy. CT 105 / VM 106 (disposable) and VM 107 (a DR artifact with a tested runbook) are deliberately unscripted.

Standing rule from items 2–3: after any change in the Keycloak admin console or the NetBird dashboard, re-run that service's export script and commit the diff — otherwise the change exists nowhere but a database.

4. srv-pbs has no provisioning directory at all — and it is the guest holding the backups. Its answer file exists only at rumba:/root/pbs-answer.toml, which embeds the root password and so cannot be committed as-is; the datastore, the prune and verify jobs and the token-ACL split are prose on the service page. A dead NAS currently takes the only copy of that configuration with it.
5. The FS2500's own configuration is captured nowhere — shares, NFS exports, users, the VMM guest. The MikroTik /export files under docs/secretzone/mikrotik/ are the model to copy (current as of 2026-08-03; restorable, just not runnable). Public DNS at Infomaniak is likewise prose only.
6. No rebuild runbook for rumba: DRP covers only ISC Learn.
7. READMEs missing for provisioning/web/ and provisioning/status/ — every other directory has one, and status/ is where it matters: the poller's frozen CSV column order and the hand-pushed Telegram token file are both undocumented there.

Email

No machine in ISC³ (the rack) can send email today (re-checked 2026-08-02): alerting is Telegram-only (srv-status), the nightly vzdump job on rumba is set to "mail on failure" but has no working transport (its queue holds days-old deferred messages — direct-to-MX on port 25 times out), and the PDU thermal email alarm (thermal protection) is blocked on the missing relay. Since 2026-08-04 Vaultwarden is affected too: its invitations, "new device" notices and emergency-access mails cannot leave the rack, so every invite has to be conveyed by hand. Note the scope: the mailbox itself is alive and in daily production use outside the rack — Moodle prod on hannibal sends through it (see below).

Decision: instead of waiting for an institutional hevs.ch relay, use the existing Infomaniak mailbox mailer@isc-vs.ch (credentials in secretzone → hannibal page, it is the Moodle sender) as an authenticated smarthost — mail.infomaniak.com:465 implicit TLS (587 STARTTLS fallback). SPF/DKIM need no change since Infomaniak hosts the isc-vs.ch zone.

Egress + auth verified 2026-08-02 — the plan is blocked on vsnet, not on us:

• From rumba, outbound TCP 25, 465 and 587 all time out (to mail.infomaniak.com and to smtp.hevs.ch/mail.hevs.ch alike). The CCR2004 is innocent — its filter chain contains only accept rules — so the SMTP block is upstream at SInf/vsnet, consistent with the egress filtering catalogue.
• No side door: mail.infomaniak.com:443 answers HTTPS (webmail), not SMTP, and Infomaniak's public REST API does mailbox management and newsletter campaigns only — no transactional send from a mailbox.
• The credentials are proven good: Moodle prod on hannibal authenticates to mail.infomaniak.com:465 (SSL, LOGIN) as mailer@isc-vs.ch daily. Its SMTP settings live in the mdl_config DB table (smtphosts/smtpuser), not in config.php — remember that when rotating the password.
• Found while digging (hannibal, not rack): the nightly www-data cron output mailed to mailer@isc-vs.ch goes out via local sendmail, unauthenticated direct-to-MX, and bounces with 550 rejected by DMARC policy — cron errors on hannibal currently report to nobody. Small separate fix: point hannibal's sendmail at the authenticated smarthost too.

Steps, in order:

1. Verify auth + outbound 465/587 from a laptop and from rumba done 2026-08-02, filtered by vsnet (see above). New first step: ask SInf/vsnet for an egress allow — TCP 465 from the Calypso subnet (or just rumba) to Infomaniak's mail servers. Fallback if refused: peer hannibal into the rack WireGuard (its egress is unrestricted, and UDP 51820 into the CCR2004 is open) and relay through it — works, but couples rack alerting to the Moodle prod VPS.
2. Proxmox rumba: pvesh create /cluster/notifications/endpoints/smtp (name infomaniak-mail), repoint the default-matcher at it, check /etc/pve/jobs.cfg for legacy mailto/mailnotification fields and switch the vzdump job to --notification-mode notification-system, then test-send (pvesh create /cluster/notifications/targets/infomaniak-mail/test).
3. PDU AP8681: check the NMC firmware supports authenticated SMTP+TLS (NMC2 needs AOS ≥ 6.8); configure it and enable email on the temperature threshold events. If the firmware is port-25-only: small relay LXC on rumba forwarding to Infomaniak instead — never open plain port 25 outbound.
4. Docs: new secretzone/smtp.md (parameters + consumer list — rotating this password then touches Moodle + PVE + PDU) and infra/services/email.md; update the rumba, thermal-protection and secrets pages; add the sidebar entry.

Follow-up candidates once the smarthost works: Synology DSM notifications, the iDRAC9 fleet (the spare R630s can only ever report via BMC email/SNMP), MikroTik /tool e-mail, Netdata, and the GitLab runbook's SMTP_* placeholders.

Server room

1. Make something nice there. Posters on the walls, screens, stuff.
2. Why is there still a box for a server in the networking lab room ? : Because we need at least one for sending back in case of support / we needed one for network labs to hide the CTF network setup
3. Rename networking lab room and change the remplaçant for Darko as well
4. Why only 10 GB for the fiber
5. Install the bought patch panel outside the rack, not inside (space will be premium; purchase logged in the journal).
6. Choose new R630 and R730 / R740 for RUMBA main. Budget 3 kFr

TODOs from Remi's papers

Calypso :

3. Change CUDA12.2 to 12.6 or 13.0 ?
4. Add the FS switch to the stack, to allow wiring all IDRACs and hosts cables to all Calypso servers
5. Migrate the current 10GB router to 25GB router ? NAS can only use ethernet 10GBx2
6. Set somewhere suitable the UID/GID plan (currently only on users.yml in Ansible playbooks) :
   1. users UIDs : 1000 : ubuntu / 1001 pmudry / 1002 remi / 1003-1100 teachers / 10000-10500 ISC students / 10500-11000 other students
   2. services : 7000-8000 services like prometheus / 8000-9000 custom groups for students / 9000-10000 researchers groups
   3. Maybe either one day integrate into HES Active Directory (no direct control of the groups and user management) or recreate a LDAP server in Rumba ?

Backups :

1. Make a coherent backup architecture : which data has to be kept how long
2. Prod backups from Hannibal to put also on the Synology SSD NAS (Rumba)
3. Teachers laptops backup space to create
4. Rumba services backups
5. Clean Moodle courses to make backups smaller : Did the auto-backups reduction option work ? Did the maximum version kept to (5?) works ?
6. Finish to test the script Rsync with ACL permissions / Extended attributes, then replace the hannibal.sh / marcellus.sh scripts on the Desktop NAS / setup on the SSD Rumba NAS, either directly from source at another moment in the night, or as a copy of the first backup to avoid more I/Os on the prod ?

Post Mortem Followup :

1. Monitoring ISC/Learn : where to install ? Rumba ? / Dashboard to create / emails alerting to setup (see the Email section above for the smarthost plan)
2. Finish the DRP wiki part : administratif:processes:drp
3. Create a PDF + paper DRP to make sure instruction are available even when nothing works : hannibal, wiki, and HES network down
4. Finish learn playbook, as a DRP system to recreate Hannibal from scratch fast in case of

Playbooks :

1. Finish slurm_calypso playbook (isc_compute and slurm_research_TODO are on the CALC@HEI todo page)
2. Finish prometheus playbook
3. Finish learn playbook, as a DRP system to recreate Hannibal from scratch fast in case of
4. Finish the k8s playbook, from the currently "manual script" playbook_a_faire_k8s_calypso.txt