ISC³ — Future global architecture
Author: Pierre-André Mudry · Date: July 2026 · Status: proposal — migration started 2026-07-31 Scope: the entire ISC fleet — programme services, student playground, NOC/SOC supervision. The document gitlab-isc-architecture.md remains the detailed reference for the GitLab service (phase 1). Acronyms are expanded in the glossary.
1. Guiding principle
One single base layer across the whole fleet: Proxmox VE. Hardware is decoupled from usage — repurposing a machine = destroying/deploying VMs, never reinstalling a bare-metal OS. Three logical zones, separated by VLANs, on two physical clusters:
| Machine | State | Since |
|---|---|---|
rumba (Dell R7920) | Proxmox VE 9.2.2, running the services — still standalone | 2026-07-31 |
carnaval0–2 (3× Dell R630) | Proxmox VE 9.2.2, clustered and quorate — the playground zone, ex-calypso8/9/10 | 2026-08-04 |
epyc0 (Gigabyte R282-Z92) | Proxmox already running, to be re-provisioned and clustered | — |
epyc1 (Gigabyte R282-Z92) | still to install | — |
Nodes are installed unattended with the reproducible procedure in
Unattended Proxmox VE install over iDRAC; the answer
files and scripts live in provisioning/pve/ in this repository. The playground cluster is
formed; rumba stays standalone until the prod cluster is, and the two clusters are deliberately
separate (§1). Note that the R630 nodes needed
a different install transport than rumba —
iDRAC8 rather than iDRAC9.
| Zone | What it hosts | Requirement |
|---|---|---|
| Services | GitLab, Moodle, public and private web servers, registry | Stability, backups, HA |
| Playground | Student K8s, GPU/CUDA, Spark/Kafka, lab VMs | Flexibility, disposable, quotas |
| Management & supervision | Proxmox/PBS/PDM, NOC (metrics), SOC (security) | Isolation, visibility |
Physically separating services and playground is the structuring choice: students cannot, even by accident, starve or compromise Moodle or GitLab.
2. Overview
3. Network
Existing, kept as-is: entry zone on the institutional network with inbound 80/443, MikroTik router at the edge, operational WireGuard VPN. The current segmentation is not redone — the table below describes the target logical zones to respect as deployments progress, to be mapped onto the existing segments (renumbering does not matter):
| Logical zone | Content | Key rules |
|---|---|---|
mgmt | Proxmox interfaces, PBS, PDM, IPMI/BMC | Reachable only from admin workstations / admin-group VPN. Never exposed to students |
srv-public | Reverse proxy (the only VM with inbound Internet traffic) | 80/443 from the MikroTik only, towards the proxy |
srv-internal | GitLab, Moodle, web VMs, databases | Reachable only through the reverse proxy + admin SSH |
playground | The whole student cluster | Internet egress OK; no flow initiated towards srv-* or mgmt; outbound 443 to GitLab allowed (runners, git) |
storage-backup | PBS traffic, possible shared storage | Internal only |
This matrix does most of the security work: a lab exercise gone wrong, a student port scan or a compromised container stays confined to the playground zone.
3ter. DNS prerequisite for *.isc.hevs.ch
The whole reverse-proxy scheme above assumes we can create service names at will. Today we cannot,
and this is a hard prerequisite rather than a detail (verified against dns1.vsnet.ch, July 2026):
| Name | What it actually is | Who controls it |
|---|---|---|
isc.hevs.ch | a CNAME to learn.isc-vs.ch → hannibal (Moodle) — not a delegated zone | vsnet / SInf |
calypso.hevs.ch | a plain A record → 153.109.29.7 (the rack) | vsnet / SInf |
isc-vs.ch | a full zone at Infomaniak, incl. learn., test., rumba. | ISC |
A CNAME creates no namespace: git.isc.hevs.ch, test.isc.hevs.ch etc. do not exist and each would
need its own vsnet request. There is no wildcard under either hevs.ch name.
Action: ask vsnet to delegate isc.hevs.ch (two NS records) to Infomaniak's nameservers, so
ISC can create every service name itself and Caddy can obtain certificates without further
coordination. Fallback if refused: a wildcard *.isc.hevs.ch → 153.109.29.7. Note that a zone apex
cannot hold a CNAME, so after delegation today's isc.hevs.ch → Moodle mapping must be recreated as
an A record (or an Infomaniak ALIAS) — coordinate with the Moodle migration.
Until then, new services can be published immediately under isc-vs.ch, which ISC already
controls, and re-pointed to an isc.hevs.ch name later with a one-line proxy change.
3bis. Remote access: VPN with SSO
NetBird runs in production on rumba (VM 109, srv-netbird) at https://vpn.isc-vs.ch, with
built-in local users instead of edu-ID (NetBird ≥ 0.62 embeds its own IdP, so hand-creating
the ~40 yearly accounts needs no Zitadel/Keycloak yet). The edge stayed untouched: the whole
control plane and relay ride the existing 443 through srv-web01; only the optional STUN UDP
port is still closed (P2P falls back to the in-rack relay). The MikroTik WireGuard remains the
emergency admin access, as designed. Details: NetBird VPN.
Phase 2 — edu-ID, in two steps. Checked in the console (August 2026): only Integrations → Identity Provider Sync (SCIM directory pull) needs a commercial licence; Settings → Identity Providers and JWT group sync are in the Community edition (table on the service page). No licence is needed and none should be bought — the sync would only pre-create users who never logged in.
- edu-ID as a second login provider — once the OIDC client is issued (§8), add it as a Generic OIDC provider in the dashboard; local accounts stay as the fallback. Cheap, and it validates the AAI process and the claims actually delivered.
- An OIDC broker (Zitadel or Keycloak, one guest on
rumba) once group mapping matters. It exists for three reasons, all of them edu-ID limitations rather than NetBird ones: it turns HES-SO affiliation/entitlement into agroupsclaim the free JWT sync can consume, it provides the device-authorization grant edu-ID lacks, and it gives the whole fleet (GitLab, Grafana, Moodle) one integration instead of five. NetBird's own embedded Dex cannot host the mapping — the combined container exposes no upstream-connector configuration.
Identified problem: the MikroTik's WireGuard works, but peer management is manual — untenable for ~40 incoming students per year.
Solution: self-hosted NetBird (open source, WireGuard-based), one VM on the prod cluster:
- Self-service enrolment: students install the client, sign in with their edu-ID (OIDC), receive their keys automatically. Zero admin intervention.
- ACLs by identity groups:
students→ playground zone only;staff→ playground + srv-internal;admin→ everything, including mgmt. - Automatic lifecycle: account disabled at the IdP = VPN access dead. Same mechanics as the yearly GitLab archiving.
- Mesh network: rules are expressed by identity, not by IP — more readable than firewall rules.
The MikroTik stays in place: edge router/firewall, and its current WireGuard is kept as an emergency admin access (2-3 static peers for the team) for the day NetBird or the IdP is unavailable. The only change at the edge: open/forward NetBird's UDP port next to the existing 80/443.
Alternatives evaluated: Firezone (equivalent, gateway model), Headscale+Tailscale (good clients, community project), Tailscale SaaS (rejected: control plane at a third party), wg-easy (rejected: does not solve the account lifecycle).
4. Services zone (PROD cluster)
- 3 nodes confirmed: the EPYC server (the one intended for this infrastructure) + 2 reliable machines from the fleet.
rumbais the first of those two and is already on Proxmox VE (2× Xeon Gold 6140, 72 threads, 187 GB RAM, 3.2 TB NVMe) — a good fit for the services zone, and it is where the teacher tooling will be redeployed as VMs/LXC. Proxmox HA enabled for critical VMs (GitLab, Moodle, reverse proxy): if a node dies, they restart elsewhere. - Note on live migration: mixing Xeon (rumba) and EPYC (epyc0/1) in one cluster works, but the VMs must use a common CPU model (e.g.
x86-64-v3) rather thanhost, otherwise migration between vendors is refused. - Single reverse proxy in front (Traefik recommended: automatic Let's Encrypt certificates per service, label-based config). Any new website = one VM/LXC + 3 lines of proxy config + one DNS entry. Application servers never expose anything directly.
- Static pages: pool them in a single lightweight nginx LXC (one vhost per site) rather than one VM per page.
- Moodle: dedicated VM (already migrated behind a reverse proxy on
isc.hevs.ch/learn— this scheme transposes as-is). - Naming convention:
srv-<service>(srv-gitlab, srv-moodle, srv-web01…), Proxmox tags per zone.
5. Playground zone (LAB cluster)
Philosophy: everything is disposable, everything starts from a template.
- Maintained cloud-init templates:
ubuntu-24.04-base,ubuntu-cuda,k8s-node. VM deployed in 30 s, student SSH key injected included. - Kubernetes: VM clusters created/destroyed per semester or per lab (3 light control-plane + N workers). Option: a small permanent "sandbox" cluster + ephemeral clusters per course.
- GPU, two modes:
- Shared LXC (default for CUDA labs): several containers share the host's GPU, each student gets their own environment, SSH/VS Code Remote connection. Good student throughput per GPU.
- VM passthrough (heavy projects, exclusivity needed): the whole GPU to one VM, native performance, slot-based reservation.
- Spark/Kafka: standard VMs, multi-node clusters if the lab requires it.
- Student access: SSH bastion (or campus VPN access) to VLAN 30 only. Scriptable provisioning (Ansible/Terraform on the Proxmox API) — which is itself teaching material for an infrastructure-as-code course.
- Quotas: Proxmox resource pools per course/cohort, so no module can cannibalize the others.
- Cluster membership rule: corosync quorum counts a powered-off node as a lost vote, so only always-on machines join a cluster (prod or lab). Lower-grade or intermittently-powered machines stay standalone, supervised through PDM (visibility + click-to-migrate of stopped guests) — a node belongs in a cluster only if its unplanned death would be an incident rather than a shrug. Practical corollaries: a node must be guest-free to join, and leaving cleanly means a reinstall — so a machine is clustered once its install is final, never while it is still a sandbox.
6. NOC / SOC
Deliberate double objective: operate the fleet and show something spectacular and honest to visitors (open days, delegations) — a natural showcase for the IT Security major.
NOC — supervision (fleet health)
| Component | Role |
|---|---|
| Prometheus | Collection: node_exporter on every host/VM, pve-exporter (Proxmox metrics), GitLab/PostgreSQL/nginx exporters |
| Grafana | Dashboards: fleet view, per-service view, playground view (CI jobs, GPUs in use, active VMs) |
| Netdata | Real-time per machine (1 s granularity), zero configuration, very visual — agent on each Proxmox host, "live" view complementing Grafana and centrepiece of the visitor display. Parent/child deployment: host agents stream to a central Netdata parent (VM on the prod cluster) — a single URL for the whole fleet, lightweight agents on the hosts (retention and queries offloaded to the parent) |
| Uptime Kuma | Simple HTTP checks of public services + status page |
| Alertmanager | Mail/Signal alerts to the technical team |
SOC — security (what is happening on the fleet)
| Component | Role |
|---|---|
| Wazuh | SIEM: agents on all service VMs, host intrusion detection, compliance, SSH attempts, file integrity |
| Loki + Promtail | Log centralization (lightweight, integrates with Grafana) |
| Suricata | Network IDS on a mirror port of the network core — flow visualization and alerts |
| CrowdSec (option) | Collaborative blocking of malicious IPs on the reverse proxy |
Visitor wall display
A dedicated Grafana dashboard in kiosk mode: real-time network flow map (Suricata), blocked attack attempts (there are always some — very compelling), student CI jobs scrolling by, GPU load, active VMs. In rotation: the Netdata view of a busy host (nothing moves as much on screen) and the Wazuh view. Reuse the work already done on the interactive HTML NOC/SOC dashboard as a skin if desired.
Sizing: the whole NOC+SOC fits in 3-4 VMs on the prod cluster (Wazuh is the hungriest: 8 vCPU / 16 GB for a fleet this size).
7. Backups and fleet management
- PBS (dedicated machine, outside the clusters): nightly for the whole prod cluster; selective for the playground (templates and infra VMs yes, disposable lab VMs no).
- PDM (small VM on the prod cluster or the PBS host): single console over both clusters — status, updates, inter-cluster migrations. Stable since 1.0 (Dec 2025; 1.1 as of Aug 2026). It connects over the API with tokens — no corosync coupling — so it also covers standalone nodes, and its migrations (same
remote-migratemechanism as the CLI: guest stopped, disks copied over the network) work between clusters and standalone nodes alike. A PDM outage costs nothing but the dashboard. - Semestrial restore test (see the GitLab document).
8. Identity
SWITCH edu-ID / OIDC as the standard for every service that supports it: GitLab, NetBird (VPN), Grafana (generic OAuth), Moodle (OIDC plugin), Wazuh. One account for everything, automatic cohort onboarding, centralized blocking on departure. Infrastructure access (Proxmox, PBS, SSH) stays on nominative admin accounts + SSH keys, outside the federation.
AAI process: one single edu-ID client, registered for the broker — not one per service. sso.isc-vs.ch (Keycloak) is the only relying party SWITCH ever sees; GitLab, Grafana, Moodle and NetBird are then clients of Keycloak, which needs no AAI paperwork at all. Registered August 2026 as hes-so_isc3_vs_oidc_sso; the full record (fields, secret, frozen values) is in the secretzone, docs/secretzone/eduid-oidc.md. This supersedes the earlier plan of requesting the GitLab and NetBird clients together.
Four edu-ID constraints to design around (from the SWITCH edu-ID OIDC documentation and the live discovery document, checked August 2026):
- Authorise on
eduPersonScopedAffiliation, never on the mere existence of an edu-ID. edu-ID accounts are personal and lifelong — they are not owned by HES-SO and do not disappear when a student leaves. What expires is the affiliation, carried in a scoped value such asstudent@hes-so.ch: the scope ties the person to HES-SO, the prefix selects the group. Getting this wrong grants alumni permanent access. Note the converse gap:student@hes-so.chis every HES-SO student, not an ISC one — the affiliation must be paired withswissEduPersonOu(requested as desired) or with a locally held ISC list, or a business student in Sierre reaches the datacenter. - No ISC-shaped groups in the token: edu-ID emits affiliations, identifiers and entitlements — nothing named
students/staff. The broker does that translation, and it reads the claims from the UserInfo endpoint (edu-ID does not put them in the ID token by default). Note that the extended attribute model is not the way to get richer data: it is a SCIM backchannel restricted to cases SWITCH reviews individually, and the core attributes already cover us. - Authorization-code flow only (
code, PKCES256, plusrefresh_token) — no device-authorization endpoint exists. Interactive SSO on a headless machine will not work; those peers keep using setup keys. - Registration through the AAI Resource Registry, no dynamic client registration, and the entry activates only after the home-organisation approver signs off — count the lead time.
9. Proposed roadmap
Each phase is useful on its own and does not require the next one.