Keycloak (SSO)

Self-hosted Keycloak (26.7.0, Apache-2.0, deployed 2026-08-03) — the identity broker of ISC³ at https://sso.isc-vs.ch. It is the single relying party that SWITCH edu-ID ever sees; every ISC service then authenticates against Keycloak instead of against the federation. Planned in the target architecture §3bis / §8.

Partly in service (August 2026)

The edu-ID client is approved and active (RRA, 2026-08-04), federated login is proven end to end (PKCE S256 on), and the institution groups hes-so / hevs / isc are wired to the claims — though only hes-so is granted so far, the org-unit claim not being released yet. Consumers: Vaultwarden since 2026-08-04, the NetBird VPN and Proxmox (rumba + carnaval) since 2026-08-05 — NetBird reading the role groups out of a groups claim, Proxmox admitting only pre-created users. NetBird admission is roster membership; since 2026-08-06 a non-rostered login is refused by this realm with an explicit message, and pre-created roster accounts link silently on first edu-ID login (open items).

At a glance

Guest	srv-keycloak — unprivileged LXC, CT 110 on rumba, 192.168.88.160, 2 cores / 4 GB / 12 GB local-lvm
Public URL	https://sso.isc-vs.ch — /realms/* public, admin console behind the admin gate (group rack-admins)
Realm	isc (the master realm is for administration only)
Database	PostgreSQL 17 in the same container
Managed from	provisioning/keycloak/ (idempotent deploy + bootstrap scripts)
Credentials	Bootstrap admin and database password in the secretzone (rumba page)

Why a broker at all

edu-ID could be pointed at each service directly, so the broker has to earn its keep. It does, on three counts — all of them limitations of edu-ID rather than of the consumers:

• It manufactures the groups. edu-ID emits eduPersonScopedAffiliation (student@hes-so.ch), never anything shaped like students / staff. Keycloak turns one into the other and hands downstream services a plain groups claim, which NetBird's free JWT group sync then consumes.
• One AAI registration for the whole fleet. GitLab, Grafana and Moodle become clients of Keycloak — no further Resource Registry paperwork, no second approval round.
• It supplies what edu-ID lacks, notably the device-authorization grant, and it insulates the fleet from a future change of federation.

Architecture

Plain HTTP on the LAN, TLS on the existing proxy — the same shape as every other guest here:

• TLS terminates on srv-web01 (sso.isc-vs.ch vhost in provisioning/web/Caddyfile), so no new inbound port. Keycloak runs with proxy-headers=xforwarded and hostname=https://sso.isc-vs.ch: without the first, every redirect it builds points at http://192.168.88.160 and the edu-ID round trip fails in a way that looks like an edu-ID fault.

• The admin console requires a login, not an address. /admin* and /realms/master* go through the admin gate on srv-web01 — Caddy forward_auth to oauth2-proxy, which requires membership of the realm group rack-admins. Works from anywhere, no VPN needed. /realms/* otherwise stays public: edu-ID's callback and every OIDC client depend on it.

• Break-glass, because the gate authenticates against this very Keycloak. A broken Keycloak would otherwise lock its own console. From 192.168.88.0/24 or the NetBird overlay the admin paths still pass without the gate: connect the VPN and point the name at the proxy with one /etc/hosts line — full recipe and the reasoning in break-glass.

  Why there is no split-horizon DNS any more (2026-08-04)

  Until the gate existed, the admin paths were protected by source IP, which meant VPN clients had to receive an internal answer for the public name — a CCR static record plus a NetBird nameserver group distributing it. That layer broke in a new way each time: RouterOS served the cached public answer until /ip dns cache flush; macOS sends only A queries to a per-domain VPN resolver, so a public CNAME riding along in the AAAA answers poisoned the cache and every lookup followed it back outside. Authenticating the path instead deleted that whole layer: the NetBird sso-split-horizon nameserver group and the vault.isc-vs.ch record are gone, and no client receives an internal answer for a public name any more.

  One static record survives, for an unrelated reason: sso.isc-vs.ch → 192.168.88.150 on the CCR2004. The rack's own services resolve the issuer themselves — oauth2-proxy and Vaultwarden both do OIDC discovery against it — and the CCR does not hairpin (verified 2026-08-04: forcing the public address from inside a container times out). Without that record their discovery fails. It is not distributed to any VPN client, so it cannot reproduce the failures above.

  If you ever reintroduce a client-facing split-horizon name: publish it as a plain A record, never a CNAME, flush the RouterOS cache, and do not "verify" with curl --resolve — that forces the internal path and skips the step that breaks.

• Health and metrics live on port 9000, deliberately not proxied.

• Native install, no container runtime: OpenJDK 21 + the Keycloak distribution under /opt/keycloak, PostgreSQL alongside. Docker inside an unprivileged LXC buys nothing here and costs storage-driver trouble.

Values that are frozen

The AAI registration pins two names. Renaming either means amending the Resource Registry entry and waiting for approval again:

Thing	Value	Why it is stuck
Realm	isc	appears in the registered callback
Identity-provider alias	eduid	idem
Public origin	https://sso.isc-vs.ch	it is also the sector identifier — changing it regenerates every pairwise sub, i.e. every user returns as a stranger

The registered callback is https://sso.isc-vs.ch/realms/isc/broker/eduid/endpoint; the registered logout URI is …/broker/eduid/endpoint/logout_response (added 2026-08-03) — both hard-code those two names. When the identity provider is created in the admin console, Keycloak displays the callback it will actually use — compare it character for character before anything else. Full registration record: docs/secretzone/eduid-oidc.md.

Configuring the edu-ID identity provider

Admin console → realm isc → Identity providers → OpenID Connect v1.0, then:

Field	Value
Alias	eduid (exactly)
Discovery endpoint	https://login.eduid.ch/.well-known/openid-configuration
Client ID	hes-so_isc3_vs_oidc_sso
Client secret	from the secretzone
Client authentication	Client secret sent as basic auth
Scopes	openid https://eduid.ch/scope/userinfo.read
Disable user info	off
Sync mode	Force — see the VPN gate & linking notes; asserted by groups-and-mappers.sh

Three of those are places where Keycloak's defaults are wrong for edu-ID, and all three fail quietly:

• Scopes default to openid profile email. SWITCH discourages those and expects userinfo.read; leave the default and the claims never arrive.
• "Disable user info" must stay off. edu-ID delivers claims from the UserInfo endpoint, not in the ID token. Switch it on and every attribute mapper finds nothing while login itself keeps working — the most confusing failure in this whole stack.
• Client authentication must match the registry (client_secret_basic). A mismatch throws invalid_client at the token exchange, after the user has successfully authenticated at edu-ID.

The provider was created 2026-08-03 with exactly these values. The scope and user-info settings are not in the Advanced settings block; they live in the OIDC-specific section, which is easy to walk past (or set them unambiguously with kcadm.sh update identity-provider/instances/eduid).

PKCE is on (S256, enabled 2026-08-04 once a login was proven to work end to end, so that a failure had one possible cause instead of two — it was deliberately off before that).

An attribute importer silently does nothing until the attribute is declared

Realm isc leaves unmanagedAttributePolicy unset, which since Keycloak 24 means disabled: only username, email, firstName and lastName are declared in Realm settings → User profile. An Attribute Importer mapper writing anything else is a no-op with no error and no log line — two logins imported nothing before this was found. Declare each imported claim first; eduPersonScopedAffiliation is declared multivalued, with view/edit restricted to admin because a self-editable affiliation is a self-service privilege grant.

Claims are also invisible until mapped — Identity provider links shows only the pairwise sub. To see the raw UserInfo JSON, drop Environment=KC_LOG_LEVEL=info,org.keycloak.social.user_profile_dump:debug into keycloak.service, restart, log in, then remove it.

Logout stops here and is not propagated to edu-ID (since 2026-08-05): the provider's Logout URL is deliberately empty, asserted by groups-and-mappers.sh. With edu-ID's end-session endpoint in there, a logout sent the browser to login.eduid.ch and left it on a "La connexion a échoué" page — edu-ID never honours post_logout_redirect_uri, so there was no way back and the last thing a user saw was an error in French. Empty means Keycloak ends its own session and returns to the client's post-logout URI.

The consequence is that the edu-ID browser session survives, so signing back in is immediate. That is correct — the session belongs to edu-ID, not to us — but on a shared machine "log out" is not the same as "leave no trace". If the endpoint is ever restored, the "Backchannel logout" toggle must stay off (its default): despite the name it makes Keycloak call that endpoint server-to-server, where edu-ID has no browser session to end. There is no single logout in the other direction either — Keycloak cannot receive an edu-ID-initiated logout (verified 2026-08-03). Details in docs/secretzone/eduid-oidc.md.

Groups, and what fills them

Three flat institution groups in realm isc, each filled from edu-ID claims by an Advanced Claim to Group mapper. All of it is in provisioning/keycloak/groups-and-mappers.sh, which is idempotent — re-run it after any claim change rather than clicking through the console.

Group	Granted when	Scope of a service using it
hes-so	affiliation matches (staff|faculty|employee|student)@hes-so.ch	anyone at HES-SO
hevs	that and eduPersonOrgUnitDN contains RORG-HEVS	HES-SO Valais
isc	that and eduPersonOrgUnitDN contains RACA-TICO-ISCO	ISC only

Flat rather than nested, because services choose their own breadth and downstream consumers match on the name (NetBird's free JWT group sync does exactly that). A person receives every group that applies — an ISC teacher is in all three — so there is no inheritance to reason about and removing one group never cascades.

Five decisions in there that should not be undone by accident:

• member@hes-so.ch is excluded. edu-ID emits it alongside the real value; it means "has any active affiliation" and covers staff and students alike, so it is broader than the student@hes-so.ch trap it sits next to. alum and affiliate are excluded too — accounts are lifelong, so alumni would otherwise keep access forever.
• A live affiliation is required for all three groups, not just hes-so. The org-unit DN alone would let a stale HES-SO directory entry keep granting access.
• syncMode=FORCE re-evaluates on every login, so losing a claim upstream also removes the group. Revocation follows HES-SO with no action here.
• The regexes are full-match, hence the .* around the DN fragments. And a DN contains commas, so never split the claim's array values on commas.
• RORG-HEVS deliberately also matches RORG-HEVS-VSEI — both are HES-SO Valais. RACA-… under ou=acad is the academic attachment, which is what makes RACA-TICO-ISCO mean ISC.

warning

hevs and isc are empty until edu-ID releases the org unit (August 2026)

Measured 2026-08-05: a federated login imports eduPersonScopedAffiliation (staff@hes-so.ch, member@hes-so.ch) but eduPersonOrgUnitDN does not arrive at all — the RR amendment adding it was made 2026-08-04 and SWITCH is not releasing it yet. The two mappers are correct and have nothing to match, so only hes-so is granted. Consumers gate on hes-so for now and tighten to isc once the claim appears; nothing else is wrong.

Role groups: the ISC list

students / staff are not derived from claims: no HES-SO claim distinguishes a person's institute reliably enough, and the yearly intake should be an explicit act. They grant admission only — all reach is entitlement groups. (Until 2026-08-06 there were four identity groups; teachers / admins carried reach of their own and were collapsed into the appliances-users / mgmt-users entitlements.) They come from a locally held list, provisioning/keycloak/roster.csv (email,groups[,nasname], one line per person, groups ;-separated), applied by the idempotent provisioning/keycloak/roster-sync.sh — so an intake is one reviewable diff rather than an afternoon in the admin console. The optional third column is the NAS home a person owns; it is unused until UID allocation exists, but the roster is the right place to record it once.

rack-admins is listed in the same file but handled differently: the sync only ever adds to it, never removes. It gates the admin console in front of this very Keycloak, so a truncated or mis-edited roster must not be able to lock everyone out of the tool needed to repair it — taking someone out stays a deliberate console action.

Resource groups, not cohort groups

Alongside the identity groups, the roster carries one group per resource — calypso-users, carnaval-users, appliances-users, mgmt-users — rather than one per cohort (students-2024, or a "3ème année" group). Both would express "the third years may reach Carnaval"; the difference is where the yearly churn lands.

Per resource, the NetBird policies are written once and never repointed: a new student joins a group that already exists, and the move-up is an edit to roster.csv, i.e. a reviewable git diff. Per cohort, the membership never changes but every policy has to be repointed at a new group name each August — in NetBird's UI, outside version control, and silently if forgotten.

The NetBird side makes this sharper still. A group only comes into existence there when a token first carries it, so a cohort group for an incoming year cannot be referenced by a policy until the new students have logged in at least once. Resource names are stable and already claimed.

The volée stays visible in roster.csv as a section comment — readable as a class list, with no mechanism behind it.

This trades one kind of work for another

Each August, the rising cohort gains a resource group: roughly one edited line per student, in the same diff that already handles arrivals and departures. That is the deliberate cost. It stops being the right trade if access ever needs to be per-course or per-project, where the group count multiplies and the file becomes an ACL matrix — at that point a volée → resources lookup table beats enumerating it per person. ::: NetBird is the first consumer; it reads the group names out of the groups claim, and since 2026-08-05 that membership is what admits someone to the VPN at all (jwt_allow_groups), not just what they may reach.

One consequence for whoever edits these names: a consumer may not be able to create a group of the same name itself. NetBird's sync ignores a claim naming a group it did not create, which is why its hand-made groups are called *-manual — so renaming a group here means renaming it there too.

What the script deliberately does not do, each for a reason worth keeping:

• It never touches hes-so / hevs / isc — those are claim-derived, see above.
• It never touches rack-admins, which gates the admin console in front of this very Keycloak: a roster edit must not be able to lock you out of the tool you would need to fix it.
• It never deletes an account. A removed line drops the group membership only; deleting the user would drop the federated identity link, i.e. the record of who this person was at edu-ID.

Two consequences of there being no SMTP on this host (smtpServer is empty, and the rack cannot send mail anyway — email egress):

• Pre-created accounts are made with emailVerified=true, which is a true statement given trustEmail on the eduid provider, not a shortcut.
• Keycloak's default way of linking a brokered login to a pre-existing account is to email the person a verification, with re-authentication by password as the only alternative — and a roster account has no password (a student hit exactly that on 2026-08-06: an unsatisfiable username/password form after a good edu-ID login). Since that day the eduid provider uses its own first-broker flow eduid-autolink (provisioning/keycloak/first-broker-autolink.sh): Create User If Unique, else Automatically set existing user — silent linking by email. Safe here because edu-ID verifies its emails, trustEmail is on, and the matched accounts are our own roster shells; do not copy it next to an IdP whose emails are unverified, that would let anyone claim any account. Auto-linking alone left the pre-created account's names empty: provider-level sync (unset = IMPORT) only applies brokered data when Keycloak creates the account, and the VERIFY_PROFILE required action then demanded first/last name from every new student — a form whose data is worthless, since edu-ID owns it (one student typed "Halllll" to prove the point). Hence, since 2026-08-06: the provider runs syncMode=FORCE (edu-ID re-applies names/email at every login, including the linking one — a fresh student's first login populated everything silently, and the "Halllll" self-corrected) and VERIFY_PROFILE is disabled. Both asserted by groups-and-mappers.sh.

Matching is on the email address, so someone whose edu-ID primary address is not their HES-SO one shows up as "logs in fine, no access" — add their real edu-ID address to the roster.

The VPN gate: refusing with a message

Since 2026-08-06 a non-rostered edu-ID login to NetBird is refused here, not by NetBird: being refused by NetBird's jwt_allow_groups meant a bare error page after a successful login, with no hint that enrollment was the issue. provisioning/keycloak/vpn-access-gate.sh (idempotent) builds the whole thing:

• realm role vpn-access, granted to the identity groups (students / staff) — membership keeps flowing from roster.csv, there is no second list;
• marker client scope vpn-gate, default on the netbird client only, excluded from the token scope — it exists so a flow condition can recognise a NetBird login;
• top-level flow vpn-access-gate, bound as the post-broker flow of the eduid provider: a conditional subflow (client scope vpn-gate ∧ user role vpn-access negated) ending in Deny access with message key vpnNotEnrolled — wording (EN/FR) in the login theme's i18n.ts, resolved server-side, so deploy the theme before the flow or denied users see the raw key — followed by a top-level REQUIRED Allow access.

danger

The trailing Allow access is load-bearing

A flow whose executions all end up disabled — every condition false, i.e. every normal login — does not count as success: the processor throws, and every edu-ID login on every client fails with invalid_user_credentials (userId=null in the events, "Invalid username or password" on screen). The first binding lacked it and broke all SSO logins on 2026-08-06 until unbound; known behaviour, keycloak#14591. Any future post-broker flow needs the same explicit success state.

Post-broker rather than a browser-flow override on the client, because a fresh brokered login never returns to the browser flow — an override there is skipped in exactly the case that matters. The post-broker flow runs on every edu-ID login; the client-scope condition keeps the deny NetBird-only, so Vaultwarden, oauth2-proxy and Proxmox are untouched. Accepted limit: a login riding an existing Keycloak SSO cookie skips the broker (and thus the gate) and falls through to NetBird's own refusal.

Where to actually test a login

A realm has no standalone login page — Keycloak only shows one when a client asks it to authenticate someone. Until NetBird is pointed here, the built-in account console is the harness:

https://sso.isc-vs.ch/realms/isc/account/

It bounces to the realm login page ("Sign in to ISC"), which carries the social-eduid button. /realms/* is public by design — edu-ID's callback and the OIDC clients need it — so this works from anywhere, VPN or not; only /admin* and /realms/master* are restricted.

Note that the account console requires PKCE: a hand-built authorization URL without code_challenge/code_challenge_method bounces back with Missing parameter: code_challenge_method. Browsers send them automatically — don't read that error as a broken realm.

Operations

• Deploy / upgrade: provisioning/keycloak/deploy-keycloak.sh, idempotent; KEYCLOAK_VERSION=… ./deploy-keycloak.sh upgrades in place, keeping the database, the rendered config and any provider jars.
• Login theme: ISC-branded (petal colors, logo, dark mode; deployed 2026-08-04), built with Keycloakify from provisioning/keycloak/theme/; deploy-theme.sh builds the jar, pushes it to providers/, runs kc.sh build + restart (the unit is --optimized, a dropped-in jar alone does nothing) and sets loginTheme=isc on the realm.
• Service: systemctl status keycloak in the container (pct exec 110 -- … from rumba, or ssh root@srv-keycloak over the VPN). Readiness: curl http://192.168.88.160:9000/health/ready.
• The JVM heap is pinned (-Xmx1536m) in the unit on purpose: a JVM in an LXC can read the host's memory through lxcfs, size its heap against rumba's 187 GB and get OOM-killed much later, with no obvious cause.
• State: PostgreSQL inside the container, covered by rumba's nightly vzdump — that job is configured all 1, so a new guest is included the same night without any action (verified 2026-08-03).
• Realm snapshot: see below — re-run it after any admin-console change.
• Losing this host means losing every federated login for whatever is wired to it. Until GitLab and the rest arrive that is only a future concern, but it is the reason the MikroTik WireGuard stays as the emergency admin path.

What survives losing this container

Asked deliberately after the NetBird store loss, whose cause was a single value existing in exactly one place. Keycloak has no equivalent. Nothing in realm isc is irreplaceable, because every part of it is either in git or re-derived on the next login:

Thing	How it comes back
Realm, groups, claim mappers	bootstrap.sh + groups-and-mappers.sh, both idempotent
Role-group membership	roster.csv + roster-sync.sh
The custom auth flows (VPN gate, auto-link)	vpn-access-gate.sh + first-broker-autolink.sh, both idempotent
Users and their edu-ID link	recreated at next login — the pairwise sub derives from the sector identifier, i.e. the frozen public origin, so the same person returns as the same subject
Imported claim attributes	re-imported at every login (syncMode=FORCE)
Client secrets	regenerated by each consumer's keycloak-client.sh, then re-pasted into that consumer
The eduid provider instance	the one piece that is console work — hence the snapshot below and the click-by-click values on this page

What a rebuild does cost: every active session and offline token dies with the old signing keys, and each consumer must be given its new client secret. Both are visible failures that get fixed in minutes — not the silent, unrecoverable kind. The realm database is nevertheless in rumba's nightly vzdump, which turns those minutes into none.

Realm snapshot

provisioning/keycloak/export-realm.sh writes realm-isc.json — a partial export, key-sorted, secrets stripped. Run it after any admin-console change and commit the diff.

bootstrap.sh creates the realm empty and groups-and-mappers.sh covers the groups and the claim mappers, but the identity-provider instance itself — the discovery import, the client id and secret, and the three settings whose defaults are wrong for edu-ID — is still console work living only in this container's database. The snapshot is what makes such a change reviewable instead of invisible.

It is not a restore path: partial export omits users and masks secrets, so recovery stays a vzdump of CT 110 or a re-click from this page. Current content: identity provider eduid, 5 mappers, groups hes-so / hevs / isc / rack-admins / students / staff and the four entitlement groups, and the clients vaultwarden, oauth2-proxy, netbird and proxmox with their groups mappers. The roster membership is not in there — users are omitted from a partial export, so provisioning/keycloak/roster.csv is the only record of who is in which role group.