Keycloak (SSO)

Self-hosted Keycloak (26.7.0, Apache-2.0, deployed 2026-08-03) — the identity broker of ISC³ at https://sso.isc-vs.ch. It is the single relying party that SWITCH edu-ID ever sees; every ISC service then authenticates against Keycloak instead of against the federation. Planned in the target architecture §7.

Partly in service (August 2026)

The edu-ID client is approved and active (RRA, 2026-08-04), federated login is proven end to end (PKCE S256 on), and the institution groups hes-so / hevs / isc are wired to the claims — though only hes-so is granted so far, the org-unit claim not being released yet. Consumers: the NetBird VPN and Proxmox (rumba + carnaval) since 2026-08-05 (Vaultwarden was the first, 2026-08-04, retired 2026-08-16) — NetBird reading the role groups out of a groups claim, Proxmox admitting only pre-created users. NetBird admission is roster membership; since 2026-08-06 a non-rostered login is refused by this realm with an explicit message — realm-wide since 2026-09-06 — and pre-created roster accounts link silently on first edu-ID login (open items).

At a glance

Guest	srv-keycloak — unprivileged LXC, CT 110 on rumba, 192.168.88.160, 2 cores / 4 GB / 12 GB local-lvm
Public URL	https://sso.isc-vs.ch — /realms/* public, admin console behind the admin gate (group role-rack-admins); a second instance, the user gate, guards the published user tools on isc/hevs
Realm	isc (the master realm is for administration only)
Database	PostgreSQL 17 in the same container
Managed from	provisioning/keycloak/ (idempotent deploy + bootstrap scripts)
Credentials	Bootstrap admin and database password in the secretzone (rumba page)

Why a broker at all

edu-ID could be pointed at each service directly. Three things the broker adds, all of them limitations of edu-ID rather than of the consumers:

• It manufactures the groups. edu-ID emits eduPersonScopedAffiliation (student@hes-so.ch), never anything shaped like students / staff. Keycloak turns one into the other and hands downstream services a plain groups claim, which NetBird's free JWT group sync then consumes.
• One AAI registration for the whole fleet. GitLab, Grafana and Moodle become clients of Keycloak — no further Resource Registry paperwork, no second approval round.
• It supplies what edu-ID lacks, notably the device-authorization grant, and it insulates the fleet from a future change of federation.

Architecture

Plain HTTP on the LAN, TLS on the existing proxy — the same shape as every other guest here:

• TLS terminates on srv-web01 (sso.isc-vs.ch vhost in provisioning/web/Caddyfile), so no new inbound port. Keycloak runs with proxy-headers=xforwarded and hostname=https://sso.isc-vs.ch: without the first, every redirect it builds points at http://192.168.88.160 and the edu-ID round trip fails in a way that looks like an edu-ID fault.

• The admin console requires a login, not an address. /admin* and /realms/master* go through the admin gate on srv-web01 — Caddy forward_auth to oauth2-proxy, which requires membership of the realm group role-rack-admins. Works from anywhere, no VPN needed. /realms/* otherwise stays public: edu-ID's callback and every OIDC client depend on it.

• Break-glass, because the gate authenticates against this very Keycloak. A broken Keycloak would otherwise lock its own console. From the NetBird overlay the admin paths still pass without the gate (the LAN /24 was dropped from the branch 2026-08-18 — src-NATed legacy WireGuard made every legacy peer a LAN address): connect NetBird and point the name at the proxy with one /etc/hosts line — full recipe and the reasoning in break-glass.

  Why there is no split-horizon DNS any more (2026-08-04)

  Until the gate existed, the admin paths were protected by source IP, which meant VPN clients had to receive an internal answer for the public name — a CCR static record plus a NetBird nameserver group distributing it. That layer broke in a new way each time (a stale RouterOS cache; macOS querying only A against a per-domain VPN resolver, so a public CNAME poisoned the answers). Authenticating the path instead deleted the whole layer: no client receives an internal answer for a public name any more.

  One static record survives, for an unrelated reason: sso.isc-vs.ch → 192.168.88.150 on the CCR2004. The rack's own services resolve the issuer themselves — oauth2-proxy does OIDC discovery against it — and the CCR does not hairpin (verified 2026-08-04: forcing the public address from inside a container times out). Without that record their discovery fails. It is not distributed to any VPN client, so it cannot reproduce the failures above.

  If you ever reintroduce a client-facing split-horizon name: publish it as a plain A record, never a CNAME, flush the RouterOS cache, and do not "verify" with curl --resolve — that forces the internal path and skips the step that breaks.

• Health and metrics live on port 9000, deliberately not proxied.

• Native install, no container runtime: OpenJDK 21 + the Keycloak distribution under /opt/keycloak, PostgreSQL alongside. Docker inside an unprivileged LXC buys nothing here and costs storage-driver trouble.

Values that are frozen

The AAI registration pins two names. Renaming either means amending the Resource Registry entry and waiting for approval again:

Thing	Value	Why it is stuck
Realm	isc	appears in the registered callback
Identity-provider alias	eduid	idem
Public origin	https://sso.isc-vs.ch	it is also the sector identifier — changing it regenerates every pairwise sub, i.e. every user returns as a stranger

The registered callback is https://sso.isc-vs.ch/realms/isc/broker/eduid/endpoint; the registered logout URI is …/broker/eduid/endpoint/logout_response (added 2026-08-03) — both hard-code those two names. When the identity provider is created in the admin console, Keycloak displays the callback it will use — compare it character for character. Full registration record: secretzone/eduid-oidc.md.

Configuring the edu-ID identity provider

Admin console → realm isc → Identity providers → OpenID Connect v1.0, then:

Field	Value
Alias	eduid (exactly)
Discovery endpoint	https://login.eduid.ch/.well-known/openid-configuration
Client ID	hes-so_isc3_vs_oidc_sso
Client secret	from the secretzone
Client authentication	Client secret sent as basic auth
Scopes	openid https://eduid.ch/scope/userinfo.read
Disable user info	off
Sync mode	Force — see the VPN gate & linking notes; asserted by groups-and-mappers.sh

Three of those are places where Keycloak's defaults are wrong for edu-ID, and all three fail quietly:

• Scopes default to openid profile email. SWITCH discourages those and expects userinfo.read; leave the default and the claims never arrive.
• "Disable user info" must stay off. edu-ID delivers claims from the UserInfo endpoint, not in the ID token. Switch it on and every attribute mapper finds nothing while login itself keeps working.
• Client authentication must match the registry (client_secret_basic). A mismatch throws invalid_client at the token exchange, after the user has successfully authenticated at edu-ID.

The scope and user-info settings are not in the Advanced settings block; they live in the OIDC-specific section, which is easy to walk past (or set them unambiguously with kcadm.sh update identity-provider/instances/eduid). PKCE is on (S256).

An attribute importer silently does nothing until the attribute is declared

Realm isc leaves unmanagedAttributePolicy unset, which since Keycloak 24 means disabled: only username, email, firstName and lastName are declared in Realm settings → User profile. An Attribute Importer mapper writing anything else is a no-op with no error and no log line — two logins imported nothing before this was found. Declare each imported claim first; eduPersonScopedAffiliation is declared multivalued, with view/edit restricted to admin because a self-editable affiliation is a self-service privilege grant.

Claims are also invisible until mapped — Identity provider links shows only the pairwise sub. To see the raw UserInfo JSON, drop Environment=KC_LOG_LEVEL=info,org.keycloak.social.user_profile_dump:debug into keycloak.service, restart, log in, then remove it.

Logout stops here and is not propagated to edu-ID (since 2026-08-05): the provider's Logout URL is deliberately empty, asserted by groups-and-mappers.sh. With edu-ID's end-session endpoint in there, a logout sent the browser to login.eduid.ch and left it on a "La connexion a échoué" page — edu-ID never honours post_logout_redirect_uri, so there was no way back and the last thing a user saw was an error in French. Empty means Keycloak ends its own session and returns to the client's post-logout URI.

The consequence is that the edu-ID browser session survives, so signing back in is immediate. That is correct — the session belongs to edu-ID, not to us — but on a shared machine "log out" is not the same as "leave no trace". If the endpoint is ever restored, the "Backchannel logout" toggle must stay off (its default): despite the name it makes Keycloak call that endpoint server-to-server, where edu-ID has no browser session to end. There is no single logout in the other direction either — Keycloak cannot receive an edu-ID-initiated logout (verified 2026-08-03). Details in secretzone/eduid-oidc.md.

Groups, and what fills them

Three flat institution groups in realm isc, each filled from edu-ID claims by an Advanced Claim to Group mapper. All of it is in provisioning/keycloak/groups-and-mappers.sh, which is idempotent — re-run it after any claim change rather than clicking through the console.

Group	Granted when	Scope of a service using it
hes-so	affiliation matches (staff|faculty|employee|student)@hes-so.ch	anyone at HES-SO
hevs	that and eduPersonOrgUnitDN contains RORG-HEVS	HES-SO Valais
isc	that and eduPersonOrgUnitDN contains RACA-TICO-ISCO	ISC only

Flat rather than nested, because services choose their own breadth and downstream consumers match on the name (NetBird's free JWT group sync does exactly that). A person receives every group that applies — an ISC teacher is in all three — so there is no inheritance to reason about and removing one group never cascades.

Five decisions in there that should not be undone by accident:

• member@hes-so.ch is excluded. edu-ID emits it alongside the real value; it means "has any active affiliation" and covers staff and students alike, so it is broader than the student@hes-so.ch trap it sits next to. alum and affiliate are excluded too — accounts are lifelong, so alumni would otherwise keep access forever.
• A live affiliation is required for all three groups, not just hes-so. The org-unit DN alone would let a stale HES-SO directory entry keep granting access.
• syncMode=FORCE re-evaluates on every login, so losing a claim upstream also removes the group. Revocation follows HES-SO with no action here.
• The regexes are full-match, hence the .* around the DN fragments. And a DN contains commas, so never split the claim's array values on commas.
• RORG-HEVS deliberately also matches RORG-HEVS-VSEI — both are HES-SO Valais. RACA-… under ou=acad is the academic attachment, which is what makes RACA-TICO-ISCO mean ISC.

hevs and isc stood empty from 2026-08-05 until the RR amendment of 2026-08-04 took effect: a federated login imported eduPersonScopedAffiliation but not eduPersonOrgUnitDN, so only hes-so could be granted and consumers had to gate on it. The claim arrives as of 2026-08-21 and both groups fill on login, which is what lets the user gate ask for isc or hevs rather than for all of HES-SO.

What the realm records

Two stores, both in the realm's own PostgreSQL database, both asserted by provisioning/keycloak/realm-audit.sh (--show prints the current values without changing anything). Read them in the admin console under Realm settings → Events, or with kcadm.sh get events -r isc and get admin-events -r isc.

Store	Holds	Retention
Login events	every event type — LOGIN, LOGIN_ERROR, LOGOUT, the identity-provider ones, credential changes	90 days
Admin events	who changed what in the realm, with the representation of the change	none — they accumulate

Admin events were off until 2026-08-28, so a change to a group membership — the thing that grants access here — left no trace at all. Login events were on but kept 7 days, shorter than the gap between something happening and somebody looking.

Every event type stays enabled rather than a curated list: with a few dozen users the store costs nothing, and curating only creates a way to have dropped the event that mattered.

Admin events have no expiration on 26.7

adminEventsExpiration is rejected as a field of RealmRepresentation — and rejecting it fails the whole update, including the settings that would otherwise apply, so do not put it back. As a realm attribute it is accepted and silently dropped. At this scale the growth is nothing; the console can clear them.

A password set or reset through the admin API is recorded as an ACTION on users/<id>/reset-password with no representation — the value itself is not stored (checked 2026-08-28 on a rotation).

Errors also reach the container journal through the jboss-logging listener at WARN (pct exec 110 -- journalctl -u keycloak), which is where a live failure is quickest to read — that is how the gate's 500 was traced on 2026-08-28. Successes are logged at DEBUG and so do not appear there; the event store is where they live.

Resisting password guessing

On since 2026-08-28, asserted by provisioning/keycloak/realm-lockout.sh (--show prints the current values). It bites the local accounts alone: an edu-ID user has no password in this realm, so no attempt against one can fail here. Until those accounts existed there was nothing to protect, which is why it was off.

Setting	Value	Why
bruteForceProtected	true
permanentLockout	false	a permanent lockout is a denial of service against a known username
failureFactor	10	Keycloak's default of 30 is lax for a public endpoint
waitIncrementSeconds / maxFailureWaitSeconds	60 / 900	the wait doubles per lockout, capped at 15 minutes
maxDeltaTimeSeconds	43200	the failure count resets after 12 hours
quickLoginCheckMilliSeconds / minimumQuickLoginWaitSeconds	1000 / 60	two attempts inside a second is a script, not a person: 60-second wait regardless of the count
passwordPolicy	length(16) and notUsername

The policy is length-only on purpose. These passwords are generated by local-users.sh, never chosen, so character-class rules buy nothing — and specialChars(1) would reject the generator's own alphanumeric output, locking the script out of the realm it manages. 16 is below the 20 it produces, so the policy can never refuse a password this repo created. No expiry either: forcing a rotation on accounts whose password lives in the secretzone only produces a value nobody records.

Clearing a lockout, which is what someone will actually need:

ssh root@rumba "pct exec 110 -- /opt/keycloak/bin/kcadm.sh delete \
  attack-detection/brute-force/users/<user-id> -r isc"

get on the same path reports numFailures and disabled without changing anything. The console does it too, under the user's Sessions tab.

Who may edit their own account

Users cannot change their own email address; everything else about their profile they can. Asserted by provisioning/keycloak/user-profile.sh (--show prints the permissions without changing anything). Set 2026-08-28, from a finding of that day's review.

Keycloak's default lets anyone edit their address in the account console, unverified. This realm links an incoming edu-ID login to an existing account by email, silently — safe while every account's address came either from edu-ID or from an administrator. The guests accounts broke that assumption: someone holding one of those passwords could set its address to a colleague's before that colleague's first edu-ID login, the login would land in the guest account, the claim mappers would fill it with isc/hevs, and the guest password would open everything those groups open. Making the address admin-only removes the one step of that chain that is ours to remove.

Nobody needs the write: edu-ID users get their address from the broker, local accounts from local-users.sh. The profile still requires an address, and an account with none that cannot set one would be stuck at login — both sources guarantee one.

Local accounts: the ones with no edu-ID

A handful of people have no edu-ID — an external partner, a guest. They get an ordinary user of realm isc in the group guests, created with provisioning/keycloak/local-users.sh add|passwd|remove|list, run inside the container:

ssh root@srv-keycloak 'bash -s' -- < provisioning/keycloak/local-users.sh add <username>

The -- is not optional: without it bash reads add as one of its own options and exits on usage. Passwords are generated, printed once and recorded in secretzone/rumba.md; passwd is how one is changed, since only the hash is kept.

Each carries a synthetic address, <username>@guests.isc-vs.ch, marked verified. That is not cosmetic: oauth2-proxy refuses a token whose email claim is empty or unverified, and the failure surfaces as a bare 500 on the callback that names neither the account nor the claim — the log line is email in id_token () isn't verified. The domain has no MX; the address exists to be well-formed and to be obviously not a person's own, which the realm would reject as a duplicate anyway.

Two rules:

• guests and nothing else. isc, hevs and hes-so are filled from edu-ID claims by the mappers above, and putting a local account in one of them grants everything those groups open. guests is admitted by the user gate and opens nothing beyond it.
• Keep the number near zero. Each is a password to rotate and a person whose departure nobody will notice — the failure mode brokering identity through edu-ID exists to avoid.
• They must not be able to rename themselves by email. They cannot, since 2026-08-28 — why.

Until 2026-08-28 these accounts lived one layer earlier, in an htpasswd file read by oauth2-proxy itself. That route never reached Keycloak, so it sat outside every control the realm has and left nothing in its logs; it also forced the gate to show its own sign-in page before the themed Keycloak one. Moving them into the realm removed both the mechanism and the extra page, and put these accounts under what the realm enforces — including the lockout and password policy turned on the same day, which had never been needed while every login was brokered to edu-ID.

Role groups: the ISC list

students / staff are not derived from claims: no HES-SO claim distinguishes a person's institute reliably enough, and the yearly intake should be an explicit act. They grant admission only — all reach is entitlement groups. (Until 2026-08-06 there were four identity groups; teachers / admins carried reach of their own and were collapsed into the vpn-rack-operators / vpn-rack-mgmt entitlements.) They come from a locally held list, provisioning/keycloak/roster.csv (email,groups[,nasname], one line per person, groups ;-separated), applied by the idempotent provisioning/keycloak/roster-sync.sh — so an intake is one reviewable diff rather than an afternoon in the admin console. The optional third column is the NAS home a person owns; it is unused until UID allocation exists, but the roster is the right place to record it once.

role-rack-admins is listed in the same file but handled differently: the sync only ever adds to it, never removes. It gates the admin console in front of this very Keycloak, so a truncated or mis-edited roster must not be able to lock everyone out of the tool needed to repair it — taking someone out stays a deliberate console action.

Resource groups, not cohort groups

Alongside the identity groups, the roster carries one group per resource — vpn-carnaval, vpn-rack-operators, vpn-rack-mgmt, plus role-pve-auditor for read-only Proxmox (how it is granted) — rather than one per cohort (students-2024, or a "3ème année" group). Both would express "the third years may reach Carnaval"; the difference is where the yearly churn lands.

Per resource, the NetBird policies are written once and never repointed: a new student joins a group that already exists, and the move-up is an edit to roster.csv, i.e. a reviewable git diff. Per cohort, the membership never changes but every policy has to be repointed at a new group name each August — in NetBird's UI, outside version control, and silently if forgotten.

The NetBird side makes this sharper still. A group only comes into existence there when a token first carries it, so a cohort group for an incoming year cannot be referenced by a policy until the new students have logged in at least once. Resource names are stable and already claimed.

The volée stays visible in roster.csv as a section comment — readable as a class list, with no mechanism behind it.

This trades one kind of work for another

Each August, the rising cohort gains a resource group: roughly one edited line per student, in the same diff that already handles arrivals and departures. That is the deliberate cost. It stops being the right trade if access ever needs to be per-course or per-project, where the group count multiplies and the file becomes an ACL matrix — at that point a volée → resources lookup table beats enumerating it per person.

NetBird is the first consumer; it reads the group names out of the groups claim, and since 2026-08-05 that membership is what admits someone to the VPN (jwt_allow_groups), not just what they may reach.

One consequence for whoever edits these names: a consumer may not be able to create a group of the same name itself. NetBird's sync ignores a claim naming a group it did not create, which is why its hand-made groups are called *-manual — so renaming a group here means renaming it there too.

What the script deliberately does not do, each for a reason worth keeping:

• It never touches hes-so / hevs / isc — those are claim-derived, see above.
• It never touches role-rack-admins, which gates the admin console in front of this very Keycloak: a roster edit must not be able to lock you out of the tool you would need to fix it.
• It never deletes an account. A removed line drops the group membership only; deleting the user would drop the federated identity link, i.e. the record of who this person was at edu-ID.

Two consequences of there being no SMTP on this host — smtpServer is empty and still unconfigured, even though the rack gained a smarthost on 2026-08-10:

• Pre-created accounts are made with emailVerified=true, which is a true statement given trustEmail on the eduid provider, not a shortcut.
• Keycloak's default way of linking a brokered login to a pre-existing account is to email the person a verification, with re-authentication by password as the only alternative — and a roster account has no password, so the student meets an unsatisfiable username/password form after a good edu-ID login. The eduid provider therefore uses its own first-broker flow eduid-autolink (provisioning/keycloak/first-broker-autolink.sh): Create User If Unique, else Automatically set existing user — silent linking by email. Safe here because edu-ID verifies its emails, trustEmail is on, and the matched accounts are our own roster shells; do not copy it next to an IdP whose emails are unverified, that would let anyone claim any account. Auto-linking alone left the pre-created account's names empty: provider-level sync (unset = IMPORT) only applies brokered data when Keycloak creates the account, and the VERIFY_PROFILE required action then demanded first/last name from every new student — a form whose data is worthless, since edu-ID owns it. Hence the provider runs syncMode=FORCE (edu-ID re-applies names and email at every login, including the linking one) and VERIFY_PROFILE is disabled. Both asserted by groups-and-mappers.sh.

Matching is on the email address, so someone whose edu-ID primary address is not their HES-SO one shows up as "logs in fine, no access" — add their real edu-ID address to the roster.

The admission gate: refusing with a message

Since 2026-08-06 a login that will not be admitted is refused here, not by the service: being refused downstream meant a bare error page after a successful login, with no hint that enrollment was the issue. It covered NetBird only until 2026-09-06, when it was generalised to every client — oauth2-proxy and Proxmox included, and any client added later. provisioning/keycloak/vpn-access-gate.sh (idempotent) builds the whole thing.

The fleet has two admission rules, so the flow tests two, told apart by which client the login is for:

Target	Realm role	Granted to	Message key
NetBird, and machine access behind it	vpn-access	students / staff — the roster	vpnNotEnrolled
Every other client — the published tools, Proxmox, future ones	isc-access	isc, plus students / staff	iscNotEnrolled

Neither role needs a list of its own: both follow group membership, which roster-sync.sh and the claim mappers already maintain. isc-access is granted to the roster groups as well as to isc, which does double duty — it lets a rostered person through when edu-ID has not released their org-unit claim, and it is how an individual exception is made (a HEI colleague added to roster.csv gets staff, and with it the role).

note

hevs is not an admission group (September 2026)

The published tools admit ISC affiliation, not the HEI at large: the rest of the school counts as external here, the future GitLab being the case that decided it. Exceptions go through roster.csv, one person at a time.

The pieces:

• marker client scope vpn-gate, default on the netbird client only, excluded from the token scope — it exists so a flow condition can tell a NetBird login from every other one;
• top-level flow vpn-access-gate, bound as the post-broker flow of the eduid provider: two conditional subflows, each client scope vpn-gate (negated in the second) ∧ user role negated, each ending in Deny access — followed by one top-level REQUIRED Allow access. Message wording (EN/FR) is in the login theme's i18n.ts and resolved server-side, so deploy the theme before the flow or denied users see the raw key.

Changing the flow's shape needs a rebuild, not an edit

The script's build branch runs only when the flow is absent, so editing it and re-running is a no-op that prints success. It compares the live shape against the one it knows and, when they differ, unbinds, deletes and rebuilds — deleting a flow takes its executions and their configs with it, which is what keeps the config aliases free for the rebuild. A bound flow cannot be deleted, hence the unbind first.

danger

The trailing Allow access is required

A flow whose executions all end up disabled — every condition false, i.e. every normal login — does not count as success: the processor throws, and every edu-ID login on every client fails with invalid_user_credentials (userId=null in the events, "Invalid username or password" on screen). This was hit in practice: the first binding lacked it and every SSO login failed until the flow was unbound. Known behaviour, keycloak#14591. Any future post-broker flow needs the same explicit success state.

Post-broker rather than a browser-flow override on the client, because a fresh brokered login never returns to the browser flow — an override there is skipped in exactly the case that matters. The post-broker flow runs on every edu-ID login. Two accepted limits: a login riding an existing Keycloak SSO cookie skips the broker, and thus the gate, and falls through to the service's own refusal; and local accounts never traverse the broker at all, so the gate does not see them either — the user gate admits them on its own.

Where to actually test a login

A realm has no standalone login page — Keycloak only shows one when a client asks it to authenticate someone. Until NetBird is pointed here, the built-in account console is the harness:

https://sso.isc-vs.ch/realms/isc/account/

It bounces to the realm login page ("Sign in to ISC"), which carries the social-eduid button. /realms/* is public by design — edu-ID's callback and the OIDC clients need it — so this works from anywhere, VPN or not; only /admin* and /realms/master* are restricted.

Note that the account console requires PKCE: a hand-built authorization URL without code_challenge/code_challenge_method bounces back with Missing parameter: code_challenge_method. Browsers send them automatically — don't read that error as a broken realm.

Operations

• Deploy / upgrade: provisioning/keycloak/deploy-keycloak.sh, idempotent; KEYCLOAK_VERSION=… ./deploy-keycloak.sh upgrades in place, keeping the database, the rendered config and any provider jars.
• Login theme: ISC-branded (petal colors, logo, dark mode; deployed 2026-08-04), built with Keycloakify from provisioning/keycloak/theme/; deploy-theme.sh builds the jar, pushes it to providers/, runs kc.sh build + restart (the unit is --optimized, a dropped-in jar alone does nothing) and sets loginTheme=isc on the realm.
• Service: systemctl status keycloak in the container (pct exec 110 -- … from rumba, or ssh root@srv-keycloak over the VPN). Readiness: curl http://192.168.88.160:9000/health/ready.
• The JVM heap is pinned (-Xmx1536m) in the unit on purpose: a JVM in an LXC can read the host's memory through lxcfs, size its heap against rumba's 187 GB and get OOM-killed much later, with no obvious cause.
• State: PostgreSQL inside the container, covered by rumba's nightly vzdump — that job is configured all 1, so a new guest is included the same night without any action (verified 2026-08-03).
• Realm snapshot: see below — re-run it after any admin-console change.
• Losing this host means losing every federated login for whatever is wired to it. Until GitLab and the rest arrive that is only a future concern, but it is the reason the MikroTik WireGuard stays as the emergency admin path.

What survives losing this container

Asked deliberately after the NetBird store loss, whose cause was a single value existing in exactly one place. Keycloak has no equivalent. Nothing in realm isc is irreplaceable, because every part of it is either in git or re-derived on the next login:

Thing	How it comes back
Realm, groups, claim mappers	bootstrap.sh + groups-and-mappers.sh, both idempotent
Role-group membership	roster.csv + roster-sync.sh
The custom auth flows (admission gate, auto-link)	vpn-access-gate.sh + first-broker-autolink.sh, both idempotent
Users and their edu-ID link	recreated at next login — the pairwise sub derives from the sector identifier, i.e. the frozen public origin, so the same person returns as the same subject
Imported claim attributes	re-imported at every login (syncMode=FORCE)
Client secrets	regenerated by each consumer's keycloak-client.sh, then re-pasted into that consumer
The eduid provider instance	the one piece that is console work — hence the snapshot below and the click-by-click values on this page

What a rebuild does cost: every active session and offline token dies with the old signing keys, and each consumer must be given its new client secret. Both are visible failures that get fixed in minutes. The realm database is nevertheless in rumba's nightly vzdump, which turns those minutes into none.

Realm snapshot

provisioning/keycloak/export-realm.sh writes realm-isc.json — a partial export, key-sorted, secrets stripped. Run it after any admin-console change and commit the diff.

bootstrap.sh creates the realm empty and groups-and-mappers.sh covers the groups and the claim mappers, but the identity-provider instance itself — the discovery import, the client id and secret, and the three settings whose defaults are wrong for edu-ID — is still console work living only in this container's database. The snapshot is what makes such a change reviewable instead of invisible.

It is not a restore path: partial export omits users and masks secrets, so recovery stays a vzdump of CT 110 or a re-click from this page. Current content: identity provider eduid, 5 mappers, groups hes-so / hevs / isc / role-rack-admins / students / staff and the four entitlement groups, and the clients oauth2-proxy, oauth2-proxy-users, netbird and proxmox with their groups mappers. The roster membership is not in there — users are omitted from a partial export, so provisioning/keycloak/roster.csv is the only record of who is in which role group.