NetBird VPN

Self-hosted NetBird (v0.76.1, BSD-3, deployed 2026-08-02) — the self-service VPN of ISC³ at https://vpn.isc-vs.ch, planned in the target architecture §3bis. Users enroll their own devices after logging in; access is governed by identity groups instead of hand-managed WireGuard peers.

Dual-path policy

The MikroTik WireGuard stays fully operational as the emergency/admin path (static peers for the team, wg-gen.sh). NetBird is the standard path for everyone else. Neither replaces the other until the architecture doc says so.

At a glance

Guest	srv-netbird — VM 109 on rumba, 192.168.88.159, 2 vCPU / 4 GB / 20 GB NVMe
Public URL	https://vpn.isc-vs.ch (dashboard + API + control plane, all on 443)
Users	Keycloak / edu-ID as an external login provider since 2026-08-05, authorized from provisioning/keycloak/roster.csv (see Identity). Local accounts (embedded Dex) stay as the admin and break-glass path
Managed from	provisioning/netbird/ (compose, config template, idempotent deploy script)
Credentials	Owner is the role account vpn-admin@isc-vs.ch (local, break-glass), not a person's federated address — password and PAT in the secretzone (rumba page)

Architecture

One combined container (netbirdio/netbird-server: management + signal + relay + STUN) plus the dashboard, behind the existing reverse proxy:

• TLS terminates on srv-web01 — the vpn.isc-vs.ch vhost in provisioning/web/Caddyfile carries everything over the already-open 443: dashboard, REST API, embedded IdP, gRPC (management + signal, h2c:// transport) and the WebSocket relay. No new inbound TCP.
• STUN (UDP 3478) is the only thing a reverse proxy cannot carry — and the uplink kills it. Verified 2026-08-03: SInf/vsnet drops inbound UDP 3478 upstream (a dst-nat rule counted zero packets), consistent with the uplink catalogue — only 80/443 TCP and UDP 51820 get in. Relay-only is the permanent mode: every client↔rack flow goes client → WebSocket relay (443) → routing peer. Since both live in the rack, this costs little for rack access — what it rules out is direct client↔client P2P, which nothing here needs. Revisit only if SInf ever opens inbound UDP 3478.
• Routing peer = the VM itself (native agent, kernel WireGuard). It advertises the rack subnets without masquerading (turned off 2026-08-03): appliances see each user's own overlay address (100.65.x.x, 1:1 to a user+device under Peers), so logs attribute actions to individuals and per-IP protections (NAS auto-block, fail2ban) act per user instead of lumping everyone — the fix for the July 2026 NAS-lockout pattern. Supporting config: a static route on the CCR2004 (100.64.0.0/10 → 192.168.88.159; the CRS326 inherits it via its default route) and an input accept for 100.64.0.0/10 on the CRS326 so its management stays reachable. Replies from 88-subnet appliances hairpin through the CCR (asymmetric first packet) — fine, no drop-invalid rules on either MikroTik. Do not allow-list 100.64.0.0/10 on the NAS: per- user auto-block is the point. NetBird's group policies still apply unchanged — they are enforced on the routing peer, before masquerading would have happened.
• No NAT hairpin from the LAN: the VM reaches its own public name through an /etc/hosts entry pointing vpn.isc-vs.ch at srv-web01 (192.168.88.150) — remember it if the proxy address ever changes.

Access model (groups → subnets)

Routes and policies are defined on the network ISC3-rack (dashboard → Networks); the built-in Default all-peers-to-all-peers policy is disabled — user devices cannot see each other.

Source group	Reaches
calypso-users	192.168.91.10/32 – .17/32 — calypso0–7
carnaval-users	192.168.91.128/26 — the carnaval lab VMs
appliances-users	192.168.88.0/24 (rumba, guests, NAS, PDU) + 192.168.91.18/32 – .20/32 (carnaval PVE hosts)
mgmt-users	192.168.90.0/24, 192.168.92.0/24 (iDRACs)
students / staff	admission + the DNS carve-out (192.168.88.1/32, UDP 53 only) — nothing else

Since 2026-08-06 the identity groups (students / staff) say only who someone is — every reach is a per-resource entitlement assigned from roster.csv. The old teachers and admins identities were exactly the first two rows' reach and became appliances-users / mgmt-users; being staff grants no appliance by itself, which is what lets a lab assistant and a teacher differ.

The api-issued *-manual groups (admins-manual / teachers-manual / students-manual — legacy names from the pre-collapse model, kept because renaming buys nothing) stay listed in every policy as the local / break-glass path; the jwt-issued groups the policies match for federated users are the entitlements plus students / staff. The naming rule below is why no group a claim will carry may ever be created by hand. replay-config.py --jwt-twins predates the collapse — verify it against this model before trusting a rebuild (open item).

note

Why per-host rather than one 192.168.91.0/24 (2026-08-05)

Until then a single resource covered the whole node subnet and every role group had it, so carnaval-users could not have meant anything different. Splitting it also dropped two things nobody had intended to grant: the carnaval PVE consoles on :8006, and 192.168.91.250, the NAS's second interface — reachable over the VPN, exporting NFS sec=sys, where a client authorises by claiming a numeric UID. .10–.17 is not a clean CIDR, hence eight host resources.

The lab range .128/26 stops at .191 for the same reason: /25 would have reached .250 again. carnaval-lab-vm.sh allocates inside it and refuses anything outside, so a VM cannot silently land where no VPN client can reach it. The CRS326 DHCP pool used to overlap the range; it was moved to .200–.240 and both DHCP servers disabled on 2026-08-05, nothing having used them since.

The 192.168.88.1/32 DNS carve-out exists because the nameserver for the .calypso names is the router, which otherwise sits in the appliances subnet students must not reach. The dns-access policy allows exactly port 53 to that one address, nothing more — and in practice UDP only (the policy rule doesn't pass TCP 53). Harmless for normal lookups — add a TCP 53 rule in the dashboard only if oversized DNS responses ever matter. calypso-internal is the only nameserver group: the sso-split-horizon one was deleted on 2026-08-04 when the admin gate replaced source-IP protection, and no client receives an internal answer for a public name any more — why that layer went.

Onboarding a user is normally a line in provisioning/keycloak/roster.csv since 2026-08-05 — no invite link and no password to hand out, the person just signs in with edu-ID (how). The dashboard path below remains for local accounts, i.e. the admin/break-glass ones: dashboard → Team → Users → Add User, two flavours:

• Invite User (preferred): name + email + Auto-assigned groups (students for students), then Create Invite Link — hand the link to the person, they set their own password (link expires after 3 days by default; regenerate under Show Invites if needed).
• Create User: same fields, but the dashboard generates the password and shows it once — copy it before closing, it cannot be retrieved later.

Every device the user enrolls inherits their auto-assigned groups, so the routes and policies apply without further clicks. Removing/blocking the account (or one device under Peers) kills that VPN access instantly. Batch onboarding is scriptable: POST /api/users/invites returns an invite link per student — one API call per line of a class list.

Identity: Keycloak as the login provider

Since 2026-08-05 the login page offers ISC SSO: NetBird is a confidential OIDC client of realm isc on Keycloak, which brokers to SWITCH edu-ID. Created by provisioning/netbird/keycloak-client.sh; groups arrive in a groups claim (jwt_groups_enabled, jwt_groups_claim_name=groups). Who is authorized to log in at all is the Keycloak role groups, i.e. provisioning/keycloak/roster.csv. Both the dashboard and the agent work through it (verified 2026-08-05), so people self-enroll their own devices with edu-ID and no setup keys are needed — which is what preserves the per-user attribution the masquerade-off design rests on.

NetBird brokers rather than redirects: both the dashboard and the agent authenticate against NetBird's own issuer (https://vpn.isc-vs.ch/oauth2), which then talks to Keycloak server to server. So Keycloak sees exactly one client, needs no device-authorization grant, and the registered callback is …/oauth2/callback (logout …/oauth2/logout/callback, a separate registration — missing it breaks logout with a bare "invalid redirect URI" page while login keeps working).

An identity is (uuid, connector) base64-encoded, e.g. CiQ…EgVsb2NhbA decoding to ba97740d-… local. A federated one carries the identity-provider id instead of local, and its uuid is the Keycloak sub — so a NetBird user is traceable to a Keycloak user with no mapping table. It also means a local and a federated login are never the same identity, whatever the email says.

danger

The account needs domain set, or every SSO login builds its own empty tenant

NetBird self-hosted is multi-tenant. Single-account mode joins a new identity to the account whose domain is netbird.selfhosted with domain_category=private — and if no account matches, it creates a new one. The 2026-08-04 account had an empty domain (created through /api/setup before this behaviour was understood), so an ISC SSO login produced a second account with the person as its owner: the dashboard reported a perfectly successful login, full admin rights, while the enrolled peer reached nothing. /api/users under the rack account's own token showed no new user, because a token cannot see into another account — only the management log named the second accountID.

The account rebuilt on 2026-08-05 has the domain set, and a federated login joins it correctly. Do not try to fix an account that lacks it by editing accounts.domain: that field is associated data for the encryption of user records, so changing it makes them undecryptable — see the incident.

Never create a group whose name a claim will carry

JWT group sync only manages the groups it created (issued=jwt). A group of the same name made by hand or through the API (issued=api) is not reused and the claim's entry is silently dropped — measured 2026-08-05: a token carrying admins, hes-so, rack-admins produced the latter two as new jwt groups and ignored admins, which already existed as api. The symptom is a federated user who logs in perfectly and reaches nothing.

Hence the layout: the api-issued groups are admins-manual / teachers-manual / students-manual, leaving the plain names free for sync, and every policy lists both. Renaming was safe because policies reference groups by id, so access never lapsed. A role group also cannot exist before some login claims it, which is why the three were minted by temporarily granting one admin all three roles in the roster. replay-config.py will not create a jwt-issued group for the same reason.

An already-enrolled device does not see a provider change until its session expires

peer_login_expiration is 86400 s (24 h) and the agent rides that token: netbird up on an enrolled peer re-authenticates nothing and prompts for nothing. A provider change is therefore invisible on existing devices, and a disabled upstream account keeps its VPN for the rest of that window — this value is the whole deprovisioning tail. To exercise a login without disturbing a working setup, netbird profile add <name> has no stored token; give it -m https://vpn.isc-vs.ch, because a new profile defaults to NetBird's cloud and will otherwise send you to login.netbird.io.

The same tail works in reverse: deleting a jwt group is futile while any pre-deletion session still refreshes — the embedded IdP re-emits its cached claims on every token refresh and JWT sync re-creates the group within minutes (seen live 2026-08-06 with the retired teachers / admins). Log the affected users out (dashboard and agents) or wait out the window, then delete.

Admission is roster membership since 2026-08-05: jwt_allow_groups is ["staff","students"] (the identity model collapsed to those two on 2026-08-06), so a token must carry an identity group — i.e. the person must be a line in roster.csv. Anyone else with an edu-ID authenticates at Keycloak (the claim mappers only label people) and since 2026-08-06 is refused by Keycloak itself with an explicit message ("not enrolled — contact the staff", themed, EN/FR): the vpn-access-gate flow denies NetBird logins whose user lacks the vpn-access role. Before that the refusal came from jwt_allow_groups alone, i.e. a bare NetBird error page after a perfectly successful edu-ID login (first hit: 2026-08-06). The NetBird check stays as defense in depth — and it is still what a user sees in the one case that skips the gate, a login riding an existing Keycloak SSO cookie (no broker round-trip, so no post-broker flow). user_approval_required is therefore off: it was a manual click per person and, worse, invisible to them — the browser said "Login successful" while the agent failed with user pending approval cannot add peers and nothing told the user an admin had to act. If it is ever turned back on, approving is POST /api/users/{id}/approve; a PUT on the user does not clear it, and a PUT silently replaces auto_groups, so send the whole list.

The local login is hidden (auth.localAuthDisabled: true in config.yaml, 2026-08-05). With it enabled, Dex showed a connector chooser that put email/password first and buried ISC SSO behind "Choose another login method"; with a single connector it redirects straight to Keycloak. Break-glass is flipping the flag back and restarting — from rumba, no VPN needed — and the owner PAT keeps working regardless, since token auth does not go through Dex.

caution

The dashboard role is manual, and a deleted user comes back as user

The admin console is the dashboard itself (vpn.isc-vs.ch) — there is no separate address. But owner / admin / user is not derived from any claim (that needs the Enterprise IdP sync), so every administrator is promoted by hand in Team → Users. A federated user is created as user on first login, which combined with the hidden local login means a fresh SSO identity can administer nothing until promoted. vpn-admin@isc-vs.ch stays owner as break-glass; people hold admin. Deleting and re-creating a user — a login test, an account rebuild — silently resets them to user, and a PUT on the user replaces auto_groups, so send the whole list.

Lazy connections are off (2026-08-05). With three peers and relay-only operation they buy nothing, and they were seen stranding a connected peer: routes correctly chosen for all five resources, handshake seconds old, yet every packet to the rack timing out while client/iface/bind/activity.go logged could not find record for address. One less code path between a peer and its routes.

What the Community edition gives us

Three features are easily confused, and only the last one is paid — checked in our own console, August 2026:

Feature	Where in the dashboard	Edition
External OIDC login provider — an extra button on the login page (Okta, Google, Microsoft, Generic OIDC)	Settings → Identity Providers	Community — only the sub claim is strictly required
JWT group sync — groups read from a token claim at every login, auto-created and auto-assigned	Settings → Groups	Community
Identity Provider Sync / SCIM — NetBird pulls the user & group directory from the IdP's management API	Integrations	Enterprise (cloud, or a commercial on-prem licence)

Since v0.62 the providers coexist: an external IdP is added next to the embedded local accounts, which stay as the fallback and the admin path. What the paid sync would buy is pre-populating users who have never logged in — worthless at ~40 accounts a year, so do not buy it.

Deprovisioning does not depend on it either: an account disabled upstream can no longer log in, and Settings → Authentication → peer session expiration bounds how long an already-enrolled device survives without a fresh token — see the 10-day window above.

The rest of the edu-ID work is on SWITCH's side, and is why the plan goes through an OIDC broker rather than pointing NetBird straight at edu-ID — see architecture §3bis. Note that the combined container hard-enables its embedded Dex and exposes no upstream-connector configuration (netbird#5335): providers are added in the dashboard (or POST /api/identity-providers), never in provisioning/netbird/config.yaml.template.

Operations

• SSH to srv-netbird goes over the legacy WireGuard (or the Proxmox console), not over NetBird (verified 2026-08-03): the VM is the routing peer, and with the Default all-peers policy disabled, traffic terminating at the peer's own addresses (192.168.88.159, its 100.65.x.x overlay IP) is dropped — subnet-resource policies only cover forwarded traffic. Kept this way on purpose (the control plane isn't exposed to its own VPN users); an mgmt-users → routing-peer policy in the dashboard would reopen it if ever needed.

danger

server.store.encryptionKey must never be empty or change

Empty, the server mints a throwaway key at every start and only logs it truncated, so the next restart makes every encrypted field undecryptable and the store is gone — no backup helps, because the key lived only in the previous process. That is exactly what happened on 2026-08-05 (incident). deploy-netbird.sh now renders it once and refuses to run against an empty one. The encrypted fields live in two databases: store.db and idp.db (the embedded IdP's own).

• State: docker volume netbird_data + /root/netbird/ on the VM (rendered config.yaml holds the relay authSecret and the store encryption key). The VM is in rumba's nightly vzdump, but that only protects the store if the key survives too — so both values are also recorded in the secretzone (rumba page). They never rotate, which is what makes a second copy safe rather than a drift risk, and it is the only thing that makes a recovered store.db readable.
• Local users have a CLI, useful when the dashboard is not an option:

  docker exec -i netbird-server /go/bin/netbird-server -c /etc/netbird/config.yaml \
      admin user change-password --email vpn-admin@isc-vs.ch --password-file -

  --password-file - reads stdin, keeping the value out of the process list; admin user reset-mfa sits beside it. admin covers only the embedded IdP — there is no account management there.
• Rebuilding the access model after a store loss: provisioning/netbird/replay-config.py <PAT> replays config-snapshot.json (groups, network, resources, policies, nameservers, a fresh setup key), then --router once the routing peer has re-enrolled. Peers re-enroll themselves.
• Config snapshot: see below — re-run it after any dashboard change.
• Update: bump the two pinned image tags in provisioning/netbird/docker-compose.yml, run ./deploy-netbird.sh (refreshes files, docker compose up -d). The agent updates via APT (pkgs.netbird.io repo).
• Client → server requirements: outbound TCP 443 only (control plane + relay). Direct P2P would additionally want outbound UDP — irrelevant, STUN is unreachable for good (see above).
• NB_SETUP_PAT_ENABLED stays in the compose: /api/setup is inert once an account exists.
• API scripting: a personal access token for the owner account is in the secretzone (rumba page). The embedded Dex offers no password grant (checked 2026-08-03), so PATs created in the dashboard are the only way to script the API. Example — the routing-peer config (where the masquerade flag lives) is GET/PUT /api/networks/d9nrorfln5ac73cej3mg/routers/d9nrorfln5ac73cej3o0 with header Authorization: Token <PAT>.
• Acceptance-tested end-to-end 2026-08-02/03 (admin and student profiles, split tunnel, masquerade-off attribution) — deployment & acceptance log.

Config snapshot

provisioning/netbird/export-config.sh writes config-snapshot.json — groups, policies, the network with its resources and routers, nameserver groups, users and setup keys (values redacted; peers excluded, they are churn). Run it after any dashboard change and commit the diff.

The whole access model above is dashboard state inside the netbird_data volume, so without this a policy edit leaves no trace anywhere outside a vzdump. It restores nothing — NetBird's API has no import counterpart.