Skip to main content

NetBird — history & operations

Dated operations on the NetBird VPN, newest first — so far the deployment and acceptance record (v0.76.1, deployed 2026-08-02). The current architecture, access model and operations live on the main page.

The admission refusal moved to Keycloak, with a message (2026-08-06)​

First real-world refusal (a student with a valid edu-ID, not yet on the roster) showed what jwt_allow_groups looks like from outside: a successful edu-ID login ending on a bare NetBird error page naming nothing. Refusal now happens at Keycloak with a themed "not enrolled — contact the staff" message (EN/FR); jwt_allow_groups stays as defense in depth. Mechanism, and the SSO-cookie case it deliberately does not cover: the VPN gate. Rollout note: the first binding lacked the flow's trailing Allow access and took down all edu-ID logins (every client) for ~20 minutes until unbound — that requirement is now a standing rule on the Keycloak page.

The node subnet split into per-resource grants (2026-08-05)​

One resource, 192.168.91.0/24, was granted to every role group — so a first-year and a third-year reached exactly the same things and carnaval-users could not have meant anything. Replaced by eight host resources for calypso0–7, a .128/26 range for the carnaval lab VMs, and the three carnaval PVE hosts in their own net-carnaval-hosts (admins and teachers, for parity with rumba's console via net-appliances — an earlier pass had folded them into the admin-only net-mgmt, which would have stopped a teacher building lab VMs for their own students). The old net-nodes group and nodes-access policy are gone. Layout and the two addressing traps are on the main page.

Two grants disappeared that nobody had meant to give students: the carnaval PVE consoles, and 192.168.91.250 — the NAS's second interface, exporting NFS sec=sys, i.e. a client authorising itself by claiming a UID.

Federated access made to work for a cohort (2026-08-05)​

The SSO login worked but granted nothing, because JWT group sync ignores a claim naming an api-issued group. Resolved by freeing the names — admins → admins-manual and so on, safe because policies reference groups by id — then minting the three jwt groups (a group cannot exist before a login claims it, so one admin temporarily held all three roles in the roster) and listing both the jwt and *-manual group in every policy. replay-config.py --jwt-twins reproduces the pairing after a rebuild and refuses to create a jwt group itself.

Admission then moved from a person to the roster: jwt_allow_groups = admins/teachers/students, and user_approval_required off. It had to go — a rehearsal of the newcomer path showed the browser saying "Login successful" while the agent failed with user pending approval cannot add peers, with nothing anywhere telling the user to wait for an admin. Same pass hid the local login (localAuthDisabled: true), which removes Dex's connector chooser so netbird up lands directly on Keycloak: verified straight through to edu-ID.

Proven with one edu-ID account, which is its limit — a second identity is needed to observe a user in no role group actually being refused, and two different roles at once (open items).

Store rebuilt after the encryption-key loss (2026-08-05)​

A restart destroyed the store — cause, impact and prevention in the incident. What the account looks like afterwards:

  • Rebuilt from config-snapshot.json with the new provisioning/netbird/replay-config.py: 8 groups, the ISC3-rack network, 5 resources, 4 policies with Default disabled, the calypso-internal nameserver group, the router with masquerade off. Peers re-enrolled themselves. The snapshot committed that same morning is the only reason this was a replay.
  • The owner is now the role account vpn-admin@isc-vs.ch, not a person's @hevs.ch address — which also keeps the break-glass identity from colliding with the same person's federated one.
  • Two values changed and the docs had to follow: the overlay range is 100.65.0.0/16 (was 100.79; the CCR's 100.64.0.0/10 route covers both, so no network change) and peer_login_expiration is 86400 s (was 864000).
  • The rebuilt account has domain=netbird.selfhosted, which the lost one did not — so a federated login now joins it instead of creating its own tenant. The thing that could not be retrofitted came free with the rebuild.

Keycloak wired in as login provider (2026-08-05)​

NetBird became an OIDC client of realm isc (provisioning/netbird/keycloak-client.sh) and the ISC SSO button appeared next to the local accounts. JWT group sync turned on with claim groups; proven from a real ID token rather than by inspection — "aud":"netbird", "groups":["hes-so","rack-admins"]. The Keycloak role groups were created and filled from the roster first, before group sync, so the claim already carried admins when it became authoritative — the other order strips the admin's own route access.

The agent login was verified the same day: a throwaway netbird profile reached the same login page and the ISC SSO button worked, so students can self-enroll with edu-ID and the per-user attribution the masquerade-off design needs is safe — no setup keys. One trap on the way: a new profile does not inherit the management URL and defaults to NetBird's cloud, so without -m https://vpn.isc-vs.ch the test lands on login.netbird.io and proves nothing.

Everything learnt that day is now a standing rule on the service page: account membership needs the account's domain set; issued=api and issued=jwt groups of the same name are different groups, so a claim naming a hand-made group is dropped; an enrolled peer rides its session token and sees no provider change until it expires; and lazy connections were stranding a connected peer (routes chosen, handshake fresh, every packet timing out) — turned off. Admission is still user_approval_required; jwt_allow_groups is deferred because local accounts carry no groups claim (open items).

Two diagnostic traps worth keeping, both of which sent the investigation the wrong way for a while. A login into a different account looks like a complete success — dashboard, owner rights and all — while /api/users under the rack account's token shows no new user, because a token cannot see into another account; only the management log names the second accountID. And the client can report a network as Selected while the route is absent from the kernel table: check netstat -rn, not the client, and netbird down && netbird up installs them.

Verified working (2026-08-02/03)​

Stack healthy (dashboard 200, API 401-auth, LE certificate valid to 2026-10-31), routing peer connected in kernel mode serving all four subnets, onboot/unless-stopped/service enablement checked end to end. External client test passed 2026-08-03 (admin and student accounts, relayed mode). Group profiles validated client-side 2026-08-03: admin reaches all four subnets with .calypso DNS; student gets exactly 192.168.91.0/24 + the DNS carve-out, with the router's non-DNS ports, NAS, PDU and iDRACs all blocked; split tunnel confirmed for both — only the rack prefixes are routed, internet egress stays on the client's own uplink.

Masquerade-off verified 2026-08-03 from an external admin client: rumba, the NAS and calypso0 each report the client's own 100.79.x.x in $SSH_CONNECTION (previously .159), and .calypso DNS still resolves through the carve-out.