Skip to main content

NetBird — history & operations

Dated operations on the NetBird VPN, newest first — the deployment and acceptance record (v0.76.1, deployed 2026-08-02) and the updates since. The current architecture, access model and operations live on the main page.

Groups renamed by kind, k8s range opened to the lab users (2026-09-08)​

Roster groups renamed in Keycloak (same ids, members kept): carnaval-users → vpn-carnaval, appliances-users → vpn-rack-operators, mgmt-users → vpn-rack-mgmt, rack-admins → role-rack-admins, pve-auditors → role-pve-auditor; vpn-rack-operators trimmed to pmudry, yacine.said, adrien.reynard. NetBird side over the API: one admin login minted the new jwt groups, the policies were repointed to them in one go (carnaval-guests-access, operators-access — the hosts policy folded into it — mgmt-access), net-carnaval became net-carnaval-guests and took the carnaval-infra resource, so vpn-carnaval reaches the k8s API/UI. Instead of a 48 h dual-name window, peer_login_expiration was set to 3600 s: every peer re-authenticated within the hour with a token carrying the new names, the old jwt groups were then deleted and the value put back to 172800 s. Consumers of the renamed rights groups (sso-realm.sh on both PVE clusters, the admin gate, NetBox oidc.py) were re-run the same evening; a login on each verified before the old groups went. Current model on the access page.

Updated to 0.78.1 (2026-09-08)​

Server 0.77.0 → 0.78.1, dashboard v2.91.1 → v2.92.0, agent 0.77.0 → 0.78.1, same procedure as 0.77.0 over VM snapshot pre-0-78-1 (delete it once the version has held a week). Store migrations clean, policies and resources unchanged, routing peer reconnected on its own. 0.78.0 changes the client's DNS resolution (NB_USE_LEGACY_DNS_RESOLUTION=true restores the old path) — the routing peer's isc3 nameserver was still served after the restart.

VPN session window raised to 48 h (2026-08-30)​

peer_login_expiration 86400 → 172800 s over the API, closing the todo item that asked for the value to be set deliberately before the first federated cohort. Expiry is computed from each peer's last_login, so enrolled devices got the longer window at once; one whose last sign-in was already older than 48 h stays expired until its next interactive login. The deprovisioning tail doubles with it — mitigation unchanged, end the session in Keycloak by hand (session window).

Updated to 0.77.0 (2026-08-16)​

Server 0.76.1 → 0.77.0, dashboard v2.90.9 → v2.91.1, agent 0.76.1 → 0.77.0, over a VM snapshot (pre-netbird-0-77-0). Store migrations ran clean and the access model came through unchanged — export-config.sh diffed identical on policies, resources, routers, nameservers and setup keys. Interruption was a few seconds per restart, twice (server, then agent).

Both restarts had to run detached on the VM, which is now the documented update procedure.

The no-hairpin hosts entry did not survive the first reboot (2026-08-15)​

rumba's reboot took rack access over NetBird down ~40 minutes — cloud-init had wiped the no-hairpin /etc/hosts entry, now persisted in the cloud-init template: incident, mechanism on the main page.

The admission refusal moved to Keycloak, with a message (2026-08-06)​

First real-world refusal (a student with a valid edu-ID, not yet on the roster) showed what jwt_allow_groups looks like from outside: a successful edu-ID login ending on a bare NetBird error page naming nothing. Refusal now happens at Keycloak with a themed "not enrolled — contact the staff" message (EN/FR); jwt_allow_groups stays as defense in depth. Mechanism, and the SSO-cookie case it deliberately does not cover: the VPN gate. Rollout note: the first binding lacked the flow's trailing Allow access and took down all edu-ID logins (every client) for ~20 minutes until unbound — that requirement is now a standing rule on the Keycloak page.

The node subnet split into per-resource grants (2026-08-05)​

One resource, 192.168.91.0/24, was granted to every role group — so a first-year and a third-year reached exactly the same things and carnaval-users could not have meant anything. Replaced by eight host resources for calypso0–7, a .128/26 range for the carnaval lab VMs, and the three carnaval PVE hosts in their own net-carnaval-hosts (admins and teachers, for parity with rumba's console via net-appliances — an earlier pass had folded them into the admin-only net-mgmt, which would have stopped a teacher building lab VMs for their own students). The old net-nodes group and nodes-access policy are gone. Layout and the two addressing traps are on the main page.

Two grants disappeared that nobody had meant to give students: the carnaval PVE consoles, and 192.168.91.250 — the NAS's second interface, exporting NFS sec=sys, i.e. a client authorising itself by claiming a UID.

Federated access made to work for a cohort (2026-08-05)​

The SSO login worked but granted nothing, because JWT group sync ignores a claim naming an api-issued group. Resolved by freeing the names — admins → admins-manual and so on, safe because policies reference groups by id — then minting the three jwt groups (a group cannot exist before a login claims it, so one admin temporarily held all three roles in the roster) and listing both the jwt and *-manual group in every policy. replay-config.py --jwt-twins reproduces the pairing after a rebuild and refuses to create a jwt group itself.

Admission then moved from a person to the roster: jwt_allow_groups = admins/teachers/students, and user_approval_required off. It had to go — a rehearsal of the newcomer path showed the browser saying "Login successful" while the agent failed with user pending approval cannot add peers, with nothing anywhere telling the user to wait for an admin. Same pass hid the local login (localAuthDisabled: true), which removes Dex's connector chooser so netbird up lands directly on Keycloak: verified straight through to edu-ID.

Proven with one edu-ID account, which is its limit — a second identity is needed to observe a user in no role group actually being refused, and two different roles at once (open items).

Store rebuilt after the encryption-key loss (2026-08-05)​

A restart destroyed the store — cause, impact and prevention in the incident. What the account looks like afterwards:

  • Rebuilt from config-snapshot.json with the new provisioning/netbird/replay-config.py: 8 groups, the ISC3-rack network, 5 resources, 4 policies with Default disabled, the calypso-internal nameserver group, the router with masquerade off. Peers re-enrolled themselves. The snapshot committed that same morning is the only reason this was a replay.
  • The owner is now the role account vpn-admin@isc-vs.ch, not a person's @hevs.ch address — which also keeps the break-glass identity from colliding with the same person's federated one.
  • Two values changed and the docs had to follow: the overlay range is 100.65.0.0/16 (was 100.79; the CCR's 100.64.0.0/10 route covers both, so no network change) and peer_login_expiration is 86400 s (was 864000).
  • The rebuilt account has domain=netbird.selfhosted, which the lost one did not — so a federated login now joins it instead of creating its own tenant.

Keycloak wired in as login provider (2026-08-05)​

NetBird became an OIDC client of realm isc (provisioning/netbird/keycloak-client.sh) and the ISC SSO button appeared next to the local accounts. JWT group sync turned on with claim groups; proven from a real ID token rather than by inspection — "aud":"netbird", "groups":["hes-so","rack-admins"]. The Keycloak role groups were created and filled from the roster first, before group sync, so the claim already carried admins when it became authoritative — the other order strips the admin's own route access.

The agent login was verified the same day: a throwaway netbird profile reached the same login page and the ISC SSO button worked, so students can self-enroll with edu-ID and the per-user attribution the masquerade-off design needs is safe — no setup keys. One trap on the way: a new profile does not inherit the management URL and defaults to NetBird's cloud, so without -m https://vpn.isc-vs.ch the test lands on login.netbird.io and proves nothing.

Everything learnt that day is now a standing rule on the service page: account membership needs the account's domain set; issued=api and issued=jwt groups of the same name are different groups, so a claim naming a hand-made group is dropped; an enrolled peer rides its session token and sees no provider change until it expires; and lazy connections were stranding a connected peer (routes chosen, handshake fresh, every packet timing out) — turned off. Admission is still user_approval_required; jwt_allow_groups is deferred because local accounts carry no groups claim (open items).

Two diagnostic traps worth keeping, both of which sent the investigation the wrong way for a while. A login into a different account looks like a success — dashboard, owner rights and all — while /api/users under the rack account's token shows no new user, because a token cannot see into another account; only the management log names the second accountID. And the client can report a network as Selected while the route is absent from the kernel table: check netstat -rn, not the client, and netbird down && netbird up installs them.

Verified working (2026-08-02/03)​

Stack healthy (dashboard 200, API 401-auth, LE certificate valid to 2026-10-31), routing peer connected in kernel mode serving all four subnets, onboot/unless-stopped/service enablement checked end to end. External client test passed 2026-08-03 (admin and student accounts, relayed mode). Group profiles validated client-side 2026-08-03: admin reaches all four subnets with .calypso DNS; student gets exactly 192.168.91.0/24 + the DNS carve-out, with the router's non-DNS ports, NAS, PDU and iDRACs all blocked; split tunnel confirmed for both — only the rack prefixes are routed, internet egress stays on the client's own uplink.

Masquerade-off verified 2026-08-03 from an external admin client: rumba, the NAS and calypso0 each report the client's own 100.79.x.x in $SSH_CONNECTION (previously .159), and .calypso DNS still resolves through the carve-out.