Vaultwarden (password vault)

Self-hosted Vaultwarden (1.37.1, AGPL-3.0, deployed 2026-08-04) at https://vault.isc-vs.ch — a Bitwarden-compatible server, so the ordinary Bitwarden desktop, mobile and browser clients work against it unchanged. It is the first service that actually consumes Keycloak: sign-in goes through the realm isc, and from there through SWITCH edu-ID.

At a glance

Guest	srv-vaultwarden — unprivileged LXC, CT 111 on rumba, 192.168.88.162, 2 cores / 2 GB / 12 GB local-lvm
Public URL	https://vault.isc-vs.ch — API public, /admin* behind the admin gate (group rack-admins)
Login	Keycloak realm isc only (SSO_ONLY), invite-only, plus a per-user master password (see below)
Runtime	the official container image under a systemd unit — upstream ships no other artifact
State	SQLite in /var/lib/vaultwarden (the whole vault), nightly consistent dump at 01:30
Managed from	provisioning/vaultwarden/ (idempotent deploy + bootstrap + Keycloak client scripts)
Credentials	/admin token in the secretzone (rumba page)

Keycloak logs you in, it does not unlock your vault

Bitwarden's model is zero-knowledge: the server stores ciphertext it cannot read, and the decryption key is derived from the master password, client-side. Keycloak proves who you are; it cannot produce that key. So every user has two secrets and both are needed:

1. edu-ID → Keycloak → Vaultwarden — authentication and the session.
2. A master password, chosen once per user, never sent to the server, and unrecoverable — lose it and the vault contents are gone. There is no admin reset that preserves the data.

Bitwarden removes the second secret with Key Connector, which is an enterprise-only, closed-source component: not available here, and not on any roadmap for Vaultwarden.

Who gets an account

SSO_ONLY=true + SIGNUPS_ALLOWED=false + INVITATIONS_ALLOWED=true: holding an edu-ID is not enough, and there is no email+master-password login to attack. An admin invites the address from /admin, and the person then signs in through Keycloak.

Nothing here can send mail — vsnet blocks outbound SMTP from the rack (why), so no SMTP_* is configured at all. Consequence: the invitation mail is never delivered and the invitee must be handed the URL by hand. Same for the "new device logged in" notices and emergency-access requests.

Who administers what

Three independent authorisation systems, all of which say "admin". They stack; none implies another.

Layer	Controls	Cannot do
rack-admins — a group in the Keycloak realm isc	whether you can reach /admin at all (the admin gate)	anything inside the panel — it only opens the door
/admin + ADMIN_TOKEN — Vaultwarden's server console (token only, no username, 20-minute sessions)	invite and delete users, delete organizations, server settings, diagnostics	read any secret. The server holds ciphertext it cannot decrypt
Organization roles — Owner / Admin / Manager / User, inside an organization	who reads and writes which shared secrets, per collection	touch personal vaults, which belong to their owner alone

The consequence worth remembering: the server console has no access to secret data. It can delete a user, never read their vault. All real access to shared secrets is organization membership, so an empty instance with a working /admin still shares nothing.

Onboarding a person

The order matters more than usual, because no mail leaves the rack.

1. Invite the address in /admin → Invite User. With SIGNUPS_ALLOWED=false this is what creates the right to have an account at all.
2. Tell them by hand — the invitation mail is never delivered. Send them https://vault.isc-vs.ch yourself.
3. They log in (ISC SSO → edu-ID) and set their master password. Do not skip ahead: an account that has never logged in has no keys, and step 5 cannot complete for it.
4. Invite them into the organization → Members → Invite member, with the collections they should see.
5. Confirm them. This is the step that looks like a no-op if you don't know about it: with mail disabled, an existing user with a master password is auto-accepted into Accepted status, and an owner still has to press Confirm in the member list before access is real.

Two failure modes that read as bugs

• Inviting an address that has no account yet from inside the organization fails with User does not exist: …. Invite through /admin first — that is why step 1 exists.
• A member stuck at Invited/Accepted sees no shared collections. They are not broken; nobody pressed Confirm (step 5).

First-time setup, once

Everything below happens in a browser at https://vault.isc-vs.ch, logged in as yourself. None of it can be scripted or done from /admin: an organization's key is generated client-side and wrapped with the creator's key, so there is no server-side path to it and the creator becomes Owner.

1. Log in (ISC SSO → edu-ID) and set your master password, if you have not already.
2. New organization → name it ISC³. The billing email it asks for is cosmetic here; use your own. Vaultwarden places no limits on organizations, members or collections.
3. Collections → create the four in the table below. Start with these rather than one flat collection: retro-fitting a split means re-filing every item by hand.
4. Settings → Policies → Account recovery administration → enable, and turn on auto-enroll so joining members are covered without having to remember a step. Read what that trades away first.
5. Move the first real secrets in, rotating each value as you go — anything that has been in docs/secretzone/ is in git history and on every clone, so copying it across preserves a leak.
6. Add the second person: onboarding, all five steps, Confirm included.
7. Set up your own browser extension and confirm the client SSO flow.
8. Write the break-glass set on paper — the vault cannot hold the credentials needed to reach the vault.

Sharing: organizations and collections

A personal vault is private by construction. Shared credentials therefore live in an organization, which owns collections, and members are granted access per collection.

An organization has to be created from a browser, by a logged-in user — its encryption key is generated client-side and wrapped with that user's key, so no script or admin token can do it, and the creator becomes Owner. Direct route: https://vault.isc-vs.ch/#/create-organization, or Settings → Organizations → New organization.

"Admin console" means two different things here

Vaultwarden's /admin (the ADMIN_TOKEN panel) can view and delete organizations but has no button to create one. Bitwarden's Admin Console — where Collections, Members and Policies live — is part of the ordinary web vault and only appears after an organization exists. Members are granted collections individually: ORG_GROUPS_ENABLED is false (upstream calls groups a beta with known issues), and ORG_CREATION_USERS is unset, so any user here may create an organization. ::: Sensible shape for this rack, one collection per blast radius:

Collection	Holds
Rack — network	MikroTik CCR/CRS, PDU, switch management
Rack — hypervisor & storage	rumba/iDRAC, PVE, PBS, the FS2500 NAS
Rack — services	Keycloak, Vaultwarden's own admin token, GitLab, Moodle
Accounts — external	Infomaniak, SWITCH/AAI registration, vendor portals

Splitting this way means a student assistant can be given Rack — network without ever seeing the hypervisor's break-glass root. One flat collection is simpler and gives everyone everything: fine for two people, not for a team that turns over yearly.

The vault cannot hold its own keys

Migrating docs/secretzone/ in here (the goal in secrets management) leaves a circular dependency: the credentials needed to reach the vault — the WireGuard/NetBird profile, rumba's root access, this instance's ADMIN_TOKEN — cannot live only inside it. That set has to stay outside, on paper or in a sealed offline copy.

Account recovery

Vaultwarden implements Bitwarden's account recovery (the ResetPassword organization policy, with optional auto-enrollment). It is the only answer to "a colleague forgot their master password", and without it that person's access is simply gone — including to shared collections.

How it behaves: an organization Owner enables the policy, members enrol (or are auto-enrolled on joining), and an Owner can then reset an enrolled member's master password. Enrolment is what makes it possible: the member's account key is wrapped for the organization at enrolment time, so a member who never enrolled cannot be recovered afterwards.

The trade-off, stated plainly: enrolling means the organization's owners can take over that account. That is a deliberate transfer of trust, and it applies to the member's personal items in that account too — not only the shared ones. For a team vault holding infrastructure credentials it is usually the right call; tell people it is on rather than letting them discover it.

Using it: browser extension and apps

The Bitwarden clients work unchanged, but the server URL has to be set before logging in — this is the step everyone misses, and the symptom is "invalid credentials" against Bitwarden's cloud.

1. Install the official Bitwarden extension (Firefox/Chrome/Edge) or app.
2. On the login screen, open the region/server selector and choose Self-hosted, then set Server URL to https://vault.isc-vs.ch. Save. (Desktop and mobile have the same setting; the extension keeps it per-profile.)
3. Log in through SSO — SSO_ONLY is set, so there is no email + master-password path.
4. Enter your master password to unlock. Every client asks for it separately: SSO authenticates the session, the master password derives the decryption key.

Client SSO, to confirm at first use

The web vault at https://vault.isc-vs.ch is the reference path and is known to work. For the official clients, the endpoint Bitwarden uses to resolve an SSO identifier from a domain (/api/organizations/domain/sso/details) is not implemented by Vaultwarden — it answers 404 — so if a client insists on an "SSO identifier" it will not be resolved for you; enter the organization identifier you set when creating the organization. Upstream also documents redirect trouble on the desktop client specifically (the browser has to hand the callback back to the app). Confirm the extension flow once and record what actually worked here.

OIDC wiring

Asserted by provisioning/vaultwarden/keycloak-client.sh, which runs on srv-keycloak and is idempotent:

Thing	Value	Why
Client	vaultwarden in realm isc, confidential, authorization-code + PKCE	—
Redirect URI	https://vault.isc-vs.ch/identity/connect/oidc-signin	not configurable — Vaultwarden derives it from its DOMAIN
SSO_AUTHORITY	https://sso.isc-vs.ch/realms/isc	must equal the issuer exactly: no trailing slash, no /.well-known/… suffix, or discovery fails on an issuer mismatch
Scopes	openid email profile offline_access	offline_access carries the refresh token that keeps a session alive past the access token
Access-token lifespan	900 s, set on the client	the realm default is 300 s and Bitwarden clients treat a token that expires in under ~5 min as already expired. Client-scoped on purpose: raising it realm-wide would also lengthen NetBird's tokens
trustEmail=true on the eduid provider	required	Keycloak marks every brokered user emailVerified=false unless the provider is trusted, and Vaultwarden refuses an SSO signup for an unverified address. edu-ID does verify the addresses it releases, so this is a true statement — the alternative, SSO_ALLOW_UNKNOWN_EMAIL_VERIFICATION, is documented upstream as an account-takeover risk. Users brokered before 2026-08-04 keep emailVerified=false and need it set by hand in the admin console

Publishing

TLS terminates on srv-web01 — vhost vault.isc-vs.ch in provisioning/web/Caddyfile; the container serves plain HTTP on the LAN and holds no certificate. The API stays public (the Bitwarden clients need /api, /identity and the /notifications WebSocket, which Caddy upgrades on its own). /admin* answers 403 outside 192.168.88.0/24 and the NetBird overlay — it holds the admin token, creates invitations and can change settings. Both sides verified on 2026-08-04: 403 from the Internet, 200 from the LAN.

/admin is gated by a login, not by the client's address: it goes through the admin gate on srv-web01, which requires membership of the Keycloak group rack-admins. It therefore works from anywhere with no VPN and no DNS special-casing. Note the two layers: the gate decides who may see the panel, the ADMIN_TOKEN is still what the panel itself asks for. A LAN/VPN break-glass branch remains for the case where Keycloak is down.

Backups

The container is in rumba's nightly 02:00 vzdump and the 03:00 PBS job, so the vault is covered with no extra action. On top of that a timer at 01:30 writes a consistent copy to /var/lib/vaultwarden/backups/db-<date>.sqlite3 (7 kept) using SQLite's own .backup API: a snapshot of a live SQLite file can restore torn, and a torn password vault is not a recoverable situation. The dumps sit inside the data directory, so they ride along in both backup jobs.

Operations

provisioning/vaultwarden/deploy-vaultwarden.sh              # (re)deploy, idempotent
VW_VERSION=1.37.2 provisioning/vaultwarden/deploy-vaultwarden.sh   # upgrade the pinned image
ssh root@rumba "pct exec 111 -- journalctl -u vaultwarden -f"      # logs
ssh root@rumba "pct exec 111 -- cat /root/vaultwarden-credentials.txt"   # /admin token
curl -s https://vault.isc-vs.ch/alive                       # liveness (returns a timestamp)

An upgrade is a tag bump: the script pulls the new image and restarts the unit; nothing in /var/lib/vaultwarden is touched.

Traps

• Docker needs nesting=1,keyctl=1 on the container. Both are set; without them dockerd will not start in an unprivileged LXC. Storage driver in use is overlay2 (checked at deploy time — if the kernel ever refuses unprivileged overlayfs, Docker silently falls back to vfs, which works but multiplies disk use per layer).
• vaultwarden hash needs a TTY. It reads the token with a terminal password prompt, so under pct exec it fails and the token would silently stay in cleartext. The bootstrap uses the argon2 CLI with Vaultwarden's own OWASP parameters (19 MiB, t=2, p=1) instead.
• /etc/vaultwarden.env is a docker --env-file, not a shell file. Quotes become part of the value and there is no expansion — a quoted SSO_SCOPES breaks the discovery request.
• "sso": "" in /api/config is a hardcoded placeholder, not a signal that SSO is off. To actually prove the chain, request the authorize endpoint and check the redirect:

  curl -sD - -o /dev/null "http://192.168.88.162/identity/connect/authorize?client_id=web&response_type=code" | grep -i location
  # → 307 to https://sso.isc-vs.ch/realms/isc/protocol/openid-connect/auth?...client_id=vaultwarden...