Skip to main content

Incident log

Cross-cutting incidents on the ISC³ rack, newest first — what happened, the impact, the recovery path and what now prevents a recurrence. The incident process describes how to handle one as it unfolds; day-to-day changes are indexed in the journal. Standing failure modes of the uplink are not incidents and live in their own catalogue, When the network lies.

August 2026 — a NetBird restart destroyed its own database​

server.store.encryptionKey was empty in provisioning/netbird/config.yaml.template, and with that field empty the combined server mints a random data-store key at every start and never persists it. It says so on the first line of its own log, truncated so the key cannot be recovered afterwards:

DataStoreEncryptionKey is not set, generating a new key
DataStoreEncryptionKey generated (fyB0g3r4...); add it to your config file
  under 'server.store.encryptionKey' to persist across restarts

The server ran 3 days without a restart, so nothing surfaced. The first restart — done on 2026-08-05 while editing the store for an unrelated reason — replaced the key, and every field written under the old one became undecryptable:

error syncing peer: decrypt user: decrypt email: decrypt: cipher: message authentication failed

Impact: total VPN outage of about 90 minutes. Peer sync failed, so clients lost every route; the API rejected all tokens with 401 and the dashboard could not authenticate. Two accounts, three peers and the whole access model were lost. Nobody but the admin was using it yet.

Recovery: the legacy MikroTik WireGuard carried admin access throughout — the dual-path policy is the reason this was recoverable at all. Then: set a persistent key, move the undecryptable databases aside, bootstrap a new owner with POST /api/setup, and replay the access model from provisioning/netbird/config-snapshot.json with provisioning/netbird/replay-config.py. Peers re-enroll themselves.

Two traps met on the way, both worth knowing before touching this store again:

  • Neither a store.db restore nor a vzdump helps. The rows are encrypted with a key that only existed in the memory of a process that has exited, so every copy of the database is equally unreadable. Only re-creating the data works.
  • The embedded IdP has a second database. Wiping store.db left the same errors, because idp.db holds the Dex users and its rows are encrypted with the same key. Both have to go, which also means the local accounts and their passwords go.

Prevention: the template now renders @STORE_ENCRYPTION_KEY@, deploy-netbird.sh generates it once and never overwrites it, and that script now refuses to run against a config whose encryptionKey is empty rather than letting the next restart do this again. export-config.sh also exports identity-providers, which had been missing — the Keycloak login provider had to be reconfigured by hand because no snapshot held it.

Read what a server logs at first boot

The warning naming this exact failure was in the log from day one of the deployment and was never read. A first-boot log is the one place a service says what it needs to survive a restart.

July 2026 — NAS auto-block locked out the whole admin team​

The CCR2004 src-NATs all legacy-WireGuard traffic towards 192.168.88.0/24 to 192.168.88.1, so the Synology's per-IP protections saw every VPN user as that single address. Accumulated failed logins from VPN clients eventually put 192.168.88.1 on the NAS auto-block list: DSM web logins were refused and SSH reset at key exchange — for every admin at once, since they all arrived from the same NATed address.

Recovery: reach DSM from a non-NATed address and remove the entry — an SSH tunnel through calypsomaster, then the DSM UI through the tunnel:

ssh -N -L 15001:192.168.88.250:5001 root@192.168.88.248   # then https://localhost:15001

Prevention: 192.168.88.1 and 192.168.88.0/24 are on the DSM allow list, so the legacy path cannot trigger the auto-block again. On the NetBird path the problem is solved at the root: masquerade is off (since 2026-08-03), every client reaches the rack with its own overlay address, and the auto-block acts per user — deliberately, so do not add 100.64.0.0/10 to the DSM allow list. The NAT quirk itself is a standing rule in the network page's caution; assume the same risk for any per-source-IP protection on the 88 subnet.

June 2026 — server-room heat event​

The room overheated around June 22–23, 2026 (confirmed temperature alarm): the PDU probe recorded a maximum of 76.4 °C on 2026-06-23 02:28, and the PDU's power peak (3.63 kW vs ~1.1 kW typical — all fans at maximum) was logged 25 hours earlier. Every SSD in the NAS recorded a 62–63 °C maximum — the lasting footprint of the event. No machine shut down automatically.

Follow-up: a layered protection policy is proposed in Thermal protection & emergency shutdown; since August 2026 the rack status page keeps a 2-year temperature history and sends Telegram alerts when the PDU probe runs hot.