Skip to main content

Incident log

Cross-cutting incidents on the ISC³ rack, newest first — what happened, the impact, the recovery path and what now prevents a recurrence. The incident process describes how to handle one as it unfolds; day-to-day changes are indexed in the journal. Standing failure modes of the uplink are not incidents and live in their own catalogue, When the network lies.

2026-09-07 — marcellus flagged as malware source, network cut by Infomaniak​

Infomaniak's abuse desk reported ov-6a5a0e.ch2.infomaniak.ch (marcellus) connecting from source port 36002 to 216.218.185.162, tagged tinba, and cut the VPS off the network. The destination is a Shadowserver sinkhole (216-218-185-162.sinkhole.shadowserver.org), so the report means a process on marcellus resolved a domain from a malware family's list and connected to it. Tinba is a Windows banking trojan; nothing Windows runs on marcellus, so the question was what relayed or imitated it.

What happened. The ikarus-sozialearbeit-vs.ch WordPress still carried the Duplicator installer from its July 2021 migration in the web root: installer.php, dup-installer/ with main.installer.php and the 15 MB dup-database__*.sql dump (site content and the admin accounts; the site has no user accounts), reachable unauthenticated since 2021. The Apache log shows GET /installer.php on 2026-09-02 at 19:02:40 UTC from 45.94.31.96, a second request one second later with a TelegramBot user agent (the link had been pasted in a Telegram chat), and the installer's bootlog and csrf files rewritten at that minute. The installer rewrites wp-config.php and re-imports the database with any admin account, which gives code execution as www-data. Its state files also date from 2026-07-06, and wp-config.php was rewritten together with dup-orig-wp-config on 2025-11-24: the installer had been run at least twice before. On 2026-09-05 21:33 and 2026-09-06 05:44, www-data dropped a 6.3 MB PyInstaller binary /tmp/python2.64 and five _MEI* extraction directories (Python 3.5 runtime, libcrypto, termios); that program made the sinkhole connection. No SSH login since 2026-08-10, no cron entry, no process left running once the network was cut.

Impact. Every marcellus site offline from the block (2026-09-07, still blocked at the time of writing): ikarus, inf1 and advpro WordPress, the LoRa/ChirpStack server, the TISC editor, Grafana. The ikarus database has been downloadable by anyone for five years; it holds public content and the admin accounts only (site-specific password), so no third-party personal data is involved.

Recovery. Investigation and containment through the Infomaniak noVNC console, the only path left (provisioning/tools/novnc-type.ps1 to type reliably from a Swiss keyboard). Payload, _MEI* directories, installer.php, the Duplicator archive and dup-installer/ moved to /root/quarantine-2026-09-07/ (kept for Infomaniak, not deleted); the ikarus vhost disabled; ProxyRequests Off in the four vhosts that had on (unrelated to the intrusion, closed while there). Unblock request to Infomaniak with the post-mortem in secretzone/marcellus-postmortem-2026-09-07.md.

Prevention. Marcellus is not to be cleaned but discarded: the intrusion predates November 2025, three WordPress sites share www-data, PHP 7.3/7.4 are end-of-life and the host has no firewall (ops todo). Hannibal, which shares the history (same admins, the same ProxyRequests on copied into its reverse-proxy vhosts), was hardened the same day: accounts pruned, SSH forwarding and root login off, Docker ports closed, 308 packages updated, automatic security updates on (hannibal, log). Standing rule for any WordPress we host: a migration installer (Duplicator, All-in-One WP Migration, wp-installer.php) is deleted the day the migration ends.

2026-09-03–08 — carnaval8's NVMe controller hung, freezing the GPU lab for five days​

The second Swissbit N-30m2 did what the first one did on 2026-08-06. On 2026-09-03 at 23:31 the card carrying fast-vm on carnaval8 started timing out (I/O tag 242 … QID 15 timeout), two controller resets failed (Device not ready, CSTS=0x1), the device was disabled at 23:37 (-19, ENODEV) and ZFS suspended the pool. The node had been in service seven hours, since gpu8 was created that afternoon.

Impact. gpu8 (VM 1168) froze with every write blocked: 1026 processes in D state, load average 1027, 877 stacked systemd-journald instances from a restart loop and 693 waiting CRON processes. It answered ping and its QEMU agent from RAM, so it read as alive, and SSH authenticated and then hung — which is why Ctrl+C did nothing (the client forwards the byte to a pty whose processes cannot take a signal; Enter ~ . is the way out). The node itself stayed up and quorate, root being on the PERC array. Nothing student-owned was on the pool: the two templates and that one lab VM, all rebuildable, homes being on the NAS.

It went five days unnoticed, against 17 h in August, for the reasons already known — no SEL entry for an add-in NVMe, and nothing alerts on pool suspension. Recovery on 2026-09-08 followed the standing sequence unchanged: kill -9, sync -f /, cold power cycle from the iDRAC. The card re-enumerated at 894.3 GB, the pool imported ONLINE and a scrub found 0 errors in 11 s.

Both cards of the pair have now hung. The August decision to keep the first one in service rested on a clean SMART report; the card that hung this time reads just as clean afterwards — Critical Warning 0x00, 0 media errors, 0 error-log entries, 0 % used at 7398 power-on hours. Power-on hours identify which card is which, since no serial had been recorded: the one now in carnaval9 is the August card, and the one that hung here is the carnaval0 card that August put "on watch". carnaval8 is powered off and the GPU lab moved to carnaval9 as gpu9; whether the cards stay in service is open.

2026-08-21 — the break-glass WireGuard peer loses 40–95 % of its return packets​

Reported as "the web is slow" on an admin laptop. The web was not slow: 394 Mbit/s, 5 ms latency, TLS in 25 ms. DNS was stalling 1 to 11 s per name, because the legacy WireGuard tunnel pushes client-dns=192.168.88.1 and Windows gives its adapter interface metric 5 against 25 for the wired link — so every lookup went to the rack resolver, over a tunnel dropping half its packets. The resolver itself answers in 25 ms; the multi-second figures were the client's retransmission ladder.

The loss is on wg13, the only peer left enabled after the fleet was turned off on 2026-08-07. It runs in dead windows of about 14 s, alternating with 10 to 15 s of normal traffic, and reaches 40 to 95 %. It is independent of packet size (flat from 32 to 1380 bytes) and of rate (46.7 % at 1 pps, 51 % at 5 pps), survives a tunnel restart, and hits every protocol — 8 of 45 TLS handshakes to the NAS failed. The RTT of surviving packets is stable at 5 ms with a mdev under 0.4 ms, so it is not congestion.

Direction is the router → client half. SNMP counters on vpn_calypso over 120 echos of 1000 bytes: inbound matches all 120 sent, outbound is far above what the 64 received replies account for. The client's tunnel adapter reports zero receive discards and zero receive errors, so the replies are not arriving and being rejected. Reproduced identically from a second ISP (phone hotspot, a different public address and NAT), which excludes the client's home path.

Ruled out by measurement: MTU and fragmentation, rate dependence, two devices sharing a key, router CPU (0 % on four cores), and the disabled test_user peer whose allowed-address is a /8. The CCR2004's own counters are clean — sfp28-2 without a single error. The remaining candidates are the router's WireGuard instance and the SInf/vsnet edge treatment of that UDP 51820 flow outbound.

Impact: the documented emergency access path is unreliable, which matters because the 2026-08-15 NetBird outage was recovered over exactly this peer. No user-facing service was affected — NetBird was up throughout.

Recovery: none needed for the reporter. Installing NetBird on the laptop resolved the symptom completely — 0 % loss to the rack, general DNS at 0.03 s — because NetBird carries client↔rack traffic over the TCP 443 relay instead of UDP 51820, and installs split-DNS NRPT rules that keep general lookups away from the rack resolver. The peer remains broken; open item in todo → VPN and identity.

Note for the next diagnosis: the uplink catalogue warns that ICMP is unreliable here, so the loss was confirmed in UDP (40 % of DNS queries) and TCP (the failed handshakes) as well; the ICMP that was measured travelled inside the tunnel, below the uplink's filter.

2026-08-15 — NetBird down after rumba's reboot: the no-hairpin hosts entry did not survive​

The first rumba reboot since NetBird's deployment (why rumba was rebooted) took the VPN's rack access down for ~40 minutes: the agent — the routing peer for every rack subnet — could not re-enroll because cloud-init (manage_etc_hosts: true) had regenerated /etc/hosts at boot, dropping the no-hairpin entry 192.168.88.150 vpn.isc-vs.ch; it dialed the public IP, which the LAN cannot reach. Recovery ran over the legacy-WireGuard break-glass, which the NetBird client kept disarming — that rule is a caution on the NetBird page.

Prevention: the entry now also lives in /etc/cloud/templates/hosts.debian.tmpl, which survives the regeneration; provisioning/netbird/fix-hairpin-hosts.sh restores both copies and re-ups the agent.

2026-08-10 — the UPS left the USB bus, NUT blind (open)​

The day after the 7 h wedge, the same link failed again: at 22:51:39 the kernel logged usb 1-13: USB disconnect and the UPS disappeared from the bus — lsusb -d 0665:5161 finds nothing and /sys/bus/usb/devices/1-13 is gone. upsc reports Data stale, nutdrv_qx loops insufficient permissions on everything, and upsmon declares the UPS dead every 5 s.

This is a different failure from 2026-08-09 and the documented recovery does not apply: a usbreset 0665:5161 needs an enumerated device, and there is none. The parent was the xHCI root hub directly (port 13 on 0000:00:14.0), so the only remote option would be resetting the whole controller — every USB device on bus 1, including the iDRAC-emulated keyboard/video — which was judged not worth it on a production hypervisor. No remote lever remains.

2026-08-11 — both ends reseated, neither brought it back. The UPS end first, then the rumba end (rear panel, left, upper). Neither produced a kernel event; the last USB line is still that disconnect. Port 13 reports state: not attached, disable=0, early_stop=no, over_current_count=0 and no xHCI error, so the host is listening and nothing is presenting itself. The kernel starts enumeration only when the device pulls up D+, so silence at replug puts the fault downstream of the port.

With both connectors reseated, what is left is the cable, the UPS's own USB port, or port 13 being electrically dead despite reporting healthy. A known-good USB device in port 13 separates the last from the first two (todo). The unit does have a DB9, so a serial link is the alternative to repairing this one.

2026-08-15 — the controller reset happened after all, and changed nothing. rumba was rebooted, which resets the whole xHCI controller — the UPS still did not enumerate. Stuck host-side controller state is ruled out; the three candidates above stand (next-time checklist).

Impact while it lasts: the UPS still protects the load, but nothing observes it. Since both of rumba's PSUs are on this UPS and NUT is monitor-only, a mains failure now means rumba runs until the battery is flat with no clean shutdown. The rack status page shows stale UPS data; thermal alerting is unaffected.

Not caused by the mail work of the same evening, though the timing invited the suspicion — the node-relay.sh run at 22:49:52 reloaded upsmon (ExecReload = upsmon -c reload, exit 0), the unit has no Requires/BindsTo on the driver, and no driver restart is logged between 22:48 and 22:54. A upsmon reload cannot touch USB.

One thing it did expose: NOTIFYFLAG NOCOMM had just been given EXEC, and upsmon re-warns NOCOMM every NOCOMMWARNTIME (300 s) — three mails in ten minutes, on course for ~120 overnight. NOCOMM is now syslog-only; COMMBAD carries the one-shot "communication lost" mail instead (email).

2026-08-09 — the UPS's USB link wedged, blinding NUT for 7 h​

On 2026-08-09 at 05:03 the rack UPS stopped answering NUT. Both of its USB interfaces died at once — nutdrv_qx looping insufficient permissions on everything ~6 500 times an hour, and every HID report returning EPROTO — while the device stayed enumerated, its node stayed correctly root:nut, and the kernel logged nothing. upsmon's last reading carried CAL, so the unit looks to have wedged during a self-test; the days before it had already been flapping Communications with the UPS re-established.

Impact was monitoring only, for 7 h 28 min: no UPS tile on the status page, no load and no autonomy figure. The UPS itself carried the load throughout (OL, 15 %, 56 min of runtime on the HID interface once readable again), and since upsmon runs powervalue 0 nothing could have been shut down by it.

Recovery was a usbreset on the device and a driver restart. The misleading error — it is a wedged USB stack, never a permissions problem — and the two checks that tell them apart in seconds are on ups-monitoring.

Telegram alerted after ~5 minutes as designed. Netdata did not and structurally cannot: its upsd_local_rackups.* charts vanish when collection stops, taking all three UPS alarms with them (why). They reappeared on their own once upsd answered again.

2026-08-06 — carnaval1's NVMe controller hung, suspending all its guest storage​

On 2026-08-06 at 18:06 the Swissbit N-30m2 carrying fast-vm on carnaval1 stopped answering. Two kernel reset attempts failed (Device not ready, CSTS=0x1), the device was disabled at 18:11:36 (-19, ENODEV) and ZFS suspended the pool. It stayed that way ~17 h, until it was noticed the next morning.

Impact was limited to that node's guests: VM 1150 (a personal GPU test VM) froze — still answering ping from RAM, every disk access hung — and a pile of zvol_tq threads plus txg_sync sat in D state, load average 38. The node itself stayed up and quorate, the cluster kept 2/3 votes, and root was never at risk: it lives on the PERC array, not the card. The pool held 17.8 GB — the two templates and one throwaway lab VM, all rebuildable from carnaval-guests.sh — so nothing student-owned and nothing worth recovering.

Not a worn-out drive. No AER errors, and SMART read after recovery is clean across the board: Critical Warning 0x00, 0 media errors, 0 error-log entries, 0 % used, 6347 power-on hours. The controller locked up and logged nothing about it. carnaval0 carries the identical card on the same firmware ARR50002, so treat this as a mode that node can hit too.

Recovery, 2026-08-07: kill -9 the kvm process, sync -f /, then a cold power cycle from the iDRAC — off, ~50 s with power removed, on. The card re-enumerated at its full 894 GB, the pool imported ONLINE, and a scrub found 0 errors in 17 s: the "5 data errors" reported while suspended were in-flight writes that never landed, not damage on disk. VM 1150 restarted with its A2 passthrough working and a clean filesystem. Full sequence and why a normal reboot is not part of it: cluster page.

Two things let it run 17 h unnoticed, both already known gaps rather than new findings: the iDRAC cannot see an add-in NVMe (no FQDD — the same reason the BIOS cannot boot it), so its SEL stayed empty, and nothing alerts on a failing disk. A ZED mail on pool suspension is the cheap half of that and is on the todo.

2026-08-06 — deleting two NetBird groups deleted five policies​

Cleaning up after the group-model collapse (2026-08-06 evening), the api-issued groups teachers-manual / students-manual were deleted in the dashboard while still listed as policy sources — and NetBird deletes every policy that references a deleted group, silently. Five of the seven policies vanished; for about an hour every VPN user could reach only what mgmt-access grants (the iDRACs, for mgmt-users) — nothing else, noticed by the two people connected that evening. Recovery: replay-config.py <PAT> recreated the five policies (and the two groups) from the committed config-snapshot.json in one run — the snapshot's first real use as a recovery path, minutes instead of re-clicking the whole access model. Prevention: the trap is a standing rule on the service page, and any future group retirement edits the policies first, then deletes the group.

2026-08-05 — a NetBird restart destroyed its own database​

server.store.encryptionKey was empty in provisioning/netbird/config.yaml.template, and with that field empty the combined server mints a random data-store key at every start and never persists it. It says so on the first line of its own log, truncated so the key cannot be recovered from it:

DataStoreEncryptionKey is not set, generating a new key
DataStoreEncryptionKey generated (fyB0g3r4...); add it to your config file
  under 'server.store.encryptionKey' to persist across restarts

The server ran 3 days without a restart, so nothing surfaced. The first restart — done on 2026-08-05 while editing the store for an unrelated reason — replaced the key, and every field written under the old one became undecryptable:

error syncing peer: decrypt user: decrypt email: decrypt: cipher: message authentication failed

Impact: total VPN outage of about 90 minutes. Peer sync failed, so clients lost every route; the API rejected all tokens with 401 and the dashboard could not authenticate. Two accounts, three peers and the whole access model were lost. Nobody but the admin was using it yet.

Recovery: the legacy MikroTik WireGuard carried admin access throughout — the dual-path policy is the reason this was recoverable. Then: set a persistent key, move the undecryptable databases aside, bootstrap a new owner with POST /api/setup, and replay the access model from provisioning/netbird/config-snapshot.json with provisioning/netbird/replay-config.py. Peers re-enroll themselves.

Two traps met on the way, both worth knowing before touching this store again:

  • Neither a store.db restore nor a vzdump helps. The rows are encrypted with a key that only existed in the memory of a process that has exited, so every copy of the database is equally unreadable. Only re-creating the data works.
  • The embedded IdP has a second database. Wiping store.db left the same errors, because idp.db holds the Dex users and its rows are encrypted with the same key. Both have to go, which also means the local accounts and their passwords go.

Prevention: the template now renders @STORE_ENCRYPTION_KEY@, deploy-netbird.sh generates it once and never overwrites it, and that script now refuses to run against a config whose encryptionKey is empty rather than letting the next restart do this again. export-config.sh also exports identity-providers, which had been missing — the Keycloak login provider had to be reconfigured by hand because no snapshot held it.

The failure was announced in the first-boot log

The warning naming this exact failure was in the log from day one of the deployment and was never read. Check a service's first-boot log for what it needs to survive a restart.

July 2026 — NAS auto-block locked out the whole admin team​

The CCR2004 src-NATs all legacy-WireGuard traffic towards 192.168.88.0/24 to 192.168.88.1, so the Synology's per-IP protections saw every VPN user as that single address. Accumulated failed logins from VPN clients eventually put 192.168.88.1 on the NAS auto-block list: DSM web logins were refused and SSH reset at key exchange — for every admin at once, since they all arrived from the same NATed address.

Recovery: reach DSM from a non-NATed address and remove the entry — an SSH tunnel through calypsomaster, then the DSM UI through the tunnel:

ssh -N -L 15001:192.168.88.250:5001 root@192.168.88.248   # then https://localhost:15001

Prevention: 192.168.88.1 and 192.168.88.0/24 are on the DSM allow list, so the legacy path cannot trigger the auto-block again. On the NetBird path the problem is solved at the root: masquerade is off (since 2026-08-03), every client reaches the rack with its own overlay address, and the auto-block acts per user — deliberately, so do not add 100.64.0.0/10 to the DSM allow list. The NAT quirk itself is a standing rule in the network page's caution; assume the same risk for any per-source-IP protection on the 88 subnet.

2026-06-22–23 — server-room heat event​

The room overheated around June 22–23, 2026 (confirmed temperature alarm): the PDU probe recorded a maximum of 76.4 °C on 2026-06-23 02:28, and the PDU's power peak (3.63 kW vs ~1.1 kW typical — all fans at maximum) was logged 25 hours earlier. Every SSD in the NAS recorded a 62–63 °C maximum — a lasting record of the event. No machine shut down automatically.

Follow-up: a layered protection policy is proposed in Thermal protection & emergency shutdown; since August 2026 the rack status page keeps a 2-year temperature history and sends Telegram alerts when the PDU probe runs hot.