Skip to main content

ISC³ network — history & operations

Dated audits and measurement campaigns on the ISC³ network, newest first. The resulting configuration and the standing rules live on the main page; this page keeps the how-we-got-there narratives and the measurement methods. Standing failure modes of the uplink are in their own catalogue, When the network lies.

The .isc3 alias domain (2026-08-16)​

The internal domain gained a parallel alias: every .calypso static record on the CCR2004 now has an .isc3 twin (41 of each), added over the RouterOS API by provisioning/network/isc3-alias.sh (idempotent, rerunnable). The NetBird calypso-internal nameserver group matches both domains — changed over the API rather than the dashboard (incident), with export-config.sh diffing only that hunk afterwards. This is an alias, not a rename: the Proxmox node FQDNs (rumba.calypso, carnaval0–2.calypso) and the DHCP domain stay calypso.

The same pass set the 72 legacy WireGuard peers' client-dns to 192.168.88.1 (isc3-alias.sh --wg-dns); they carried 172.30.7.1, which cannot answer the internal names. That field only feeds configs the router generates — already-issued client confs keep their local DNS = line until reissued (todo). wg-gen.sh now emits DNS = 192.168.88.1, isc3, calypso, and the 13 deploy-script call sites pass --searchdomain "isc3 calypso" (PVE joins the list into one search line; verified against PVE::LXC::Setup::Base::set_dns on rumba). Existing guests keep search calypso — bare names resolve, so nothing to migrate.

CRS326 from RouterOS 6.49.19 to 7.23.3 (2026-08-09)​

Four hops in one afternoon — 6.49.19 → 6.49.20 (security fix) → 7.21.5 → 7.23.3 — each one reboot, ~40 s of downtime apiece for the nodes and every iDRAC, with RouterBOOT flashed to match at the end. The v6 → v7 jump is the one that mattered: on a v6 device only the testing channel offers v7 at all, and landing on it flips the channel back to stable, which then serves 7.23.3 like any other point release. That last hop changed nothing in the config but a name=client1 on the dhcp-client. The v7 config conversion came through intact: 25 bridge ports with their two pvids, the LACP bond, the seven addresses, nine firewall rules, both (disabled) DHCP servers, RoMON, and no rule or address flagged invalid. What the export gained is v7 boilerplate — ingress-filtering and path-cost on every bridge port, and default /routing ospf, /ip smb, /interface lte sections.

Two things had to be fixed first, and both were mis-diagnosed at length before they were:

  • The switch could not reach the Internet — not egress, NAT or the SInf uplink, but its own input chain ending in drop with no established,related accept, so replies to connections it opened were dropped by itself. The rule now lives on the network page.
  • The upgrade was called impossible on free-space arithmetic — 2.0 MiB free of 16 MiB against a 12.1 MiB package. Wrong: the updater stages the package, and free space afterwards was unchanged. Rule on the backup & restore page.

Only stable/long-term were checked at first, and both answer "already up to date" on a v6 device — v7 is offered exclusively on the testing channel, which is the whole reason WinBox showed no package. Landing on v7 flips it back to stable, which then offers 7.23.3 normally.

CCR2004 firmware upgrade (2026-08-09)​

RouterOS 7.20.4 → 7.23.3 and RouterBOOT 7.20.2 → 7.23.3, in that order (the bootloader is flashed by /system routerboard upgrade and only takes effect on the next reboot). Two reboots, each under a minute of downtime, run at 01:30 on a Sunday with one WireGuard peer enabled.

Everything came back: 74 Wireguard peers, 8 NAT rules, 65 static DNS entries, the LACP bond, and end-to-end proof through the whole chain (vault.isc-vs.ch answering 200 from the Vaultwarden LXC via the srv-web01 reverse proxy). The .rsc diff across the upgrade is almost entirely export syntax — switch1/switch2 naming, suppress-hw-offload and /port no longer exported. One change is worth knowing:

  • fasttrack-connection lost its hw-offload=yes — the parameter no longer exists in 7.23 (/ip/firewall/filter/get … hw-offload errors), offload having become implicit. The 2026-08-02 fix is intact: the rule counted 1 141 packets in the ten minutes after the reboot, and the dummy passthrough rule shows traffic bypassing the chain.

The CRS326 made the same jump later that day, from 6.49.19, so both MikroTiks now run 7.23.3.

Performance audit (2026-08-02)​

The audit found the trunk links healthy — every inter-switch link at 10 G, the LACP bond up, node→node TCP at full 1 GbE wire speed (110 MB/s measured) — but traffic between the 91 and 88 subnets was software-routed by the CRS326's single 800 MHz core (RouterOS 6, no L3 offload on this switch chip), with the return path additionally hairpinning through the CCR2004. Measured cost: 38.7 MB/s calypso0→rumba versus 110 MB/s node→node on the same links. All NFS home traffic (nodes ↔ NAS) shared that ceiling.

The fix — the on-link routes now documented on the main page — was applied the same day. After the change, calypso0's ARP entry for the NAS is direct (dev eno1 … REACHABLE) and an 800 MB NFS write with conv=fsync went from the routed ceiling to 61.5 MB/s — the remaining gap to wire speed is NFS sync-write protocol overhead, not the network.

Two router-side changes were applied from the same audit (counters verified): the fasttrack-connection rule and the layer-3-and-4 bond hash policy, both described in the storage fast path. Setting transmit-hash-policy on the live CCR2004 bond re-formed the LACP aggregate; with lacp-rate=30secs that meant ~30–60 s where everything behind the bond (CRS326, all nodes) was unreachable — observed live on 2026-08-02. The later CRS326-side change caused no observable blip, and re-setting an unchanged value is a no-op — but plan any future hash/bond edits for a quiet window.

The results and their practical corollary are on the main page. Method notes, because this network and the test endpoints both lie:

  • speedtest-style clients and anything ICMP-based are useless here (see the failure catalogue); plain curl -4 over 443 is the reliable probe.
  • Cloudflare's endpoint caps each request just under 100 MB and rate-limits by transferred bytes per window: past the budget it returns an instant 1-byte HTTP 429, which makes a naive script report wrong speeds (or 26 B/s). Always print %{http_code} and treat sub-second "transfers" as invalid. The budget takes tens of minutes to refill; the upload budget is separate from the download one.
  • proof.ovh.net throttles ~3.5 Mbit/s per connection — useless for a multi-gigabit link. mirror.init7.net serves ~1.5 Gbit/s per stream but fair-queues aggregates to ~2.7 Gbit/s, below the uplink ceiling.

The runnable probe is provisioning/network/uplink-speedtest.sh.