Skip to main content

ISC³ network — history & operations

Dated audits and measurement campaigns on the ISC³ network, newest first. The resulting configuration and the standing rules live on the main page; this page keeps the how-we-got-there narratives and the measurement methods. Standing failure modes of the uplink are in their own catalogue, When the network lies.

Performance audit (2026-08-02)​

The audit found the trunk links healthy — every inter-switch link at 10 G, the LACP bond up, node→node TCP at full 1 GbE wire speed (110 MB/s measured) — but traffic between the 91 and 88 subnets was software-routed by the CRS326's single 800 MHz core (RouterOS 6, no L3 offload on this switch chip), with the return path additionally hairpinning through the CCR2004. Measured cost: 38.7 MB/s calypso0→rumba versus 110 MB/s node→node on the very same links. All NFS home traffic (nodes ↔ NAS) shared that ceiling.

The fix — the on-link routes now documented on the main page — was applied the same day. After the change, calypso0's ARP entry for the NAS is direct (dev eno1 … REACHABLE) and an 800 MB NFS write with conv=fsync went from the routed ceiling to 61.5 MB/s — the remaining gap to wire speed is NFS sync-write protocol overhead, not the network.

Two router-side changes were applied from the same audit (counters verified): the fasttrack-connection rule and the layer-3-and-4 bond hash policy, both described in the storage fast path. Setting transmit-hash-policy on the live CCR2004 bond re-formed the LACP aggregate; with lacp-rate=30secs that meant ~30–60 s where everything behind the bond (CRS326, all nodes) was unreachable — observed live on 2026-08-02. The later CRS326-side change caused no observable blip, and re-setting an unchanged value is a no-op — but plan any future hash/bond edits for a quiet window.

The results and their practical corollary are on the main page. Method notes, because this network and the test endpoints both lie:

  • speedtest-style clients and anything ICMP-based are useless here (see the failure catalogue); plain curl -4 over 443 is the reliable probe.
  • Cloudflare's endpoint caps each request just under 100 MB and rate-limits by transferred bytes per window: past the budget it returns an instant 1-byte HTTP 429, which makes a naive script report absurd speeds (or 26 B/s). Always print %{http_code} and treat sub-second "transfers" as invalid. The budget takes tens of minutes to refill; the upload budget is separate from the download one.
  • proof.ovh.net throttles ~3.5 Mbit/s per connection — useless for a multi-gigabit link. mirror.init7.net serves ~1.5 Gbit/s per stream but fair-queues aggregates to ~2.7 Gbit/s, below the uplink ceiling.

The runnable probe is provisioning/network/uplink-speedtest.sh.