Skip to main content

ISC³ network

ISC³ lives in an isolated separate network inside the school, reachable via the VPN. This page maps the appliances, subnets and physical connectivity.

Topology​

Two MikroTik devices run the network: a CCR2004-1G-12S+2XS (core router: WAN, NAT, Wireguard hub) and a CRS326-24G-2S+ (access switch for the nodes and their iDRACs), joined by a 2×10 G LACP bond. Everything is bridged into a single flat L2 domain (no VLAN filtering) — the subnets below are logical separations only, routed on the two MikroTiks. The same routers also carry the Rumba subnets.

Topology verified by logging into both routers (July 2026), NAS and PDU ports re-verified against the CCR2004 bridge host table in August 2026 — the NAS is on sfp-sfpplus6/7 (it was previously documented as sfp+3) and the PDU on sfp-sfpplus10.

note
sfp-sfpplus10 runs at 100 Mb, and that is correct

The PDU's management NIC is 10/100 only. Don't "fix" it.

Subnets​

SubnetUsageGateway
192.168.88.0/24Network appliances (NAS, PDU), calypsomaster, rumba, epyc0/epyc1192.168.88.1 (CCR2004)
192.168.89.0/24VPN192.168.89.1 (CCR2004)
192.168.90.0/24Server iDRACs (out-of-band)192.168.90.1 (CRS326)
192.168.91.0/24Servers192.168.91.1 (CRS326)
192.168.92.0/24Rumba iDRAC (out-of-band)192.168.92.1 (CCR2004)
192.168.93.0/24legacy Rumba servers — unused since the July 2026 reinstall192.168.93.1 (CCR2004)
192.168.94.0/24legacy Rumba LXC containers — unused since the July 2026 reinstall192.168.94.254 (CCR2004)
Legacy-WireGuard users share one source IP (NetBird users no longer do)

The CCR2004 src-NATs all legacy WireGuard traffic towards 192.168.88.0/24 to 192.168.88.1 (rule "NAT Wireguard -> Interne"). Appliances on that subnet therefore cannot tell those VPN users apart — and per-IP protections lump everyone together. This once locked the whole admin team out of the NAS; 192.168.88.1 and 192.168.88.0/24 are since on the DSM allow list so that particular block cannot recur. Keep this NAT quirk in mind for any other per-source-IP security feature on the 88 subnet.

NetBird runs with masquerade off (since 2026-08-03): every NetBird client reaches the rack with its own overlay address (100.65.x.x), so logs attribute actions to individuals and per-IP protections act per user. The plumbing that makes this work: a static return route on the CCR2004 (100.64.0.0/10 → 192.168.88.159, covers the CRS326 too via its default route) and an input-chain accept for 100.64.0.0/10 on the CRS326 (its input otherwise drops non-88/91 sources, which would hide the switch's management UI). Replies from 88-subnet appliances hairpin through the CCR (first packet is asymmetric); harmless because neither MikroTik has a drop invalid/default-drop forward rule. The NAS auto-block therefore applies per user on the NetBird path — deliberately, so do not add 100.64.0.0/10 to the DSM allow list.

Storage fast path (nodes ↔ NAS)​

Traffic between the 91 and 88 subnets is software-routed by the CRS326's single 800 MHz core (RouterOS 6, no L3 offload on this switch chip), with the return path additionally hairpinning through the CCR2004 — a ~39 MB/s ceiling, measured by the 2026-08-02 performance audit. Since everything is one flat L2 domain, the fix applied then was on-link routes so both ends ARP each other directly and the packets never touch a router CPU (61.5 MB/s NFS writes, 108–113 MB/s reads per node — before/after numbers on the history page and the NAS history page).

What remains of it today: the NAS holds a static 192.168.91.250 on eth3, directly in the node subnet, and that is the address every node and lab VM must mount (how) — nothing in the 91 subnet routes to 192.168.88.250 any more. The NAS-side script /usr/local/etc/rc.d/nas-fastpath.sh (a 192.168.91.0/24 dev eth2 route re-added at DSM boot) is still installed; the per-node /32 route and its nas-fastpath.service went with the 2026-08-28 node rebuild. Whether the NAS-side script is still needed is a todo.

Applied from the same audit (2026-08-02, counters verified): a fasttrack-connection rule now heads the CCR2004 forward chain (established/related traffic skips the slow conntrack path), and both ends of the bond use transmit-hash-policy=layer-3-and-4 so flows spread across both 10 G links (layer-2 hashing pinned ~everything to one link per direction).

caution
A router's own replies need established,related in input, not just forward

The CRS326 had no Internet for an unknown length of time — not egress, NAT or the SInf uplink, but its own input chain ending in drop with no established,related accept, so answers to connections it opened were dropped by itself. The forward chain did have that rule, which is why everything behind the switch was online while the switch was not: NTP never synced and check-for-updates timed out, both looking exactly like an upstream block. Found and fixed 2026-08-09. When a MikroTik alone cannot reach out, read its input chain before blaming the uplink.

Changing the bond hash policy can restart the bond

Setting transmit-hash-policy on the live CCR2004 bond re-formed the LACP aggregate — observed live on 2026-08-02 as ~30–60 s of total unreachability for everything behind the bond. Plan any future hash/bond edits for a quiet window.

The CRS326's old DHCP/DNS liabilities are closed: the pool that spanned the whole /24 was narrowed and then both DHCP servers disabled outright (2026-08-05, nothing used them), and its device DNS points at the CCR since 2026-08-09 instead of the retired 172.30.7.1. Both MikroTiks run RouterOS 7.23.3 with matching RouterBOOT since 2026-08-09 — the CCR from 7.20.4, the CRS from 6.49.19 — and the CRS's DNS points at the CCR. The CCR's pre-WireGuard leftovers are gone: Dude disabled 2026-08-05, the L2TP server, its IPsec secret, the vpn PPP account and the wifx.ch L2TP client deleted 2026-08-09. Remote access is NetBird; the legacy WireGuard fleet was turned off on 2026-08-07 and keeps one enabled break-glass peer.

The break-glass WireGuard peer loses 40–95 % of its return packets (August 2026)

Measured 2026-08-21 on wg13, the only enabled peer: the client's packets reach the CCR2004, the router answers, and 40 to 95 % of the replies never arrive — in dead windows of about 14 s. Loss is independent of packet size and rate, the RTT of what survives is stable at 5 ms, and the router's own counters and CPU are clean. Reproduced from two different ISPs, so the client's home path is not involved. Until it is fixed, the documented emergency access does not work reliably: assume ssh and the Proxmox UI will time out over it. Story and measurements in the incident.

Addressing​

The eleven carnaval nodes — carnavalN is the ex-calypsoN chassis and kept its address (since 2026-08-28; carnaval5 is powered off):

NodeIPiDRAC
carnaval0 … carnaval10192.168.91.10 … .20 (.91.(10+N))192.168.90.10 … .20 (.90.(10+N))

The calypsomaster node is not available for connection to students. It still carries a kubeadm control plane, but nothing runs on it — its former workers are carnaval nodes (conversion plan):

NodeIP
calypsomaster192.168.88.248
calypsomaster iDRAC192.168.90.248

Other appliances on the network:

ApplianceIP
nas192.168.88.250 (plus 192.168.91.250 on its second 10 G port — why)
pdu192.168.88.10
router C (FS S3600-48T4S switch)192.168.88.2

The Rumba Proxmox node also lives on this subnet (rumba, 192.168.88.51, statically configured on vmbr0, attached to the CCR2004 on sfp+5), as does epyc1 (192.168.88.46, configured statically on the host — a Proxmox VE standalone node since August 2026 — with its BMC on 192.168.88.12). epyc0 (192.168.88.47) is in repair as of August 2026 and offline. Both reach the CCR2004 through the FS S3600-48T4S switch (router C) on sfp+12. See the rack layout page. The Tango workstation pair also lives here — the racked node on a dynamic 192.168.88.43 rather than the reserved .20 / .21 (why), the other momentarily out of the rack.

DNS​

The CCR2004 (192.168.88.1) is the DNS server for the whole network (DHCP domain isc3, upstream 8.8.8.8, remote requests allowed — it answers VPN clients too). Machines with a static DNS entry can be reached by name, bare or as <name>.isc3:

NameIP
calypso-master / calypsomaster192.168.88.248
calypsomaster-idrac192.168.90.248
rumba192.168.88.51
rumba-idrac192.168.92.10
srv-web01192.168.88.150
nas192.168.88.250
pdu192.168.88.10
epyc0192.168.88.47
epyc1192.168.88.46
epyc1-bmc192.168.88.12
carnaval0 … carnaval10192.168.91.10 … .20
Two records per host

The CCR has no domain-suffix handling: rumba and rumba.isc3 are two separate /ip dns static records. Add both when registering a host, or the bare name will resolve while the FQDN silently does not (which is how the carnaval entries were first added, 2026-08-04). provisioning/network/carnaval-dns.sh writes both for a carnaval node.

.isc3 replaced the .calypso domain on 2026-08-28 (decision) — the .calypso twins and the calypsoN names still exist on the router until removed (todo). The PVE hosts carry <name>.isc3 in /etc/hosts (rewritten 2026-08-28, certificates reissued).

The SInf gateway (172.30.7.1) is the upstream WAN hop; it is no longer used as a DNS server.

Inbound from the Internet​

Publicly, calypso.hevs.ch resolves to 153.109.29.7, which HES-SO/SInf maps to the CCR2004's WAN address 172.30.7.2. Only two ports are published, both by dst-nat on the CCR2004:

PortForwarded toServes
TCP 80192.168.88.150ISC landing page — redirected to HTTPS by Caddy, and used for ACME challenges
TCP 443192.168.88.150Same page over TLS (Let's Encrypt certificate for calypso.hevs.ch)

UDP 51820 (Wireguard) is handled by the router itself, not by a forward.

UDP 51820 is the only inbound UDP the uplink lets through

Verified 2026-08-03 during the NetBird deployment: a dst-nat for UDP 3478 (STUN) counted zero packets across a client reconnect and a manual probe from outside — the filter is upstream at SInf/vsnet, not on the CCR2004. Inbound, the rack is reachable on exactly TCP 80/443 + UDP 51820, nothing else; adding a dst-nat for any other port is a no-op until SInf opens it. Details in When the network lies.

These two rules also protect the router

Both dst-nat rules shadow the MikroTik's own web interface: disabling them makes WebFig reachable from the Internet on calypso.hevs.ch. To publish a different service, retarget them (or better, add a vhost on srv-web01, which already terminates TLS) — never disable them.

Until 2026-07-31 they pointed at 192.168.94.13, the reverse-proxy container of the pre-migration Rumba. That container disappeared with the reinstall, which left both ports pointing into a void until they were retargeted.

Outbound to the Internet (what the upstream filters)​

The SInf/vsnet uplink filters outbound traffic from the whole subnet — in ways that make ping, dig and bare ssh misleading. The full failure catalogue with symptoms, wrong diagnoses and the diagnostic toolbox is on its own page: When the network lies. Summary of the facts, verified from rumba (2026-08-01 during the ISC Learn restore, SMTP re-checked 2026-08-02):

  • ICMP to the Internet is dropped — ping 8.8.8.8 failing is normal, not a symptom.
  • Standard ports work: outbound 80/443 (and 22, per the GitLab tests) are fine.
  • Non-standard ports are dropped: outbound TCP 20002 (hannibal's SSH) times out, while the same host answers instantly from other networks. Test method: curl portquiz.net:443 (works) vs curl portquiz.net:20002 (hangs) — portquiz listens on every port, so the difference is the uplink, not the target.
  • SMTP: only 465 to Infomaniak — 25/465/587 were all dropped (2026-08-02) until SInf granted an egress-allow on request; verified 2026-08-10, it covers the whole 192.168.88.0/24 but only TCP 465 to Infomaniak's mail IPs. 25 and 587 remain dropped, as does 465 to any other provider. See Email.
  • IPv6 is a trap: DNS returns AAAA records but there is no IPv6 route, so clients that prefer IPv6 fail with Network is unreachable. Force IPv4 (ssh -4, curl -4).

Workaround when a machine inside must reach a filtered port outside: a reverse tunnel through an admin laptop that sees both networks, e.g. ssh -f -N -R 127.0.0.1:2222:target.example:20002 root@rumba — sustained ~20–25 MB/s during the 293 GB ISC Learn pull. Asking SInf to open a specific egress port is the long-term alternative.

Measured from rumba with parallel curl -4 transfers over HTTPS against speed.cloudflare.com (nearest PoP, so this is the local uplink capacity, not long-haul throughput), cross-checked the same day against mirror.init7.net. Two runs a few hours apart gave consistent numbers:

Direction1 TCP streamAggregate ceiling
Download~1.2–1.6 Gbit/s~4.3 Gbit/s (8 streams ≈ 16 streams — flat)
Upload~0.6–0.75 Gbit/s~1.3–1.4 Gbit/s (4 streams; more doesn't help)

The link is asymmetric (~4.3 down / ~1.3 up), by upstream shaping or provisioning. Practical corollary: a single long-distance transfer should be expected to run at the single-stream figures, and the ~20–25 MB/s of the ISC Learn pull above was the laptop-relay bottleneck, not the uplink (~25× headroom downstream).

Both this network and the speed-test endpoints lie in their own ways (Cloudflare's silent HTTP 429 budget above all) — the method notes and the exact scripts are in History & operations.