ISC³ network
ISC³ lives in an isolated separate network inside the school, reachable via the VPN. This page maps the appliances, subnets and physical connectivity.
Topology
Two MikroTik devices run the network: a CCR2004-1G-12S+2XS (core router: WAN, NAT, Wireguard hub) and a CRS326-24G-2S+ (access switch for the nodes and their iDRACs), joined by a 2×10 G LACP bond. Everything is bridged into a single flat L2 domain (no VLAN filtering) — the subnets below are logical separations only, routed on the two MikroTiks. The same routers also carry the Rumba subnets.
Topology verified by logging into both routers (July 2026), NAS and PDU ports re-verified against the
CCR2004 bridge host table in August 2026 — the NAS is on sfp-sfpplus6/7 (it was previously
documented as sfp+3) and the PDU on sfp-sfpplus10.
sfp-sfpplus10 runs at 100 Mb, and that is correctThe PDU's management NIC is 10/100 only. Don't "fix" it.
Subnets
| Subnet | Usage | Gateway |
|---|---|---|
192.168.88.0/24 | Network appliances (NAS, PDU), calypsomaster, rumba, epyc0/epyc1 | 192.168.88.1 (CCR2004) |
192.168.89.0/24 | VPN | 192.168.89.1 (CCR2004) |
192.168.90.0/24 | Server iDRACs (out-of-band) | 192.168.90.1 (CRS326) |
192.168.91.0/24 | Servers | 192.168.91.1 (CRS326) |
192.168.92.0/24 | Rumba iDRAC (out-of-band) | 192.168.92.1 (CCR2004) |
192.168.93.0/24 | legacy Rumba servers — unused since the July 2026 reinstall | 192.168.93.1 (CCR2004) |
192.168.94.0/24 | legacy Rumba LXC containers — unused since the July 2026 reinstall | 192.168.94.254 (CCR2004) |
The CCR2004 src-NATs all legacy WireGuard traffic towards 192.168.88.0/24 to 192.168.88.1 (rule "NAT Wireguard -> Interne"). Appliances on that subnet therefore cannot tell those VPN users apart — and per-IP protections lump everyone together. This once locked the whole admin team out of the NAS; 192.168.88.1 and 192.168.88.0/24 are since on the DSM allow list so that particular block cannot recur. Keep this NAT quirk in mind for any other per-source-IP security feature on the 88 subnet.
NetBird runs with masquerade off (since 2026-08-03): every
NetBird client reaches the rack with its own overlay address (100.65.x.x), so logs attribute
actions to individuals and per-IP protections act per user. The plumbing that makes this work: a
static return route on the CCR2004 (100.64.0.0/10 → 192.168.88.159, covers the CRS326 too via
its default route) and an input-chain accept for 100.64.0.0/10 on the CRS326 (its input
otherwise drops non-88/91 sources, which would hide the switch's management UI). Replies from
88-subnet appliances hairpin through the CCR (first packet is asymmetric); harmless because
neither MikroTik has a drop invalid/default-drop forward rule. The NAS auto-block therefore
applies per user on the NetBird path — deliberately, so do not add 100.64.0.0/10 to the
DSM allow list.
Storage fast path (nodes ↔ NAS)
Traffic between the 91 and 88 subnets is software-routed by the CRS326's single 800 MHz core
(RouterOS 6, no L3 offload on this switch chip), with the return path additionally hairpinning
through the CCR2004 — a ~39 MB/s ceiling that all NFS home traffic (nodes ↔ NAS) used to share.
The 2026-08-02 performance audit that found and
measured this is in the audits page. Since everything is one flat L2 domain, the fix (applied
2026-08-02) is on-link routes so both ends ARP each other directly and the packets never touch
a router CPU:
# NAS (Synology, sudo):
ip route replace 192.168.91.0/24 dev eth2 src 192.168.88.250
# each calypso node (sudo):
ip route replace 192.168.88.250/32 dev eno1
After the change, NFS writes went from the routed ceiling to 61.5 MB/s — the remaining gap to wire speed is NFS sync-write protocol overhead, not the network (before/after measurements in the history page).
Reads were first measured on 2026-08-03, when the route was accidentally lost and restored: 108–113 MB/s per node on the on-link path (the node's own 1 GbE port at wire speed) against 74–84 MB/s shared between all nodes when routed, with the CRS326's single core pinned at 97–100 %. Numbers and method in the NAS history page.
The routes are persisted (2026-08-02): a nas-fastpath.service systemd oneshot unit on each
node re-adds the /32 at boot (benign if it fails — traffic falls back to the slow routed path),
and /usr/local/etc/rc.d/nas-fastpath.sh on the NAS re-adds the node-subnet route at DSM boot.
Rollback is ip route del plus removing those two files.
Two known weaknesses, neither worth fixing: the NAS script runs only at DSM boot, so any runtime
network event drops the route for good (the node half survives, since eno1 keeps its address); and
calypso5–calypso10 were powered off when it was rolled out, so they have neither the route nor the
unit — provisioning/network/followups.exp is idempotent and catches them up if needed.
The real fix is already half done: eth3 on the NAS now holds a static 192.168.91.250, so once the
rebuilt Proxmox nodes mount that address (how a node must mount it)
the NAS is simply in the node subnet and none of this is needed. Note the per-node /32 is about how a
node reaches 192.168.88.250 — so it only goes away when the mounts are re-pointed, not merely
because the NAS gained an address.
Applied from the same audit (2026-08-02, counters verified): a fasttrack-connection rule now
heads the CCR2004 forward chain (established/related traffic skips the slow conntrack path), and
both ends of the bond use transmit-hash-policy=layer-3-and-4 so flows spread across both
10 G links (layer-2 hashing pinned ~everything to one link per direction).
Setting transmit-hash-policy on the live CCR2004 bond re-formed the LACP aggregate — observed
live on 2026-08-02 as ~30–60 s of total unreachability
for everything behind the bond. Plan any future hash/bond edits for a quiet window.
Still open (see the todo): CRS326 DHCP pool spans the whole /24 and hands
nodes 8.8.8.8 as DNS (so they cannot resolve .calypso names), device DNS points at the retired
172.30.7.1, RouterOS 6, and the CCR leftovers (Dude, L2TP server, the wifx.ch L2TP client).
Addressing
The 11 worker nodes currently running and accessible to students for their labs (hardware specs are on the Calypso page):
| Node | IP |
|---|---|
calypso0 | 192.168.91.10 |
calypso1 | 192.168.91.11 |
calypso2 | 192.168.91.12 |
calypso3 | 192.168.91.13 |
calypso4 | 192.168.91.14 |
calypso5 | 192.168.91.15 |
calypso6 | 192.168.91.16 |
calypso7 | 192.168.91.17 |
The last three addresses of that range now belong to the carnaval
playground cluster, which kept the addresses of the calypso8 – calypso10 machines it was built
from (2026-08-04):
| Node | IP | iDRAC |
|---|---|---|
carnaval0 | 192.168.91.18 | 192.168.90.18 |
carnaval1 | 192.168.91.19 | 192.168.90.19 |
carnaval2 | 192.168.91.20 | 192.168.90.20 |
The calypsomaster node hosts the Kubernetes control plane and is not available for connection to students:
| Node | IP |
|---|---|
calypsomaster | 192.168.88.248 |
calypsomaster iDRAC | 192.168.90.248 |
Other appliances on the network:
| Appliance | IP |
|---|---|
nas | 192.168.88.250 (plus 192.168.91.250 on its second 10 G port — why) |
pdu | 192.168.88.10 |
| router C (FS S3600-48T4S switch) | 192.168.88.2 |
The Rumba Proxmox node also lives on this subnet (rumba, 192.168.88.51, statically configured on vmbr0, attached to the CCR2004 on sfp+5), as do epyc0 / epyc1 (192.168.88.47 / .46, static DHCP reservations) — see the rack layout page. The Pagode workstation pair also lives here (192.168.88.20 / .21) when racked; the machines are momentarily removed.
DNS
The CCR2004 (192.168.88.1) is the DNS server for the whole network (DHCP domain calypso, upstream 8.8.8.8, remote requests allowed — it answers VPN clients too). Machines with a static DNS entry can be reached by name, both bare and with the .calypso suffix:
| Name | IP |
|---|---|
calypso-master / calypsomaster | 192.168.88.248 |
calypsomaster-idrac | 192.168.90.248 |
rumba | 192.168.88.51 |
rumba-idrac | 192.168.92.10 |
srv-web01 | 192.168.88.150 |
nas | 192.168.88.250 |
pdu | 192.168.88.10 |
epyc0 | 192.168.88.47 |
epyc1 | 192.168.88.46 |
calypso0 … calypso7 | 192.168.91.10 … .17 |
carnaval0 / 1 / 2 | 192.168.91.18 / .19 / .20 |
The CCR has no domain-suffix handling: rumba and rumba.calypso are two separate
/ip dns static records. Add both when registering a host, or the bare name will resolve while
the FQDN silently does not (which is how the carnaval entries were first added, 2026-08-04).
The SInf gateway (172.30.7.1) is the upstream WAN hop; it is no longer used as a DNS server.
Inbound from the Internet
Publicly, calypso.hevs.ch resolves to 153.109.29.7, which HES-SO/SInf maps to the CCR2004's WAN
address 172.30.7.2. Only two ports are published, both by dst-nat on the CCR2004:
| Port | Forwarded to | Serves |
|---|---|---|
| TCP 80 | 192.168.88.150 | ISC landing page — redirected to HTTPS by Caddy, and used for ACME challenges |
| TCP 443 | 192.168.88.150 | Same page over TLS (Let's Encrypt certificate for calypso.hevs.ch) |
UDP 51820 (Wireguard) is handled by the router itself, not by a forward.
Verified 2026-08-03 during the NetBird deployment: a dst-nat for UDP 3478 (STUN) counted zero packets across a client reconnect and a manual probe from outside — the filter is upstream at SInf/vsnet, not on the CCR2004. Inbound, the rack is reachable on exactly TCP 80/443 + UDP 51820, nothing else; adding a dst-nat for any other port is a no-op until SInf opens it. Details in When the network lies.
Both dst-nat rules shadow the MikroTik's own web interface: disabling them makes WebFig
reachable from the Internet on calypso.hevs.ch. To publish a different service, retarget them
(or better, add a vhost on srv-web01, which already terminates TLS) — never disable them.
Until 2026-07-31 they pointed at 192.168.94.13, the reverse-proxy container of the pre-migration
Rumba. That container disappeared with the reinstall, which left both ports pointing into a void
until they were retargeted.
Outbound to the Internet (what the upstream filters)
The SInf/vsnet uplink filters outbound traffic from the whole subnet — in ways that make ping,
dig and bare ssh actively misleading. The full failure catalogue with symptoms, wrong
diagnoses and the diagnostic toolbox is on its own page:
When the network lies. Summary of the facts, verified from rumba
(2026-08-01 during the ISC Learn restore,
SMTP re-checked 2026-08-02):
- ICMP to the Internet is dropped —
ping 8.8.8.8failing is normal, not a symptom. - Standard ports work: outbound 80/443 (and 22, per the GitLab tests) are fine.
- Non-standard ports are dropped: outbound TCP 20002 (hannibal's SSH) times out, while the
same host answers instantly from other networks. Test method:
curl portquiz.net:443(works) vscurl portquiz.net:20002(hangs) — portquiz listens on every port, so the difference is the uplink, not the target. - SMTP is dropped (25/465/587, verified 2026-08-02): no machine in the rack can reach an external smarthost or MX directly — the CCR2004 has no drop rules, the filter is upstream. Email from ISC³ therefore needs a vsnet egress-allow or a relay outside the filter; plan and options in the todo, Email section.
- IPv6 is a trap: DNS returns AAAA records but there is no IPv6 route, so clients that prefer
IPv6 fail with Network is unreachable. Force IPv4 (
ssh -4,curl -4).
Workaround when a machine inside must reach a filtered port outside: a reverse tunnel through an
admin laptop that sees both networks, e.g.
ssh -f -N -R 127.0.0.1:2222:target.example:20002 root@rumba — sustained ~20–25 MB/s during the
293 GB ISC Learn pull. Asking SInf to open a specific egress port is the long-term alternative.
Uplink bandwidth (measured 2026-08-01)
Measured from rumba with parallel curl -4 transfers over HTTPS against
speed.cloudflare.com (nearest PoP, so this is the local uplink capacity, not long-haul
throughput), cross-checked the same day against mirror.init7.net. Two runs a few hours apart
gave consistent numbers:
| Direction | 1 TCP stream | Aggregate ceiling |
|---|---|---|
| Download | ~1.2–1.6 Gbit/s | ~4.3 Gbit/s (8 streams ≈ 16 streams — flat) |
| Upload | ~0.6–0.75 Gbit/s | ~1.3–1.4 Gbit/s (4 streams; more doesn't help) |
The link is clearly asymmetric (~4.3 down / ~1.3 up), by upstream shaping or provisioning. Practical corollary: a single long-distance transfer should be expected to run at the single-stream figures, and the ~20–25 MB/s of the ISC Learn pull above was the laptop-relay bottleneck, not the uplink (~25× headroom downstream).
Both this network and the speed-test endpoints lie in their own ways (Cloudflare's silent
HTTP 429 budget above all) — the method notes and the exact scripts are in
History & operations.