Skip to main content

When the network lies — SInf uplink failure catalogue

The Calypso uplink is filtered by SInf/vsnet in ways that make the standard diagnostic reflexes — ping, dig, a bare ssh — actively misleading: they fail (or lie) for reasons that have nothing to do with the thing you are debugging. This page catalogues every such failure actually hit while operating the rack (July–August 2026), what it looked like, what it really was, and the reflex to adopt. Most entries cost real time before the cause was found; the point of this page is to never pay twice.

The five reflexes (TL;DR)​

  1. Never trust ping — ICMP to the Internet is dropped. Test reachability with TCP.
  2. Never trust dig — port 53 is intercepted; answers come from the local resolver's cache, negative entries included. Verify DNS with DNS-over-HTTPS.
  3. Always force IPv4 (ssh -4, curl -4) — AAAA records exist, an IPv6 route does not.
  4. Assume outbound non-standard ports are dropped until portquiz.net:<port> proves otherwise. That includes all SMTP ports (25/465/587) — no email leaves the rack directly.
  5. Inbound is TCP 80/443 + UDP 51820, full stop. A new dst-nat on the CCR2004 changes nothing until SInf opens the port upstream (verified 2026-08-03: UDP 3478 → zero packets). Design new services to ride 443 through srv-web01, or the WireGuard/NetBird tunnels over what's already open.

The catalogue​

1. ping fails to any Internet host​

  • Looks like: the whole network is down; a freshly created guest seems to have no connectivity.
  • Actually: ICMP to the Internet is filtered upstream for the whole subnet. apt, curl, everything else works fine.
  • Reflex: curl -4 -sI https://debian.org | head -1, or a raw TCP probe: timeout 3 bash -c 'echo > /dev/tcp/host/443' && echo open.

2. dig lies (port-53 interception)​

  • Looks like: a DNS record you just created "does not exist", even when querying dig @8.8.8.8 or dig @1.1.1.1 explicitly.
  • Actually: the network intercepts port 53 — any dig @server is answered by the local resolver from cache, negative entries included: a name probed before it existed keeps looking absent for up to an hour after it goes live.
  • Cost: this interacts viciously with Let's Encrypt — enabling a Caddy vhost before the name really resolves burns the 5 failed-validations per hostname per hour limit and delays the certificate (July 2026, static-sites deployment).
  • Reflex: 
    curl -s -H 'accept: application/dns-json' \
      "https://cloudflare-dns.com/dns-query?name=foo.isc-vs.ch&type=A"

3. The IPv6 AAAA trap — Network is unreachable​

  • Looks like: ssh: connect to host …: Network is unreachable after zero seconds — reads like a routing catastrophe or a dead target.
  • Actually: public DNS returns AAAA records, clients prefer IPv6, and the subnet has no IPv6 route. The target is fine.
  • Cost: the first attempt at the 293 GB ISC Learn pull died instantly on this (2026-08-01); a curl to a dual-stacked site can also just hang.
  • Reflex: -4 on ssh/curl/wget/rsync's transport, or fix per-host in /etc/gai.conf (prefer IPv4). Note the error text differs from the egress filter below — unreachable = IPv6 trap (instant), timed out = filtered port (slow).

4. Outbound non-standard TCP ports are silently dropped​

  • Looks like: the remote side is blocking you. ssh -4 -p 20002 learn.isc-vs.ch from rumba times out, while the same command works from a laptop on another network — everything points at a firewall on the target or its hoster.
  • Actually: the uplink drops outbound TCP to non-standard ports. Verified 2026-08-01 with portquiz (an echo server listening on every port): curl -4 portquiz.net:443 → instant 200, curl -4 portquiz.net:20002 → timeout. Ports proven to work outbound: 80, 443, 22 (GitLab runner tests), plus UDP 51820 (WireGuard) and 53/123 implicitly.
  • SMTP is filtered too (verified 2026-08-02 from rumba): outbound TCP 25, 465 and 587 all time out — to mail.infomaniak.com and to smtp.hevs.ch/mail.hevs.ch alike. The CCR2004 has no drop rules, so the block is upstream. Consequences: the rack cannot use any external smarthost directly (see the Email plan in the todo), and rumba's postfix silently defers everything — a mailnotification setting on a PVE job is a no-op here.
  • Cost: ~45 minutes auditing the innocent target (hannibal's ufw — inactive, fail2ban — no bans, nftables — INPUT ACCEPT) and wrongly suspecting an Infomaniak cloud firewall, before testing the egress itself.
  • Reflex: test your own egress first — curl -4 --max-time 5 portquiz.net:<port> — before diagnosing anyone else's firewall.

5. Cross-site SSH/replication is impossible without a relay — the structural problem​

The combination is what really hurts (this is the one to remember when designing anything cross-site):

  • Outbound: rumba cannot dial hannibal's SSH (20002 — filtered, see #4). Had SSH been on 22 it would have worked; the "security" port move backfires inside this network.
  • Inbound: only 80/443 (dst-nat to srv-web01) and UDP 51820 reach Calypso from outside — no direct SSH into the rack from another site, ever, except over the VPN. Other inbound UDP is dropped too: verified 2026-08-03 with a dst-nat for UDP 3478 during the NetBird STUN test — zero packets matched across a client reconnect and a manual probe.

Consequences, lived on 2026-08-01: the 293 GB ISC Learn restore data had to flow through a user-run reverse tunnel on an admin laptop that saw both networks (ssh -f -N -R 127.0.0.1:2222:learn.isc-vs.ch:20002 root@rumba, ~20–25 MB/s sustained, ~3 h). Workable once, not an architecture. For anything recurring (backup replication, PBS sync, monitoring of external boxes), the realistic options are:

  1. Standard ports on the far end (22/443) — cheapest fix, decide it at service-creation time;
  2. WireGuard site-to-site — the far side peers with the CCR2004 (UDP 51820 is open);
  3. A SInf request to open a specific egress port — slow, but permanent.

6. The backup NAS is on another island​

  • Looks like: isc-ds923.synology.me / 10.32.2.108 down — port 22 unreachable.
  • Actually: the DS923 (the only off-site copy of ISC Learn until 2026-08-01) sits on the school intranet, which is not routed from Calypso or the VPN. Nothing is wrong with it; there is simply no path.
  • Cost: the DR exercise could not restore from the backups — it had to pull read-only from prod instead. The DS923→restore path therefore remains untested, and refreshing the rack's own copy depends on prod being alive: exactly what a backup is supposed not to do.
  • Mitigation: an on-site copy now exists (rumba:/hdd/backup/hannibal-mirror/); a real DS923↔rack replication (e.g. via the FS2500) is on the todo list.

Copy-paste diagnostic toolbox​

# TCP reachability (ping is useless here)
timeout 3 bash -c 'echo > /dev/tcp/HOST/PORT' && echo open || echo closed/filtered

# Is it MY egress or THEIR firewall?  portquiz listens on all ports
curl -4 -s --max-time 5 portquiz.net:PORT && echo egress-ok || echo egress-filtered

# DNS truth, bypassing the interceptor
curl -s -H 'accept: application/dns-json' \
  "https://cloudflare-dns.com/dns-query?name=NAME&type=A"

# IPv6-trap-proof connection
ssh -4 ... / curl -4 ... / rsync -e "ssh -4 ..." ...

# uplink bandwidth (~4.3 Gbit/s down / ~1.3 up) — full method & the Cloudflare-429 trap:
# isc3_network-history.md § Uplink bandwidth measurement
curl -4 -s -o /dev/null -w '%{speed_download} B/s (HTTP %{http_code})\n' \
  'https://speed.cloudflare.com/__down?bytes=90000000'

Related: ISC³ network (topology, inbound rules, the outbound summary, measured uplink bandwidth), ISC Learn restore runbook (where most of this was learned the hard way).