Skip to main content

When the network lies — SInf uplink failure catalogue

The rack's SInf uplink is filtered by SInf/vsnet in ways that make the standard diagnostic reflexes — ping, dig, a bare ssh — misleading: they fail (or lie) for reasons that have nothing to do with the thing you are debugging. This page catalogues every such failure hit while operating the rack (July–August 2026), what it looked like, what it really was, and the reflex to adopt. Most entries cost real time before the cause was found; this page records them so the diagnosis is not repeated.

The five reflexes (TL;DR)​

  1. Never trust ping — ICMP to the Internet is dropped. Test reachability with TCP.
  2. Never trust dig — port 53 is intercepted; answers come from the local resolver's cache, negative entries included. Verify DNS with DNS-over-HTTPS.
  3. Always force IPv4 (ssh -4, curl -4) — AAAA records exist, an IPv6 route does not.
  4. Assume outbound non-standard ports are dropped until portquiz.net:<port> proves otherwise. SMTP is the one negotiated exception: TCP 465 and 587 to Infomaniak's mail IPs only (opened by SInf on request, August 2026 — see Email); both ports stay dropped to any other host, and 25 is dropped everywhere.
  5. Inbound is TCP 80/443 + UDP 51820. A new dst-nat on the CCR2004 changes nothing until SInf opens the port upstream (verified 2026-08-03: UDP 3478 → zero packets). Design new services to ride 443 through srv-web01, or the WireGuard/NetBird tunnels over what's already open.

The catalogue​

1. ping fails to any Internet host​

  • Looks like: the whole network is down; a freshly created guest seems to have no connectivity.
  • Actually: ICMP to the Internet is filtered upstream for the whole subnet. apt, curl, everything else works fine.
  • Reflex: curl -4 -sI https://debian.org | head -1, or a raw TCP probe: timeout 3 bash -c 'echo > /dev/tcp/host/443' && echo open.

2. dig lies (port-53 interception)​

  • Looks like: a DNS record you just created "does not exist", even when querying dig @8.8.8.8 or dig @1.1.1.1 explicitly.
  • Actually: the network intercepts port 53 — any dig @server is answered by the local resolver from cache, negative entries included: a name probed before it existed keeps looking absent for up to an hour after it goes live.
  • Cost: this interacts badly with Let's Encrypt — enabling a Caddy vhost before the name really resolves burns the 5 failed-validations per hostname per hour limit and delays the certificate (July 2026, static-sites deployment).
  • Reflex: 
    curl -s -H 'accept: application/dns-json' \
      "https://cloudflare-dns.com/dns-query?name=foo.isc-vs.ch&type=A"

3. The IPv6 AAAA trap — Network is unreachable​

  • Looks like: ssh: connect to host …: Network is unreachable after zero seconds — reads like a routing failure or a dead target.
  • Actually: public DNS returns AAAA records, clients prefer IPv6, and the subnet has no IPv6 route. The target is fine.
  • Cost: the first attempt at the 293 GB ISC Learn pull died instantly on this (2026-08-01); a curl to a dual-stacked site can also just hang.
  • Reflex: -4 on ssh/curl/wget/rsync's transport, or fix per-host in /etc/gai.conf (prefer IPv4). Note the error text differs from the egress filter below — unreachable = IPv6 trap (instant), timed out = filtered port (slow).

4. Outbound non-standard TCP ports are silently dropped​

  • Looks like: the remote side is blocking you. ssh -4 -p 20002 learn.isc-vs.ch from rumba times out, while the same command works from a laptop on another network — everything points at a firewall on the target or its hoster.
  • Actually: the uplink drops outbound TCP to non-standard ports. Verified 2026-08-01 with portquiz (an echo server listening on every port): curl -4 portquiz.net:443 → instant 200, curl -4 portquiz.net:20002 → timeout. Ports proven to work outbound: 80, 443, 22 (GitLab runner tests), plus UDP 51820 (WireGuard) and 53/123 implicitly.
  • SMTP was filtered too, and is the one restriction we got lifted: 25/465/587 all timed out on 2026-08-02, then SInf opened 465 to Infomaniak on request (verified 2026-08-10) and 587 to Infomaniak as well (verified from rumba 2026-08-11 — details and scope on the Email page). Still dropped: 25 to anything, and 465/587 to any other host (smtp.gmail.com, smtp.hevs.ch, portquiz.net) — so there is a second port to the same provider, but no second provider. rumba's postfix was repointed at the relay by node-relay.sh (Email); a host that has not had that run still direct-to-MXes on port 25 and silently defers, and a legacy mailnotification field on a PVE job remains a no-op — only notification-mode notification-system reaches the smarthost.
  • Cost: ~45 minutes auditing the innocent target (hannibal's ufw — inactive, fail2ban — no bans, nftables — INPUT ACCEPT) and wrongly suspecting an Infomaniak cloud firewall, before testing the egress itself.
  • Reflex: test your own egress first — curl -4 --max-time 5 portquiz.net:<port> — before diagnosing anyone else's firewall.

5. Cross-site SSH/replication is impossible without a relay — the structural problem​

The combination is what matters (this is the one to remember when designing anything cross-site):

  • Outbound: rumba cannot dial hannibal's SSH (20002 — filtered, see #4). Had SSH been on 22 it would have worked; the "security" port move backfires inside this network.
  • Inbound: only 80/443 (dst-nat to srv-web01) and UDP 51820 reach the rack from outside — no direct SSH into the rack from another site except over the VPN. Other inbound UDP is dropped too: verified 2026-08-03 with a dst-nat for UDP 3478 during the NetBird STUN test — zero packets matched across a client reconnect and a manual probe.

Consequences, lived on 2026-08-01: the 293 GB ISC Learn restore data had to flow through a user-run reverse tunnel on an admin laptop that saw both networks (ssh -f -N -R 127.0.0.1:2222:learn.isc-vs.ch:20002 root@rumba, ~20–25 MB/s sustained, ~3 h). That worked as a one-off. For anything recurring (backup replication, PBS sync, monitoring of external boxes), the realistic options are:

  1. Standard ports on the far end (22/443) — cheapest fix, decide it at service-creation time;
  2. WireGuard site-to-site — the far side peers with the CCR2004 (UDP 51820 is open);
  3. A SInf request to open a specific egress port — slow, but permanent.

6. The backup NAS is on another island​

  • Looks like: isc-ds923.synology.me / 10.32.2.108 down — port 22 unreachable.
  • Actually: the DS923 (the only off-site copy of ISC Learn until 2026-08-01) sits on the school intranet, which is not routed from the rack or the VPN. Nothing is wrong with it; there is no path.
  • Cost: the DR exercise could not restore from the backups — it had to pull read-only from prod instead. The DS923→restore path therefore remains untested, and refreshing the rack's own copy depends on prod being alive.
  • Mitigation: an on-site copy now exists (rumba:/hdd/hannibal-mirror/); a real DS923↔rack replication (e.g. via the FS2500) is on the todo list.

Copy-paste diagnostic toolbox​

# TCP reachability (ping is useless here)
timeout 3 bash -c 'echo > /dev/tcp/HOST/PORT' && echo open || echo closed/filtered

# Is it MY egress or THEIR firewall?  portquiz listens on all ports
curl -4 -s --max-time 5 portquiz.net:PORT && echo egress-ok || echo egress-filtered

# DNS truth, bypassing the interceptor
curl -s -H 'accept: application/dns-json' \
  "https://cloudflare-dns.com/dns-query?name=NAME&type=A"

# IPv6-trap-proof connection
ssh -4 ... / curl -4 ... / rsync -e "ssh -4 ..." ...

# uplink bandwidth (~4.3 Gbit/s down / ~1.3 up) — full method & the Cloudflare-429 trap:
# isc3_network-history.md § Uplink bandwidth measurement
curl -4 -s -o /dev/null -w '%{speed_download} B/s (HTTP %{http_code})\n' \
  'https://speed.cloudflare.com/__down?bytes=90000000'

Related: ISC³ network (topology, inbound rules, the outbound summary, measured uplink bandwidth), ISC Learn restore runbook (where most of this was learned the hard way).