NAS

The rack's shared storage: a Synology FlashStation FS2500 (nas, 192.168.88.250), a 1U all-flash 12-bay NAS at U10 in the rack. It serves the Calypso student homes over NFS (jobs can run on any node against a common filesystem) and shares for Rumba.

At a glance

Model	Synology FlashStation FS2500 — 12-bay all-flash, 8 GB RAM
DSM	7.2.1-69057 Update 12
Volumes	1.7 TB Btrfs on RAID 5 (Calypso homes) + 6.7 TB Btrfs on RAID 6 (free), one hot spare
Network	two 10 GbE links since 2026-08-03 — see Network
Access	DSM web UI (https://192.168.88.250:5001) and SSH, over the VPN — credentials in the secretzone

Network

A second link was cabled on 2026-08-03 — the narrative, including the transceiver swap it took to reach 10 Gb, is in the history page. State after it (August 2026):

Interface	DSM	Speed	Address	Switch port	Notes
eth2	LAN 3	10 GbE	192.168.88.250/24 (DHCP reservation)	CCR2004 sfp-sfpplus6	DSM, SSH, rumba/PBS backups, calypsomaster
eth3	LAN 4	10 GbE	192.168.91.250/24 static, no gateway	CCR2004 sfp-sfpplus7	Cabled 2026-08-03; Calypso node NFS, directly in the node subnet
eth0, eth1	LAN 1, 2	1 GbE	—	—	Onboard Realtek RJ45, unused

One dedicated 10 G port per subnet, so node NFS and rumba's backup stream do not share a port. eth3 is deliberately given no gateway: that is what keeps it out of DSM's default-gateway election (see the danger box below). DSM also builds a per-interface policy-routing table for it, so egress follows the source address — replies from .88.250 leave on eth2, replies from .91.250 on eth3, with no conflict between the two 192.168.91.0/24 routes in the main table.

Verified from calypso0 (August 2026): 192.168.91.250 is on-link (dev eno1, no gateway), answers showmount, and accepts NFS on tcp/2049.

Measured throughput (August 2026, raw TCP, 8.4 GB single stream):

Path
rumba → NAS (eth2)	1.1 GB/s	8.9 Gbit/s — the backup direction
NAS (eth2) → rumba	938 MB/s	7.5 Gbit/s
node ↔ NAS (eth3)	108–113 MB/s	the node's 1 GbE port at wire speed, not a NAS limit

DSM labels the ports LAN 1–4 against eth0–eth3. When editing one, go by the address it currently shows rather than the label — the 10 GbE pair is easy to mix up, and picking the wrong one cuts off the DSM session you are working in.

eth2/eth3 are the two onboard 10 GbE ports (amd-xgbe); eth0/eth1 are separate 1 GbE Realtek NICs. They are not bonded, deliberately: bonding would give the pair one shared address and forfeit the per-subnet split, and redundancy is explicitly not the goal here — throughput and simplicity are. The remaining step is re-pointing the node mounts, in todo → NAS item 2.

Never fill in a gateway on a second interface

DSM re-elects the default-gateway interface on every link or DHCP event (/etc/iproute2/config/gateway_v4_priority), putting the most recently addressed one first — which sends all NAS egress, including the NFS return path, out of whichever port was touched last. eth3 is immune only because it is static with the gateway field blank. Keep it that way, and note eth0/eth1 are still BOOTPROTO=dhcp, so cabling either of them re-opens the same hole.

Related: 192.168.88.250 is not configured on the NAS at all — it is a DHCP reservation on the CCR2004 keyed to eth2's MAC, so that address follows the port, not the box.

Until the nodes are rebuilt, the fast-path route can vanish

Nodes still mounting 192.168.88.250 depend on the on-link storage fast path, which is re-added only by /usr/local/etc/rc.d/nas-fastpath.sh — at DSM boot and nowhere else. Any link flap deletes it (losing an address drops every route referencing it) and nothing puts it back, silently costing ~40 % of read throughput. After physical work behind the NAS, check ip route get 192.168.91.10 from 192.168.88.250 says dev eth2, and if not:

sudo ip route replace 192.168.91.0/24 dev eth2 src 192.168.88.250

This disappears once the nodes mount 192.168.91.250 instead — see mounting.

How a node must mount it

This is the rule for every current and future Calypso node — the rebuilt fleet will be Proxmox nodes, so it applies to a PVE host or to whatever guest actually consumes the share:

1. Use 192.168.91.250 — never nas, never 192.168.88.250. The 88 address is reserved for DSM, rumba and PBS backups; using it from a node puts node traffic back on the wrong port and re-creates the need for a routing workaround.
2. Omit sync. The exports have been async since 2026-08-05, which took writes from 65 to 102–109 MB/s — wire speed on a 1 GbE node, matching reads. A client-side sync on top of that costs almost nothing (measured), so the pre-rebuild nodes' rw,sync,soft is not worth an emergency edit — just leave sync out of anything new.
3. Define the mount exactly once. The pre-rebuild nodes had it twice in /etc/fstab (/exports and /exports/).
4. Do not install nas-fastpath.service, or any 192.168.88.250/32 route. Those existed only to work around the NAS being in a different subnet, which is no longer true.
5. Use NFSv4.1 — v3 is not a fallback. The homes directory carries a Synology ACL. Over v4.1 it is honoured and the directory reads drwxrwxrwx; over v3 only the POSIX mode is visible, it is 000, and every user is denied. Both measured 2026-08-04.

showmount -e 192.168.91.250 reports one export, /volume1/calypso_homes, permitted to 192.168.88.248/32 and 192.168.91.0/24.

Host-level mount (/etc/fstab, or a systemd .mount unit):

192.168.91.250:/volume1/calypso_homes/homes  /exports  nfs  rw,soft,vers=4.1,_netdev  0 0

danger

The share root's guest ACL entry is what makes this mount possible — never remove it

The node rule squashes root to guest (uid 1025), and an NFSv4 mount of a subdirectory makes the server walk /volume1 → calypso_homes → homes as that squashed identity. guest therefore needs traverse rights on the share root, held by a single non-inheriting entry added 2026-08-05 (why):

user:guest:allow:r-x---a-R-c--:---n   on /volume1/calypso_homes

Remove it and every new mount fails with access denied by server while the already-mounted ones keep working — so the breakage stays invisible until the next reboot. Keep it non-inheriting (---n): propagating it would hand guest read access to homes and to every student directory.

caution

async exports: a crash can lose work the client thinks is safe

Both rules have carried async since 2026-08-05 — the NAS acks a write once it is in RAM, not on flash. That is where the write throughput came from, and it is the normal NFS trade, but it means a NAS crash or power cut can lose recently written student work, with no fsync from the client able to prevent it. The UPS does not cover the NAS (it powers rumba and nothing else), so the exposure is a building power cut, not just a kernel panic. Revisit if calypso_homes ever holds something that cannot be redone — protecting it is a separate open item.

note

ls as root on a fresh mount says "Permission denied" — that is root_squash, not a fault

Access it as a user whose UID matches the owner; the homes are mode 750, and NFS authorises by numeric UID. See NFS exports.

If instead the share is registered as Proxmox storage (for guest disks or dumps), add it with pvesm rather than per-node fstab, so it lands in the cluster-wide /etc/pve/storage.cfg and every node inherits it:

pvesm add nfs <storage-id> --server 192.168.91.250 \
    --export /volume1/calypso_homes/homes --options vers=4.1,soft

This share is the student homes — it is not a general-purpose store

There is exactly one export, and it holds the only copy of the student homes. Registering it as a Proxmox iso,vztmpl storage would create ISO and template directories inside calypso_homes, which is why the carnaval cluster deliberately has no NAS storage. Shared ISOs need their own export first.

The export allows the whole node subnet, with root squashed

192.168.91.0/24 is permitted with root_squash, so a PVE host mounting this as root cannot write as root. That is deliberate — see NFS exports. If a future guest genuinely needs unsquashed access, add a specific /32 rule for it rather than relaxing the subnet rule.

Disks and arrays

All twelve bays are populated with SATA SSDs. Layout after the 2026-08-02 rebuild:

Bays	Disks	Array	Tolerates	Backs
sata1–5	5× Synology SAT5210 — 480 GB each	RAID 5 (md2)	1 failure	Volume 1
sata6–11	6× Toshiba THNSN81Q92CSE — 1.92 TB each	RAID 6 (md3)	2 failures	Volume 2
sata12	1× Toshiba THNSN81Q92CSE — 1.92 TB	hot spare	—	both pools (vg1 + vg2), auto-replacement on

The DSM system partition (md0, 8 GB) is mirrored (RAID 1) across all twelve SSDs, so the OS survives any disk loss.

Disk health

SMART read on every disk (August 2026):

Set	Power-on hours	Wear	Errors
Volume 1 — 5× SAT5210	≈ 15 700 h (1.8 y)	1 % used — ≈ 8 TB written per disk	none
Volume 2 — 7× Toshiba THNSN8 (6 + spare)	≈ 64 500 h (7.4 y)	≈ 1.5 % erase-count wear	none (35 unexpected power losses, power-loss protection intact)

The Toshibas predate the NAS by years: seven identical drives with identical history is a correlated-failure profile, which is exactly why Volume 2 was moved to double parity plus a spare (same reasoning as Rumba's RAIDZ2 choice). Wear is not the concern — age is. Every one of them also recorded a 62–63 °C maximum, the footprint of the June 2026 heat event.

Data scrubbing

A recurring scrub schedule is enabled on both pools (vg1, vg2) in Storage Manager → Data Scrubbing, anchored 2026-08-02. The first-ever verification runs and their results are in the history page.

Don't schedule a scrub for a hot month

Scrubbing reads every disk at full speed. During the June 2026 heat event the rack drew 3.6 kW with all fans at maximum and these disks logged 62–63 °C, so keep the quarterly window in a cool month and outside teaching hours.

Volumes and shares

Volume	Filesystem	Usable	Used (Aug 2026)	Shares
Volume 1	Btrfs	1.7 TB	428 GB (26 %)	calypso_homes (student homes), homes (DSM user homes)
Volume 2	Btrfs	6.7 TB	~35 GB (Aug 2026)	none, and by decision none coming (2026-08-04): the volume is dedicated to the VMM repository with the PBS VM (32 GB system + 5 TiB thin datastore disks). The old rumba_shares/rumba_backup shares died with the rebuild

NFS exports

All exports are sec=sys (identity is whatever UID the client claims). State after the 2026-08-02 hardening:

Export	Allowed clients	Root	Purpose
/volume1/calypso_homes	192.168.91.0/24	squashed	Student homes (71 as of July 2026), mounted on all Calypso nodes
/volume1/calypso_homes	192.168.88.248/32	not squashed	calypsomaster — home provisioning

These two rules are now the only exports on the device: the /volume2/rumba_shares export disappeared with the Volume 2 rebuild.

Both rules are rw, crossmnt and — deliberately — sync; the node rule squashes root to guest with anonuid=1025,anongid=100. Read the live truth with sudo cat /etc/exports over SSH; DSM's Squash dropdown maps to root_squash/no_root_squash and "Enable asynchronous" to async.

The masks, the root squashing and the homes-directory ACL are the result of the 2026-08-02 hardening; the share root's non-inheriting user:guest:allow:r-x entry, without which no new node mount can be created, was added 2026-08-05 and is described with the mounting rules. One caveat survives the hardening: the share root still carries an inherited fd delete-capable ACL entry, so any new top-level folder created there inherits delete rights.

Never propagate ACLs to the student homes

The 71 home directories are plain POSIX 750 directories with no ACL. Applying a shared-folder ACL recursively would replace that with the inherited group:users:allow:rwx… entry and let every student read every other student's home. Always apply permission changes to the folder itself only.

VPN users share one source IP

The core router NATs all legacy-WireGuard traffic to 192.168.88.1, so DSM's per-IP protections see every one of those VPN users as a single address — this once auto-blocked the whole admin team. The full story and recovery path are in the incident log.

How the export is mounted on the Calypso nodes (paths, ~/nas_home symlink) is described on the Calypso page.

Next steps

Open items for the NAS are tracked in the ops todo → NAS.