NAS
The rack's shared storage: a Synology FlashStation FS2500 (nas, 192.168.88.250), a 1U all-flash 12-bay NAS at U10 in the rack. It serves the student homes over NFS to the carnaval lab VMs and shares for Rumba.
Disk, volume, SMART and UPS events are emailed through the rack relay
(192.168.88.163:25, sender mailer@isc-vs.ch), verified end to end. nas-dsm-email.sh sets the
transport; the recipient must be added in Control Panel → Notification → Email — DSM 7 stores it
as a separate recipient profile that the config file's eventmails does not feed, so it cannot be
scripted that way.
At a glance
| Model | Synology FlashStation FS2500 — 12-bay all-flash, 8 GB RAM |
| DSM | 7.2.1-69057 Update 12 |
| Volumes | 1.7 TB Btrfs on RAID 5 (student homes) + 6.7 TB Btrfs on RAID 6 (carnaval library, PBS datastore — see Volumes and shares), one hot spare |
| Network | two 10 GbE links since 2026-08-03 — see Network |
| Access | DSM web UI (https://192.168.88.250:5001) and SSH, over the VPN — credentials in the secretzone |
Network
A second link was cabled on 2026-08-03 — the narrative, including the transceiver swap it took to reach 10 Gb, is in the history page. State after it (August 2026):
| Interface | DSM | Speed | Address | Switch port | Notes |
|---|---|---|---|---|---|
eth2 | LAN 3 | 10 GbE | 192.168.88.250/24 (DHCP reservation) | CCR2004 sfp-sfpplus6 | DSM, SSH, rumba/PBS backups, calypsomaster |
eth3 | LAN 4 | 10 GbE | 192.168.91.250/24 static, no gateway | CCR2004 sfp-sfpplus7 | Cabled 2026-08-03; node/lab-VM NFS, directly in the node subnet |
eth0, eth1 | LAN 1, 2 | 1 GbE | — | — | Onboard Realtek RJ45, unused |
One dedicated 10 G port per subnet, so node NFS and rumba's backup stream do not share a port.
eth3 is deliberately given no gateway: that is what keeps it out of DSM's default-gateway
election (see the danger box below). DSM also builds a per-interface policy-routing table for it, so
egress follows the source address — replies from .88.250 leave on eth2, replies from .91.250 on
eth3, with no conflict between the two 192.168.91.0/24 routes in the main table.
Verified from a node in the 91 subnet (August 2026): 192.168.91.250 is on-link (dev eno1, no gateway),
answers showmount, and accepts NFS on tcp/2049.
Measured throughput (August 2026, raw TCP, 8.4 GB single stream):
| Path | ||
|---|---|---|
rumba → NAS (eth2) | 1.1 GB/s | 8.9 Gbit/s — the backup direction |
NAS (eth2) → rumba | 938 MB/s | 7.5 Gbit/s |
node ↔ NAS (eth3) | 108–113 MB/s | the node's 1 GbE port at wire speed, not a NAS limit |
DSM labels the ports LAN 1–4 against eth0–eth3. When editing one, go by the address it currently
shows rather than the label — the 10 GbE pair is easy to mix up, and picking the wrong one cuts off
the DSM session you are working in.
eth2/eth3 are the two onboard 10 GbE ports (amd-xgbe); eth0/eth1 are separate 1 GbE
Realtek NICs. They are not bonded, deliberately: bonding would give the pair one shared address and
forfeit the per-subnet split, and redundancy is not the goal here — throughput and simplicity are.
The remaining step is re-pointing the node mounts, in
todo → NAS item 1.
DSM re-elects the default-gateway interface on every link or DHCP event
(/etc/iproute2/config/gateway_v4_priority), putting the most recently addressed one first — which
sends all NAS egress, including the NFS return path, out of whichever port was touched last.
eth3 is immune only because it is static with the gateway field blank. Keep it that way, and
note eth0/eth1 are still BOOTPROTO=dhcp, so cabling either of them re-opens the same hole.
Related: 192.168.88.250 is not configured on the NAS — it is a DHCP reservation on the
CCR2004 keyed to eth2's MAC, so that address follows the port, not the box.
How a node must mount it
This is the rule for everything in the node subnet that mounts the share — a carnaval PVE host or the lab VM that actually consumes it:
- Use
192.168.91.250— nevernas, never192.168.88.250. The88address is reserved for DSM,rumbaand PBS backups; using it from a node puts node traffic back on the wrong port and re-creates the need for a routing workaround. - Omit
sync. The exports have beenasyncsince 2026-08-05, which took writes from 65 to 102–109 MB/s — wire speed on a 1 GbE node, matching reads. A client-sidesyncon top of that costs almost nothing (measured), so the pre-rebuild nodes'rw,sync,softis not worth an emergency edit — just leavesyncout of anything new. - Define the mount exactly once. The pre-rebuild nodes had it twice in
/etc/fstab(/exportsand/exports/). - Do not install
nas-fastpath.service, or any192.168.88.250/32route. Those existed only to work around the NAS being in a different subnet, which is no longer true. - Use NFSv4.1 — v3 is not a fallback. The
homesdirectory carries a Synology ACL. Over v4.1 it is honoured and the directory readsdrwxrwxrwx; over v3 only the POSIX mode is visible, it is000, and every user is denied. Both measured 2026-08-04.
showmount -e 192.168.91.250 reports one export, /volume1/isc3_homes, permitted to
192.168.88.248/32 and 192.168.91.0/24.
Host-level mount (/etc/fstab, or a systemd .mount unit):
192.168.91.250:/volume1/isc3_homes/homes /exports nfs rw,soft,vers=4.1,_netdev 0 0
guest ACL entry is what makes this mount possible — never remove itThe node rule squashes root to guest (uid 1025), and an NFSv4 mount of a subdirectory makes the
server walk /volume1 → isc3_homes → homes as that squashed identity. guest therefore needs
traverse rights on the share root, held by a single non-inheriting entry added 2026-08-05
(why):
user:guest:allow:r-x---a-R-c--:---n on /volume1/isc3_homes
Remove it and every new mount fails with access denied by server while the already-mounted ones
keep working — so the breakage stays invisible until the next reboot. Keep it non-inheriting
(---n): propagating it would hand guest read access to homes and to every student directory.
ls as root on a fresh mount says "Permission denied" — that is root_squash, not a faultAccess it as a user whose UID matches the owner; the homes are mode 750, and NFS authorises by
numeric UID. See NFS exports.
If instead the share is registered as Proxmox storage (for guest disks or dumps), add it with
pvesm rather than per-node fstab, so it lands in the cluster-wide /etc/pve/storage.cfg and every
node inherits it:
pvesm add nfs <storage-id> --server 192.168.91.250 \
--export /volume1/isc3_homes/homes --options vers=4.1,soft
There is exactly one export, and it holds the only copy of the student homes. Registering it as a
Proxmox iso,vztmpl storage would create ISO and template directories inside isc3_homes,
which is why the carnaval cluster deliberately has no NAS storage. Shared
ISOs need their own export first.
192.168.91.0/24 is permitted with root_squash, so a PVE host mounting this as root cannot write as
root. That is deliberate — see NFS exports. If a future guest genuinely needs
unsquashed access, add a specific /32 rule for it rather than relaxing the subnet rule.
Disks and arrays
All twelve bays are populated with SATA SSDs. Layout after the 2026-08-02 rebuild:
| Bays | Disks | Array | Tolerates | Backs |
|---|---|---|---|---|
sata1–5 | 5× Synology SAT5210 — 480 GB each | RAID 5 (md2) | 1 failure | Volume 1 |
sata6–11 | 6× Toshiba THNSN81Q92CSE — 1.92 TB each | RAID 6 (md3) | 2 failures | Volume 2 |
sata12 | 1× Toshiba THNSN81Q92CSE — 1.92 TB | hot spare | — | both pools (vg1 + vg2), auto-replacement on |
The DSM system partition (md0, 8 GB) is mirrored (RAID 1) across all twelve SSDs, so the OS survives any disk loss.
Disk health
SMART read on every disk (August 2026):
| Set | Power-on hours | Wear | Errors |
|---|---|---|---|
| Volume 1 — 5× SAT5210 | ≈ 15 700 h (1.8 y) | 1 % used — ≈ 8 TB written per disk | none |
| Volume 2 — 7× Toshiba THNSN8 (6 + spare) | ≈ 64 500 h (7.4 y) | ≈ 1.5 % erase-count wear | none (35 unexpected power losses, power-loss protection intact) |
The Toshibas predate the NAS by years: seven identical drives with identical history is a correlated-failure profile, which is why Volume 2 was moved to double parity plus a spare (same reasoning as Rumba's RAIDZ2 choice). Wear is not the concern — age is. Every one of them also recorded a 62–63 °C maximum, the footprint of the June 2026 heat event.
Data scrubbing
A recurring scrub schedule is enabled on both pools (vg1, vg2) in Storage Manager → Data Scrubbing, anchored 2026-08-02. The first verification runs and their results are in the history page.
Volumes and shares
| Volume | Filesystem | Usable | Used (Aug 2026) | Shares |
|---|---|---|---|---|
| Volume 1 | Btrfs | 1.7 TB | 428 GB (26 %) | isc3_homes (student homes; calypso_homes until 2026-08-28), homes (DSM user homes) |
| Volume 2 | Btrfs | 6.7 TB | ~262 GB (Aug 2026) | pve-library — the carnaval shared library, added 2026-08-16, revising the 2026-08-04 "no shares here" decision — plus the VMM repository with the PBS VM (32 GB system + 5 TiB thin datastore disks). The old rumba_shares/rumba_backup shares died with the rebuild |
NFS exports
All exports are sec=sys (identity is whatever UID the client claims). State after the 2026-08-02 hardening:
| Export | Allowed clients | Root | Purpose |
|---|---|---|---|
/volume1/isc3_homes | 192.168.91.0/24 | squashed | Student homes (66 as of August 2026 — the UID register owns the count), mounted on all Calypso nodes |
/volume1/isc3_homes | 192.168.88.248/32 | not squashed | calypsomaster — home provisioning |
/volume2/pve-library | 192.168.91.0/27 | not squashed | Carnaval shared library (added 2026-08-16): ISOs, snippets, CUDA template dumps. No-squash because PVE root writes the dumps; the /27 covers current and future carnaval nodes while excluding the lab VMs |
The /volume2/rumba_shares export disappeared with the Volume 2 rebuild; /volume2/pve-library arrived 2026-08-16.
The two homes rules are rw, crossmnt and — since 2026-08-05 — async; the node rule squashes root to guest with anonuid=1025,anongid=100. Read the live truth with sudo cat /etc/exports over SSH; DSM's Squash dropdown maps to root_squash/no_root_squash and "Enable asynchronous" to async.
The masks, the root squashing and the homes-directory ACL are the result of the 2026-08-02 hardening; the share root's non-inheriting user:guest:allow:r-x entry, without which no new node mount can be created, was added 2026-08-05 and is described with the mounting rules. One caveat survives the hardening: the share root still carries an inherited fd delete-capable ACL entry, so any new top-level folder created there inherits delete rights.
The home directories are plain POSIX 750 directories with no ACL. Applying a shared-folder ACL recursively would replace that with the inherited group:users:allow:rwx… entry and let every student read every other student's home. Always apply permission changes to the folder itself only.
How the export is mounted in the lab VMs (/exports/<name>, ~/nas_home symlink) is described on the labs page.
Next steps
Open items for the NAS are tracked in the ops todo → NAS.