NAS (FS2500) — history & operations

Dated operations on the NAS, newest first. Current state — arrays, volumes, exports and their standing rules — lives on the main page; finished work across the whole rack is indexed in the ops journal.

Exports switched to async, 2026-08-05

"Enable asynchronous" was ticked on both calypso_homes NFS rules, so /etc/exports now reads async instead of sync and the NAS acks a write once it is in RAM rather than on flash. This was the open speed item; it turned out to be the whole of it.

Measured from carnaval0 (1 GbE), writing into a scratch directory in the export:

	Sequential 1 GiB	500 × 4 KiB files
Before (sync export, sync client)	65 MB/s	175 files/s
After, client still sync	102 MB/s	300 files/s
After, client sync dropped too	109 MB/s	292 files/s
Local disk, same test	—	451 files/s

Sequential writes now reach the node's 1 GbE wire speed, matching reads (108–113 MB/s), and small-file rate went from 32 % of local to ~65 %.

The export was the entire bottleneck: dropping the client's sync on top buys nothing (102 → 109 MB/s, small files unchanged within noise). The client-side half of that todo item was therefore closed without doing it — new mounts simply omit sync, and the nodes' fstab gets tidied whenever they are rebuilt.

The durability cost is real and now carries a standing caution on the main page: a NAS crash or power cut can lose writes the client believes are committed, and the UPS gives the NAS no ride-through.

Student-home mounts repaired, 2026-08-05

Since the 2026-08-02 hardening no new NFSv4.1 mount of /volume1/calypso_homes/homes could be created — access denied by server from the carnaval nodes and from calypso0 itself — while the three live Calypso mounts, made before it, kept working. Any node reboot would therefore have lost its student homes, and lab VMs came up without ~/nas_home.

Cause. Squashing root to guest was the half of the hardening that did it. An NFSv4 mount of a subdirectory makes the server walk /volume1 → calypso_homes → homes as the squashed identity, and guest (uid 1025) had no entry in the share-root ACL — the same reason a mount of the parent succeeded but read d---------. DSM's per-rule "Allow users to access mounted subfolders", the obvious suspect, was already enabled (crossmnt is in /etc/exports on both rules) and was never involved.

Fix, preferred over relaxing the squash because it keeps the hardening intact: one non-inheriting r-x entry for guest on the share root only — user:guest:allow:r-x---a-R-c--:---n on /volume1/calypso_homes. It is now a standing rule on the main page. Relaxing the rule to no_root_squash instead was rejected: it would undo the hardening for the whole node subnet, and buys nothing, since a student with sudo in a lab VM can reach any UID regardless of squash — which is the real exposure.

Verified from carnaval0: fresh vers=4.1 mount succeeds, ls as root still denied (squash intact), uid 10020 reads its own 750 home, the three live mounts undisturbed. The exact fstab form the Calypso nodes carry (hostname nas, no explicit vers=) also mounts and negotiates 4.1, so a node reboot is safe again.

Second network link cabled, 2026-08-03

A second cable was fitted to the NAS and the addressing reworked so that each subnet gets its own dedicated 10 GbE port: eth2 keeps 192.168.88.250 (DSM, SSH, rumba/PBS backups) and eth3 was given a static 192.168.91.250/24 with no gateway, putting the NAS directly in the Calypso node subnet. Bonding the two ports was considered and rejected — redundancy is not the priority here, and a bond gives the pair a single shared address, forfeiting the split.

Consequences worth knowing:

• node NFS and rumba's backup stream no longer share a port;
• the per-node nas-fastpath.service units and the NAS rc.d boot script become unnecessary rather than merely repaired, once the rebuilt nodes mount the new address — see how a node must mount it. Pre-rebuild nodes keep working unchanged on 192.168.88.250 in the meantime (DSM routes egress by source address, so the two addresses coexist — details on the main page).

Measurements

The 2026-08-02 audit only ever measured writes, so these are the first read and rumba-path figures on record.

Path	Rate	Bounded by
rumba → NAS (eth2)	1.1 GB/s (8.9 Gbit/s)	nothing on the network — the backup direction is not network-bound
NAS → rumba (eth2)	938 MB/s (7.5 Gbit/s)	as above
node read (eth3)	108–113 MB/s	the node's own 1 GbE port, at wire speed
node write (eth3)	65 MB/s	the export's sync, not the network
node small files	175 files/s (vs 549 on local disk)	sync again

The two write rows are superseded

Both were sync-bound. Since 2026-08-05 the exports are async and writes reach 102–109 MB/s / ~300 files/s — quote those, not these.

Method: rumba figures are raw TCP over an 8.4 GB single stream; node reads are five consecutive 1 GiB dd runs with iflag=direct so the client cache cannot flatter them.

For contrast, while the on-link route was briefly missing that morning, reads fell back to being software-routed through the CRS326 at 97–100 % CPU, giving 74–84 MB/s shared between all nodes rather than per node. That is the cost the fast path (and now the 91 address) avoids.

Rebuild of Volume 2, 2026-08-02

Volume 2 was recreated from RAID 5 over seven disks to RAID 6 over six disks plus a hot spare. The trigger was the disk-health readout: seven 7.4-year-old identical drives behind single parity, with no spare anywhere in the chassis, is a configuration where a second failure during a rebuild loses the volume. It was done while the volume was still empty (verified — nothing used it) because DSM cannot convert RAID 5 → RAID 6 without a free bay. Capacity went from 11 TB to 6.7 TB at no practical cost.

What this bought:

• Volume 2 survives two simultaneous disk failures instead of one.
• The spare (sata12) protects both pools with auto-replacement on — so a Volume 1 disk failure now rebuilds automatically too. (Synology RAID F1 was rejected: still single-parity, and the risk here is age, not wear.)

The old shared folders (rumba_backup, rumba_shares — both unused) were deliberately not recreated: a future share can be given root squash from the start instead of inheriting the no_root_squash that had to be removed from Volume 1 the same day.

Hardening of the NFS exports, 2026-08-02

Three problems were found and fixed on calypso_homes:

• The calypsomaster rule was written 192.168.88.248/24, and the /24 mask admitted the whole 192.168.88.0/24 subnet — which includes 192.168.88.1, the address every legacy-WireGuard user is NATed to (see the source-IP caution). Combined with no_root_squash, any VPN client could mount all student homes as root. Tightened to /32.
• The node rule (192.168.91.0/24) also had no_root_squash, while students on the nodes are members of the docker group — i.e. root-equivalent locally, hence root on the NAS. Root is now squashed to guest on that rule; calypsomaster keeps no_root_squash for provisioning, which is what the /32 tightening makes acceptable.
• The ACL on /volume1/calypso_homes/homes granted group:users delete and delete-child — so any docker-capable student could rename or delete another student's home directory (contents stayed safe, each home being 750). Both rights were removed from that directory, deliberately without propagating to sub-folders.

All three fixes were verified from calypso0 (rename denied, root read denied, student access to their own home unaffected). One caveat survives: the share root still carries an inherited delete-capable ACL entry, so any new top-level folder inherits delete rights — flagged with the standing rules on the main page.

First data scrubbing, 2026-08-02

Until 2026-08-02 nothing had ever been verified: Volume 1 had never been scrubbed in 1.8 years, and neither md array had run a consistency check. Both halves were then run on Volume 1 and a recurring schedule was enabled:

Check	Result (2026-08-02)
Btrfs scrub, /volume1	432.85 GiB verified in 3 min 20 s, 0 errors
RAID 5 parity check, md2	mismatch_cnt=0 after a full pass (17 min)
Schedule	enabled on both pools (vg1, vg2) in Storage Manager → Data Scrubbing, anchored 2026-08-02

Volume 2 was skipped (empty, and recreated later the same day); the schedule survived the recreation.