Skip to main content
Part of CALC@HEI, the HEI research computing infrastructure — governance in transition, currently operated by the ISC staff.

Incident log

Cross-cutting incidents on the CALC@HEI cluster, newest first — what happened, the impact, the recovery path and what now prevents a recurrence.

2026-08-23 — GrpWall cancelled premium jobs mid-run​

What happened. premium_dance_qos carried GrpWall=3-00:00:00 and standard_dance_qos GrpWall=1-00:00:00, and 22 associations carried the same values — 14 account rows and 8 user rows. GrpWall is an aggregate wall-time budget shared by everything under it, not a per-job walltime; when it runs out SLURM cancels the jobs already running. On every row the value equalled that row's MaxWall, so a group's entire budget was one job's maximum length.

Impact. loic.azzalini's job 18441 was cancelled 46 minutes into a 2-day limit and recorded TIMEOUT. Resubmissions pended on QOSGrpWallLimit, ran briefly as the budget decayed, and were cancelled again — each attempt spending budget that recovers only at the 7-day PriorityDecayHalfLife. The QOS was at 4199 of 4320 minutes when found. Reported by the user after four days.

The fault was tier-wide, not one user's: the QOS budget is shared by all six premium groups, and mattech stood at 2023 of 4320 on its own account row. The association rows also bind standby_qos, the one limit standby is meant never to inherit.

Why nothing caught it. The QOS objects predate the migration onto them — until 2026-08-19 every association ran under QOS=normal, so the GrpWall on those objects was inert. provision.py --sync-qos reads MaxTRESPA, MaxTRESPJ, MaxWall and Priority only, so it never saw the field. The 2026-08-21 pass that moved walltimes off the associations queried MaxWall alone and left GrpWall behind on every row it had touched. Every account carrying one was hand-provisioned; the simlab* accounts, created by provision.py, were clean.

Origin. The add-user recipe in secretzone/calc/chacha.md set GrpWall and MaxWall to the same value, treating the two as synonyms.

Recovery. GrpWall=-1 on both QOS, then apply_priority_policy.sh --step maxwall --apply for the 22 associations. Cleared limits apply to pending jobs immediately; nothing had to be cancelled.

Prevention. GrpWall is now forbidden rather than merely unmanaged — see Aggregate wall time for the rule and the two tools that enforce it. The recipe that introduced it has been corrected.

2026-08-20 — slurmctld SIGSEGV, 3 h 30 without scheduling​

What happened. slurmctld died at 12:10 with SIGSEGV in a worker thread (worker[4]: segfault at 40 … in libc.so.6, a null dereference). The log stops at 12:02:42 with no fatal: or error: line, the state save at 12:05 completed normally, and no core dump was written — there is no backtrace and the trigger is unknown. Slurm 24.11.0.

Impact. No scheduling for 3 h 30. squeue returned Unable to contact slurm controller. Running jobs were unaffected: slurmd kept them alive on both nodes and their exit statuses were collected when the controller came back, so nothing was lost and accounting stayed complete.

Why it lasted 3 h 30. The packaged unit sets no Restart=, so a crash is terminal and nothing retried. The outage lasted until someone ran squeue.

Recovery. systemctl start slurmctld. State replayed from StateSaveLocation, running jobs were re-attached, and the ~30 completions from the outage window were reconciled from the nodes in one burst.

Prevention. A Restart=on-failure drop-in (calc/provisioning/slurm/systemd/slurmctld-restart.conf, 3 restarts per 10 min) — see Controller resilience.

Found while investigating. / was at 99 % (4.3 G free of 436 G), and StateSaveLocation lives on it. Had it filled before the crash, the checkpoint would have been unusable and the whole queue lost. One home held 266 G because 14 accounts on Chacha and 3 on Disco had no quota at all — see the quota backfill below.

2026-08-20 — 17 accounts provisioned without any disk quota​

What happened. Quotas were assigned by running add_quota.sh by hand, once per user, tracked only in /root/users_quota.list on Chacha. That file had not been updated since 2025-02-28, so every account created in the 18 months since was unlimited: 14 on Chacha, 3 on Disco. Nobody hit a soft limit, so no grace warning ever fired and the drift stayed invisible until / nearly filled.

Impact. / reached 99 % on Chacha, putting slurmctld's state directory at risk (above). No data was lost.

Recovery. The 266 G home was moved to /data by its owner, taking / to 42 %. Quotas were then backfilled on both nodes from a versioned list.

Prevention. users_quota.list and sync_quotas.sh are now in the repo and run from the Ansible common role, which fails the play if any account lacks a quota — see Disk quotas.

Trap for the next person. Usage must be read with repquota, not quota -u <user>: the latter prints nothing for a filesystem where the user has no quota yet, which is exactly the case during a backfill. A safety check built on it passes vacuously for every account it is meant to protect. This mistake was made during this backfill and briefly put five users over their hard limit on /data.