Skip to main content
Part of CALC@HEI, the HEI research computing infrastructure — governance in transition, currently operated by the ISC staff.

SLURM — history & operations

Dated operations on the SLURM configuration, newest first. Current state — partitions, limits, QOS and their standing rules — lives on the main page; finished work across CALC@HEI is indexed in the ops journal.

Aggregate wall-time limits removed, 2026-08-26​

GrpWall cleared from premium_dance_qos, standard_dance_qos and 22 associations (14 account rows, 8 user rows). It had been cancelling premium jobs mid-run since the 2026-08-19 migration onto the tier QOS — see the incident for how it was found and why nothing caught it for four days.

Values before the clear: both QOS at their own MaxWall (3 d premium, 1 d standard), the QOS budget 4199 of 4320 minutes spent, mattech 2023 of 4320 and photocast 697 of 4320 on their account rows. Only Dance was affected; the Chacha and Disco QOS never carried one.

Two tools now keep it out, and neither existed for this field before: provision.py --sync-qos clears GrpWall on any QOS carrying one, and apply_priority_policy.sh --step maxwall clears it from associations alongside MaxWall. --sync-qos also warns about the other 14 limit fields accounts.yaml cannot express, which it previously did not read at all — the blind spot that let this run. See Aggregate wall time.

Thread defaults and billing weights fixed, 2026-08-21​

Investigating why uptime on Chacha read load average 120+ against 96 threads: cpu.stat on every job's cgroup showed usage staying inside its own cpuset, so cgroup enforcement was not the cause. The oversubscription was inside the cpuset — jobs run through Apptainer (the trunkx-app.sif / run_calibration_sweep.py pattern most jobs here use) started PyTorch worker pools sized from the node's physical core count (48 on Chacha) rather than SLURM_CPUS_PER_TASK, since none of OMP_NUM_THREADS, MKL_NUM_THREADS or OPENBLAS_NUM_THREADS was set anywhere. A 6-CPU job (job 18101), correctly held to its 6-CPU cpuset by cgroups, ran 4 worker processes of 46 threads each — 40 spurious runnable threads per job, and 250-300k context switches/second measured via vmstat.

Fixed with a TaskProlog exporting the four thread-count variables from SLURM_CPUS_PER_TASK, for whichever a job hasn't already set itself — see Thread defaults. Verified end to end with a real 2-CPU test job: both the batch step and an srun step inside it picked up OMP_NUM_THREADS=2 etc. Installed on both nodes, config diff-verified identical between them before scontrol reconfigure.

A sacct pull of 21 days / 1672 finished jobs made while investigating found cluster CPU efficiency at 65% (24815 CPU-h reserved, 16106 used, 8709 wasted), spread evenly across efficiency bands rather than concentrated in a few jobs — a resource-sizing habit, not the thread issue above (GPU jobs billed no worse than CPU-only ones). Tracing why no one had a reason to size requests correctly found TRESBillingWeights="Mem=1.0" on Dance is per megabyte against CPU=1.0 per core — a GB-hour cost 1024× a CPU-hour, so fairshare was effectively a memory-only metric (the largest CPU-hour waster, 2236 CPU-h over 21 days, carried close to zero fairshare cost for it) — and Chacha/Disco carried no weights at all, so the same job billed ~1000× less there than on Dance. See Billing weights for the fix and the numbers behind the new weights.

RawUsage was reset to 0 on every account and user association afterward, using the real, enumerated account list — the billing scale dropped ~3600×, and at the 7-day PriorityDecayHalfLife new usage would have taken ~83 days to outweigh the stale numbers otherwise.

Neither change touched running jobs: TaskProlog only applies at task launch, and TRESBillingWeights/RawUsage only affect scheduling priority, nothing enforced.

Job priority put into service, 2026-08-21​

Job priority was job size and nothing else. PriorityWeightFairshare and PriorityWeightAge were 0, PriorityWeightTRES was set, and PriorityDecayHalfLife was 10 seconds, so a pending job scored cpu/160×1000 + mem/1281564×1000 — 209 for a typical one. The 750/250 tier weights were configured and documented but contributed nothing, and sprio -l reported FAIRSHARE, AGE, JOBSIZE and QOS all zero.

Applied step by step with provisioning/slurm/apply_priority_policy.sh (dry-run by default, --apply to execute, --step to run one at a time), then provision.py --sync-qos --apply:

  1. RawShares=100 on the 82 associations and group accounts still at 1, four SIMLab groups among them. This had to come first: enabling the fairshare weight against a 1-vs-100 split would have starved them.
  2. MaxWall moved off 81 associations onto the QOS — see Where a walltime belongs.
  3. Priority=1000 on the seven in-limits QOS, standby_qos left at 0.
  4. The priority block written into /etc/slurm/slurm.conf on both nodes (timestamped backup first) and scontrol reconfigure.
  5. Per-job MaxTRES derived from accounts.yaml through the new job_share, replacing values that matched no hardware — see QOS and Limits.

Step 3 was not inert on its own, unlike steps 1 and 2: PriorityWeightQOS was already 1000, so in-limits jobs gained priority the moment the QOS carried it.

The Priority=1000 / 0 split was set by hand in step 3 and then moved into accounts.yaml the same day, so --sync-qos now converges it with the rest and warns about any live QOS the file declares nowhere.

Verified afterwards on a pending job: priority 1001202 = QOS 1000000 + fairshare 1075 + age 127. Fairshare now separates users — yiwei.jia ranks last, against mattech's EffectvUsage of 0.949.

RawUsage was deliberately not reset. The values were inherited from the 10-second half-life, but mattech climbed from 4.6 M to 188 M within the hour under the 7-day one, so a resetusage would have changed nothing the new decay does not.

note
scontrol reconfigure re-execs slurmctld

On Slurm 23.11 and later a reconfigure replaces the daemon rather than re-reading the file — MainPID went 1338586 → 1510140 here, while ActiveEnterTimestamp stayed put. Settings that the manual says need a restart, PriorityDecayHalfLife among them, do take effect; equally, a reconfigure carries a restart's risk profile rather than a config reload's. Running jobs survived it.