SLURM configuration

This section details the current SLURM configuration for CALC@HEI.

Installation

SLURM has been installed from tarball, version : 24.11.0

All official plugins installed :

• libnvidia-ml
• TODO

Install / upgrade process TODO

Architecture

Chacha

• Client (slurm-smd-client)
• Worker (slurm-smd,slurm-smd-slurmd)
• Controller (slurm-smd-slurmctld)
• Accounting DB (slurm-smd-slurmdbd)

Disco

• Client (slurm-smd-client)
• Worker (slurm-smd,slurm-smd-slurmd)

Schema

TODO

Controller resilience

slurmctld runs on Chacha only — there is no SlurmctldHost backup. The packaged unit ships without Restart=, so a crash leaves the cluster with no scheduler until someone notices. A drop-in supplies one (August 2026):

# /etc/systemd/system/slurmctld.service.d/restart.conf
[Service]
Restart=on-failure
RestartSec=10s

Source: calc/provisioning/slurm/systemd/slurmctld-restart.conf, which also caps restarts at 3 per 10 minutes so a reproducible crash stays down and visible instead of looping.

A dead controller does not stop running jobs

slurmd keeps allocations alive across a controller outage and reports their exit status when it returns, so a crash costs scheduling, not work in flight. The failure is silent from the compute side — the first symptom is squeue returning Unable to contact slurm controller, which is why the 2026-08-20 SIGSEGV ran 3 h 30.

danger

StateSaveLocation shares the root filesystem

/var/spool/slurmctld is on / on Chacha, alongside /home. If / fills, the controller cannot checkpoint and a subsequent restart loses the queue. / reached 99 % on 2026-08-20 from a single unquota'd home. Per-user quotas are what hold this line — see Disk quotas.

Partitions

Dance

This is the default partition : is is currently composed of Chacha and Disco.

Chacha and Disco

These partitions can be used to restrain an account to use only one server.

Associations and partitions

Every user holds one association per partition — Chacha, Disco and Dance — each carrying that partition's QOS as its default, so a job picks up the right caps from the partition it runs on (August 2026). SLURM matches the association of the partition a job is submitted to: a user without a row for that partition is rejected at submission with Invalid account or account/partition combination specified, whatever their QOS list says.

Multi-partition requests are not supported

--partition=Chacha,Disco fails with Multiple partition job request not supported when a partition is set in the association (ESLURM_PARTITION_ASSOC). SLURM refuses a partition list whenever the association it resolves has a partition set, which is the case for every account here. Adding a partition-less association alongside does not lift it — verified August 2026. Submit to one partition; Dance already spans both nodes.

Which partition to submit to is a scheduling choice, not just a limits one: Dance carries the 3-day MaxWall but shares one queue across both nodes, while Chacha and Disco have their own shorter queues and the 8-hour cap. A deep Dance backlog can put a new job weeks out while the same job on Disco starts the same day.

Limits model

SLURM has four objects that matter for limits. "Tier" and "group" are our words, not SLURM's — both are accounts, at different depths of a single tree.

Our word	SLURM object	What it is
queue	partition	the set of nodes a job is submitted to — Chacha, Disco, Dance
tier	account — premium_rs, standard_rs	fairshare weight and which QOS its members may use; holds no resources
group	account — mattech, students, …	a project: identity, accounting, and the members sharing one allowance
—	association	the row joining (cluster, account, user, partition); where defaults and limits are stored
—	QOS	a policy label carried by a job: priority, walltime, preemption, limits

A QOS is not a queue. A job goes to one partition and runs under one QOS; the two are independent. Users hold a list of permitted QOS with one default and pick another per job with --qos=.

How a job's limits are computed

For every job SLURM checks both the association chain — the user's row, then its account, then each parent up to root — and the job's QOS. The most restrictive value wins, and AccountingStorageEnforce=associations,limits,qos means all of it is enforced. Two consequences:

• A limit on a tier account applies to every group beneath it, combined — which is why tier accounts carry none, see Accounts.
• An association limit binds a job whatever QOS it runs under, so a cap on the group account also applies to that group's standby_qos jobs. No QOS flag lifts an association limit.

Where an allowance belongs

An allowance is group-wide: a pool the members of one project share. Three fields can express a pool and only one of them is group-wide.

Field	Scope	Use
GrpTRES on a QOS	all accounts using that QOS, combined — tier-wide	never
MaxTRESPA on a QOS	each account separately, counting only jobs under that QOS	the allowance
GrpTRES on a group account	every job of that group, any QOS	backstop only

MaxTRESPA is what makes standby_qos work: it counts only jobs running under its own QOS, so with the group accounts' GrpTRES cleared a --qos=standby_qos job sits outside every allowance. What bounds it is the hardware and its own 3-day MaxWall.

caution

normal bounds nothing but walltime

normal carries no TRES limits and is not preemptible, so a job under it ignores the group allowance entirely. Its only ceiling is the 1-day MaxWall set 2026-08-21. Two associations still grant it, both deliberately — see below.

caution

Clearing a TRES limit needs -1 per TRES on a QOS

sacctmgr modify account X set GrpTRES=-1 clears an account's limits, but the same form on a QOS fails with slurmdb_format_tres_str: no TRES id found for -1. A QOS wants every TRES named: GrpTRES=cpu=-1,mem=-1,node=-1,gres/gpu=-1,gres/shard=-1.

The allowance lives on the QOS alone. Applied 2026-08-19 by provisioning/slurm/migrate_group_limits.sh then clear_account_limits.sh (both dry-run by default, --apply to execute, both idempotent): the six QOS carry MaxTRESPA, every association defaults to its tier's QOS, and no association — account or user — carries GrpTRES or MaxTRES any more.

normal was dropped from 91 user associations at the same time. Two rows still grant it and are meant to: root, and yannic.delisle's partition-less mattech row, which has no DefaultQOS and would be unable to submit without it. That row is unreachable in practice — his per-partition rows win for every partition — and is due for removal, see the ops todo.

Where a walltime belongs

On the QOS, for the same reason. An association MaxWall binds a job whatever QOS it runs under, so the 4-hour walltime that sat on 81 associations also capped those users' standby_qos jobs at 4 hours — the one limit standby was never meant to inherit. Cleared 2026-08-21 with MaxWall=-1, which an association accepts as-is (a QOS would demand -1 per TRES). The tier QOS carry the walltimes in QOS and Limits; standby_qos and normal belong to no tier and take theirs from qos_maxwall in accounts.yaml.

standby_qos is set to 3 days, matching the longest in-limits window (premium_dance_qos), so standby is never the more privileged QOS on time.

Aggregate wall time is never set

GrpWall is not a walltime. MaxWall bounds one job; GrpWall bounds the sum of wall time across every job of an association or QOS and everything beneath it — a budget that decays at PriorityDecayHalfLife, not a ceiling that throttles. When it runs out SLURM cancels the jobs already running and records them TIMEOUT, so a group loses work in flight.

It is set nowhere on this cluster, at any level, and accounts.yaml cannot express it. Two tools hold that:

• provision.py --sync-qos treats it as forbidden rather than managed — it emits set GrpWall=-1 for any QOS carrying one, declared in accounts.yaml or not.
• apply_priority_policy.sh --step maxwall clears it from associations in the same pass as MaxWall.

caution

A TIMEOUT far short of Timelimit is a Grp limit, not a walltime

A job cancelled this way records State=TIMEOUT with an Elapsed well under its own Timelimit, and nothing in the job record names the limit that killed it. The two fields disagreeing is the signature. Resubmissions then pend with Reason=QOSGrpWallLimit or AssocGrpWallLimit, which is what names the level. scontrol show assoc_mgr qos=<name> flags=qos prints GrpWall=limit(usage).

Limits other than GrpWall that accounts.yaml cannot express are warned about by --sync-qos, not cleared: some are deliberate, such as the MaxSubmitJobsPU=0 on jail that locks an account out.

Accounts

Accounts have been created in 2 groups :

• Premium Researchers (premium_rs) :
• Standard Researchers (standard_rs) : All users who can't participate financially to the project. Students are also part of this group

There are 2 other groups : Test and temp : Test is only for administration purpose, and temp is a locked group either to migrate someone from another account (can't delete an account when someone has it as a default account) or to disallow someone to run jobs (MaxSubmitJob=0)

TODO

caution

Tier accounts carry no GrpTRES

premium_rs and standard_rs group project accounts for QOS and fairshare; the resource caps belong on the project accounts below them. SLURM enforces GrpTRES up the whole lineage, so a cap on a tier applies to all of its groups combined.

premium_rs carried cpu=88,gres/gpu=4,gres/shard=192,mem=1000G,node=2 — one project's allowance — until 2026-08-19, holding its six groups to 88 of the cluster's 160 CPUs and 2 nodes between them. Cleared; standard_rs never had one. provisioning/accounts.yaml sets no GrpTRES, so nothing reinstates it.

QOS and Limits

Six tier QOS objects, one per (tier, partition), plus standby_qos and the legacy normal. MaxTRESPA is the allowance one group may hold on that partition — see Limits model for why that field and not another.

QOS	Priority	MaxWall	MaxTRESPA
premium_chacha_qos	1000	8 h	cpu=48, gpu=1, shard=96, mem=515782M, node=1
premium_disco_qos	1000	8 h	cpu=32, gpu=1, shard=80, mem=125000M, node=1
premium_dance_qos	1000	3 d	cpu=80, gpu=2, shard=176, mem=640782M, node=2
standard_chacha_qos	1000	4 h	cpu=24, gpu=1, shard=48, mem=257891M, node=1
standard_disco_qos	1000	4 h	cpu=16, gpu=1, shard=40, mem=62500M, node=1
standard_dance_qos	1000	1 d	cpu=40, gpu=1, shard=88, mem=320391M, node=2
normal	1000	1 d	none
standby_qos	0	3 d	none, see Standby QOS

MaxTRES, the per-job ceiling, equals MaxTRESPA on all six tier QOS: one job may hold its group's entire allowance. The priority column is what Scheduling turns into the QOS factor — it separates in-limits work from standby, not premium from standard, which is fairshare's job.

None of those numbers are typed anywhere. accounts.yaml declares a share per tier — premium 1/2 of a partition, standard 1/4 — and a job_share, 1/1 on both tiers, and provision.py --sync-qos multiplies them by the partition's live capacity from scontrol show partition. Add memory or a node and re-running moves both caps to match. cpu, mem, gpu and shard scale; node is set to the partition's node count rather than scaled, so a group on Dance can still span both machines; gpu and shard never round below 1. MaxWall and Priority come from the same file — maxwall and priority per tier, and a top-level qos block for the QOS belonging to no tier. Nothing on a QOS is set by hand.

The per-job values job_share replaced on 2026-08-21 were unrelated to the allowances: premium_disco_qos allowed a single job mem=1000G and node=2 on a one-node partition with 250 G, while three others bound tighter than the allowance (premium_dance_qos at cpu=44 against 80). Deriving them removes the second set of hand-typed numbers that could drift.

Premium and standard are the only two levels — every group in a tier gets the same allowance. Six premium groups × 80 CPUs is 480 against Dance's 160, and that oversubscription is deliberate: the QOS bounds any one group, the hardware bounds the total, and fairshare (750 / 250) arbitrates.

Memory defaults

DefMemPerCPU=3900 (MB) in slurm.conf on every node, set 2026-08-19.

SelectTypeParameters=CR_CORE_MEMORY makes memory a consumable resource, so a job submitted without --mem is charged the entire memory of its node — 1031564 MB on Chacha. 3900 MB is Disco's per-CPU share (64 × 3900 = 249600 ≤ 250000), so a job claiming a whole node without naming memory still fits the smaller of the two. DefMemPerCPU and DefMemPerNode are mutually exclusive; scontrol show config prints only the one in force.

Jobs submitted through sbatch --wrap=… carry no #SBATCH header, so they take this default — worth an explicit --mem when the value matters.

Over-limit jobs pend, they are not rejected

Group limits are shared, so a job exceeding its account's GrpTRES stays pending with Reason=AssocGrpMemLimit (or …CpuLimit, …NodeLimit) rather than being refused at submit — including when the job alone could never fit the cap on an idle cluster. Anything chained behind it with --dependency=afterok: then reports (Dependency), which hides the cause. scontrol show assoc_mgr account=<acct> flags=assoc prints limit(usage) per TRES for the account and its parents, which is the quickest way to find the binding one.

Thread defaults

TaskProlog=/etc/slurm/task_prolog.sh on both nodes (August 2026) exports OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS and NUMEXPR_NUM_THREADS from SLURM_CPUS_PER_TASK, for whichever the job hasn't already set itself. It runs before the batch step, so it reaches jobs that never call srun — the Apptainer pattern most jobs here use — not just srun steps.

Without it, PyTorch/OpenBLAS/MKL size their default thread pool from the node's physical core count (48 on Chacha, 32 on Disco) rather than the cpuset a job is confined to: a 6-CPU job started 46-thread workers, held to its 6 CPUs by the cgroup but adding 40 spurious runnable threads to the load average and ~250-300k context switches/second of scheduler overhead for no throughput gain.

Source: calc/provisioning/slurm/install_task_prolog.sh (dry-run by default, --apply to execute, idempotent).

Standby QOS

standby_qos lets a group work past its allowance on the understanding that the job is interrupted as soon as someone inside their limits needs the room. Three mechanisms carry that, and all three have to hold:

• No allowance of its own. It carries no MaxTRESPA, and since the association limits were cleared on 2026-08-19 nothing else caps it either, so a standby job takes whatever the hardware has idle.
• It never outranks in-limits work. Priority=0, against the 1000 the seven in-limits QOS carry, scores a standby job 0 on a factor worth 1000000. Everything else it can earn — a perfect fairshare and a fully aged job — tops out at 110000, so no standby job outranks an in-limits one at any age or any fairshare. See Scheduling.
• It reserves nothing. The NoReserve flag stops the backfill scheduler holding resources for a pending standby job, so standby never delays an in-limits job to make room for itself.

Its MaxWall is 3 days, matching the longest in-limits window (premium_dance_qos) so standby is never more privileged on time. That ceiling is what ends a standby job on a quiet cluster, where nothing comes along to preempt it — and it matters because PreemptMode=REQUEUE restarts a preempted job from zero.

The preemption itself is still unverified on this cluster — the sanity-check in the ops todo matters more now that standby can fill the machine. Preempted standby jobs are requeued (PreemptMode=requeue), not killed outright, but they do start over unless the application checkpoints itself.

It's granted to every account/user alongside their normal QOS (see Provisioning below) — a user opts in per job with --qos=standby_qos (see SLURM examples), nothing runs under it by default.

Set up 2026-08-12: PreemptType=preempt/qos and PreemptMode=REQUEUE in slurm.conf on every node, plus the QOS object itself and its Preempt= relationship to normal/premium_*_qos/standard_*_qos, created by calc/provisioning/slurm/setup_standby_qos.sh. Preempt= had to be set on normal too, not just the premium/standard QOS objects — see the Slurm accounts todo item: several accounts still run live under QOS=normal with hand-set GrpTRES rather than the per-partition QOS objects, so preemption would have silently not applied to them otherwise.

Scheduling

PriorityType=priority/multifactor, with three factors weighted (August 2026):

Factor	Weight	What it separates
QOS	1000000	in-limits jobs from standby_qos
Fairshare	100000	tiers, and groups within a tier
Age	10000	ties the two above leave, capped at PriorityMaxAge=7-00:00:00

PriorityWeightJobSize, PriorityWeightPartition and PriorityWeightAssoc are 0, and PriorityWeightTRES is unset. The weights sit an order of magnitude apart so each factor only breaks the ties the one above it leaves; the arithmetic that makes standby strictly subordinate is in Standby QOS.

QOS priority is normalised against the highest value in the system, so the seven QOS at Priority=1000 each contribute the full 1000000 and standby_qos at 0 contributes nothing. Premium and standard share the same QOS priority on purpose: the difference between the tiers is fairshare, not a queue-jump. Since the QOS weight is ten times the fairshare weight, raising one tier above the other would put its jobs first whatever their fairshare.

Fairshare is Fair Tree (the default — PriorityFlags=NO_FAIR_TREE would turn it off), so usage is compared against siblings at each level of the account tree rather than globally. The tier accounts hold premium_rs 750 and standard_rs 250, every group account holds 100, and all 92 user associations hold 100. PriorityDecayHalfLife=7-00:00:00 sets how fast recorded usage fades.

Billing weights

Fairshare usage accrues in billing — TRES collapsed to one number by TRESBillingWeights on the partition — not in raw CPU-seconds. Until 2026-08-21, Dance carried Mem=1.0, which is per megabyte, against CPU=1.0 per core: a GB-hour cost 1024× a CPU-hour, so billing was effectively a memory-only metric and reserving CPUs you don't use carried no fairshare cost at all. Chacha and Disco carried no weights, so the same job billed on CPU count alone there — roughly 1000× less than the identical job on Dance.

Weights now normalise each TRES to what a full node costs, per partition (CPU stays the unit): Chacha CPU=1.0,Mem=0.095G,gpu=48,shard=0.5 · Disco CPU=1.0,Mem=0.262G,gpu=32,shard=0.4 · Dance CPU=1.0,Mem=0.128G,gpu=40,shard=0.455. Typed GPU TRES (gres/gpu:nvidia_h100_nvl etc.) are weighted 0 — Slurm bills the generic and typed GPU TRES separately, and summing both double-counted every GPU job under the old weights.

RawUsage was reset to 0 on every account and user association the same day rather than left to decay: the old billing scale was ~3600× the new one, and at the 7-day PriorityDecayHalfLife new usage would have taken ~83 days to outweigh the stale numbers.

Source: calc/provisioning/slurm/install_billing_weights.sh (dry-run by default, --apply to execute, idempotent). Nothing besides fairshare reads billing here — PriorityWeightTRES is unset and no QOS or association limit references it — so a wrong weight costs priority accuracy only, nothing enforced.

caution

A new association defaults to RawShares=1

sacctmgr add user without fairshare= creates the row at 1, a 100× disadvantage against every other user — invisible while PriorityWeightFairshare is 0, decisive once it is not. provision.py passes fairshare=100 on every user it adds; a row created by hand needs it too.

sprio -l breaks a pending job's priority into its factors, which is the quickest way to see which one is deciding. A change to any weight takes up to PriorityCalcPeriod=00:05:00 to appear there, and only pending jobs are recomputed — a running job keeps the priority it started with.

Provisioning

Accounts and users are declared in provisioning/accounts.yaml (groups → tier → per-partition QOS, plus the user → group mapping) and applied with provisioning/slurm/provision.py, run on Chacha where slurmdbd lives. --check diffs the file against the live sacctmgr state without changing anything; plain / --apply plans or executes the sacctmgr commands to add whatever's missing.

Users get an association on every partition by default; an optional partitions: [dance] on a user entry restricts them to a subset — see Associations and partitions. Writes need sudo on Chacha: root is the only AdminLevel=Administrator in the accounting DB, and sacctmgr refuses everything else with Access/permission denied. Reads, including --check and the dry runs, work unprivileged.

--sync-qos converges every QOS onto accounts.yaml: MaxTRESPA from share, the per-job MaxTRES from share × job_share, MaxWall and Priority. The six tier QOS take their walltime from the tier's maxwall; standby_qos, normal and jail are declared in the top-level qos block with a priority and an optional walltime. It is idempotent and reports QOS limits already match accounts.yaml when there is nothing to do.

It also warns, without changing anything, about a live QOS that accounts.yaml declares nowhere — nothing would manage its limits, and left at Priority=0 it ranks with standby — and about one declared in the file that does not exist live, since creating a QOS is a deliberate act rather than a reconciliation.

A user moved to another group in accounts.yaml keeps their old association until --prune-extra removes it; --fix-accounts repoints their DefaultAccount, without which their jobs keep charging the account they were moved off. --prune-extra reads squeue and holds back any association a user still has queued jobs against, so a consolidation done while jobs run needs a second pass once the queue drains. Changing a group's tier is not propagated at all: the account keeps its old parent and gains the new tier's QOS on top of the old ones. Edit the YAML for new projects/users instead of running add_slurmuser.sh by hand. accounts.yaml's top-level standby_qos field is granted to every account/user the same way — --fix-accounts --apply backfills it onto associations that predate the field (see Standby QOS).

To offboard a user, delete their entry from accounts.yaml and run --prune / --prune --apply to remove the live association — see How to remove a user.

accounts.yaml is SLURM-only — it doesn't manage the Linux account or SSH key. Those are Ansible's job (provisioning/ansible/), including on Chacha/Disco — see Ansible.

User creation

Users fill their form from the page Requesting access , then their information is used to create their SSH access, and their SLURM user in its project account, according to the SLA / QOS we can provide : Premium or Standard.

Several users can share the same project account to work as a team. (Limits are applied both to each user for maximum limits and the group for group limits)

The staff creates and configure user accounts.

User creation process

Backups

Cluster configuration

Currently, all the SLURM configuration is manually backuped (files, DB)

TODO : automate and redirect to backup server when its ready

User data

For user data, currently compute users have to request some space on the filer01.hevs.ch server from the Sinf : there is a request form in "Demande de service" in "Comptes et accès > Obtention d'accès réseau" . More documentation to see with the Sinf here, generally for researchers the filesystem to ask for is the fs_projets.

Consider all space on CALC@HEI to be short-lived, it is not made to store data as backups : only temporary to compute and get results.