Skip to main content
Part of CALC@HEI, the HEI research computing infrastructure — governance in transition, currently operated by the ISC staff.

SLURM configuration

This section details the current SLURM configuration for CALC@HEI.

Installation​

SLURM has been installed from tarball, version : 24.11.0

All official plugins installed :

  • libnvidia-ml
  • TODO

Install / upgrade process TODO

Architecture​

Chacha​

  • Client (slurm-smd-client)
  • Worker (slurm-smd,slurm-smd-slurmd)
  • Controller (slurm-smd-slurmctld)
  • Accounting DB (slurm-smd-slurmdbd)

Disco​

  • Client (slurm-smd-client)
  • Worker (slurm-smd,slurm-smd-slurmd)

Schema​

TODO

Partitions​

Dance​

This is the default partition : is is currently composed of Chacha and Disco.

Chacha and Disco​

These partitions can be used to restrain an account to use only one server.

More details of partitions usage TODO

Accounts​

Accounts have been created in 2 groups :

  • Premium Researchers (premium_rs) :
  • Standard Researchers (standard_rs) : All users who can't participate financially to the project. Students are also part of this group

There are 2 other groups : Test and temp : Test is only for administration purpose, and temp is a locked group either to migrate someone from another account (can't delete an account when someone has it as a default account) or to disallow someone to run jobs (MaxSubmitJob=0)

TODO

QOS and Limits​

Current limits on QOS and accounts applied to each project account :

  • premium_rs :
    • MaxCPUs=44
    • MaxNodes=2
    • MaxTRES=gres/gpu=1,gres/shard=96,cpu=44,mem=500G
    • GrpTRES=gres/gpu=1,gres/shard=96,cpu=44,mem=500G
    • GrpWall=3-00:00:00
    • MaxWall=3-00:00:00
  • standard_rs :
    • MaxCPUs=24
    • MaxNodes=1
    • MaxTRES=gres/gpu=1,gres/shard=96,cpu=24,mem=256G
    • GrpTRES=gres/gpu=1,gres/shard=96,cpu=24,mem=256G
    • GrpWall=1-00:00:00
    • MaxWall=1-00:00:00

TODO : complete / modify until limits are finished

Scheduling​

Fairshare is one way of priorizing jobs in the job queue.

  • premium_rs :
    • Fairshare : 750
  • standard_rs :
    • Fairshare : 250

Then on each account, everyone has a 100 fairshare : SLURM is just making calculation then on the percentage using both the share in each group, and the parent account premium / standard.

Provisioning​

Accounts and users are declared in provisioning/accounts.yaml (groups → tier → per-partition QOS, plus the user → group mapping) and applied with provisioning/slurm/provision.py, run on Chacha where slurmdbd lives. --check diffs the file against the live sacctmgr state without changing anything; plain / --apply plans or executes the sacctmgr commands to add whatever's missing. Edit the YAML for new projects/users instead of running add_slurmuser.sh by hand.

To offboard a user, delete their entry from accounts.yaml and run --prune / --prune --apply to remove the live association — see How to remove a user.

accounts.yaml is SLURM-only — it doesn't manage the Linux account or SSH key. Those are Ansible's job (provisioning/ansible/), including on Chacha/Disco — see Ansible.

User creation​

Users fill their form from the page Requesting access , then their information is used to create their SSH access, and their SLURM user in its project account, according to the SLA / QOS we can provide : Premium or Standard.

Several users can share the same project account to work as a team. (Limits are applied both to each user for maximum limits and the group for group limits)

The staff creates and configure user accounts.

User creation process

Backups​

Cluster configuration​

Currently, all the SLURM configuration is manually backuped (files, DB)

TODO : automate and redirect to backup server when its ready

User data​

For user data, currently compute users have to request some space on the filer01.hevs.ch server from the Sinf : there is a request form in "Demande de service" in "Comptes et accès > Obtention d'accès réseau" . More documentation to see with the Sinf here, generally for researchers the filesystem to ask for is thefs_projets.

Consider all space on CALC@HEI to be short-lived, it is not made to store data as backups : only temporary to compute and get results.