Using CALC@HEI

Welcome to CALC@HEI, the HEI research computing infrastructure.

Everyone shares the same hardware and submits jobs through the same queue. This page explains how that sharing works, before you submit your first job.

Rules of the game

• Submit your work as an Apptainer job through SLURM. See How to run a job.
• This is a best-effort service: there is no guaranteed uptime or support. Search the docs first, and help improve them when you can.

How SLURM works

SLURM is the scheduler. You describe what your job needs — CPUs, memory, GPU, how long it runs — and SLURM queues it, starts it once the resources are free, and enforces those limits while it runs.

A few terms explain most of what you see in squeue and sacct:

• Account — your project. Limits and priority are set per account, not per user, so everyone in your account shares the same pool.
• Partition — the nodes a job runs on: Chacha, Disco, or Dance (both nodes, one shared queue). Pick one.
• QOS (Quality of Service) — the rules a job runs under: how long it can run, how much of your account's allowance one job can use, and its priority. Your account has a default QOS per partition, so you usually don't need to set one yourself.
• Fairshare — SLURM remembers how much CPU, memory and GPU your account has used lately, and lowers your priority the more you've used compared to others. That memory fades over about a week. It is not a budget you can run out of, and it never stops you from submitting — it only changes your place in the queue.

What decides when your job starts

Three things, in order: whether the job is within your account's normal allowance or running under standby_qos (see below); your account's fairshare standing; and how long the job has waited. Heavy recent use never gets a job refused — it can only push it later in the queue.

Asking for more than your account's allowance doesn't get the job refused either. It just waits — squeue shows a reason like Reason=AssocGrpCpuLimit — until your account's usage drops or your other jobs finish.

Standby: extra room when the cluster is idle

If your account is already at its cap but the cluster has free capacity, add --qos=standby_qos. It has no cap of its own, but it can be stopped and requeued the moment an in-limits job needs the resources back. Only use it for jobs that can restart from scratch. See SLURM examples for the sbatch flags.

For the exact numbers and how they're worked out, see SLURM configuration.

Students today

Students carrying out projects with us share one SLURM account, students. It sits in the standard_rs tier, alongside simlab, digitallifescience and generalaccess. That tier's priority weight is 250, against 750 for premium accounts, so on a busy cluster premium jobs go first. On an idle cluster it makes no difference: student jobs start right away either way.

The tier sets these limits, one per partition, for the whole students account — not per student, so jobs from several students at once share the same cap:

Partition	Walltime	One job may hold
Chacha	4 h	24 CPUs, 1 GPU, 48 shards, ~252 GB
Disco	4 h	16 CPUs, 1 GPU, 40 shards, ~61 GB
Dance	1 d	40 CPUs, 1 GPU, 88 shards, ~313 GB

A job that asks for more than this waits in the queue rather than failing. standby_qos is available too, for jobs that can tolerate being restarted.

How to credit individual students within that shared allowance is still an open question — see the ops todo.

Common pitfalls

Why won't my job start?

Run squeue -u $USER and look at the REASON column:

• Priority — other jobs are ahead of yours right now. See fairshare above.
• Resources — the cluster is full. Your job starts as soon as something frees up.
• AssocGrpCpuLimit, AssocGrpMemLimit, AssocGrpGRES — your account is already at its allowance (see Students today, or your own account's limits). The job waits until another job from your account finishes, or you add --qos=standby_qos.
• QOSMaxWallDurationPerJobLimit — you asked for more walltime than your QOS allows. This job will never start as submitted; cancel it and resubmit with a shorter --time.
• Dependency — it is waiting on another job you chained it to.

scontrol show job <jobid> gives the full detail behind any of these.

I asked for too little

• Too little --mem: the job is killed the moment it crosses that limit, not slowed down. sacct -j <jobid> --format=State,ExitCode shows OUT_OF_MEMORY when this is what happened.
• Too little --time: the job is cancelled at that point with State=TIMEOUT, whatever it was doing.
• Too few --cpus-per-task for code that assumes many threads: it still runs, just slower — see thread defaults for why that no longer oversubscribes the node.

I asked for too much

A job never fails for asking for more than it needs. It just holds resources it never uses, which pushes other jobs from your account further back in the queue (see what decides when your job starts) and lowers your account's fairshare for room it never needed. Size requests from what a job actually used, not a round number — see below.

Checking what a past job actually used

sacct -j <jobid>.batch --format=JobID,Elapsed,TotalCPU,MaxRSS,State

The .batch line carries the real numbers; the plain job ID row can under-report TotalCPU. Compare TotalCPU against Elapsed × --cpus-per-task for CPU efficiency, and MaxRSS against the --mem you asked for. Check a job or two before scaling up a sweep — a large gap there is usually spurious threads or an oversized reservation, and both cost your account priority for nothing in return.

Where to go next

1. Requesting access — who to contact and what to prepare
2. Connecting — SSH to the compute nodes
3. Submitting jobs — the full workflow, from container to results
4. SLURM job examples — serial, MPI, GPU, interactive
5. Building Apptainer containers — package your environment
6. Storage and quotas — where to put data, and for how long
7. FAQ — common questions and pitfalls

The machines themselves — Chacha, Disco and the SLURM configuration — are described in the CALC@HEI overview.