Skip to main content
Part of CALC@HEI, the HEI research computing infrastructure — governance in transition, currently operated by the ISC staff.

Ops journal

Completed actions, newest first. This is where finished items from the todo land, so that page only ever contains live work. Each entry links to the page holding the details — the journal is the index, not the archive.

  • 2026-09-16 — Glorymary Givi TrunX shared quota raised from 100G to 200G — increased only the shared_hard limit for glorymary.givi in users_quota.list. TrunX simulations fan out into many jobs and temporarily fill shared_datasets; with the previous 100G limit, data had to be manually offloaded to cloud storage between simulation runs. / remains at 20G and local /data remains at 100G; only the shared storage quota on Disco is increased.

  • 2026-09-11 — yacine.said given cluster-admin sudo — ansible-playbook users.yml -e 'h=dance' created the Linux account and granted passwordless sudo (sudonopass group) on Chacha and Disco; config already carried him in the uid_admin cohort (provisioning/ansible/conf/users/uid-admin.yml) with his key in conf/authorized_keys/, so this run was the first to apply it live. No SLURM association — he administers the cluster, not submits jobs — see Ansible roles.

  • 2026-08-26 — aggregate wall-time limits removed — GrpWall on the two Dance QOS and 22 associations was cancelling running jobs and recording them TIMEOUT; it is an aggregate budget for a whole group, not the per-job walltime the add-user recipe treated it as — incident, history, Aggregate wall time.

  • 2026-08-22 — user-facing SLURM concepts documented — Using CALC@HEI replaced its stale, date-bound phase timeline with a plain-language explanation of accounts, partitions, QOS and fairshare, plus a table of the students account's current per-partition allowances — the students running project work with us, not a course.

  • 2026-08-21 — thread defaults and billing weights fixed — Apptainer jobs were spawning thread pools sized off the node's physical core count instead of their SLURM allocation, adding 40 spurious runnable threads per job to the load average (TaskProlog now exports OMP_NUM_THREADS etc. from SLURM_CPUS_PER_TASK); separately, TRESBillingWeights on Dance priced a GB-hour at 1024× a CPU-hour, so fairshare was effectively a memory-only metric — history, Thread defaults, Billing weights.

  • 2026-08-21 — job priority was job size and nothing else — the fairshare and age weights were 0 and PriorityDecayHalfLife was 10 seconds, so the documented 750/250 tier design contributed nothing to the queue order. Shares levelled, MaxWall and the per-job MaxTRES moved off the associations onto the QOS, and standby_qos pinned below every in-limits QOS. Every QOS value now comes from accounts.yaml — history, Scheduling.

  • 2026-08-20 — slurmctld SIGSEGV, 3 h 30 without scheduling — a null dereference in a worker thread killed the controller; the packaged unit has no Restart=, so nothing retried until someone ran squeue. Running jobs were untouched and the restart recovered cleanly from StateSaveLocation. A Restart=on-failure drop-in now covers it — incident, Controller resilience.

  • 2026-08-20 — 17 accounts had no disk quota — /root/users_quota.list had not been touched since 2025-02-28, so every account created since was unlimited; one home reached 266 G and took / to 99 %, with slurmctld's state directory on the same filesystem. The list and an idempotent sync_quotas.sh are now versioned and enforced by the Ansible common role. Three users on Chacha /data are over the new targets and under a 7-day grace — incident, Disk quotas.

  • 2026-08-20 — martin.barry's data removed after departure — ~907 G across both nodes (556 G Chacha /data/datasets, 337 G Disco /data/shared, homes and Apptainer caches). Access, roster and SLURM entries had already been revoked; this was the retention decision the offboarding process defers. Chacha /data went 97 % → 77 %, Disco /data/space 66 % → 61 %. The locked account itself is still present on both hosts.

  • 2026-08-19 — Group allowances now live on the QOS alone — the account-level GrpTRES/MaxTRES backstop is gone: 18 account rows and 14 user rows cleared, and normal dropped from 91 associations so it cannot be used to bypass the allowance. The per-user caps turned out to bind — students were held to 24 CPUs each against a group allowance of 40 — and they applied to standby_qos too, so standby only became genuinely unlimited once they went. Limits model, Standby QOS.

  • 2026-08-19 — QOS allowances derived from the hardware — every cap was hand-typed and three of them described hardware that does not exist (premium_disco_qos allowed 1000 G on a 244 GiB node, two QOS allowed 2 nodes on single-node partitions). accounts.yaml now declares a share per tier — premium half a partition, standard a quarter — and provision.py --sync-qos multiplies it by the live scontrol show partition capacity, so adding memory or a node moves the caps by itself. Premium on Dance went 44 → 80 CPU, memory 1000 G → 625 G. QOS and Limits.

  • 2026-08-19 — Group allowances moved onto the tier QOS — GrpTRES on a QOS is one pool shared by every account using it, so the six premium groups were competing for a single 44-CPU ceiling that sat permanently saturated while accounts still on normal ran unbounded. The six QOS objects now carry MaxTRESPA instead, which caps each group separately, and all 31 Dance associations default to their tier's QOS. Group caps stay on the accounts as a backstop until step 3 — Limits model.

  • 2026-08-19 — Dance queue deadlock: memory default and the premium_rs tier cap — a 68-job pipeline sat behind an sbatch --wrap="true" barrier which, with no DefMemPerCPU set, was charged Chacha's entire memory and so exceeded its account cap; underneath, premium_rs held its six groups to 88 of the cluster's 160 CPUs. DefMemPerCPU=3900 set on both nodes and the tier cap cleared — Memory defaults, Accounts.

  • 2026-08-19 — Per-partition SLURM associations — jobs to Chacha or Disco were rejected for every user but one: accounts.yaml scoped each association to Dance, and SLURM matches the association of the partition submitted to. Users now hold one association per partition with that partition's QOS as default, applied live (86 + 11 sacctmgr commands, additive). The same run carried the group consolidation: simlab-core/-trunx/-hydro created, five accounts folded into generalaccess, no user lost, and 10 of the 11 superseded associations removed with the new --prune-extra (the 11th holds while its owner's jobs drain). Partition lists stay unsupported — see Associations and partitions.

  • 2026-08-19 — Closed items moved off the todo page — five items were sitting there struck through instead of here: the Apptainer cache relocation — APPTAINER_CACHEDIR under /data/apptainer/<user>/.apptainer and APPTAINER_TMPDIR=/tmp, existing .apptainer dirs migrated and symlinked from each home, both GPUs exposed in the gres/slurmd configs, job start/end mails, the billing-credit resource partitioning, and the sshfs/VS Code guidance now on runjob plus the .vscode sweep. Their completion dates were never recorded.

  • 2026-08-18 — CALC@HEI logo — a mark of its own, built from the ISC palette: five arcs in the orientation colours around a magenta die, wordmark in DM Sans. Nine SVGs in static/img/calc/; used on the CALC home page, the governance banner and the ISC³ home card.

  • 2026-08-12 — RFQ review comments addressed — pmudry's review comments on the EPYC compute node RFQ docs (PR #12) applied: Xeon-based Dell/HPE models swapped for their EPYC equivalents (XE7745/R7725, DL385), GPU node RAM repopulated across all 24 memory channels (24×16 GB instead of 8×32 GB), H200/CUDA 12.x compatibility flagged as unverified, Genoa DDR5-4800 cap noted. A general (non-inline) comment was missed in the first pass: the Réseau/NIC lines conflated Ethernet GbE with the existing Mellanox MQM8700 switch, which is InfiniBand HDR — respecced as InfiniBand ConnectX-6 (HDR or HDR100 via splitter cable) in both docs. RFQ docs: calc/roadmap/compute/.

  • 2026-08-12 — Standby QOS shipped — standby_qos, an opportunistic/preemptible QOS with no limits of its own so users can run past their account's caps on idle capacity, requeued when a normal/premium/standard job needs the resources back. accounts.yaml/provision.py grant it to every account/user; setup_standby_qos.sh and the slurm.conf preemption settings applied live and verified (Preempt=standby_qos on normal too, since some accounts still run under it rather than the per-partition QOS). Standby QOS

  • 2026-08-11 — Wording pass over the CALC pages — flattened emphasis used to insist across 10 files (the **NOTE : whole sentence** pattern, you **MUST**), no facts or commands touched. Rule in the root CLAUDE.md; findings filed on the ISC³ todo.

  • 2026-08-11 — CALC operations TODO renamed — What still needs to be fixed was renamed to Open points / TODOs in the CALC operations sidebar. Open points / TODOs

  • 2026-08-10 — Offboarded remi (uid 1002) and pamela.delgado (uid 1005), both flagged STILL LIVE by uid-liveness.py — departed in uid-map.csv but still had working SSH access on Chacha/Disco. Neither had roster/SLURM/quota entries, so revoking access via remove_sshuser.sh on both hosts (Disco first) was the whole fix, same as kilian.leger below.

  • 2026-08-10 — Offboarded kilian.leger (uid 10500 on Chacha/Disco, never logged in, no roster/SLURM/quota entries) via remove_sshuser.sh on both hosts. Also fixed uid-liveness.py false-flagging his account as oscar.esteban "still live" — same number, unrelated person, dance's uid namespace is disconnected from the NAS register (see uids.md).

  • 2026-08-09 — Drafted requirement/BOM documents for two new AMD EPYC compute nodes (CPU-only, 192 cores/6 GB per core; GPU, 2× H100 or H200 NVL/4 GB per core), following the storage server doc's Typst template. See calc/roadmap/compute/ and todo for the RFQ next steps.

  • 2026-08-08 — Fixed kevin.voisin owning parent directories under Chacha's /data/disk01 and /data/space instead of root:root — two directory levels each (the mountpoint and the tier subdirectory), though every individual user's own leaf directory checked out fine throughout (his UID 25301 is a known drift-prone one). Disco's equivalent tree was unaffected. Ran fix_data_mount_ownership.sh (see scripts.md), verified all four paths now root:root and kevin.voisin's own leaf directories untouched. Still open: the duplicate /data/apptainer tree on Chacha — see todo.

  • 2026-08-06 — Unlinked accounts.yaml and deploy.sh.

  • 2026-08-05 — users.md now requires a PR, approved by an admin, for every onboarding/offboarding — the diff is the audit trail for who was granted or revoked access. Prompted by Lucien Troillet's onboarding below going through a PR for the first time.

  • 2026-08-05 — Onboarded Lucien Troillet to CALC@HEI per users.md: uid 25525 added to uid-others.yml, SSH key added, and an accounts.yaml entry added (group simlab, partition dance). provision.py --validate passed locally. Applying to the live systems (Ansible, provision.py --apply, quotas) is left for the user to run — see todo.

  • 2026-08-05 — Started executing the UID register design: added provisioning/uid/uid-map.csv, a committed source of truth for UIDs that 01_users.yml now reads to override conf/users/uid-others.yml's id for any name it lists. Seeded with the 18 confirmed chacha/disco UIDs that made users.yml -e 'h=dance' unsafe to run — guillaume.bessard stays out pending a live getent check. Also restored francisco.ribeiro/sebastien.borloz to uid-others.yml (accidentally dropped in commit 620e370).

    First attempt also "fixed" uid-isc3-2526.yml's louis.heredero from 25304 to 10020 on the strength of uid-allocation.md's claim that 10020 is live "everywhere else" — wrong for chacha/disco specifically, since that claim was about NAS-home-ownership and Calypso, not dance. A real --check --diff against h=dance caught it: live UID on both chacha and disco is 25304, and applying 10020 would have usermod'd a live account. Reverted to 25304 for dance's sake; Calypso's number needs its own separate check before users.yml ever runs against calypso. See todo — running the playbook for real is still open.

  • 2026-08-05 — Root's crontab on Chacha and Disco was updated by hand to call rm_vscode.sh at its new /root/provisioning/nodes/scripts/ path instead of the flat /root/rm_vscode.sh — the only admin script with an automated caller. calc/provisioning/deploy.sh now re-asserts that cron line idempotently on every run, and gained PRUNE_LEGACY=1 to remove the old flat /root/*.sh copies (including add_sshuser.sh/add_cifs_mount.sh) once redeployed. See scripts.md and todo — still needs to actually be run against Chacha/Disco.

  • 2026-08-05 — Reconciled the accounts.yaml/Ansible user rosters: added the 19 accounts.yaml users missing from Ansible's conf/users/uid-others.yml (UIDs 25506–25524) and conf/authorized_keys/, recovering their real keys byte-for-byte from the pre-revert ssh_key backfill instead of re-deriving them. The Ansible-only names (kevin.voisin, marco.caporizzi, olivier.amacker) are left as-is by design. Still open: running ansible-playbook users.yml -e 'h=dance' for real, and redeploying provisioning/ to /root via deploy.sh — both need a working SSH session to Chacha/Disco that this session doesn't have. See todo.

  • 2026-08-05 — Reconciled the duplicate SSH-key/account provisioning that emerged once provisioning/ansible/ was consolidated into this repo: Ansible is now the trusted mechanism for Linux accounts, SSH keys and storage on Chacha/Disco, matching the rest of the fleet. Reverted the ssh_key field and provision.py --ssh-key from accounts.yaml (SLURM-only again) and deleted add_sshuser.sh / add_cifs_mount.sh (fully superseded by Ansible's users.yml playbook). SLURM policy stays in provision.py, deliberately not folded into Ansible — different kind of logic (sacctmgr diffing, not desired-state convergence), chacha-only, and already scoped to CALC@HEI ahead of a possible future pool split. See scripts.md and todo for the roster-reconciliation work still open.

  • 2026-08-05 — Backfilled ssh_key in provisioning/accounts.yaml for 19 of the ~30 users, pulling real key material from /root/authorized_keys/<username> on Chacha/Disco (identical on both hosts) and verifying each against the live files. The other 11 have no key file on either host, so nothing to backfill for them yet. Old /root/authorized_keys/<username> files not removed and provisioning/ not redeployed — still open, see todo.

  • 2026-08-05 — Made provisioning/accounts.yaml (moved out of slurm/, it's no longer SLURM-only) the source of truth for SSH access too: users can carry an ssh_key, and add_sshuser.sh reads it via the new provision.py --ssh-key mode, falling back to the older /root/authorized_keys/<username> file for anyone not yet migrated. No existing user's key was backfilled — see todo.

  • 2026-08-05 — Added calc/provisioning/deploy.sh to push the provisioning/ tree to /root/provisioning on Chacha and Disco. SSH access there is a personal account with sudo, not root, so it stages via rsync under the user's home first, then sudo cps into place; additive only, like provision.py. Not yet run for real — see todo.

  • 2026-08-04 — Added a user offboarding path: remove_sshuser.sh (locks the account, revokes the SSH key, leaves data untouched) and a --prune mode on provisioning/slurm/provision.py (removes SLURM users no longer in accounts.yaml). Documented as the How to remove a user checklist. Not yet deployed to /root or run for real — see todo.

  • 2026-08-04 — Deleted add_slurmuser.sh (superseded by provisioning/slurm/) and batch_add_sshusers.sh (never worked) from the repo and from /root on both Chacha and Disco. Updated users.md to document the accounts.yaml + provision.py workflow instead. See scripts.md.

  • 2026-08-04 — Repatriated the /root admin scripts from Chacha and Disco into provisioning/nodes/scripts/ for versioning. Confirmed /root is a local filesystem on each host, not shared — the two copies were kept in sync by hand. Found and fixed 5 correctness bugs (broken group-check test, dead code, a non-idempotent sudoers append, a Chacha-only hardcoded path, a wrong chmod) and 2 hosts' worth of deployment drift. See scripts.md for the full list; todo for what's still open.

  • 2026-08-04 — Added provisioning/slurm/ (accounts.yaml + provision.py) as the declarative source of truth for SLURM accounts/users on Chacha/Disco, replacing manual sacctmgr/add_slurmuser.sh calls for new provisioning. See SLURM configuration. Its --check mode found existing drift, tracked in todo.