Skip to main content
Part of CALC@HEI, the HEI research computing infrastructure — governance in transition, currently operated by the ISC staff.

Open points / TODOs

Open points specific to the CALC@HEI research infrastructure (Chacha, Disco, Mambo and the planned compute expansion). Extracted from the ISC³ todo page when the two documentations were separated.

Apptainer​

  1. Quota enforcement is now in place on / and on the data filesystems of both nodes, from a versioned list — see Disk quotas. Superseded by the item below.

Disk quotas (2026-08-20)​

Backfilled after 17 accounts were found with no quota at all (incident). Still open:

  1. What a fair /data allowance actually is, is undecided. The 100 G default came from the 2025 list and was never revisited. Three users were over it and under grace on 2026-08-20; rather than have them delete 418 G between them, their limits were raised to usage +25 % — alexandre.veuthey 400 G, loic.christen 300 G, yiwei.jia 300 G — because Chacha /data had dropped to 77 % (651 G free) once martin.barry's data went, so the 100 G figure no longer had scarcity behind it. That records what people hold, not what they are entitled to: ~35 other users live with 100 G. Decide whether 100 G is the policy, and if so give the three of them real notice.
  2. Disco has no one in grace: the shared_hard and data_hard columns were rebuilt from the live per-user limits rather than a flat default, which is what the two apparent violations there turned out to be (see items 7 and 8).
  3. alexandre.veuthey's home is 27 G against a 30 G interim limit, of which 15 G is a .cache/uv package cache. Drop him to the 20 G default once it is pruned.
  4. Five decommissioned accounts (kevin.voisin, marco.caporizzi, jeremy.duc, olivier.amacker, alexandre.venturi — removed from the uid lists 2026-08-11) still have home directories on Chacha, at ~0 bytes. Deliberately absent from users_quota.list; remove the homes. martin.barry's locked account is in the same state on both hosts — his data is gone but the account itself was left in place, see journal.
  5. guillaume.bessard is not in the list because his live UID is unconfirmed (see the reconciliation item below); he keeps his hand-set 10 G/20 G until it is resolved.
  6. loic.azzalini holds 2277 G on Disco's /data/space — 53 % of what is used on that 7 T filesystem, and he had no limit there at all. The list records 2900 G, which is his current usage plus headroom, so the backfill does not break him. That number is not an approved allowance and no one has discussed it with him; the 1 TB agreed on 2026-08-20 covered Chacha's /data only.
  7. data_hard is one column for two filesystems — /data on Chacha and /data/disk01 on Disco — whose per-user limits drifted apart while they were set by hand. simon.duerr was 20 G on one and 1000 G on the other. Handled for him with a value that suits both, but the next divergence needs per-host columns in users_quota.list and a sync_quotas.sh that reads them.
  8. The shared_hard column reproduces the per-user /data/space limits as found live on 2026-08-20 (620 G, 300 G, 250 G, 1000 G, …). Nobody knows why each is what it is. Worth deciding whether they are policy or accident before the next person flattens them.
  9. A second NFS mount of Disco's /data/shared sits on Chacha at /tmp/toto, alongside the real one at /data/shared. Someone's leftover test; it will survive until reboot. Unmount it.
  10. sudo rm under /data/shared fails on Chacha — it is Disco's export and NFS squashes root. Deletions of shared data have to run on Disco, the same constraint add_quota.sh already carries for shared quotas.

Slurm controller (2026-08-20)​

  1. Core dumps are still off on Chacha, which is why the 2026-08-20 SIGSEGV (incident) left no backtrace. Two causes: the unit inherits LimitCORE soft 0, and core_pattern pipes to apport, which discards crashes from unpackaged binaries — Slurm is built from tarball. Two drop-ins are written and not yet installed, calc/provisioning/slurm/systemd/slurmctld-coredump.conf and 60-slurm-coredump.conf. systemd-coredump was rejected: on jammy it pulls a systemd upgrade and removes the apport and ubuntu-server metapackages. Revisit at the next maintenance window.
  2. slurmctld cannot chdir to its own log directory — error: chdir(/var/log): Permission denied on every start (five since 2026-08-19). It runs as SlurmUser=slurm and /var/log is root:syslog 775, so the working directory a core would land in is unwritable. Fixing LimitCORE and core_pattern per item 1 is not enough on its own; give it a writable WorkingDirectory= in the drop-in.
  3. Nothing prunes /var/crash once those drop-ins are installed — cores land there uncompressed and unrotated (slurmctld RSS is ~36 MB).
  4. slurmctld runs on Chacha only, with no SlurmctldHost backup — a Restart= drop-in covers a crash but not the host going down.

Slurm on chacha or disco​

  1. The billing-credit partitioning itself works, but it still needs real jobs to see how to tweak it. ✔ tweaked 2026-08-21 against 21 days of real sacct data — see Billing weights. Still open: discuss how to allocate credits for students specifically.
  2. For the future jump server, need to test how to restrict ssh access to other servers : via SLURM they might recreate their authorized_keys by running a job writing a .ssh/authorized_keys on the server the job is run. (change .ssh/ permission disabling them to chmod this dir?)

Data directory ownership on Chacha (2026-08-08)​

/data/disk01, /data/disk01/apptainer, /data/space and /data/space/datasets on Chacha were owned by kevin.voisin instead of root:root — fixed, see journal. Still open:

  1. Chacha also carries an older, separate flat /data/apptainer tree (root-owned) alongside /data/disk01/apptainer — looks like an unfinished migration to Disco's naming convention left two parallel trees. Not touched by the fix above; needs its own investigation (dedup? which one is actually in use?) before doing anything about it.
  2. No record anywhere of how the ownership got set this way (no journal entry near the directories' Apr 23 2026 mtime) — worth asking around if it recurs.

Admin scripts (2026-08-04)​

provisioning/nodes/scripts/ now holds the versioned, bug-fixed copies (see scripts.md). Not yet actioned:

  1. calc/provisioning/deploy.sh can now push the whole provisioning/ tree to /root/provisioning on both hosts (rsync
    • sudo cp), but hasn't been run for real yet. Decided: admins point at the new /root/provisioning/nodes/scripts/ path rather than a symlink. Root's crontab on both hosts was updated by hand to call rm_vscode.sh there instead of the flat /root/rm_vscode.sh — the only script with an automated (hourly) caller, everything else is admin-run by hand — and deploy.sh now re-asserts that cron line idempotently on every run so a future redeploy can't silently drop it. deploy.sh PRUNE_LEGACY=1 (see its header) removes the old flat /root/*.sh copies once the new tree is confirmed in place, including /root/add_sshuser.sh and /root/add_cifs_mount.sh (deleted from the repo 2026-08-05, superseded by Ansible, see Ansible/accounts.yaml reconciliation below). Still needs a human to actually run it — this session has no working SSH credentials to chacha.hevs.ch/disco.hevs.ch.

User offboarding (2026-08-04)​

Added remove_sshuser.sh and provision.py --prune (see users.md) but neither has been run for real yet. Not yet actioned:

  1. Deploy remove_sshuser.sh to /root on Chacha and Disco — it only exists in the repo so far, like the rest of provisioning/nodes/scripts/ (see Admin scripts above).
  2. Dry-run remove_sshuser.sh and provision.py --prune against a real (test) user before relying on them for an actual offboarding.

New user: Lucien Troillet (2026-08-05)​

Onboarding per users.md, simlab / dance / researcher tier. Repo side is done (see journal) — not yet actioned, to be run by a human with real access:

  1. ansible-playbook users.yml -e 'h=dance' from provisioning/ansible/ — creates the Linux account, deploys the SSH key, sets up the apptainer/dataset/results/shared_datasets dirs.
  2. ./provision.py --apply from calc/provisioning/slurm/, run from Chacha — creates the SLURM association.
  3. ./add_quota.sh lucien.troillet 20G 100G 100G, run from Disco then Chacha (researcher tier).
  4. Send the onboarding email (template), CC Pierre-André.

Ansible/accounts.yaml reconciliation (2026-08-05)​

accounts.yaml briefly grew an ssh_key field (2026-08-05) as a stopgap source of truth for SSH access, in parallel with the separately-maintained Ansible isc_compute roster. Once provisioning/ansible/ was consolidated into this repo the same day, that duplication became untenable — decided: Ansible is the trusted mechanism for Linux accounts, SSH keys and storage on Chacha/Disco (as for the rest of the fleet); accounts.yaml is SLURM-only. The ssh_key field, provision.py --ssh-key, add_sshuser.sh and add_cifs_mount.sh were reverted/deleted accordingly — see scripts.md. Not yet actioned:

  1. Roster reconciliation — done, but the UIDs it introduced are wrong, see item 4. The 19 accounts.yaml users missing from Ansible got entries in conf/users/uid-others.yml (UIDs 25506–25524) and conf/authorized_keys/, keys recovered byte-for-byte from the pre-revert ssh_key backfill (commit ec1de97). The reverse gap is expected, not a bug: kevin.voisin, marco.caporizzi, olivier.amacker stay Ansible-only since not everyone with a Linux account needs a SLURM association.

  2. Leave /root/authorized_keys/<username> on Chacha/Disco alone for now — it's dead as of the revert (nothing reads it), not worth cleaning up until the playbook run below confirms nothing still depends on it.

  3. Run ansible-playbook users.yml -e 'h=dance' for real. A real --check --diff (2026-08-05, -e ansible_user=gregory.mermoud --ask-become-pass) got past the UID checks clean — see item 4 — with only benign, non-UID changed entries. guillaume.bessard (item 4) is still unresolved and stays excluded from the roster either way. Worth one more --check --diff immediately before the real run, in case anything drifted live in the meantime.

  4. UIDs assigned in item 1 collided with reality for 18 of the 19 users — fixed 2026-08-05 via provisioning/uid/uid-map.csv, the first slice of the design in the UID register. A --check --diff dry run of users.yml -e 'h=dance' had shown 18 of the 19 already have live Linux accounts — an older, separate ad-hoc batch (personal group == UID, the plain-useradd pattern), nothing to do with the 25xxx cohort scheme. chacha and disco agreed with each other on every one of those 18 (this isn't the louis.heredero-style cross-host mismatch — see below). Applying users.yml unmodified would have usermod -u all 18 onto the wrong UID, orphaning ownership of their home dir, apptainer cache and datasets. 01_users.yml now reads uid-map.csv and overrides uid_others's id for any name it lists, before the merge/uniqueness check. Confirmed against a real ansible-playbook users.yml -e 'h=dance' --check --diff (2026-08-05, both chacha and disco reachable): all 18 came back either ok (already live-correct) or changed on a non-UID field only — the create users diff for yiwei.jia, cedric.travelletti, gregory.mermoud, theophile.gentilhomme, laura.finarelli, steve.devenes, darko.petrovic, francisco.ribeiro and sebastien.borloz said changed, but the later "Get user's info" task in the same run confirms their live UID already matches uid-map.csv exactly — the diff is shell/GECOS, not a UID renumber. No usermod in the plan for any of the 18. uid-others.yml also had francisco.ribeiro, guillaume.bessard and sebastien.borloz missing entirely — an accidental deletion in commit 620e370, unrelated to that commit's stated purpose. Restored francisco.ribeiro and sebastien.borloz. Still open: guillaume.bessard — the dry-run read him as new on both hosts but also surfaced a stray uid 10001, so his live UID is unclear; deliberately left out of both uid-others.yml and uid-map.csv rather than guessed. Needs a direct getent passwd guillaume.bessard on both hosts before he can be added back.

    A first attempt at this also touched uid-isc3-2526.yml's louis.heredero, changing 25304 → 10020 on the strength of uid-allocation.md's claim that 10020 is his live UID "everywhere else" — wrong for chacha/disco specifically, since that claim was about NAS-home-ownership and Calypso, not dance. The same --check --diff run caught it: create users showed changed for him on both hosts, and "Get user's info" confirmed live UID 25304 on both. uid_isc3_2526 is shared between calypso_uids and dance_uids (roles.yml) with no per-host-group override the way uid_others now has via the CSV — so whatever this file says can only be right for one side if Calypso and dance really disagree. Reverted to 25304, correct for dance; Calypso's real number for louis.heredero is still unconfirmed and needs its own check (not via this CSV, which doesn't cover uid_isc3_2526) before users.yml ever runs against calypso. The wider NAS-vs-Calypso drift for the rest of this cohort (kevin.voisin, marco.caporizzi, jeremy.duc, olivier.amacker — see uid-allocation.md's own numbers) is untouched by this change: those UIDs are already live on Calypso, so correcting them is a separate live-renumbering job (re-chowning files outside the home dir too), deliberately out of scope here.

Job priority (2026-08-21)​

Multifactor priority is in service — QOS, fairshare and age all carry weight, see Scheduling and the history entry. Still open:

  1. QOS priorities are not managed from accounts.yaml. ✔ done 2026-08-21: --sync-qos converges Priority too, and warns about any live QOS the file declares nowhere — see Provisioning.
  2. The standby ordering has only been checked arithmetically. Confirm it on a live pending standby job queued behind an in-limits one — a running job keeps the priority it started with, so the standby jobs in the queue during the change still read their old values.

Standby QOS (2026-08-12)​

standby_qos is live — slurm.conf preemption settings, setup_standby_qos.sh's QOS/Preempt= wiring, and the standby_qos grant to every account/user (provision.py --apply && --fix-accounts --apply) all confirmed applied 2026-08-12. Still open:

  1. Sanity-check with a real preemption: submit a long standby_qos job, then a normal-QOS job that needs the same resources, confirm the standby job gets requeued rather than just trusting the config.
  2. /home/gregory.mermoud/management/ on chacha is a manually-placed copy of provisioning/accounts.yaml + provisioning/slurm/, not a git checkout — it had already drifted from the repo before this change (an extra note about dion.osmani/kevin.voisin sitting directly under premium_rs, not yet copied back). Worth turning into a real clone at some point so --apply runs are provably running the reviewed version.

Slurm accounts (new provisioning tool, 2026-08-04)​

provisioning/accounts.yaml + provision.py is now the source of truth for accounts/users (see SLURM configuration). --check against the live cluster found:

  1. Move allowances to MaxTRESPA on the tier QOS. Decided 2026-08-19: an allowance is group-wide, premium or standard, with no per-group values. GrpTRES on a QOS is one pool shared by every account using it (tier-wide), which is why premium_dance_qos sits saturated at cpu=44(44) while the accounts still on normal run against their own caps — see Limits model. The order matters, nothing may be left unbounded:

    1. On the six QOS: clear GrpTRES, set the same numbers as MaxTRESPA.
    2. Point the 14 Dance rows still on normal or unset at their tier's QOS. ✔ done 2026-08-19 via provisioning/slurm/migrate_group_limits.sh --apply; premium_dance_qos went from cpu=44(44) saturated to no group pool at all.
    3. Clear GrpTRES/MaxTRES on the group accounts, paired with removing normal from the association QOS lists.
    4. Clear the redundant per-user GrpTRES on the Dance user rows. ✔ done 2026-08-19 via clear_account_limits.sh --apply: 18 account rows and 14 user rows cleared, normal dropped from 91 associations. The per-user caps mattered more than expected — students were held to 24 CPUs individually against a group allowance of 40, and they bound standby_qos jobs, so standby was not actually unlimited until they went.

    Then decide the three QOS values that match no hardware, and teach provision.py to manage the QOS numbers from accounts.yaml. ✔ done 2026-08-19: accounts.yaml declares a share per tier (premium 1/2 of a partition, standard 1/4) and provision.py --sync-qos multiplies it by the live scontrol show partition capacity, so no QOS number is typed anywhere — see QOS and Limits.

  2. Reconcile the shard numbers against the hardware. ✔ done 2026-08-19: every shard cap is now derived from the partition's live count (Chacha 192, Disco 160, Dance 352), which also settled the old claim that Disco had 80. The stale 80 in slurm-examples is corrected too.

  3. Decide the simlab tier: accounts.yaml says standard_rs, live it is still parent=premium_rs carrying both tiers' QOS, so alexandre.veuthey and nachiket.chondhikar default to standard on Chacha/Disco but premium on Dance, with premium reachable everywhere via --qos=. Either revert the tier in the file, or apply the three subtractive commands (modify account simlab set parent=standard_rs, qos-=premium_* on the account and on both users) — provision.py only ever adds, so it will not do this.

  4. Remove the 11 associations left under simlab, dymonet, optival, vortexdynamics and omniperso after the 2026-08-19 consolidation. ✔ done 2026-08-19, 10 of 11 via provision.py --prune-extra --apply. Still open: marc.gillioz's simlab row, held back because 21 array jobs of his were charging it. His queue is empty as of 2026-08-21, so --prune-extra --apply will take it now. Then the emptied accounts (dymonet, optival, omniperso, vortexdynamics, eventbasedcpc, pmpm) can go too; simlab stays, alexandre.veuthey and nachiket.chondhikar are still in it.

  5. yannic.delisle keeps a partition-less association under mattech alongside their per-partition rows. Harmless — the per-partition rows win for every partition — but --check reports it each run, and since 2026-08-19 it is one of only two rows still granting the limit-free normal QOS (it has no DefaultQOS, so dropping normal would leave it unable to submit). Removing it means dropping and re-adding all their mattech associations, so it needs a quiet moment with no jobs of theirs queued.

  6. dion.osmani and kevin.voisin sit directly under premium_rs (no project account) with standard-tier limits live, not premium. ✔ no such associations exist as of 2026-08-21.

  7. Remove the test2 admin account — dropped from accounts.yaml already, still present live. test and the stray and user are already gone.

  8. premium_rs still carries MaxTRESPJ, the per-job half of the tier cap. ✔ done 2026-08-19, cleared with the other account limits in item 1.

Dance New (compute expansion)​

  1. Check spares for EPYC servers / order some discs, fans, power supplies
  2. Finish the Mellanox switch IP configuration, to put in the new Sinf subnet 10.5.1.148/24 / GW 10.5.1.1 / DNS 10.130.0.11,10.130.1.11
  3. Test and configure BeeGFS on the EPYC 48TB storage
  4. Configure the storage infiniband network
  5. Create SLURM Test Partition : using shard on Disco, or put the current Rumba Dell 7920 with all 3 Nvidia RTX GPUS, and put the test partition on it
  6. Change the creation script to make the "Test" partition the default partition when a researcher arrives on the ISC Compute, then when they are ready to run assign the "Dance" partition
  7. Apply Data quota on all ISC compute users, not the case for everyone yet
  8. Check the BeeGFS quota mecanism to migrate EXT4 quota from current Disco/Chacha to the new Epyc storage
  9. Script a wrapper on Apptainer to check execution context and refuse to run directly bare-bone : same for python or other execcutable to avoid run out of SLURM
  10. Migrate current NVMe data disks from Disco/Chacha to BeeGFS when it will be tested and ready also on EPYC
  11. Automate file deletion for Standard (Premium too?) researchers to avoid having scratch partition with old tests files / Set in meeting what TTL we want : 2 weeks standard TTL ? More for Premium ?
  12. Rename the Filesystem : datasets -> workspace? local_workspace? chacha_workspace ? local_scratch ? / shared -> network_workspace ? remote_workspace ? remote_scratch ?
  13. Migrate Prometheus from Chacha to the new EPYC server / Add some alerting on common checks, disks, jobs outside of slurm etc...
  14. Move NVMe disks ? Disco 7TB to Chacha ? / NVMe 3TB from Calypso storage to Disco ? Format as BeeGFS
  15. Check for a Modules installation ? Or Apptainer is already fine ? : Install LMOD to allow dynamic lib loading : where to put the terabytes of libraries for Dance ?

Note : the EPYC servers and the Mellanox switch are physically part of the ISC³ rack and are (for now) also part of the ISC³ Proxmox plans — who ends up owning them is a Phase 2 governance question.

Documentation​

  1. Rewrite Connecting to CALC@HEI once the jump host is ready — it currently tells students to SSH straight to Chacha and Disco from the school network or the HEVS VPN.
  2. calc/using/start.md still calls a phase "initial" whose dates ended in 2025; the three phases need re-dating by someone who knows where the roadmap stands. ✔ replaced 2026-08-22 with a plain-language SLURM concepts/scheduling section and a section on the current students account configuration — no more phase dates to keep current.

Backups​

  1. Backups for Disco and Chacha : to define (see the ISC³ backups architecture for the rest of the fleet).

Playbooks​

  1. Finish the isc_compute system playbook, especially the Mambo login node part, and new configs to avoid direct connection on compute nodes
  2. Finish the slurm_research_TODO playbook (redo all installation part from compilation install, separate install from config)

The Ansible environment itself is currently shared with ISC³ — see Ansible.