What still needs to be fixed

Open points specific to the CALC@HEI research infrastructure (Chacha, Disco, Tango and the planned compute expansion). Extracted from the ISC³ todo page when the two documentations were separated.

Apptainer

Make that everyone exports (after adaptation of course) : export APPTAINER_CACHEDIR=/scratch/gpfs/$USER/APPTAINER_CACHE export APPTAINER_TMPDIR=/tmp ✔ done : Created a /data/apptainer/user.name/.apptainer, migrated the current .apptainer users dirs and added a symlink in their home.

1. To prevent quota explosion : Installed the quota package on disco and chacha :. should we enforce quota on FS level ? All the people who said they would use less than 100GB are using more than what they said (ex martin.barry at 350GB) : Applied quota on / filesystem, not yet on datasets.

Slurm on chacha or disco

1. Make both GPUs available in gres/slurmd confs ✔ done
2. Make emails working for start/end of jobs, use an emailer ✔ done
3. Find how to do the ressource partitioning with billing credits by user / account ✔ done (but still needs tests and real jobs to see how to tweak)
4. Discuss how to allocate credits for users : what about students ?
5. Note everywhere to either remove sshfs for VScode, and give links to properly configure it or no VScode at all : Noted on runjob and started script to check for .vscode in homedirs : auto-rm in crontab directly ? Done
6. For the future jump server, need to test how to restrict ssh access to other servers : via SLURM they might recreate their authorized_keys by running a job writing a .ssh/authorized_keys on the server the job is run. (change .ssh/ permission disabling them to chmod this dir?)

Admin scripts (2026-08-04)

provisioning/nodes/scripts/ now holds the versioned, bug-fixed copies (see scripts.md). Not yet actioned:

1. provisioning/deploy.sh can now push the whole provisioning/ tree to /root/provisioning on both hosts (rsync
   • sudo cp), but hasn't been run for real yet. Decided: admins point at the new /root/provisioning/nodes/scripts/ path rather than a symlink. Root's crontab on both hosts was updated by hand to call rm_vscode.sh there instead of the flat /root/rm_vscode.sh — the only script with an automated (hourly) caller, everything else is admin-run by hand — and deploy.sh now re-asserts that cron line idempotently on every run so a future redeploy can't silently drop it. deploy.sh PRUNE_LEGACY=1 (see its header) removes the old flat /root/*.sh copies once the new tree is confirmed in place, including /root/add_sshuser.sh and /root/add_cifs_mount.sh (deleted from the repo 2026-08-05, superseded by Ansible, see Ansible/accounts.yaml reconciliation below). Still needs a human to actually run it — this session has no working SSH credentials to chacha.hevs.ch/disco.hevs.ch.

User offboarding (2026-08-04)

Added remove_sshuser.sh and provision.py --prune (see users.md) but neither has been run for real yet. Not yet actioned:

1. Deploy remove_sshuser.sh to /root on Chacha and Disco — it only exists in the repo so far, like the rest of provisioning/nodes/scripts/ (see Admin scripts above).
2. Dry-run remove_sshuser.sh and provision.py --prune against a real (test) user before relying on them for an actual offboarding.

Ansible/accounts.yaml reconciliation (2026-08-05)

accounts.yaml briefly grew an ssh_key field (2026-08-05) as a stopgap source of truth for SSH access, in parallel with the separately-maintained Ansible isc_compute roster. Once provisioning/ansible/ was consolidated into this repo the same day, that duplication became untenable — decided: Ansible is the trusted mechanism for Linux accounts, SSH keys and storage on Chacha/Disco (as for the rest of the fleet); accounts.yaml is SLURM-only. The ssh_key field, provision.py --ssh-key, add_sshuser.sh and add_cifs_mount.sh were reverted/deleted accordingly — see scripts.md. Not yet actioned:

1. Roster reconciliation — done, but the UIDs it introduced are wrong, see item 4. The 19 accounts.yaml users missing from Ansible got entries in conf/users/uid-others.yml (UIDs 25506–25524) and conf/authorized_keys/, keys recovered byte-for-byte from the pre-revert ssh_key backfill (commit ec1de97). The reverse gap is expected, not a bug: kevin.voisin, marco.caporizzi, olivier.amacker stay Ansible-only since not everyone with a Linux account needs a SLURM association.

2. Leave /root/authorized_keys/<username> on Chacha/Disco alone for now — it's dead as of the revert (nothing reads it), not worth cleaning up until the playbook run below confirms nothing still depends on it.

3. Run ansible-playbook users.yml -e 'h=dance' for real once item 4 below is fixed. Blocked on two things: no working SSH credentials to chacha.hevs.ch/disco.hevs.ch in this session (publickey/password both rejected — must run by hand), and the UID conflict in item 4, which would actively break live accounts if applied as currently written.

4. UIDs assigned in item 1 collide with reality for 18 of the 19 users. A --check --diff dry run of users.yml -e 'h=dance' (2026-08-05) showed 18 of the 19 already have live Linux accounts — an older, separate ad-hoc batch (personal group == UID, the plain-useradd pattern), nothing to do with the 25xxx cohort scheme. chacha and disco agree with each other on every UID confirmed below (this isn't the known louis.heredero-style cross-host mismatch). Applying users.yml unmodified would usermod -u all 18 onto the wrong UID, orphaning ownership of their home dir, apptainer cache and datasets. Confirmed live UIDs (both hosts unless noted) vs. the wrong value currently in uid-others.yml:

   user	wrong (assigned)	live UID
   jaime.barrancohernandez	25506	1017
   mauro.leidi	25507	1016
   yannick.bovier	25508	1024
   yiwei.jia	25509	1010
   loic.azzalini	25510	1008
   cedric.travelletti	25511	1006
   gregory.mermoud	25512	1004
   marta.rende	25513	10008
   theophile.gentilhomme	25514	1018
   simon.duerr	25515	1020
   laura.finarelli	25516	1009
   steve.devenes	25517	1012
   gregoire.castelain	25518	1023
   darko.petrovic	25519	1022
   roland.willa	25520	1021
   francisco.ribeiro	25521	10009 (chacha only — missing on disco)
   martin.tavernier	25523	10010
   sebastien.borloz	25524	10002 (chacha only — missing on disco)
   guillaume.bessard	25522	unclear — dry run read as new on both hosts but also surfaced a stray 10001; needs a direct getent passwd guillaume.bessard check on both hosts, don't trust the dry-run log here

   Fix: rewrite uid-others.yml to the live UIDs above before running users.yml for real; for francisco.ribeiro/sebastien.borloz use the chacha UID on disco too so the account matches across both hosts.

Slurm accounts (new provisioning tool, 2026-08-04)

provisioning/accounts.yaml + provision.py is now the source of truth for accounts/users (see SLURM configuration). --check against the live cluster found:

1. Every association is still on QOS=normal with hand-set GrpTRES instead of the per-partition QOS objects that already exist (premium_dance_qos, standard_disco_qos, etc.) — those objects have the correct numbers (e.g. Disco's real 80-shard cap), the hand-set values don't (some standard_rs rows have shard=96, unreachable on Disco). Needs a decision on when to cut over — it's a live limits change for people with running jobs, not a mechanical add.
2. dion.osmani and kevin.voisin sit directly under premium_rs (no project account) with standard-tier limits live, not premium — give them a real project account.
3. Remove the test/test2 admin account and the stray and user under premium_rs (leftover typo, share=1, no real association) — dropped from accounts.yaml already, still present live.

Dance New (compute expansion)

1. Check spares for EPYC servers / order some discs, fans, power supplies
2. Finish the Mellanox switch IP configuration, to put in the new Sinf subnet 10.5.1.148/24 / GW 10.5.1.1 / DNS 10.130.0.11,10.130.1.11
3. Test and configure BeeGFS on the EPYC 48TB storage
4. Configure the storage infiniband network
5. Create SLURM Test Partition : using shard on Disco, or put the current Rumba Dell 7920 with all 3 Nvidia RTX GPUS, and put the test partition on it
6. Change the creation script to make the "Test" partition the default partition when a researcher arrives on the ISC Compute, then when they are ready to run assign the "Dance" partition
7. Apply Data quota on all ISC compute users, not the case for everyone yet
8. Check the BeeGFS quota mecanism to migrate EXT4 quota from current Disco/Chacha to the new Epyc storage
9. Script a wrapper on Apptainer to check execution context and refuse to run directly bare-bone : same for python or other execcutable to avoid run out of SLURM
10. Migrate current NVMe data disks from Disco/Chacha to BeeGFS when it will be tested and ready also on EPYC
11. Automate file deletion for Standard (Premium too?) researchers to avoid having scratch partition with old tests files / Set in meeting what TTL we want : 2 weeks standard TTL ? More for Premium ?
12. Rename the Filesystem : datasets -> workspace? local_workspace? chacha_workspace ? local_scratch ? / shared -> network_workspace ? remote_workspace ? remote_scratch ?
13. Migrate Prometheus from Chacha to the new EPYC server / Add some alerting on common checks, disks, jobs outside of slurm etc...
14. Move NVMe disks ? Disco 7TB to Chacha ? / NVMe 3TB from Calypso storage to Disco ? Format as BeeGFS
15. Check for a Modules installation ? Or Apptainer is already fine ? : Install LMOD to allow dynamic lib loading : where to put the terabytes of libraries for Dance ?

Note : the EPYC servers and the Mellanox switch are physically part of the ISC³ rack and are (for now) also part of the ISC³ Proxmox plans — who ends up owning them is a Phase 2 governance question.

Backups

1. Backups for Disco and Chacha : to define (see the ISC³ backups architecture for the rest of the fleet).

Playbooks

1. Finish the isc_compute system playbook, especially the Tango login node part, and new configs to avoid direct connection on compute nodes
2. Finish the slurm_research_TODO playbook (redo all installation part from compilation install, separate install from config)

The Ansible environment itself is currently shared with ISC³ — see Ansible.