Admin scripts

Root-owned admin scripts for CALC@HEI, historically living only as files in /root/ on Chacha and Disco. As of 2026-08-04 they're mirrored in this repo at provisioning/nodes/scripts/ for versioning — edit there and redeploy, rather than editing in place on the servers. calc/provisioning/deploy.sh pushes them to /root/provisioning/nodes/scripts/ on both hosts — that's the path admins and cron should use now, not the old flat /root/*.sh layout. rm_vscode.sh is the only script with an automated caller (root's hourly crontab); deploy.sh re-asserts that cron line on every run, and PRUNE_LEGACY=1 ./deploy.sh removes the old flat copies once the new tree is confirmed in place.

/root is a local ext4 filesystem on each host's own root LV, not a shared mount — the two servers each keep an independent copy. Checked by md5sum on 2026-08-04: the eleven scripts common to both hosts were byte-identical, i.e. kept in sync by hand rather than actually shared.

Deployment drift found 2026-08-04

• check_vscode.sh and check_apptainer_instances.sh exist on Chacha but were missing from /root on Disco.
• migrate_apptainer_caches.sh exists on Chacha only and was undocumented here until now.
• dance_config.cfg (a sacctmgr load dump) and slurm_disco.conf (a diff-comparison reference file, despite the name) are Chacha-only by design and aren't scripts — left out of the repatriation.

Removed 2026-08-04

• add_slurmuser.sh — superseded by calc/provisioning/accounts.yaml + provision.py (see SLURM configuration and user creation). Deleted from the repo and from /root on both Chacha and Disco. It also had a broken group-membership check (if [ ! id -nGz "$1" | grep -qzxF 'slurm' ] ; then — a pipeline invalid inside [ ], failing on every run and always falling through to usermod -aG slurm, harmless only because that's idempotent).
• batch_add_sshusers.sh — never worked (the CSV/argument format it assumed never matched add_sshuser.sh's actual (username, uid) calling convention). Deleted from the repo and from /root on both hosts.

Removed 2026-08-05

• add_sshuser.sh and add_cifs_mount.sh — superseded by provisioning/ansible/ (see Ansible), which was consolidated into this repo the same day. Ansible's users.yml playbook already creates the Linux account, deploys the SSH key from conf/authorized_keys/<name>.pub, sets up the apptainer/dataset/results/ shared_datasets dirs and symlinks, and grants every user a blanket mount -t cifs //*/* /home/<user>/* sudoers rule unconditionally — making both scripts redundant for standard onboarding. accounts.yaml dropped the ssh_key field it briefly carried; it's SLURM-only again. See Managing compute users for the new onboarding flow.

Correctness issues found and fixed in the repo copy (2026-08-04)

These were bugs in the scripts as deployed; fixing them in the repo doesn't change the live copy on the servers until redeployed.

• add_group.sh — group directories were created chmod 670, which gives the group rwx but the owner (root) no execute bit. This only "worked" because root bypasses permission checks; a non-root process relying on the mode bits as documented would not have been able to traverse it. Changed to 770. Doesn't retroactively fix already-created group directories.
• migrate_apptainer_caches.sh — hardcoded /data/apptainer, which is Chacha's path only; running it on Disco (where the apptainer cache lives under /data/disk01/apptainer) would have migrated to the wrong place with no warning. Added a hostname branch to pick the right path per host.

Users scripts

Scripts ran without arguments print the usage help. Onboarding (Linux account creation, SSH keys, CIFS mount rights) is now Ansible's job — see Ansible and Managing compute users. The scripts below cover what Ansible doesn't: quotas, project-group storage, and offboarding.

add_group.sh

Adds a group directory for a project with several users

./add_group.sh project 9011 "user.name1 user.name2"

It will create a shared directory in /data/shared/groups/project that will be available from both servers to user.name1 and user.name2 and a symlink to this directory on each one home dir /home/user.name/group/

NOTE : needs to be run first on Disco because Chacha can't modify the NFS directories that are shared by Disco

remove_sshuser.sh

Revokes a user's SSH/login access without touching their data — locks the account (usermod -L, shell → /usr/sbin/nologin) and moves aside the user's ~/.ssh/authorized_keys. Ansible has no removal logic, so it also prints a reminder to drop the user from the Ansible roster (provisioning/ansible/conf/users/*.yml and conf/authorized_keys/) — otherwise the next playbook run silently re-creates access.

./remove_sshuser.sh jean.machin

Home dir, datasets and apptainer cache are left in place — see How to remove a user for the full offboarding checklist (quotas, SLURM, data retention).

fix_data_mount_ownership.sh

One-off fix for /data/disk01, /data/disk01/apptainer and their space equivalents on Chacha, found owned by a regular user instead of root:root at two directory levels — see todo. Only chowns those four specific paths, never recursive, so per-user leaf directories underneath keep their existing ownership.

./fix_data_mount_ownership.sh

Chacha-only — Disco's equivalent tree was checked and is correctly root:root. Idempotent — skips any path already root:root or missing.

Disk quotas

Quotas are declared for every user in calc/provisioning/nodes/users_quota.list — one line per user, <user> <root_hard> <data_hard> <shared_hard> — and applied by sync_quotas.sh, which picks the right mounts for the host it runs on (/ and /data on Chacha; /, /data/disk01 and /data/space on Disco). Soft limits are derived at 90 % of the hard limit.

./sync_quotas.sh              # dry run: what would change, plus who is over
./sync_quotas.sh --apply

Idempotent, and it refuses to hand anyone a limit below what they already hold, which would leave their next write failing with EDQUOT. For a deliberate reduction of someone already above the target, --grace sets the soft limit to the target and the hard limit just above current usage: the 7-day grace countdown starts and quota warns them, but writes keep working. --force overrides the refusal outright and will break a running job.

Every run ends with a quota-pressure report — OVER, SOFT (in grace) and FREE (no limit at all) per user per mount.

The Ansible common role deploys both files and runs the check on every pass, failing the play if any account is missing a quota. It never runs --apply: narrowing a quota should be a deliberate act, not a side effect of a routine playbook run.

caution

Read usage with repquota, not quota -u

quota -u <user> prints nothing for a filesystem where the user has no quota yet. Any check built on it silently reads their usage as zero — for exactly the accounts a backfill exists to fix. repquota reports usage whether or not a limit is set. This cost five users a broken /data quota on 2026-08-20.

add_quota.sh

Superseded by sync_quotas.sh above, which covers every user from the versioned list rather than one user per invocation. Kept for one-off manual overrides.

./add_quota.sh user.name 10G 200G 100G

You can also unset a quota : you just need to specify a unit like 0M to unset a quota.

./add_quota.sh user.name 0M 0M 0M

NOTE : Running twice doesn't affect the quota used.

Scripts list

• add_group.sh : Adds a group directory for a project with several users NOTE : needs to be run first on Disco because Chacha can't modify the NFS directories that are shared by Disco
• add_quota.sh : Adds a quota for a single user — superseded by sync_quotas.sh, kept for one-off overrides
• sync_quotas.sh : Applies users_quota.list to every user on the host; idempotent, refuses to narrow a limit below current usage, reports who is over
• remove_sshuser.sh : Revokes a user's SSH/login access (lock account, revoke key) without touching their data
• backup_slurm_data.sh : Backups the Slurm configuration files and backups the SLURM database : be advised it stops/starts the Mysql service to backup the DB
• check_apptainer_instances.sh : Checks if a process was launched with Apptainer outside of SLURM. Present on Chacha; missing from Disco's /root.
• check_vscode.sh : Checks any VScode process running. Present on Chacha; missing from Disco's /root.
• fix_data_mount_ownership.sh : One-off fix for /data/disk01//data/space directory ownership on Chacha. Run 2026-08-08.
• install_fastfetch.sh : Extracts fastfetch.tar.gz (must already be present in /root) into /usr/local/bin.
• migrate_apptainer_caches.sh : One-off migration of a user's .apptainer cache dir to the per-user directory under the node's apptainer mount. Chacha only.
• rm_vscode.sh : Ran automatically every hour by root's crontab (5 * * * *, crontab -u root -l) on both hosts, to remove any VScode cache dirs on every user. deploy.sh keeps this entry pointed at /root/provisioning/nodes/scripts/rm_vscode.sh.
• users_quota.list : The declared quota for every user, versioned in calc/provisioning/nodes/; consumed by sync_quotas.sh

Add new compute user

Managing compute users