Skip to main content

Secrets management

Where the ISC³ credentials live, who can read them, and where we want to take this.

Current situation​

Sensitive values (BMC/iDRAC logins, PDU access, service credentials, …) live in secretzone/ at the root of this repository (ISC³) and in secretzone/calc/ for the CALC@HEI machines — the latter kept as its own subfolder so it can move out with the rest of the CALC documentation. Since 2026-08-10 the folder sits outside both Docusaurus trees, so it can never end up on the published site regardless of configuration (before that, non-publication hung on one exclude line in docusaurus.config.ts; the patterns remain as a guard should a secrets folder reappear under docs/). Access control is simply: whoever can read the private GitHub repository can read the secrets.

Limitations of the current scheme:

  • No granularity — repo access is all-or-nothing; there is no way to share one credential without sharing all of them.
  • Plaintext in git history — every clone carries every secret ever committed; removing a value from the file does not retract it.
  • No audit trail — nobody knows who consulted what, or when.

Target scheme (proposal)​

One scheme for everything, human-held and machine-consumed alike: SOPS with age keys, over the files already in secretzone/. Until 2026-08-13 the plan split human-held credentials into a Vaultwarden instance; that split is gone — one system, nothing to run, patch or back up — and the deployed instance was retired on 2026-08-16 without ever holding a shared secret (journal).

  • The secretzone/ pages become YAML files whose values sops encrypts in place; structure and non-secret context (hosts, user names, quirks) stay plaintext, so the files remain diffable and reviewable in git. Doc pages reference credentials by file and key name; the encrypted files hold the values.
  • Each admin holds an age keypair whose private half never leaves their machine. A .sops.yaml at the repo root maps paths to the public keys allowed to read them, so subfolders act as zones: someone can be given one zone and nothing else, provided they have read access to this (private) repository.
  • Onboarding is one added public-key line plus sops updatekeys; offboarding is the removal plus rotating the values in the leaver's zones — nothing stops a former reader from decrypting revisions they already cloned.
  • A dedicated break-glass keypair is a recipient on every rule; its secret half exists only in the sealed offline copy (below), so the whole store is recoverable from the repo plus that one key.
  • The Ansible playbooks read the files through the community.sops collection. Preferred over ansible-vault, whose single shared passphrase has no per-user identity — one person leaving means rotating for everyone.

What this gives up compared to a password manager: reading a credential takes a laptop with the repo, sops and one's age key — no phone, no browser autofill — and there is still no audit trail of who decrypted what. Accepted 2026-08-13 as the right trade for a small admin team.

Break-glass set​

The disaster recovery plan has a bootstrap problem : credentials needed to rebuild the datacenter must not live only in a service running in that datacenter. Keep a sealed offline copy (paper or encrypted USB, in a safe) of:

  • BMC / iDRAC access for the core machines
  • Hypervisor and storage root access
  • The break-glass age secret key — with a repo clone, it decrypts the whole store. It goes on paper, generated and sealed per the break-glass paper procedure; a hardware token is not usable for this role, the page says why.

Migration checklist​

  • Decide: Vaultwarden on Rumba vs. institutional service — self-hosted on rumba, deployed 2026-08-04; superseded 2026-08-13 by the SOPS-only plan above, instance retired 2026-08-16
  • Move secretzone/ out of the docs tree, to the repo root — 2026-08-10, HTTP exposure now structurally impossible instead of resting on an exclude config line
  • Each admin generates an age keypair; commit .sops.yaml (zones + the break-glass recipient) and just recipes for edit / extract / rekey
  • Convert the secretzone/ pages to encrypted YAML, rotating each value as it moves — the old values remain in git history and on every clone
  • Point the provisioning scripts and Ansible (community.sops) at the encrypted files
  • Write and seal the break-glass set, including the age secret key on paper
  • Retire the plaintext pages (keep the non-secret prose in the regular pages)