ISC Learn — disaster recovery

ISC Learn runs on the hannibal Infomaniak VPS: three directories under /srv/www/learn.isc-vs.ch/ and one MySQL database. Everything below assumes that inventory, which is set out in the from-scratch runbook.

Which path, for which failure

Failure	Restore path	Time
Corrupted files or a bad plugin, hannibal otherwise healthy	Replace the affected files in place from a backup copy — rumba:/hdd/hannibal-mirror/ is the nearest one, the DS923 the other	minutes to hours, depending on what is replaced
Bad change, and a data snapshot was taken just before it	Restore the Infomaniak data snapshot. It lands on a new disk (/dev/sdc) rather than rolling /dev/sdb back, so the files then have to be moved into place	~1 h to restore, after the ~2 h 30 the snapshot itself takes to create
Moodle unusable, hannibal reachable, no recent good copy	Restore onto a second vhost on hannibal, then repoint the learn.isc-vs.ch CNAME at it	hours
hannibal lost or unreachable	Cut over to the standing mirror on rumba (below)	see the cutover steps
No usable backup and no mirror	Full rebuild on a new server from a filesystem backup: from-scratch runbook	≈ 2 h 40 min, measured on the 2026-08-01 run

caution

The Infomaniak system snapshot is not a restore path

Repeated restore tests in 2025 left the VM with an unbootable system disk. They were run on Hasdrubal, a temporary DR-test VM destroyed since. Use the data snapshot only — and note a data snapshot is typically weeks old unless it was taken deliberately before a change, so it is a last resort, not a first one.

The standing mirror on rumba

A full working mirror runs as VM 107 srv-learn-test at http://learn-test.isc3/learn (VPN only), restored from hannibal on 2026-08-01 and verified down to byte-exact file downloads. It is deliberately not production-ready, and that is what a cutover has to undo:

1. Refresh the data. The mirror is as old as its last refresh (2026-09-05, data of the 2026-09-04 dump). The staging copy at rumba:/hdd/hannibal-mirror/ (~320 GB) takes a delta pull, so a refresh is about an hour, not another three-hour transfer — as long as hannibal is still reachable (procedure).
2. Publish it through the srv-web01 reverse proxy under a public name, with Caddy terminating TLS, and repoint the learn.isc-vs.ch CNAME. Never point a production name at the mirror outside a real cutover.
3. Remove the mail muzzle, deliberately, and record it. The mirror is triple-muzzled precisely so it cannot mail students: ufw denies outbound SMTP, no MTA is installed, and $CFG->noemailever = true. On a real cutover it must mail students again — pointed straight at mail.infomaniak.com:465 with the mailbox credentials, exactly as hannibal does, not at the rack relay, whose recipient allowlist is built for alerting.
4. Re-enable Moodle cron, which the mirror also runs without.

Since 2026-08-10 outbound 465 from the rack works, so the muzzle is the only thing preventing a restored Moodle from mailing real students. Details and the exact commands are in the runbook's step 0.

Falling back to hannibal afterwards

Once hannibal is healthy again, the direction reverses and the mirror is the source. Schedule a window, then: dump the database and the data files from the mirror, transfer both to hannibal's original locations, restore the dump, let Moodle finish the upgrade and plugin self-checks from the GUI, clear the caches and restart php8.3-fpm and apache2, then verify before repointing the name back.

Longer term

A migration to managed hosting is proposed, which would retire this runbook's premise. Until it happens, the open items — a second reachable copy, and the paper form of this plan — are in the todo.