Restore ISC Learn on a new server from scratch

Tested end-to-end 2026-08-01: hannibal → VM 107 srv-learn-test on rumba, resulting in a working mirror at http://learn-test.isc3/learn (login, course pages and byte-exact file downloads verified). Total wall clock ≈ 2 h 40 min, dominated by the 293 GB moodle_data transfer. The historical 4.4→4.5 upgrade log that used to sit here is in the history.

What ISC Learn actually is

Three directories under /srv/www/learn.isc-vs.ch/ on hannibal, plus one MySQL database:

Piece	Size (July 2026)	Content
moodle_isc/	~450 MB	Moodle code + plugins + config.php (5.2 since 2026-09-05: web root is moodle_isc/public, CLI scripts and config.php at the tree root)
moodle_data/	~293 GB (~130 GB since the 2026-09-05 purge)	dataroot: filedir/ (the real course files), caches, sessions
db_backups/	~1.5 GB	nightly automysqlbackup dumps (06:25), the restore source for the DB
DB moodle_isc	~315 MB in MySQL	everything else: users, courses, grades

The nightly dumps under db_backups/automysqlbackup/daily/moodle_isc/ mean you never run mysqldump against prod during a restore — a ≤24 h-old dump is already in the filesystem copy. Hannibal also hosts other vhosts (ingegamez, files, in-egalite, the hub at isc.hevs.ch/, marks in docker) and a WordPress DB — not restored by this runbook, though their files, the docker volumes and the host config (Shibboleth SP, TLS, fail2ban, crontabs) sit in the same rumba copy since 2026-09-05 (refresh).

Step 0 — safety rails (do these BEFORE any data lands)

A restored Moodle believes it is the real site: its cron will email real students within minutes. Triple muzzle, in this order, on the fresh guest — and note that since 2026-08-10 the muzzle is the only thing standing in the way: TCP 465 out of the rack now works (email), where it used to be dropped upstream. Do not skip a rail because "the uplink blocks mail anyway". Two things do still help by accident: the mirror is not in the rack relay's mynetworks, and that relay only accepts admin recipients — but neither is a substitute for the rails, since the mirror can reach Infomaniak directly.

On a real disaster recovery, this section inverts

The day this mirror becomes production it must mail students. Point it straight at mail.infomaniak.com:465 with the mailbox credentials, exactly as hannibal does — not at the rack relay, whose recipient allowlist is built for alerting. Remove the muzzle deliberately, as a step of the cutover, and record that it was removed.

1. ufw deny out 25/tcp; ufw deny out 465/tcp; ufw deny out 587/tcp; ufw deny out 2525/tcp; ufw --force enable
2. Do not install an MTA (no sendmail/postfix/opendkim — the prod package list includes them; skip).
3. $CFG->noemailever = true; in config.php before the first page load (step 6).
4. No system cron entry for Moodle until the muzzle is verified.

Prod itself is touched read-only (rsync pull as ubuntu@learn.isc-vs.ch, which has passwordless sudo for reading www-data files). No DNS change ever points a production name at the mirror.

Step 1 — the guest

Per the proxmox-guest skill. What was used and works: a VM (matches prod's Ubuntu 24.04 — kernel isolation, faithful stack), 8 vCPU / 16 GB RAM, 40 GB OS disk on local-lvm (NVMe — MySQL lives here), 400 GB data disk on hdd-vm (RAIDZ2 spinners — fine for moodle_data, which is streaming I/O), mounted ext4 at /srv. Ubuntu 24.04 cloud image + cloud-init, VPN-only, internal DNS learn-test.isc3 on the CCR2004.

Step 2 — getting the data out of hannibal (the network traps)

Hannibal's sshd listens on port 20002. Two traps cost real time (2026-08-01):

• IPv6 AAAA trap: learn.isc-vs.ch publishes AAAA records and the rack has no IPv6 route — a bare ssh from rumba dies with Network is unreachable. Always force ssh -4.

• vsnet egress filter: even with -4, outbound TCP 20002 from the rack subnet times out — the upstream filters non-standard ports (curl portquiz.net:443 works, :20002 doesn't). There is no firewall on hannibal itself (INPUT ACCEPT; fail2ban does ban on 20002 since 2026-08-07, so don't fumble ten logins in ten minutes). Workaround: a reverse tunnel through an admin machine that reaches both sides:

  # on the admin Mac (VPN up, key in ssh-agent):
  ssh -f -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
      -R 127.0.0.1:2222:learn.isc-vs.ch:20002 root@rumba

Then pull from rumba, staging on the ZFS pool (the staging copy doubles as an on-site backup):

# small pieces first — enough to restore the whole site skeleton:
rsync -a --numeric-ids --rsync-path="sudo rsync" \
  -e "ssh -p 2222 -o HostKeyAlias=hannibal-tunnel" \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/moodle_isc \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/db_backups \
  /hdd/hannibal-mirror/srv/learn.isc-vs.ch/

# then the big one (~3 h at the ~20-25 MB/s the tunnel sustains):
rsync -a --numeric-ids --partial --info=progress2 --rsync-path="sudo rsync" \
  -e "ssh -p 2222 -o HostKeyAlias=hannibal-tunnel" \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/moodle_data \
  /hdd/hannibal-mirror/srv/learn.isc-vs.ch/

Also grab /etc/apache2 /etc/php /etc/mysql /etc/redis for reference. Notes: rsync exit code 24 (files vanished) is normal against a live Moodle — sessions/caches churn; those dirs get purged anyway. Don't trust rsync's percentage: it grows as discovery proceeds. du on hannibal's btrfs and the logical bytes rsync moves disagree; plan for ~300 GB. While the pull runs, repeated rsync passes staging→VM overlap the two hops so the final delta is minutes.

Step 3 — the stack (Ubuntu 24.04)

apt install apache2 php8.3-fpm php-curl php-zip php-gd php-mbstring php-xml php-soap \
    php-intl php-mysql php-tidy php-igbinary php-redis redis-server mysql-client git zip unzip pigz
a2enmod proxy_fcgi setenvif rewrite headers expires && a2enconf php8.3-fpm && a2dissite 000-default

• PHP 8.3 matches prod exactly. Moodle needs max_input_vars = 5000 (plus sane upload/memory limits) in a conf.d/90-moodle.ini for both fpm and cli.
• MySQL: prod runs Oracle's 8.4 LTS, but repo.mysql.com's GPG key was expired upstream (Aug 2026) — Ubuntu's 8.0.46 works with the plain-SQL dumps. Note the difference and move on. Set port = 3366 in mysqld.cnf (prod convention, and config.php expects it).
• Skipped from the prod package list: sendmail/opendkim (muzzle), docker-ce, shibboleth (SSO can't work on a different hostname anyway — login is manual accounts), certbot (no TLS on the VPN-only mirror), fail2ban.

Step 4 — the database

Do not restore the sys/mysql system-database dumps (the old runbook did) — importing system tables across MySQL versions is fragile. Create the app user fresh instead; the password in secretzone/hannibal.md (mysql moodle user) matches prod's config.php, so the copied config keeps working:

CREATE DATABASE moodle_isc DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'moodle_isc'@'localhost' IDENTIFIED BY '<secretzone: mysql moodle user>';
GRANT ALL PRIVILEGES ON moodle_isc.* TO 'moodle_isc'@'localhost';

zcat db_backups/automysqlbackup/daily/moodle_isc/<latest>.sql.gz | mysql -u root moodle_isc

Import of the ~263 MB dump: ≈ 15 min on the NVMe (24 min on prod hardware per the old notes). Verify: 508 tables, SELECT FROM_UNIXTIME(MAX(timecreated)) FROM mdl_logstore_standard_log; should be within a day of the dump.

Step 5 — code, data, vhost

rsync -a moodle_isc db_backups root@<vm>:/srv/www/learn.isc-vs.ch/   # from staging
rsync -a --delete --numeric-ids moodle_data root@<vm>:/srv/www/learn.isc-vs.ch/
chown -R moodle:www-data /srv/www/learn.isc-vs.ch/moodle_isc
chown -R www-data:www-data /srv/www/learn.isc-vs.ch/moodle_data

The vhost is a plain HTTP copy of the /learn block from hannibal's isc.hevs.ch.conf (Alias "/learn" → moodle_isc/public since Moodle 5.2, SetHandler proxy:unix:/run/php/php8.3-fpm.sock); serving under the same /learn path avoids any URL rewriting in the DB. Deployed as /etc/apache2/sites-available/learn-test.conf on the mirror.

The mirror is also where a Moodle upgrade is rehearsed first: build the new tree beside the old one with prod's config.php and the 23 non-standard plugins, point the vhost at public/, swap the directories under CLI maintenance, run admin/cli/upgrade.php --non-interactive, then purge caches. The 5.0.1 → 5.2.2+ rehearsal on 2026-09-05 took 93 s for the DB step and caught two things prod then avoided: Moodle 5.2 refuses MySQL 8.0 (the mirror had Ubuntu's; it now runs Oracle's 8.4 like hannibal), and a theme archive unpacked on a Mac ships ._* AppleDouble files into the tree (find . -name '._*' -delete). Rollback on the mirror is the PVE snapshot taken before the rehearsal (qm rollback 107 <name>).

Step 6 — config.php (three changes only)

$CFG->wwwroot   = 'http://learn-test.isc3/learn';  // was https://isc.hevs.ch/learn
$CFG->noemailever = true;                             // the mirror must never send mail

…and one that lives in the DB, not the file — prod sets it and it silently breaks login on a plain-HTTP mirror (the Secure session cookie is never sent back, login just loops):

sudo -u www-data php admin/cli/cfg.php --name=cookiesecure --set=0

Everything else (dbhost localhost:3366, dbpass, dataroot) is already correct in the copied file. Keep the pristine prod copy as config.php.hannibal-orig.

Step 7 — caches and first load

cd /srv/www/learn.isc-vs.ch/moodle_data
rm -rf ./cache/ ./temp/ ./trashdir/ ./sessions/ ./localcache/ ./muc/
systemctl restart php8.3-fpm apache2

First page load rebuilds caches (~10 s); warm loads ≈ 0.15 s.

Step 8 — verification checklist (what "it works" means)

1. curl -w "%{http_code}" http://learn-test.isc3/learn/ → 200, title ISC / Learn.
2. Login: the admin account is admin_moodle_isc (the secretzone's administrateur is stale, and its prod password doesn't match either). Reset it on the mirror only: sudo -u www-data php admin/cli/reset_password.php --username=admin_moodle_isc --password=… --ignore-password-policy (mirror password: see learn-test in secretzone/rumba.md).
3. Open a real course page — verified with course 129 105.1 Summer school 1.
4. Download a real file via pluginfile.php and compare the byte count with mdl_files.filesize — proves filedir/ integrity end-to-end.
5. Check nothing can email: no MTA, noemailever=1, ufw status shows the SMTP denies.

Scripted-login gotcha for future automation: the login page contains the logintoken input four times — grab only the first or Moodle logs invalid login token as failure reason 3, indistinguishable from a wrong password in the event log. Debug logins via mdl_logstore_standard_log (user_login_failed, other.reason: 3 = bad password or bad token) and note that login_failed_count_since_success only increments for real password failures.

Timings summary (2026-08-01 run)

Phase	Duration
VM create + stack install	~25 min
moodle_isc + db_backups pull	~3 min
DB import (263 MB dump)	~15 min
moodle_data pull (293 GB via tunnel)	~3 h (overlapped with everything above)
staging → VM final delta + caches	~10 min
Prod → verified working mirror	≈ 2 h 40 min wall clock

Refreshing the mirror

The mirror is as old as its last refresh; the whole thing is incremental except the database. Script: provisioning/drp/learn-mirror-refresh.sh, run on rumba with the tunnel open and the agent forwarded (copy it to rumba and run it as a file, scp … root@rumba:/tmp/ && ssh -A root@rumba bash /tmp/learn-mirror-refresh.sh; not bash -s < …, whose inner ssh calls swallow the rest of the script after the pull phase; usage in its header). Phases:

1. Pull hannibal → staging, a delta rsync. Five weeks of drift (2026-07-31 → 2026-09-04) moved 37 GB in about an hour through the tunnel, mostly the weekly course auto-backups under moodle_data/backup/. A dropped tunnel kills the rsync; rerun it, --partial resumes. The pull ends with a ZFS snapshot hdd/hannibal-mirror@hannibal-<date> (since 2026-09-05), so what the --delete removes from the copy stays readable under /hdd/hannibal-mirror/.zfs/snapshot/hannibal-<date>/…; the script keeps 8 weeks of them plus the first of each month for a year. Since 2026-09-05 the pull also carries the rest of the host (< 2 GB): /etc/shibboleth (the SP key SWITCH knows isc.hevs.ch by — same entityID and key on a rebuilt host means nothing to re-register), /etc/letsencrypt, fail2ban, the crontabs, root's keys, the other vhost roots and the docker volumes. The mirror VM still installs none of that; it is there for a rebuild of hannibal itself.
2. Database: full reimport of the latest nightly dump (≈ 5 min on the NVMe, 508 tables).
3. Files staging → VM (≈ 120 MB/s on the rack network).
4. Fixup — the sync and the reimport erase the four mirror-only settings, so they are reapplied every time: wwwroot and noemailever in config.php (the code sync brings prod's copy back), cookiesecure=0 and the mirror admin password (both live in the DB). Then the step 7 cache purge and restart. The host-level muzzle (no MTA, ufw, no cron) is untouched by a refresh; the script prints it anyway.

Last refresh: 2026-09-05, data as of the 2026-09-04 dump.