Restore ISC Learn on a new server from scratch

Tested end-to-end 2026-08-01: hannibal → VM 107 srv-learn-test on rumba, resulting in a fully working mirror at http://learn-test.calypso/learn (login, course pages and byte-exact file downloads verified). Total wall clock ≈ 2 h 40 min, dominated by the 293 GB moodle_data transfer. This page replaces the previous untested info-dump; the historical 4.4→4.5 upgrade log it contained is preserved on its own page.

What ISC Learn actually is

Three directories under /srv/www/learn.isc-vs.ch/ on hannibal, plus one MySQL database:

Piece	Size (July 2026)	Content
moodle_isc/	~450 MB	Moodle 4.5 code + plugins + config.php
moodle_data/	~293 GB	dataroot: filedir/ (the real course files), caches, sessions
db_backups/	~1.5 GB	nightly automysqlbackup dumps (06:25), the restore source for the DB
DB moodle_isc	~315 MB in MySQL	everything else: users, courses, grades

The nightly dumps under db_backups/automysqlbackup/daily/moodle_isc/ mean you never run mysqldump against prod during a restore — a ≤24 h-old dump is already in the filesystem copy. Hannibal also hosts other vhosts (ingegamez, wiki, files, marks, in-egalite) and a WordPress DB — not covered by this runbook.

Step 0 — safety rails (do these BEFORE any data lands)

A restored Moodle believes it is the real site: its cron will email real students within minutes. Triple muzzle, in this order, on the fresh guest:

1. ufw deny out 25/tcp; ufw deny out 465/tcp; ufw deny out 587/tcp; ufw deny out 2525/tcp; ufw --force enable
2. Do not install an MTA (no sendmail/postfix/opendkim — the prod package list includes them; skip).
3. $CFG->noemailever = true; in config.php before the first page load (step 6).
4. No system cron entry for Moodle until the muzzle is verified.

Prod itself is touched read-only (rsync pull as ubuntu@learn.isc-vs.ch, which has passwordless sudo for reading www-data files). No DNS change ever points a production name at the mirror.

Step 1 — the guest

Per the proxmox-guest skill. What was used and works: a VM (matches prod's Ubuntu 24.04 — kernel isolation, faithful stack), 8 vCPU / 16 GB RAM, 40 GB OS disk on local-lvm (NVMe — MySQL lives here), 400 GB data disk on hdd-vm (RAIDZ2 spinners — fine for moodle_data, which is streaming I/O), mounted ext4 at /srv. Ubuntu 24.04 cloud image + cloud-init, VPN-only, internal DNS learn-test.calypso on the CCR2004.

Step 2 — getting the data out of hannibal (the network traps)

Hannibal's sshd listens on port 20002. Two traps cost real time (2026-08-01):

• IPv6 AAAA trap: learn.isc-vs.ch publishes AAAA records and calypso has no IPv6 route — a bare ssh from rumba dies with Network is unreachable. Always force ssh -4.

• vsnet egress filter: even with -4, outbound TCP 20002 from the calypso subnet times out — the upstream filters non-standard ports (curl portquiz.net:443 works, :20002 doesn't). There is no firewall on hannibal itself (INPUT ACCEPT; note: its fail2ban watches port 2002, a typo — it protects nothing). Workaround: a reverse tunnel through an admin machine that reaches both sides:

  # on the admin Mac (VPN up, key in ssh-agent):
  ssh -f -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
      -R 127.0.0.1:2222:learn.isc-vs.ch:20002 root@rumba

Then pull from rumba, staging on the ZFS pool (the staging copy doubles as an on-site backup):

# small pieces first — enough to restore the whole site skeleton:
rsync -a --numeric-ids --rsync-path="sudo rsync" \
  -e "ssh -p 2222 -o HostKeyAlias=hannibal-tunnel" \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/moodle_isc \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/db_backups \
  /hdd/backup/hannibal-mirror/srv/learn.isc-vs.ch/

# then the big one (~3 h at the ~20-25 MB/s the tunnel sustains):
rsync -a --numeric-ids --partial --info=progress2 --rsync-path="sudo rsync" \
  -e "ssh -p 2222 -o HostKeyAlias=hannibal-tunnel" \
  ubuntu@127.0.0.1:/srv/www/learn.isc-vs.ch/moodle_data \
  /hdd/backup/hannibal-mirror/srv/learn.isc-vs.ch/

Also grab /etc/apache2 /etc/php /etc/mysql /etc/redis for reference. Notes: rsync exit code 24 (files vanished) is normal against a live Moodle — sessions/caches churn; those dirs get purged anyway. Don't trust rsync's percentage: it grows as discovery proceeds. du on hannibal's btrfs and the logical bytes rsync moves disagree; plan for ~300 GB. While the pull runs, repeated rsync passes staging→VM overlap the two hops so the final delta is minutes.

Step 3 — the stack (Ubuntu 24.04)

apt install apache2 php8.3-fpm php-curl php-zip php-gd php-mbstring php-xml php-soap \
    php-intl php-mysql php-tidy php-igbinary php-redis redis-server mysql-client git zip unzip pigz
a2enmod proxy_fcgi setenvif rewrite headers expires && a2enconf php8.3-fpm && a2dissite 000-default

• PHP 8.3 matches prod exactly. Moodle needs max_input_vars = 5000 (plus sane upload/memory limits) in a conf.d/90-moodle.ini for both fpm and cli.
• MySQL: prod runs Oracle's 8.4 LTS, but repo.mysql.com's GPG key was expired upstream (Aug 2026) — Ubuntu's 8.0.46 works fine with the plain-SQL dumps. Note the difference and move on. Set port = 3366 in mysqld.cnf (prod convention, and config.php expects it).
• Skipped from the prod package list: sendmail/opendkim (muzzle), docker-ce, shibboleth (SSO can't work on a different hostname anyway — login is manual accounts), certbot (no TLS on the VPN-only mirror), fail2ban.

Step 4 — the database

Do not restore the sys/mysql system-database dumps (the old runbook did) — importing system tables across MySQL versions is fragile. Create the app user fresh instead; the password in docs/secretzone/hannibal.md (mysql moodle user) matches prod's config.php, so the copied config keeps working:

CREATE DATABASE moodle_isc DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'moodle_isc'@'localhost' IDENTIFIED BY '<secretzone: mysql moodle user>';
GRANT ALL PRIVILEGES ON moodle_isc.* TO 'moodle_isc'@'localhost';

zcat db_backups/automysqlbackup/daily/moodle_isc/<latest>.sql.gz | mysql -u root moodle_isc

Import of the ~263 MB dump: ≈ 15 min on the NVMe (24 min on prod hardware per the old notes). Verify: 508 tables, SELECT FROM_UNIXTIME(MAX(timecreated)) FROM mdl_logstore_standard_log; should be within a day of the dump.

Step 5 — code, data, vhost

rsync -a moodle_isc db_backups root@<vm>:/srv/www/learn.isc-vs.ch/   # from staging
rsync -a --delete --numeric-ids moodle_data root@<vm>:/srv/www/learn.isc-vs.ch/
chown -R moodle:www-data /srv/www/learn.isc-vs.ch/moodle_isc
chown -R www-data:www-data /srv/www/learn.isc-vs.ch/moodle_data

The vhost is a plain HTTP copy of the /learn block from hannibal's isc.hevs.ch.conf (Alias "/learn" → moodle_isc, SetHandler proxy:unix:/run/php/php8.3-fpm.sock); serving under the same /learn path avoids any URL rewriting in the DB. Deployed as /etc/apache2/sites-available/learn-test.conf on the mirror.

Step 6 — config.php (three changes only)

$CFG->wwwroot   = 'http://learn-test.calypso/learn';  // was https://isc.hevs.ch/learn
$CFG->noemailever = true;                             // the mirror must never send mail

…and one that lives in the DB, not the file — prod sets it and it silently breaks login on a plain-HTTP mirror (the Secure session cookie is never sent back, login just loops):

sudo -u www-data php admin/cli/cfg.php --name=cookiesecure --set=0

Everything else (dbhost localhost:3366, dbpass, dataroot) is already correct in the copied file. Keep the pristine prod copy as config.php.hannibal-orig.

Step 7 — caches and first load

cd /srv/www/learn.isc-vs.ch/moodle_data
rm -rf ./cache/ ./temp/ ./trashdir/ ./sessions/ ./localcache/ ./muc/
systemctl restart php8.3-fpm apache2

First page load rebuilds caches (~10 s); warm loads ≈ 0.15 s.

Step 8 — verification checklist (what "it works" means)

1. curl -w "%{http_code}" http://learn-test.calypso/learn/ → 200, title ISC / Learn.
2. Login: the admin account is admin_moodle_isc (the secretzone's administrateur is stale, and its prod password doesn't match either). Reset it on the mirror only: sudo -u www-data php admin/cli/reset_password.php --username=admin_moodle_isc --password=… --ignore-password-policy (mirror password: see learn-test in docs/secretzone/rumba.md).
3. Open a real course page — verified with course 129 105.1 Summer school 1.
4. Download a real file via pluginfile.php and compare the byte count with mdl_files.filesize — proves filedir/ integrity end-to-end.
5. Check nothing can email: no MTA, noemailever=1, ufw status shows the SMTP denies.

Scripted-login gotcha for future automation: the login page contains the logintoken input four times — grab only the first or Moodle logs invalid login token as failure reason 3, indistinguishable from a wrong password in the event log. Debug logins via mdl_logstore_standard_log (user_login_failed, other.reason: 3 = bad password or bad token) and note that login_failed_count_since_success only increments for real password failures.

Timings summary (2026-08-01 run)

Phase	Duration
VM create + stack install	~25 min
moodle_isc + db_backups pull	~3 min
DB import (263 MB dump)	~15 min
moodle_data pull (293 GB via tunnel)	~3 h (overlapped with everything above)
staging → VM final delta + caches	~10 min
Prod → verified working mirror	≈ 2 h 40 min wall clock