Skip to main content

Hannibal — the ISC Learn server

Infomaniak VPS hannibal.isc-vs.ch (185.143.103.222), Ubuntu 24.04, 4 vCPU / 12 GB RAM, /srv 250 GB btrfs-compressed (171 GB used, September 2026). It runs ISC Learn and a few co-tenant sites. Facts below re-verified August–September 2026, read-only unless dated otherwise. Credentials and stale-account warnings: secretzone/hannibal.md. Dated work: history & operations; open items: ops todo → ISC Learn.

Access​

  • SSH: ssh -4 -p 20002 ubuntu@learn.isc-vs.ch, key-based only (PasswordAuthentication no), passwordless sudo. From inside the rack network the port is unreachable — the uplink filters outbound non-standard ports; learn.isc-vs.ch also publishes AAAA records and the rack has no IPv6 route, hence -4.
  • Break-glass: lose the key and the only way back in is the Infomaniak console (VNC or serial, both getties run) — credentials in secretzone/hannibal.md (password re-verified against the shadow hash 2026-09-07).
  • sshd hardening (since 2026-09-07, drop-in /etc/ssh/sshd_config.d/50-isc-hardening.conf): no root login, no forwarding of any kind (TCP, agent, X11), 3 authentication attempts — a key holder cannot use the box as a SOCKS relay. Re-open forwarding per user with a Match block at the end of sshd_config if an admin needs a tunnel: drop-ins are included at the top, so a Match there would swallow the main file.

Accounts (2026-09-07)​

  • ubuntu — shared, NOPASSWD sudo; keys: pmudry and the two DS923 backup keys.
  • pmudry, yacine.said — admins, sudo + docker (yacine.said also sudonopass, UID 1105 as in the UID register).
  • marks_dev / marks_prod — run the Marks-crawler containers, docker group, no authorized key left: nobody can log in (todo).
  • pim, leny.bressoud and remi were deleted on 2026-09-07 with their keys (log); what was removed is kept in /srv/removed-accounts-2026-09-07/.

The docker group is root-equivalent: membership is an admin decision.

Network exposure​

  • Infomaniak firewall in front of the VPS lets only 443 and 20002 through (verified from outside 2026-09-07); the host itself has no ufw/INPUT policy.
  • Docker-published ports (8080/8443 Snipe-IT, 8501/8502 Marks-crawler) are the second layer: Snipe-IT binds 127.0.0.1 only, and the systemd unit isc-docker-ports.service drops those four ports on ens3 in the DOCKER-USER chain at boot. Apache reaches them all on localhost.
  • fail2ban (August 2026): jails sshd, apache-auth, apache-overflows; the sshd one bans after 10 failures in 10 min for 20 min, on port 20002, IPv4 and IPv6, via nftables. About 1 200 failed SSH attempts a day reach the port (September 2026) — don't fumble ten logins in ten minutes during a restore.
  • Reverse-proxy vhosts (marks, marks.dev, inventory): ProxyRequests Off. On had been copied into all three; with mod_proxy_connect loaded that is an open proxy, and scanners were probing for it (CONNECT attempts in the logs, August and September 2026).

Updates​

unattended-upgrades installs the Ubuntu security origin daily without rebooting (/etc/apt/apt.conf.d/52-isc.conf, since 2026-09-07). Docker CE and MySQL come from their own repos and are applied by hand with provisioning/hannibal/system-update.sh, which also reports whether a kernel reboot is pending (--reboot does it, about two minutes of Learn downtime).

Layout on disk​

Everything lives in /srv (btrfs, compressed):

PathWhat
/srv/www/learn.isc-vs.ch/{moodle_isc,moodle_data,db_backups}ISC Learn — layout
/srv/www/isc.hevs.ch/hubdocroot of the ISC hub, an rsync of /srv/git/isc-hub-dist (below)
/srv/www/<name>the other vhosts' roots (ingegamez, files, in-egalite, marks); /srv/www/wiki.isc-vs.ch is the retired DokuWiki, deletable (todo)
/srv/docker/inventorySnipe-IT compose project (moved from a user home 2026-09-07; same project name, so the inventory_* volumes were untouched)
/srv/.snapshots/read-only btrfs snapshots taken before changes; each pins the space it protects until deleted
/srv/plugin-updates/<date>/, /srv/upgrade-52/previous plugin directories and upgrade staging, kept for rollback
/srv/removed-accounts-2026-09-07/homes and keys of the deleted accounts

Nightly DB dumps at 06:25 (automysqlbackup) land in db_backups/; manual dumps and operation logs go to db_backups/manual/.

Apache​

Apache listens on 443 only (port 80 is free for the certbot challenge). Vhosts in /etc/apache2/sites-available/:

VhostServes
isc.hevs.ch.confthe hub at / (below), Moodle at the /learn Alias → moodle_isc/public, the Shibboleth <Directory …/auth/shibboleth/index.php> block
learn.isc-vs.ch301 to https://isc.hevs.ch/learn/ (own vhost since 2026-09-08; before that the name fell through to files.isc-vs.ch, under a certificate that did not cover it)
files.isc-vs.chstatic files, among them hesso.xml, the Gradescope SSO metadata, and isc-learn-static/ (teacher photos for the Learn frontpage)
ingegamez.isc-vs.chWordPress, student project site (co-tenants)
in-egalite.ch (+ www)static site
marks.isc-vs.ch, marks.isc-vs.devreverse proxy to the Marks-crawler containers (8501/8502)
inventory.isc-vs.ch, isc-inventory.isc-vs.chreverse proxy to Snipe-IT (8080/8443)

The Moodle vhost in detail (since 2026-09-08, log):

  • Compression: the vhost declares its own AddOutputFilterByType BROTLI_COMPRESS list, which overrides the server-level one — SVG had to be added there or it went out raw.
  • Cache policy, Header rules in the vhost (the old mod_expires block capped HTML at 5 days and is gone): HTML no-cache, /assets/ one year immutable, unhashed static files one week, Moodle's revved endpoints (theme/{font,styles,javascript,yui_combo,image}, lib/{requirejs,javascript}) 90 days immutable. For pluginfile.php: Header always edit Cache-Control "max-age=[0-9]{1,3}\b" "max-age=3600" — Moodle serves user files with max-age=0 or 10 and no ETag, so every page view re-downloaded every image. The {1,3} bound leaves Moodle's own long values (86400, 5184000) untouched; always is required because headers set by PHP-FPM land in err_headers_out, where plain Header edit does not look. Nothing of this may sit at server scope: conf-enabled/isc-hub.conf used to, and put a 7-day cache on files.isc-vs.ch/hesso.xml.
  • X-Sendfile: mod_xsendfile, XSendFilePath /srv/www/learn.isc-vs.ch/moodle_data, $CFG->xsendfile = 'X-Sendfile'. No measurable gain (the cost of pluginfile.php is Moodle's ~75 ms bootstrap), kept because it frees the PHP worker during large downloads and adds byte-range support.
  • PHP-FPM pool (pool.d/www.conf): pm.start_servers 4, pm.min_spare_servers 3, pm.max_spare_servers 8, pm.max_children 20 (never reached in a week); slowlog on with request_slowlog_timeout = 3s — the unit is seconds, 500ms parses as minutes and logs nothing.

The ISC hub at the root of isc.hevs.ch​

Since 2026-08-22 isc.hevs.ch/ serves the ISC hub, so the address bar reads the institutional name (why, and what it costs). DocumentRoot /srv/www/isc.hevs.ch/hub in the *:443 block; the /learn Alias wins over DocumentRoot, so Moodle and $CFG->wwwroot are untouched. Content is a read-only clone of the isc-hub repo's deployed branch in /srv/git/isc-hub-dist, rsynced to the docroot by isc-hub-pull.timer every 5 minutes; a build reaches the site within one tick. The hub routes on real paths (/etudiant-es), so the block carries FallbackResource /index.html, scoped to that <Directory>. The branch is an orphan carrying the built tree only, read with a repo deploy key (/root/.ssh/isc-hub-deploy, read-only), so GitHub has no write path into this host. Installer: provisioning/web/hannibal-hub/install.sh.

Checks: curl -sI https://isc.hevs.ch/ (200, the hub), curl -sI https://isc.hevs.ch/learn/ (200, MoodleSession cookie on path=/learn/).

The git clone must stay out of the docroot

Ubuntu's Apache denies only ^\.ht, so a .git directory below DocumentRoot publishes the whole repository over HTTP — history included, on a private repo. That is why the clone lives in /srv/git and the docroot is an rsync of it. Check the body of https://isc.hevs.ch/.git/config, not its status: FallbackResource answers 200 with index.html for every path that is not a file, so a status check passes whether or not anything leaks — which is also why the block 404s dotfiles explicitly.

Co-tenants​

None of them is covered by the Moodle DR mirror; their files, the docker volumes and the host config are in the rumba copy since 2026-09-05 (backups).

  • ingegamez.isc-vs.ch (WordPress, three admin accounts) runs under the same www-data and the same PHP-FPM pool as Moodle, so a compromise there is a compromise of Learn's user. WordPress core, all plugins and all themes auto-update (auto_update_plugins / auto_update_themes, set 2026-09-07); the code-snippets plugin lets any admin run PHP. Moving the site off hannibal is the real fix (todo).
  • Snipe-IT (ISC Inventory): app + MariaDB in Docker, /srv/docker/inventory, ports 8080/8443 on 127.0.0.1.
  • Marks-crawler prod/dev: Docker, 8501/8502, accounts marks_prod / marks_dev.
  • files.isc-vs.ch: hesso.xml is the Gradescope SAML metadata — Gradescope SSO breaks if it moves without notice.
  • in-egalite.ch: static.
  • The DokuWiki that served wiki.isc-vs.ch was archived and switched off in August 2026; the name redirects to this documentation from srv-web01.

Certificates​

Two certbot lineages, both standalone (Apache never listens on 80, so the port is free for the challenge): learn.isc-vs.ch alone, and isc.hevs.ch carrying the 10 other names (files, hannibal, in-egalite + www, ingegamez, inventory, isc-inventory, marks.isc-vs.ch, marks.isc-vs.dev). A renewal-hooks/deploy/reload-apache hook reloads Apache after a renewal (added 2026-08-10): mod_ssl never re-reads its files, and until then a renewed certificate was only picked up by the weekly logrotate reload.

The shared lineage​

A name that moves off hannibal must leave the lineage

isc.hevs.ch is one lineage covering ten names, and certbot fails the whole lineage on a single failed authorization. The moment a name is repointed elsewhere — decommissioned, moved to the rack, handed to someone else — it stops answering hannibal's HTTP-01 challenge and the next renewal takes all ten vhosts down with it, silently, 30 days before expiry. It happened once, with wiki.isc-vs.ch in August 2026 (log).

Re-issue with the current list minus the departed name; a --dry-run first proves every authorization before spending a real one:

sudo certbot certonly --cert-name isc.hevs.ch --standalone -n \
  -d isc.hevs.ch -d files.isc-vs.ch -d hannibal.isc-vs.ch \
  -d in-egalite.ch -d www.in-egalite.ch -d ingegamez.isc-vs.ch \
  -d inventory.isc-vs.ch -d isc-inventory.isc-vs.ch \
  -d marks.isc-vs.ch -d marks.isc-vs.dev

Backups​

  • DS923 (Energypolis, 23N312, school intranet — unreachable from the rack or the VPN): nightly rsync pull as ubuntu with the two DS923 keys, --delete-before into one directory; versions, if any, are Synology snapshots nobody has verified (backups).
  • rumba hdd/hannibal-mirror (since 2026-08-01, ~320 GB): the staging copy of the DR mirror, a delta pull by hand through a tunnel from an admin Mac, covering the whole host since 2026-09-05 (/etc/shibboleth, /etc/letsencrypt, fail2ban, crontabs, root's keys, the other vhost roots and the docker volumes — enough to rebuild hannibal itself). Each pull ends with a ZFS snapshot, 8 weeks kept plus the first of each month for a year (refresh).
  • Infomaniak snapshots: the data snapshot (2 h 30 to create, lands on a new disk on restore) is a valid last resort before a change; the system snapshot is not a restore path — repeated restore tests in 2025 left the VM unbootable (DRP).

Mail​

Moodle mails through mailer@isc-vs.ch at mail.infomaniak.com:465. The host's own sendmail sends the www-data cron output direct-to-MXes and bounces with 550 rejected by DMARC policy, so cron errors on Moodle prod report to nobody (todo).