Skip to main content

ISC Learn — history & operations

Dated operations on ISC Learn and its server hannibal, newest first. Current state is on the ISC Learn and hannibal pages; the marcellus log is on External VPS.

2026-09-09 — frontpage: ambient background, spacing, menu​

Ambient background ported from the hub landing page (the layer under <body> needs the theme's white #region-main and .pagelayout-content made transparent), hero frame and graph paper a shade darker, course and « Mes cours » blocks shown at 0.7 s instead of 1.3 s (per-image fade instead of waiting for every image), petal mark moved to the bottom-right corner, one 44 px gap between all blocks, typewriter no longer wiping its first word, the edu-ID button opening the header's login window, « ISC Hub » as a root menu entry and « En savoir plus » pointing at the landing page. Six deploys of frontpage-2a/deploy.sh from the Mac (now portable) and one menu script; standing description on the frontpage section.

2026-09-09 — theme AJAX blocks requested at HTML parse time​

The frontpage's semester tabs, blog row and « Mes cours », and the course panel of the toggle sidebar, were empty containers New Learning filled by AJAX once first.js had run — 0.6 s after the HTML here, after load on a throttled mobile. Two inline scripts now send the same web-service requests at parse time and inject the same markup: loader.html on the builder page (the theme's frontpage loaders are neutralised and tab clicks handled by the script) and a block of header.html for course pages (the theme's call still follows and re-renders the same panel). Frontpage blocks at 0.5 s instead of 0.7 s on a fast connection, 4.5 s instead of 32 s under PageSpeed mobile throttling; course panel 0.8 s instead of 1.3 s, 4.9 s instead of 32 s; the course header image, lazy-loaded by the theme at ~2 s, set at 0.8 s the same way. The 2026-09-05 theme-file patch rendering the course panel inline was retired the same day (provisioning/learn/mb2nl-tgsdb-chome-restore.sh, patched file kept beside the original as lib_toggle_sidebar.php.isc-patched-2026-09-09); the deploy script and apply-frontpage-2a.php carry the scripts, deploy.sh now also runs from macOS. Standing description and what can break it: early fetch; numbers: performance.

2026-09-09 — 2026-27 students registered​

First run of the semester scripts from ISC-HEI/moodle-isc-admin-scripts (the yearly-archive repository, renamed): 33 accounts created (auth=shibboleth, no forced password change) and 371 per-module enrolments in the 15 autumn courses, from the secretariat's workbook. Per-student enrolments replace the year cohorts. Left for a person: the bachelor thesis 330.1 (labelled S6 in the workbook), three X? marks, four modules without a course (todo). Procedure: registering students.

2026-09-09 — file-system repository aliases moved to personal instances​

The 2026-27 courses 101.1, 101.2 and 205.1 had their 350 "File system" aliases spread over nine repository instances belonging to archived courses (two of them still serving the 2022 folder mui); they now go through two instances in pmudry's user context (101, 205.1), which course copies inherit and which no course deletion can convert to frozen copies. Standing rule on the service page.

2026-09-08 → 09 — second performance pass: Apache, PHP-FPM, content, theme​

Server side, in sites-available/isc.hevs.ch.conf — the standing configuration is on the hannibal page:

  • SVG added to the vhost's own brotli list (the ISC logo 12 446 → 2 926 bytes, 53.8 kB across the five site SVG).
  • mod_expires block removed (it capped HTML at 5 days, so returning visitors saw a stale site); Header rules instead, and pluginfile.php responses rewritten to max-age=3600.
  • conf-enabled/isc-hub.conf deleted — it held those rules at server scope, on all seven vhosts.
  • PHP-FPM pool: start/min/max spare 2/1/3 → 4/3/8, max_children 20 unchanged; slowlog at 3 s.
  • X-Sendfile enabled: 78 ms before, 77 ms after on a 110 kB file — kept for the freed worker and byte ranges, not for speed.
  • learn.isc-vs.ch got its own vhost (301 to isc.hevs.ch/learn/); it had none and fell through to files.isc-vs.ch, answering "Nothing to see here" under a certificate that did not cover it.

Moodle side:

  • Two frontpage blog images 5776×3848 (3.3 MB) and 4000×2664 (2.7 MB) → 1600 px WebP at 265 and 182 kB; page weight 8 507 → 2 774 kB. An LTI tool icon pointing at challengeme.online (a 4664×4664 PNG for a small logo, third-party fetch) removed.
  • Theme patch: the toggle sidebar's course header rendered inline instead of by AJAX ~2.1 s later (a theme-file patch, lost on every update; retired 2026-09-09 for the early fetch). Trap kept from it: never call theme_mb2nl_external::course_chome() from a page render, its validate_context() resets the page output.
  • Toggle sidebar drawn right at first paint (2026-09-09): the unitless --tgsdb_cheight: 0 left the column at content height (670 px in a 900 px viewport) until tgsdb.js ran; fixed in CSS alone, appended to the Custom CSS setting with provisioning/learn/mb2nl-tgsdb-css.sh, measured identical before and after the JS runs.
  • Frontpage typewriter script repaired (2026-09-09): a hand save in the builder editor on 2026-09-07 had replaced every […] of the hero script by a "no shortcodes" notice, a JS syntax error on every load. Repaired in place with provisioning/learn/frontpage-hero-js-2026-09-09.php, other hand edits kept; the script is now bracket-free.
  • $CFG->enable_read_only_sessions tried twice and reverted (Moodle refuses to start while any session-mode MUC definition resolves to the session store).

Result: PageSpeed mobile 46 → 53, LCP 20.2 → 13.0 s; course page 361 ms median server render; isc.hevs.ch/landing/ 81 → 99. What remains and what was ruled out: performance.

2026-09-07 — hannibal: accounts pruned, sshd and Docker hardened, 308 updates​

Triggered by the marcellus compromise (same admins, same history). Audit first (read-only), then three scripts in provisioning/hannibal/:

  • accounts-2026-09-07.sh — deleted pim (sudo, sudonopass), leny.bressoud (docker) and remi (sudo, no key since 2024) with their homes; pim's three keys removed from ubuntu, marks_dev and marks_prod; yacine.said added as admin. The Snipe-IT compose project lived in leny's home: moved to /srv/docker/inventory (same project name, volumes untouched), ports bound to 127.0.0.1, container recreated — a few seconds of downtime. Everything removed is in /srv/removed-accounts-2026-09-07/. pim's home server had been pulling a full rsync of / and /srv nightly at 01:00 as ubuntu; that copy stops with his key.
  • harden.sh — ProxyRequests Off in marks, marks.dev and inventory; sshd drop-in (no root, no TCP/agent/X11 forwarding, 3 attempts, idle timeout); DOCKER-USER drop of 8080/8443/8501/8502 on ens3 persisted by isc-docker-ports.service; unattended-upgrades reinstalled (it had been removed), security origin only, no automatic reboot.
  • system-update.sh — 308 packages. The kernel reboot is pending (todo).

Verified from outside: only 443 and 20002 answer, CONNECT through the reverse-proxy vhosts is refused, Snipe-IT, marks and Learn answer 200. Break-glass checked: the ubuntu console password in secretzone/hannibal.md matches the shadow hash, both getties (tty1, ttyS0) are active.

Afternoon, after a read-only check for the marcellus pattern (no leftover installer, dump or dropped file): the ingegamez WordPress got auto-updates for every plugin and theme and its seven pending plugin updates were applied through the WordPress updater API (no wp-cli on the host; a first attempt left the auto_updater.lock option behind, which the second run had to delete); the Moodle code tree lost its 2 076 group-writable and 1 257 world-writable paths, all in the web-installed New Learning plugins. Learn and ingegamez answered 200 throughout.

2026-09-06 — six plugin updates​

Moodle's plugin manager had been reporting updates since the 5.2 upgrade: FilterCodes 2.7.2 → 3.0.1, Components for Learning (tiny_c4l) 3.1.0 → 3.3.0, Redis tool, Bulk enrol and Static page v5.0 → v5.2-r1, Mailtest 3.1.7 → 3.1.8. Run on the VM 107 mirror at 16:16, then on hannibal at 18:23 with provisioning/learn/plugin-updates.sh hannibal: btrfs snapshot /srv/.snapshots/pre-plugins-2026-09-06, zips from moodle.org checked against the published md5, previous plugin directories kept in /srv/plugin-updates/2026-09-06/prev/, upgrade.php in 39 s with no maintenance mode (four new FilterCodes settings at their defaults). Verified: pages 200, theme CSS unchanged, cron running, no PHP error. Log: hannibal:/srv/plugin-updates/2026-09-06/run-1823.log.

2026-09-06 — new frontpage design​

Design « 2a » from a Claude Design hand-off, reworked on the VM 107 mirror during the afternoon (PVE snapshot pre-frontpage-2a taken first) and applied to hannibal at 16:40 on a Sunday: no downtime, one cache purge. The hand-off was a Boost child theme; New Learning reads its settings from the active theme, so a child theme loses them all — the design went into the theme's Custom CSS and a builder page instead, which also survives theme updates. Kept from the old page: the semester course tabs with their images, the blog row and the footer; removed: the resources carousel, the « philosophie » text and the typewriter line, which now lives in the hero. Standing description and traps: frontpage.

2026-09-06 — content touch-ups and course images​

All rehearsed on VM 107, applied on hannibal with scripts in provisioning/learn/, each with --revert:

  • Teaching-team page (Mermoud professeur ordinaire, Zahno on processor architecture, Andrea Guerrieri added for Digital design, Xavier Barmaz removed) and the « En savoir plus » menu item pointing at the hub's prospective-students page — team-menu-2026-09-06.php.
  • Navigation root reads « Infos » (was « Infos... »), its dropdown opens with an « ISC Hub » link — menu-infos-2026-09-06.php.
  • Phone pass: semester tabs as a 3 × 2 grid of pills, the notification bell hidden below 820 px with a « Notifications » entry added to the user menu, the footer button « Plus d'infos sur la formation » pointing at the hub — footer-usermenu-2026-09-06.php plus the frontpage-2a CSS redeployed (course-code line on every course listing, header logo proportions on phones).
  • Course images for 200.3, 304.2 and 305.2: three flat SVG illustrations in the style of the Digital design card (ISC palette, no third-party photo), set as course overview files with course-images.php; 304.2 had been showing the Robot control picture.

2026-09-06 — mirror refreshed​

VM 107 on the 06:25 dump and the day's files, PVE snapshot pre-refresh-2026-09-06 taken first, frontpage and content scripts re-applied. Two fixes to provisioning/drp/learn-mirror-refresh.sh: moodle_data/trashdir is excluded (89 GB of purged course backups were being copied), and the script must run from a file on rumba, not bash -s, since its inner ssh calls swallow stdin (runbook).

2026-09-05 — first performance pass​

Read-only survey first: TTFB 150 ms, HTTP/2, brotli, 90-day immutable asset caching, PHP-FPM never at its 20-worker cap in 7 days. Four server-side leftovers, all changed online with no downtime and one user connected: InnoDB buffer pool 128 MB for a 567 MB database (resized to 2 GB, persisted in mysqld.cnf); OPcache at PHP defaults (128 MB, 10 000 files, revalidate 2 s) for a 51 000-file tree (256 MB / 30 000 / 60 s, PHP-FPM reload); logstore_standard never purged (1.87 M rows, 372 MB of the 567 — retention set to 365 days, the rows before 2025-09-06 deleted through Moodle's own cleanup task, 708 680 rows left; safety net: the same-evening pre-upgrade dump); and Redis 7.0 installed but unused — sessions and the MUC application cache moved to it (provisioning/learn/learn-redis-enable.sh, rehearsed on the mirror an hour earlier with a real login). Pages 200 in 0.11–0.16 s before and after: on a single request these settings change nothing measurable, they matter on heavy pages and under load. Standing settings: service page; scripts provisioning/learn/learn-perf-2026-09-05.sh and learn-redis-enable.sh, log db_backups/manual/perf-2026-09-05-2346.log.

2026-09-05 — Moodle 5.0.1 → 5.2.2+​

Done at 23:15 on a Saturday, 118 s of maintenance page, no user connected. Moodle 5.0 security support ends 2026-10-05; 5.3 LTS is released that day and is the next step, at a holiday. Rehearsed the same afternoon on the mirror (VM 107), which is now also on 5.2.2+; the rehearsal caught two things prod then avoided — Moodle 5.2 refuses MySQL 8.0 (the mirror had Ubuntu's, it now runs Oracle's 8.4 like hannibal), and a theme archive unpacked on a Mac ships ._* AppleDouble files into the tree.

Sequence (provisioning/learn/moodle-upgrade-cutover-2026-09-05.sh): read-only btrfs snapshot /srv/.snapshots/pre-upgrade-52-2026-09-05 and a --single-transaction dump (db_backups/manual/moodle_isc_pre-upgrade-52_2026-09-05.sql.gz); new tree built beside the live one from the archive tested on the mirror (Moodle 5.2.2+ build 20260903 plus the 23 non-standard plugins: New Learning 12.2.7 and companions from the vendor package, VPL 4.5.0, CodeRunner 5.10.1 and behaviour 1.4.7 from GitHub, the rest carried over) with prod's config.php; the three moodle_isc paths in isc.hevs.ch.conf pointed at moodle_isc/public (Alias, <Directory>, Shibboleth block; previous file kept as .pre52); CLI maintenance, directory swap, admin/cli/upgrade.php --non-interactive in 116 s (29 new settings at their defaults), cache purge, PHP-FPM restart. Verified: pages 200, Shibboleth entry redirects to the SWITCH WAYF, theme width 1600 px, cron running, no PHP or Apache error. Old code kept at moodle_isc.50 for a week (todo); the year-move-up scripts that sat in the old web root went with it, so they are no longer reachable over HTTP.

Two findings kept on the service page: the public/ layout, and the New Learning 12.2.7 page-width bug with its one-line patch. Left open: Moodle's router check. Log: hannibal:/srv/upgrade-52/upgrade-2026-09-05-2315.log.

2026-09-05 — course backups purged​

moodle_data was 303 GB logical, of which 280 GB were automated course backups: 5 versions of every course, stored twice (directory + course file area, block-shared on btrfs), and never pruned for the 116 archived courses because Moodle only prunes what it backs up. Applied rule: 3 versions for the 25-26 courses and for active ones, 1 for 22-23 to 24-25. Removed 410 file-area entries through the File API and 390 directory .mbz from pre-generated lists, checked file by file against the rumba copy first; 195 / 199 kept, backup_auto_max_kept 5 → 3. Lists and logs: hannibal:/srv/www/learn.isc-vs.ch/db_backups/manual/purge-2026-09-05/. About 92 GB physical, released once Moodle's trash task runs and the read-only snapshot /srv/.snapshots/pre-purge-2026-09-05 is deleted (todo). Same day, the end dates of 25-26 / 330 - TB and 25-26 / 105.1 - SS were set to 2026-09-04 23:00 (update_course(), which requires startdate in the object). Also the same day the frozen Aug-2024 Moodle on marcellus was deleted after checking every course exists here under Archives (marcellus log).

2026-09-05 — mirror refreshed, rumba copy versioned​

VM 107 now carries the 2026-09-04 dump and moodle_data as of the day; five weeks of drift were a 37 GB delta pull through the tunnel, about an hour. The steps became provisioning/drp/learn-mirror-refresh.sh and a runbook section. The rumba copy moved into its own dataset hdd/hannibal-mirror (block-cloned, 4 min, no extra space; why not hdd/backup); the first snapshot hannibal-2026-09-05 holds the pre-purge course backups, and the script snapshots after every pull with an 8-weeks-then-monthly retention. The pull now also carries the rest of the host (/etc/shibboleth, /etc/letsencrypt, fail2ban, crontabs, root's keys, the other vhost roots, the docker volumes — under 2 GB).

2026-08-22 — isc.hevs.ch serves the hub instead of redirecting to it​

The address bar keeps the institutional name. DocumentRoot /srv/www/isc.hevs.ch/hub, fed from the isc-hub repo's deployed branch by a 5-minute pull timer, read with a read-only deploy key (provisioning/web/hannibal-hub/install.sh); a build propagates within one tick. The hub dropped the # from its URLs the same day, so the block carries FallbackResource /index.html; /learn is unaffected. The fleet links moved to isc.hevs.ch, making it the canonical of the two names. Standing state: hannibal; the trade against host-independence: migration plan.

2026-08-21 — the root of isc.hevs.ch 301s to the hub​

Execution of the 2026-08-11 decision, four items:

  1. DNS: hub → CNAME isc-hei.github.io. in our Infomaniak zone.
  2. The hub repo: base '/isc-hub/' → '/', public/CNAME = hub.isc-vs.ch, the absolute og:/twitter: URLs. lighthouserc.json was the tripwire — it audited http://localhost:4173/isc-hub/ and its workflow gates the deploy, so at base / nothing published until it was changed. Pages is a workflow deployment, so the domain is set with gh api -X PUT repos/ISC-HEI/isc-hub/pages -f cname=hub.isc-vs.ch (that PUT clears https_enforced; set it back with a second PUT … -F https_enforced=true once the certificate is approved), plus gh api -X PATCH repos/ISC-HEI/isc-hub -f homepage=https://hub.isc-vs.ch/, since public/repos.json is regenerated daily from that field.
  3. The isc-hei.github.io/isc-hub/ URL retired in 15 links across 7 repos (isc-hub, isc-tbs, isc-calendar, isc-curriculum, isc-typst-tb-descriptions, isc-marks-mail, isc-shirts, and this site's navbar and index). GitHub's automatic redirect keeps old links working and is the one part that dies if the hub ever leaves GitHub. Repointed again to https://isc.hevs.ch/ the next day.
  4. Hannibal: RedirectMatch permanent ^/$ https://hub.isc-vs.ch/ in the *:443 block — replaced by the DocumentRoot the next day. The certificate needed nothing: isc.hevs.ch stays a SAN on the existing lineage.

2026-08-10 — the shared certificate re-issued before it failed​

wiki.isc-vs.ch had been retired to a redirect on srv-web01 while still a SAN on the isc.hevs.ch lineage; the renewal due around 2026-09-21 would have failed all ten names silently. Re-issued without it (ten names, all verified from outside), and the missing deploy hook that reloads Apache added. Standing rule and command: hannibal. Marcellus had the same problem, fixed the same evening (marcellus log). The DokuWiki itself was archived off hannibal the same day (where it is, the redirect vhost).

2026-08-07 — fail2ban fixed, facts refreshed, migration plan written​

The sshd jail rejected on port 2002 while sshd listens on 20002, so ~20 000 failed attempts in 30 days had produced 14 bans that blocked nothing. Fixed and verified on IPv4 and IPv6; ubuntu given a console break-glass password since SSH is key-only. Half of audit finding F-OPS-3 — the missing host firewall stays open (todo). Same day, prod facts refreshed read-only (Moodle 5.0.1 then, moodledata 303 GB of which 134 GB course backups, VPL/Jobe on Infomaniak Jelastic, both public names CNAMEs to hannibal.isc-vs.ch — the older "isc.hevs.ch → learn.isc-vs.ch" description was wrong) and the migration to managed hosting proposed.

2026-08-01 — disaster-recovery mirror built on rumba​

Full restore hannibal → VM 107 srv-learn-test, ≈ 2 h 40 min wall clock, dominated by the 293 GB moodle_data transfer through a laptop tunnel at 20–25 MB/s (the rack cannot dial port 20002, and learn.isc-vs.ch publishes AAAA the rack cannot route). Verified down to byte-exact pluginfile.php downloads. The steps and the traps became the from-scratch runbook; the staging copy stayed on rumba as the first on-site backup of ISC Learn. Two findings went to the ops todo: the DS923 had been the only copy, and the rumba_backup share on the FS2500 was an empty stub.

2025-04 — Moodle 4.4.2 → 4.5.4​

Historical notes from the April 2025 upgrade of production, kept verbatim: the block of cp -a lines was the de-facto inventory of the third-party plugins until the 5.2 upgrade rebuilt the tree from packages.

### New Migration from 4.4.2 to 4.5.4

# 9H00 : Maintenance mode
cd /srv/www/learn.isc-vs.ch/moodle_isc/
sudo -u www-data /usr/bin/php admin/cli/maintenance.php --enable

# Retrieve package : DONE
#cd /srv/migration_moodle
#wget https://download.moodle.org/download.php/direct/stable405/moodle-4.5.4.tgz
#tar -xzf moodle-4.5.4.tgz
#mv /srv/migration_moodle/moodle /srv/migration_moodle/moodle_isc

cd /srv/migration_moodle

# Backup moodle Data
cp -a /srv/www/learn.isc-vs.ch/moodle_data /srv/www/learn.isc-vs.ch/moodle_data_4.4.2

# Backup moodle config
cp -a /srv/www/learn.isc-vs.ch/moodle_isc /srv/migration_moodle/moodle_isc_4.4.2

# 9H10 : Backup DB
mysqldump -u root -p -C -Q -e --create-options moodle_isc > /srv/migration_moodle/moodle_backup_$(date +%Y-%m-%d).sql

#### Start moving files : Put aside the old moodle_isc and replace with the 4.5.4 moodle_isc directory
mv /srv/www/learn.isc-vs.ch/moodle_isc /srv/www/learn.isc-vs.ch/moodle_isc_4.4.2
cp -a /srv/migration_moodle/moodle_isc /srv/www/learn.isc-vs.ch/

# To ease reading for the copies
export migration="/srv/www/learn.isc-vs.ch/moodle_isc_4.4.2"
export target="/srv/www/learn.isc-vs.ch/moodle_isc"

cp -a $migration/config.php  $target/
cp -a $migration/theme/mb2nl $target/theme/
cp -a $migration/mod/vpl $target/mod/

chown -R moodle:www-data $target

### The upgrade part :
cd $target
sudo -u www-data /usr/bin/php8.3 admin/cli/upgrade.php

# All plugins copies
cp -a $migration/local/mb2builder $target/local/
cp -a $migration/local/mb2slides $target/local/
cp -a $migration/local/mb2coursenotes $target/local/
cp -a $migration/local/mb2megamenu $target/local/
cp -a $migration/local/mb2notices $target/local/
cp -a $migration/local/mb2reviews $target/local/
cp -a $migration/local/staticpage $target/local/
cp -a $migration/lib/editor/atto/plugins/mb2shortcodes $target/lib/editor/atto/plugins/
cp -a $migration/lib/editor/atto/plugins/c4l $target/lib/editor/atto/plugins/
cp -a $migration/lib/editor/atto/plugins/code $target/lib/editor/atto/plugins/
cp -a $migration/lib/editor/atto/plugins/templates4u $target/lib/editor/atto/plugins/
cp -a $migration/filter/mb2shortcodes $target/filter/
cp -a $migration/filter/synhi $target/filter/
cp -a $migration/filter/syntaxhighlighter $target/filter/
cp -a $migration/filter/filtercodes $target/filter/
cp -a $migration/report/coursesize $target/report/
cp -a $migration/blocks/mb2azcourses $target/blocks/
cp -a $migration/question/behaviour/adaptive_adapted_for_coderunner $target/question/behaviour/
cp -a $migration/question/type/coderunner $target/question/type/
cp -a $migration/admin/tool/redis $target/admin/tool/
cp -a $migration/local/mailtest $target/local/

systemctl restart php8.3-fpm.service && systemctl restart apache2

cd /srv/www/learn.isc-vs.ch/moodle_isc/
sudo -u www-data /usr/bin/php admin/cli/maintenance.php --disable

# Erreur "cache definition does not exist local_mb2builder/pagedata" → vider le cache :
cd /srv/www/learn.isc-vs.ch/moodle_data/
rm -rf ./cache/ ./temp/ ./trashdir/ ./sessions/ ./localcache/ ./muc/

# Finir l'upgrade dans le GUI sans le plugin "stockphotos", puis le désinstaller
mkdir /srv/migration_moodle/quarantine_faulty_plugin
mv /srv/www/learn.isc-vs.ch/moodle_isc_4.4.2/repository/stockphotos /srv/migration_moodle/quarantine_faulty_plugin/

# Test Switch-Edu ID : connexion OK
############# Upgrade terminée   #######