ISC GitLab — Architecture and Deployment Plan

Author: Pierre-André Mudry · Date: August 2026 (revised 2026-08-26) · Status: design aligned to the retained architecture; deployment is phase 2 of its plan d'exécution

This document covers the GitLab service. Hardware, zones and phases are defined once, in the architecture page; what was learnt from the July 2026 evaluation instance is in the rumba history.

---

1. Context and sizing

Self-hosted GitLab instance for the ISC programme:

• ~30 permanent staff
• ~40 new students per year on a 3-year curriculum → ~120 active students
• Total: ~150 active users, zero growth (the graduating cohort is archived annually)

GitLab's reference architecture for 1,000 users is 8 vCPU / 16 GB RAM. The production cluster (pve-01/pve-02, two dual-EPYC 7443 nodes with ≈ 5.8 TiB of nvme_pool each, plus pve-03 and gpu-01) is oversized for GitLab alone, so GitLab is one VM among the production core guests; the rest of the estate serves CI runners and the other services.

2. General architecture

Every runner → GitLab arrow is outbound: a runner polls GitLab over HTTPS and GitLab never connects to a runner. No inbound port is needed on runner machines, which is what lets them sit in lab-virtual, on carnaval, or anywhere on the campus network.

Structural decisions

Decision	Choice	Rationale
Hypervisor	Proxmox VE 9.x, the production cluster	Snapshot before every GitLab upgrade, PBS backups, the same management plane as every other service
GitLab distribution	GitLab CE, Omnibus package	PostgreSQL/Redis/nginx bundled, simple upgrades. No Helm/K8s at this scale
Storage	nvme_pool — ZFS mirror vdevs on the PM1735	IOPS for PostgreSQL and Git; the same storage ID on every production node so the guest migrates
Availability	pvesr replication pve-01 ↔ pve-02, HA group restricted to those two	No shared storage, no Ceph; a 5–15 min RPO is acceptable for this service
Publication	Behind the srv-web01 Caddy proxy, names in isc-vs.ch	The only inbound path is 80/443 to the proxy; hevs.ch is not our zone, isc-vs.ch is (landing page)
Authentication	OIDC client of Keycloak	edu-ID is already federated through the broker; one AAI registration for the fleet, no new SWITCH paperwork
Runners	Separate VMs in lab-virtual, never on the GitLab VM, never in srv-internal	CI runs arbitrary student code: isolation, CPU/RAM ceilings, zone rules
Backups	pbs-01 nightly + gitlab-backup	Two restore paths: whole VM, or a single project

Placement and high availability

The gitlab VM belongs to a Proxmox HA group restricted to pve-01 and pve-02. Guests use cpu: x86-64-v3, which every production node provides (all four are AMD), so a CPU-only guest live-migrates anywhere in the cluster. The GPU runner VMs hold a passed-through card and are therefore pinned to gpu-01 and excluded from HA groups — they are rebuilt from a playbook, not migrated or backed up.

Git over SSH

The uplink admits TCP 80/443 and UDP 51820 inbound, nothing else (inbound rules). Consequence: from the Internet, Git works over HTTPS only; git@ SSH access is available over the NetBird VPN, where the VM's LAN address is reachable directly. Document HTTPS + personal access token as the default in the student instructions.

3. Group structure

isc/
├── staff/            ← staff members
├── teaching/         ← courses, modules, solutions
└── students/
    ├── 2026/         ← incoming cohort
    ├── 2025/
    └── 2024/         ← archived wholesale at graduation

Annual archiving is a single operation on isc/students/<year>: projects archived (read-only), accounts blocked. Active volume growth ≈ zero. Group membership comes from the Keycloak groups claim (students, staff), the same roles the VPN uses.

4. Backup strategy

1. pvesr replication pve-01 ↔ pve-02. Not a backup (corruption replicates too) but the availability mechanism: if a node dies, the VM restarts on the other with at most the replication interval of lag.
2. Proxmox Backup Server on pbs-01 (Dell R740xd all-NVMe, in 23N307 — the other building): nightly vzdump of the VM, deduplicated, verified weekly, 14 d / 8 w / 6 m retention. The PVE token is DatastoreBackup-only, so a compromised node cannot prune or delete history. Until pbs-01 is racked, the existing srv-pbs on the FS2500 takes the job.
3. gitlab-backup create (nightly cron) plus /etc/gitlab/ (secrets — without this directory the backup is unusable), shipped off the VM with proxmox-backup-client into pbs-01. Covers restoring a single project, migrating to a new VM, changing version.
4. ZFS snapshots on the dataset (24 hourly / 7 daily) — operator error.

Restore test each semester: gitlab-backup restore into a throwaway VM (~30 min).

5. Scalability

• GitLab: no scaling needed at 150 users — a 6× margin before reaching the 1k reference architecture.
• Runners: stateless. If CI load grows, register another runner with the same tags.
• Storage: per-namespace quotas plus a default artifact expiry (2 weeks), otherwise the pool fills up. nvme_pool grows by add-in NVMe cards, not bays: both R282-Z92 U.2 backplanes are defective (Epyc).

5b. Runners

Principle

A runner needs outbound HTTPS reachability to GitLab and a registration token — no inbound port, no GitLab-side configuration. Job routing happens through tags: a job declares the tags it needs and GitLab dispatches it to the first available runner carrying them. Two runners with the same tags form a pool; stopping one is transparent.

Procedure per Linux runner

# 1. Docker
curl -fsSL https://get.docker.com | sh

# 2. gitlab-runner
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | bash
apt install -y gitlab-runner

# 3. Create the runner in GitLab: Admin → CI/CD → Runners → New instance runner
#    → pick the tags (e.g. linux,docker) → copy the token (glrt-...)

# 4. Register
gitlab-runner register --non-interactive \
  --url "https://git.isc-vs.ch" --token "glrt-XXXX" \
  --executor docker --docker-image "ubuntu:24.04"

Then in /etc/gitlab-runner/config.toml: set concurrent to the machine's capacity (rule of thumb: core count ÷ 4).

GPU runners

• gpu-01 (G242-Z11, 3 × RTX PRO 4500 Blackwell 32 GB, phase 3): one GPU passed through whole per runner-gpu-0x VM — gitlab-runner, Docker executor, NVIDIA Container Toolkit, gpus = "all" in [runners.docker], tags linux,gpu,cuda. Spare GPUs serve course inference.
• Interim until gpu-01 lands: carnaval2's Tesla T4 passed through to a first runner VM (whole-card passthrough, VM not LXC — one passthrough mode per node). The T4 returns to student CUDA afterwards.

macOS and Windows runners

Linux covers most of the curriculum; two cases do not: Xcode/Swift builds and signing (iOS and macOS courses) and x86 Windows builds (.NET Framework, MSVC, Windows-only tooling). Neither can run in a Linux container. GitLab offers hosted macOS/Windows runners on gitlab.com only, not to a self-managed instance, so both are ours to provide.

	macOS	Windows
Where it runs	Apple hardware only — the macOS licence allows virtualisation solely on a Mac, two guests per host. A Mac mini (Apple silicon) in rack-B, 1 U on a shelf, ~CHF 700–1 000	A VM on the production cluster, lab-virtual zone, from the school's Windows licence — Windows Server for docker-windows, or Windows 11 for a plain shell runner
Executor	tart through the custom executor: an ephemeral macOS VM cloned from a base image per job, destroyed afterwards. Two jobs in parallel per Mac, the licence ceiling	docker-windows with Windows containers (mcr.microsoft.com/windows/servercore) — the only Windows executor that isolates student code. shell (PowerShell) if a job needs the full desktop stack (MSVC, WPF, installers)
Isolation	Per-job VM, host untouched	Containers: acceptable for student jobs. Shell: the job runs as the runner's user on the VM — staff projects only, as a group runner on isc/teaching; snapshot the VM and roll back weekly
Tags	macos, arm64, xcode	windows, and windows-shell for the unisolated one
Upkeep	Xcode updates by rebuilding the tart base image; Apple ID for signing kept in a staff-only CI variable	Windows Update in a monthly window with the GitLab upgrade; Docker Desktop is not licensed for this — use the Docker CE engine for Windows Server or Mirantis runtime

Rules of thumb before buying either: a Windows or macOS need that is only compiling is often a Linux job with a cross toolchain (mingw-w64, .NET SDK on Linux, Flutter builds for Android); what genuinely needs the platform is Xcode, code signing, MSVC and GUI tests. Start with one Windows VM (no hardware cost) and buy the Mac mini when a course actually asks for it. Both runners register like any other — outbound HTTPS to git.isc-vs.ch, no inbound port — so the Mac can sit on the campus network if rack space or the VLAN plan is a problem.

Security rules (identical everywhere)

1. Never mount /var/run/docker.sock into jobs — CI runs arbitrary student code.
2. No shell executor for student jobs on any platform — Docker on Linux and Windows, tart VMs on macOS. The Windows shell runner serves trusted staff projects, restricted via a group runner on isc/teaching rather than instance-wide.
3. Runners live in lab-virtual: outbound Internet and 443 to GitLab, nothing toward srv-*/mgmt/oob. A runner that disappears breaks nothing: remove it from the admin list.

6. Operations

• Upgrades: fixed monthly window, Proxmox snapshot first, following the official upgrade path across major versions.
• Container registry on registry.isc-vs.ch, a second Caddy vhost proxied to the VM.
• Pages: one wildcard entry (*.pages.isc-vs.ch) rather than a record per project; the wildcard certificate needs a DNS-01 challenge against the Infomaniak zone, which the proxy does not do today — an open item before Pages is enabled.
• Mail: through srv-mail, the rack's Postfix relay — plain SMTP on the LAN, no credential in gitlab.rb.
• Software cost: CHF 0.

7. Prerequisites

Already in place (August 2026):

Need	Covered by
edu-ID sign-on	Keycloak — the OIDC client to SWITCH is approved; GitLab is registered as a client of the isc realm, redirect URI https://git.isc-vs.ch/users/auth/openid_connect/callback
Public names and TLS	isc-vs.ch at Infomaniak, certificates on the Caddy proxy — add git and registry A records to the existing public address and two vhosts
Outbound mail	srv-mail relay
Off-site backup	srv-pbs now; pbs-01 (ordered) later
Admin access	NetBird VPN

Still to do, in the order of the architecture's phases:

1. Phase 1 — the cluster. epyc1 runs PVE standalone today; epyc0 is in repair (Epyc). GitLab waits for the two-node cluster with the QDevice on mgmt-01.
2. Phase 2 — GitLab + CPU runner on the cluster, runner-gpu-01 on carnaval2. Address plan and DNS entries per the proxmox-guest skill; VM in srv-internal, runners in lab-virtual.
3. Phase 3 — gpu-01 joins the cluster; runners leave the T4.

The install runbook is provisioning/gitlab/install-runbook.md (revised with this page, 2026-08-26): it starts once the cluster exists and covers VMs, GitLab, proxy publication, runners, backups and the admin settings.