Skip to main content

GitLab ISC — Architecture and deployment plan

Author: Pierre-André Mudry · Date: July 2026 · Status: proposal

1. Context and sizing​

Self-hosted GitLab instance for the ISC programme:

  • ~30 permanent staff members
  • ~40 new students per year, 3-year curriculum → ~120 active students
  • Total: ~150 active users, zero growth (yearly archiving of the graduating cohort)

The GitLab reference architecture for 1,000 users is 8 vCPU / 16 GB RAM. The planned server (dual-EPYC, 1 TB RAM, 40 TB NVMe) is therefore vastly oversized for GitLab alone. Consequence: we virtualize — GitLab occupies only a fraction of the machine, the rest serves CI runners and future services.

2. General architecture​

Note: all runner → GitLab arrows are outbound. A runner initiates the connection and polls GitLab over HTTPS; GitLab never connects to the runners. Consequence: no inbound port to open on runner machines — they can be anywhere on the campus network.

Structuring choices​

DecisionChoiceRationale
HypervisorProxmox VE (free, AGPLv3)Snapshots before every GitLab upgrade, one-click VM backups, future flexibility
GitLab distributionGitLab CE, Omnibus packagePostgreSQL/Redis/nginx/Let's Encrypt included, simple upgrades. No Helm/K8s at this scale
StorageZFS, vdevs in mirrors (not RAIDZ)Better IOPS for PostgreSQL and Git, fast resilver, extension by pairs
RunnersSeparate VM, never on the GitLab OSCI executes arbitrary student code: isolation + CPU/RAM caps
AuthenticationOIDC via SWITCH edu-IDZero-touch onboarding in September, every student arrives with an edu-ID

3. Group organisation​

isc/
├── staff/            ← staff members
├── teaching/         ← courses, modules, solutions
└── students/
    ├── 2026/         ← incoming cohort
    ├── 2025/
    └── 2024/         ← archived in bulk at graduation

The yearly archiving = one operation on isc/students/<year>: projects archived (read-only), accounts blocked. Active volume growth ≈ zero.

4. Backup strategy (3 layers)​

  1. Proxmox Backup Server (separate machine) — nightly backup of the whole GitLab VM, deduplicated, incremental, verified. Restore = one click, the VM comes back identical. This is the disaster-recovery path.
  2. gitlab-backup create (nightly cron) + copy of /etc/gitlab/ (secrets — without this folder the backup is unusable), exported off the VM. Covers: restoring a single project, migrating to a new VM, version changes.
  3. Automatic ZFS snapshots (24 hourly / 7 daily) — a free safety net against handling mistakes.

Restore test: once per semester, gitlab-backup restore into a disposable VM (~30 min).

5. Scalability​

  • GitLab: will never need scaling at 150 users. ×6 headroom before reaching the 1k reference architecture.
  • Runners: stateless. CI load going up → install gitlab-runner on an extra machine, register it with the same tags, done. GitLab sees no difference.
  • Storage: per-namespace quotas + default artifact expiry (2 weeks), otherwise the 40 TB melt away. ZFS pool extension by adding mirror pairs.

5bis. Adding runners on existing bare-metal machines​

Principle​

A runner needs only one thing: reaching GitLab over outbound HTTPS. No inbound port, no GitLab-side configuration other than creating a token. Any available machine can therefore become a runner in ~10 minutes, wherever it sits on the campus network.

Job routing is done by tags: a student job declares the tags it needs, GitLab dispatches it to the first available runner carrying those tags.

Two runners with the same tags (e.g. the runner-01 VM and a linux,docker bare-metal machine) automatically form a pool: GitLab balances the load between them, and stopping one is transparent (jobs go to the other).

Procedure per Linux machine​

# 1. Docker
curl -fsSL https://get.docker.com | sh

# 2. gitlab-runner
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | bash
apt install -y gitlab-runner

# 3. Create the runner in GitLab: Admin → CI/CD → Runners → New instance runner
#    → pick the tags (e.g. linux,docker) → copy the token (glrt-...)

# 4. Register
gitlab-runner register --non-interactive \
  --url "https://git.isc.hevs.ch" --token "glrt-XXXX" \
  --executor docker --docker-image "ubuntu:24.04"

Then in /etc/gitlab-runner/config.toml: adjust concurrent to the machine's capacity (simple rule: number of cores ÷ 4).

Special cases​

  • Machine with a GPU: install the NVIDIA Container Toolkit, then in config.toml, section [runners.docker]: gpus = "all". Tags: linux,gpu,cuda. Student jobs use it with tags: [gpu] and a CUDA image.
  • Windows machine (e.g. builds requiring Windows x86): install gitlab-runner.exe as a service, shell executor (PowerShell) or docker-windows. Tag: windows.
  • Machine shared with another purpose (e.g. the PBS host): cap concurrent = 2 and set a Docker memory limit (memory = "8g" in [runners.docker]) so the main service is not starved.

Security rules (identical everywhere)​

  1. Never mount /var/run/docker.sock in jobs — CI executes arbitrary student code.
  2. No shell-executor runner on Linux for student jobs (the code would run directly on the machine's OS) — Docker executor only. The Windows shell exception must only serve trusted staff projects, restricted via a group runner on isc/teaching rather than instance-wide.
  3. A runner that disappears or breaks doesn't break anything: remove it from the list in the GitLab admin, that's all.

6. Operations​

  • Upgrades: fixed monthly window, Proxmox snapshot before, follow the official upgrade path between major versions.
  • Container registry enabled (registry.<domain>) — heavy student usage expected.
  • Runners: Docker executor, tags (linux, gpu, …), instance-wide shared fleet.
  • Software costs: 0 CHF. Option: Proxmox subscription (~€230/year for 2 sockets) for the enterprise repository — useful but not necessary.

7. Prerequisites before installation — how-to​

Five steps to complete before launching the runbook. No. 3 (SWITCH) has an administrative delay: start with it.

7.1 Reserve two IP addresses and the DNS entries​

What to do: ask the HES-SO Valais-Wallis IT service for:

  • 2 static IPs on the server VLAN: one for the gitlab VM, one for the runner-01 VM (+ note the gateway, netmask, DNS servers — these values go into the runbook's placeholder table).
  • 2 DNS A records pointing to the gitlab VM's IP:
    • git.isc.hevs.ch → gitlab VM IP
    • registry.isc.hevs.ch → same IP

How to verify: from your workstation, nslookup git.isc.hevs.ch must answer the right IP. Without this, Let's Encrypt cannot issue the certificates and the installation will block at phase 3.

7.2 Request the firewall openings​

What to do: ticket to the network service with this table:

SourceDestinationPortUsage
Campus network (+ Internet if external access wanted)gitlab VM80/tcpHTTPS redirect + Let's Encrypt challenge
samegitlab VM443/tcpWeb UI, API, registry
samegitlab VM22/tcpGit over SSH (git clone git@...)
runner-01 VMgitlab VM443/tcpThe runner contacts GitLab
Proxmox hostPBS machine8007/tcpBackups

Decision to make: GitLab reachable from the Internet or campus/VPN only? For students working from home, Internet access is strongly recommended (that is how a GitLab normally operates). Port 80 must be open from the Internet at minimum for Let's Encrypt, even if you then restrict 443.

7.3 Register the OIDC client with SWITCH edu-ID ⏱️ do this first​

What to do: GitLab must be declared as a "client application" with SWITCH so that the edu-ID login button works.

  1. Contact the AAI representative of HES-SO Valais-Wallis (every SWITCH institution has a designated person; the IT service knows who it is). That person has access to the SWITCH registration portal.
  2. Request the registration of an OpenID Connect client with these parameters:
    • Service name: GitLab ISC
    • Redirect URI: https://git.isc.hevs.ch/users/auth/openid_connect/callback (exactly this URL, with the FQDN chosen in 7.1)
    • Scopes: openid, profile, email
    • Flow: authorization code
  3. In return, you receive three values to keep safe: issuer URL, client_id, client_secret. These are the runbook's OIDC_* placeholders.

Delay: count a few days to a few weeks depending on responsiveness. Meanwhile, the installation can proceed anyway — the local root login works, and OIDC is enabled afterwards with a simple gitlab.rb change.

7.4 Obtain an institutional SMTP relay​

What to do: ask the IT service for the internal mail relay parameters (host, port, possible authentication) and the authorization for the gitlab VM to send through it, with a sender address like gitlab-noreply@hevs.ch. Without SMTP, no notifications and no invitation mails — GitLab works, but degraded.

7.5 Designate the backup machine and install Proxmox​

PBS machine: pick one of your available machines (the one planned as a future runner can do both). Criterion: enough disk for ~3× the useful size of the GitLab VM (deduplication helps a lot). Install Proxmox Backup Server on it from the official ISO (proxmox.com → Downloads), guided installation in 10 minutes. Note its IP and the fingerprint shown in its web UI (Dashboard) — the runbook's PBS_HOST placeholders.

Proxmox VE on the dual-EPYC server:

  1. Download the Proxmox VE ISO (proxmox.com → Downloads) and copy it to a USB stick (balenaEtcher or dd), or mount it through the server's IPMI/BMC interface (virtual media — more comfortable, no need to move).
  2. Boot on it, guided installation: pick a dedicated disk for the system (or a ZFS mirror pair via the installer) — do not use the NVMe drives intended for the data pool, they will be configured in phase 1 of the runbook.
  3. Fill in the management IP, gateway, DNS, root password, email.
  4. Check access to the web UI: https://<server-ip>:8006.

Once these five points are settled, fill in the placeholder table at the top of the runbook and launch Claude Code on the Proxmox host.