Email — outbound mail from the rack

ISC³ has no mail server. Everything in the rack that needs to send mail hands it to srv-mail, a Postfix relay in CT 112, which holds the only copy of the credential and does the authenticated leg to Infomaniak. The mailbox is mailer@isc-vs.ch — the same one Moodle prod uses on hannibal. SPF/DKIM need no upkeep: Infomaniak hosts the isc-vs.ch zone.

Until August 2026 nothing here could send: vsnet dropped every SMTP port (catalogued in When the network lies), so alerting was Telegram-only and the nightly backup's "mail on failure" was a silent no-op. SInf opened the egress on request; relay and PVE notifications live since 2026-08-10.

Why a relay and not credentials per service

• Some devices cannot do authenticated TLS. The APC NMC and the iDRAC8s of the spare R630s are effectively port-25-plaintext only. Since the egress allow is 465-only, without a relay they could not send anything. This is the blocking reason, and it covers the thermal alarm.
• One copy of the password. It would otherwise end up in PVE, the PDU, Keycloak, DSM, the iDRACs and the MikroTiks — seven places, one of them a student-facing production Moodle, which in practice means it never gets rotated.
• Queueing. An appliance that sends an alert during an Infomaniak hiccup loses it. The relay retries for a day.
• Sender rewriting. Infomaniak refuses any envelope sender but the mailbox itself, and most appliances cannot be told what to use. The relay rewrites unconditionally.
• One log. /var/log/postfix.log on srv-mail is the only record of what the rack has sent.

Parameters

For a rack device — no credential, no TLS, nothing to rotate:

Server	srv-mail.isc3 = 192.168.88.163, port 25
Auth / TLS	none — LAN only, and the source address is the authorisation
Sender	anything; rewritten to mailer@isc-vs.ch
Recipients	rack-admins@isc-vs.ch — the group address, see below

The relay's own upstream leg — mail.infomaniak.com (83.166.143.44, .45) on 465, implicit TLS, AUTH PLAIN as mailer@isc-vs.ch, password in secretzone/smtp.md.

The egress allow is destination-scoped — do not design around a fallback

SInf's rule permits SMTP to Infomaniak's mail IPs and nothing else. Verified 2026-08-10: 465 open, 25 and 587 timing out. SInf then opened 587 as well — verified from rumba on 2026-08-11: 465 and 587 reach 83.166.143.44/.45, while 587 to smtp.gmail.com, smtp.hevs.ch and portquiz.net still times out, as does 25 to anything. So there is a STARTTLS port to the same provider but no second provider, and if Infomaniak ever renumbers mail.infomaniak.com outbound mail stops until SInf extends the rule. The relay stays on 465/implicit TLS. The 465 allow covers the whole 192.168.88.0/24 — for 587 only rumba has been tested — so a new consumer needs no request to SInf, only a line in the relay's allowlist.

Who may send, and to whom

Three independent layers, all in provisioning/mail/deploy-relay.sh. Change them there and re-run; the script is idempotent.

Layer	Mechanism	Why
Which hosts may submit	mynetworks, an explicit address list	The rack is one flat L2 domain, so a future test guest must not inherit the right to send
Which recipients are accepted	check_recipient_access allowlist: the group address and the individual admin addresses	Even an abused relay reaches only the admin addresses
How many	smtpd_client_message_rate_limit = 30 per minute per client	The realistic failure is a flapping probe emptying itself into the mailbox Moodle also uses

One group address, expanded on the relay

Every consumer writes to rack-admins@isc-vs.ch; nothing in the rack names a person. The address is hosted nowhere — virtual_alias_maps on srv-mail expands it to the current admin addresses and the mail leaves already addressed to them. virtual(5) rewrites every recipient whatever the domain, so isc-vs.ch does not have to be a local domain on the relay.

Who receives rack mail is therefore one line, ADMINS in provisioning/mail/deploy-relay.sh; edit it, re-run, and every consumer follows without being touched. Two ordering facts: the allowlist is checked at RCPT TO, before expansion in cleanup, so the group address needs its own OK line; and the individual addresses stay on the allowlist, so a consumer that still names one directly keeps working.

The PVE and PBS endpoints do not name the group either — they target root@pam, and that user record carries the group address; notify-smtp.sh and pbs-notify.sh assert the field. On a node, node-relay.sh writes root: rack-admins@isc-vs.ch into /etc/aliases, so zed, smartd, cron, upsmon and any daemon added later follow the admin list with no further work. Appliances and the notification systems never mail root — they carry their own recipient field, which is why the group address exists rather than a second entry in each.

caution

Never put 192.168.88.1 or the whole /24 in mynetworks

The CCR2004 src-NATs legacy WireGuard clients to 192.168.88.1, so listing that address would make every VPN user an authorised sender as @isc-vs.ch, with valid SPF and DKIM — a credible phishing vector from our own domain. Same reasoning as the NAS lockout, in the other direction. Only one legacy WireGuard client remains (the rest moved to NetBird, unmasqueraded since 2026-08-03), but the rule stands.

The recipient allowlist has a consequence: this relay is for infrastructure alerting, not for user mail. Moodle prod is unaffected — it lives outside the rack and mails students directly with its own settings, which is why the password still exists in mdl_config too. Consolidation is deliberately partial.

Consumers

Consumer	State	Notes
rumba PVE notifications	✅ via relay	endpoint rack-relay, failures only; no credential on the hypervisor
rumba host: ZFS (zed), smartd, cron, UPS (upsmon)	✅ via relay	provisioning/mail/node-relay.sh — see the warning below
PBS srv-pbs	✅ via relay	verify / GC / prune failures; provisioning/mail/pbs-notify.sh, plus its own host mail
PDU AP8681 thermal alarm	✅ via relay	provisioning/mail/pdu-email.sh; thresholds 45 °C high / 50 °C max. Still names a person — ops todo
rumba iDRAC9	✅ via relay	hardware faults (PSU, fans, predictive disk failure); provisioning/mail/idrac-email.sh, 192.168.92.10
NAS FS2500 DSM	✅ via relay	disk / volume / SMART / UPS events. Transport from nas-dsm-email.sh; the recipient is UI-only — DSM 7 keeps it as a separate profile that eventmails does not feed, so it still names a person (ops todo)
Moodle prod (hannibal, outside the rack)	✅ direct	settings in the mdl_config table, not config.php; cannot use the relay
Keycloak, MikroTiks	❌ open by choice	ops todo — Keycloak is user-facing mail and needs the recipient allowlist widened

Purge the queue when you fix a host's transport

A machine that has been deferring mail for days returns the old messages to sender as soon as a working transport appears — so fixing it produces a confusing bounce quoting the previous failure (connect to hevs-ch.mail.protection.outlook.com:25: Connection timed out), delivered through the new relay. Run mailq and postsuper -d ALL deferred as part of the fix; node-relay.sh now prints what is pending instead of leaving it to surprise you. Nodes also get maillog_file = /var/log/postfix.log — a PVE node ships no rsyslog, and without it the host keeps no mail log, which makes this kind of after-the-fact question unanswerable.

zed and smartd were enabled all along, with no working transport

zfs-zed and smartd are on by default on a PVE node and both mail root. With no root alias and no relayhost, a degraded RAIDZ2 pool, a scrub error or a SMART pre-failure warning went to a local mailbox nobody opens — 21 stale vzdump reports were sitting in the queue when the relay went in. If you add a node, run node-relay.sh on it or it starts out the same way.

Rotating the Infomaniak password now touches two places: srv-mail (re-run the deploy script) and Moodle prod. Consumers of the relay are unaffected — none of them holds a credential.

Proxmox notifications

provisioning/pve/notify-smtp.sh configures a node idempotently — endpoint, matcher and vzdump jobs. It defaults to the relay and needs no password:

ssh root@rumba 'bash -s -- --test' < provisioning/pve/notify-smtp.sh
VIA_RELAY=0 SMTP_PASSWORD=… ssh root@rumba 'bash -s' < provisioning/pve/notify-smtp.sh  # direct, fallback

Three things about PVE that cost time if unknown:

• A job's legacy mailnotification failure field is ignored by the endpoint system. Each vzdump job needs notification-mode notification-system; otherwise it falls back to local sendmail, which direct-to-MXes on port 25 and defers forever — unless the host's own postfix has been pointed at the relay, which node-relay.sh now does on rumba, so /usr/sbin/sendmail works too.
• The builtin default-matcher targets mail-to-root, a sendmail target — which is why the rack looked configured for months while nothing could arrive. It now points at rack-relay with match-severity warning,error: success reports stay in the task log on purpose, since two nightly backup jobs would otherwise mail every day until nobody read the mailbox.
• pvesh create /cluster/notifications/targets/<name>/test bypasses matchers, so it proves the endpoint only, and it is silent on success. To exercise the matcher path, run a real backup.

Header rewriting for remote clients

Postfix rewrites header addresses only for clients matching local_header_rewrite_clients, whose default covers locally-submitted mail only. A rack device is a remote client, so its envelope sender was rewritten to mailer@isc-vs.ch while From: kept e.g. root@rumba.isc3 — and Infomaniak answers 550 5.7.1 Sender mismatch. It is set to permit_mynetworks here, which is exactly the set of hosts allowed to submit. Symptom if it ever regresses: PVE and anything that sets its own From keeps working, while sendmail, cron and appliances bounce.

Verifying

ssh root@rumba 'pct exec 112 -- tail -20 /var/log/postfix.log'   # every mail the rack sent
ssh root@rumba 'pct exec 112 -- mailq'                           # stuck mail
ssh root@rumba 'pct exec 112 -- postconf mynetworks'             # who may submit

A successful hop reads status=sent (250 2.0.0 Ok: queued as …). A refusal is explicit: Client host rejected: Access denied (layer 1) or Recipient address rejected (layer 2).

Open items: ops todo → Email.

Known gap

The relay runs on rumba, like srv-status which sends the Telegram alerts. If rumba is down — or overheating, the June 2026 scenario — nothing gets out on any channel. The rack still has no alert path independent of itself; a relay does not change that. Closing it needs something outside the rack (a channel via hannibal, or an external heartbeat that alerts on absence of signal). Telegram therefore stays the primary channel, email the secondary.