Epyc
Two dual-socket AMD EPYC servers (Gigabyte R282-Z92) intended for the future Proxmox
infrastructure — see Future work. epyc1 (U34) runs
Proxmox VE as a standalone node; epyc0 (U8) is in repair as of August 2026 and offline, expected back on the bench in
September 2026 (as of 2026-09-01). A third R282-Z92, bought new in August 2026, is held as an unracked cold spare
for the pair: 2 × EPYC 7302 (16 c Rome, 3.0/3.3 GHz, 128 MB L3, 155 W), 24 × U.2 + 2 × 2.5"
bays, 2 × 1600 W, no RAM and no drives. Its delivery date is not recorded
(todo). Dated
operation logs are in Epyc — history & operations.
At a glance
epyc1 | |
|---|---|
| Role | Proxmox VE node, standalone (own cluster CA — not the Rumba cluster) |
| Machine | Gigabyte R282-Z92-00 |
| CPU / RAM | 2× AMD EPYC 7443 (24 cores each, 48 total) · 480 GiB |
| Firmware | see BIOS and firmware |
| Hypervisor | PVE 9.2.2, kernel 7.0.2-6-pve (August 2026) |
| Network | 192.168.88.46, static on the host · BMC 192.168.88.12 |
| Rack | U34–35 (see rack layout) |
| Access | VPN, then SSH as root or https://192.168.88.46:8006 |
Memory
Both nodes run DDR4-3200 RDIMM, populated differently. The spare chassis holds none: the shelf carries 2 × 32 GB after the August 2026 harvest, so pressing the spare into service means moving the failed node's DIMMs across — which works for a board or CPU failure and not for a memory one.
epyc0 holds 16 × 32 GB, one per channel — 8 Samsung M393A4K40DB3-CWE and 8 Micron
18ASF4G72PZ-3G2E1, all in the Dimm1 slots, running at 3200 MHz (BMC Redfish, 2026-08-29). It
held 1 TB in all 32 slots until 16 modules were pulled for pve-03 and epyc1
(log).
epyc1 mixes 32 GB Samsung and 16 GB Kingston modules. The mixing rule from the board manual is
what makes its layout look wrong at first glance:
AMD's rule, in Gigabyte's population guideline, is that all eight channels per socket carry the
same capacity, and that DIMMs sharing a channel match in module type, DRAM width and density. The
mixed 32/16 GB stock satisfies it by giving every one of the 16 channels 32 GB — either one
32 GB module, or a pair of 16 GB. Where a channel holds a single 32 GB module, its second slot
stays empty by design: on epyc1 that is DIMM_P0_A0, P0_E0, P1_I0 and P1_M0. They are not
faulty.
epyc1 reports 480 GiB, not 512: channel P on socket P1 is not enumerated — both
DIMM_P1_P0 and DIMM_P1_P1, exactly the missing 32 GB. Setup, dmidecode and the BMC all report
those two slots as absent with no SPD data at all, identical to an empty slot, while every other
populated slot returns its part number. Since both slots of one channel fail together while the
other seven channels on that socket work, the shared element is the channel itself — its routing or
its pin field in socket P1.
Firmware does not treat the channel as merely empty. It records the absence as a memory fault and a deliberate map-out, in two places, on every boot:
| Source | What it reports |
|---|---|
| BIOS, via BERT | severity fatal · fru_text: DIMM Locate: P1H0 · section_type: memory error · error_status: Storage error in DRAM memory · node:1 card:4 module:0 · error_type: 15, physical memory map-out event |
| BMC, via SEL | Memory Device Disabled · Event_Data_2 = 0x40 (card 4, module 0) · Event_Data_3 = 1 (node 1) |
Both name socket 1, card 4, module 0. P1H0 is the same channel as the board's P: AMD letters the
channels A–H per socket, Gigabyte silkscreens CPU1's as I–P, so socket 1's eighth channel is AMD's
H and the board's P.
DIMM_P0_A1, which is healthyEvery one of these events prints Sensor_Name : DIMM_P0_A1. That is the static SDR label for memory
sensor #19, not the failing part — DIMM_P0_A1 enumerates normally as a 32 GB Samsung and passes
every check. Read the Event_Data bytes, not the sensor name.
The record looks stale rather than live. A channel whose DIMMs return no SPD was never trained, so it cannot be producing a DRAM storage error now, and the entry survives reboots, a BIOS defaults reset and a BIOS setting change — which fits a persisted NVRAM record replayed through BERT at each boot.
Everything reachable without opening the chassis has been eliminated — the modules, the seating,
the firmware and DRAM Soft Post Package Repair (log). What
remains is physical and is listed in the todo: a CMOS/NVRAM clear, the
only thing that clears a persisted map-out, then a CPU swap between sockets — if the fault follows
the CPU it is the CPU, if it stays with the socket it is the board.
Storage
epyc1 has two pools (August 2026):
| Pool | Layout | Usable | Serves |
|---|---|---|---|
rpool | mirror, 2× Samsung MZ7L3480HCHQ 480 GB SATA SSD | 441 GiB | boot · PVE local, local-zfs |
nvme_pool | RAIDZ1, 3× Samsung MZPLJ3T2HBJR (PM1735) 3.2 TB NVMe | 5.68 TiB | PVE nvme_pool — VM disks and containers |
nvme_pool is ashift=12, autotrim=on, compression=lz4, atime=off, registered in PVE with
blocksize 16k and content rootdir,images. It survives one drive failure. RAIDZ1 over a 3-way
mirror: an odd drive count rules out striped mirrors, and a 3-way mirror would spend 6.4 TB of the
9.6 TB on redundancy; the 16K volume block size keeps RAIDZ1 padding waste low on a 3-wide vdev.
The layout is reproducible from provisioning/pve/epyc1-nvme-pool.sh. Measured 4.7 GB/s write and
7.8 GB/s read with direct I/O (August 2026) — roughly two orders of magnitude past what the host's
1 GbE link can carry, which is the point of the network note below.
Scrub and trim run monthly from the zfsutils-linux cron. A fourth NVMe, a consumer Samsung 980
500 GB, is wiped and unallocated.
The drives are second-hand and their hours are worth knowing: 11,459 power-on hours on each PM1735 and 11,897 on the boot mirror (August 2026), against 0% endurance used, no reallocated sectors and no media errors. Age, not wear, is the thing to watch — the same correlated-failure reasoning as the NAS Toshibas.
The U.2 backplanes are defective on both machines
The 24 hot-swap U.2 bays cannot be used as planned. Both epyc0 and epyc1 have a per-bay
defect on the backplane PCB — affecting different bays on each machine — so the PM983 U.2 drives
bought for them stayed out, and epyc1 runs on add-in-card NVMe instead. The architecture no longer
depends on the bays: the 19 TB tank target was dropped on 2026-08-13 in favour of four add-in-card
NVMe per node, so this is an inventory problem rather than a blocker
(§5.2).
How the fault is localised, which is worth knowing before re-opening a chassis: with power only and no data cables connected, the LEDs of the bad bays blink while the good ones stay steady, and only the steady-LED bays go on to enumerate and train once cabled. Cables and riser cards are not the cause; Gigabyte declined to supply CPLD firmware and the BIOS and BMC updates it advised instead changed nothing.
The LED test is the signal to work from because a bad bay leaves no trace in the operating system,
and no amount of error reporting would change that: a drive that never enumerates never trains a
link, so there is no link partner and nothing to report. PCIe AER is also off on this machine
(why, and how to enable it), but enabling it would not have helped here for the same reason.
Do not expect dmesg to corroborate a bad bay.
Reference points for whoever picks this up:
| Bays on CPU_0 | NVME_#00–#11 (root ports 00:02.x, 20:06.x, 40:02.x, 60:06.x) |
| Bays on CPU_1 | NVME_#12–#23 (root ports 80:02.x, a0:06.x, c0:02.x, e0:06.x) |
| Backplane CPLDs | BPB_CPLD1 v13, BPB_CPLD2 v10 — two of them, matching the two 12-bay groups |
| Board / BIOS | MZ92-FS0-00 rev 100 · BIOS M30 (2026-06-23) · BMC 12.61.39 |
Bay LEDs, per-bay power sequencing and presence detect are CPLD functions, so the symptom fits a
CPLD or its group's power circuitry as well as it fits a dead PCB — untested, because the firmware
is not obtainable. Gigabyte's own BIOS and BMC packages contain no CPLD image, and the BMC blob
ships encrypted (rom.ima_enc), so there is nothing to extract locally either.
A cable bypass is possible: the board carries SlimLine SAS connectors (SLSAS_0–SLSAS_3, the
family sold as SlimSAS SFF-8654) and bays #10–#11 are already fed by an onboard NVMe cable
rather than through a card. Running SFF-8654-to-U.2 cables straight to the drives skips the
backplane, at the cost of hot-swap and tidy cabling.
The 24-bay configuration is built from add-in cards that occupy almost
every slot: CNV3024 in both x16 FHHL slots, CNV3022 in an x8 on each CPU, CNVO134 in the
OCP 3.0 slot and CNVO022 in the OCP 2.0 mezzanine. In that configuration only two x8 FHHL slots
remain free, one per CPU. So bays and add-in-card NVMe compete for the same slots.
What epyc1 actually carries (August 2026): the three PM1735 at x8 each on CPU0, the
ConnectX-6 at x16 on CPU1, and the spare Samsung 980 in the onboard M.2 slot. Restoring
the 24-bay path would displace both the NVMe that carries nvme_pool and the InfiniBand card, which
is why the storage decision and the fabric decision cannot be taken separately.
Which card feeds which bays, for correlating a bad bay with its feeder:
| Feeder | Bays | Feeder | Bays | |
|---|---|---|---|---|
CNV3024 | #00–#03 | CNVO134 | #12–#15 | |
CNV3022 | #04–#05 | CNV3024 | #16–#19 | |
CNVO134 | #06–#09 | CNV3022 | #20–#21 | |
| onboard cable | #10–#11 | CNVO022 | #22–#23 |
Open items: ops todo → epyc.
Proxmox migration requires a storage of the same ID to exist on the target node. epyc1's pool is
nvme_pool; whatever epyc0 ends up with should carry the same storage ID, even if its disk layout
differs, or guest configs need editing by hand on every migration. Asymmetric layouts are otherwise
fine here — there is no Ceph and no shared storage (§5.1).
zfs-zed is running and mails root, but postfix has no relayhost and there is no root alias, so
a nvme_pool degradation shows only in local mail and the web UI. Open item:
ops todo → epyc.
Network
epyc1 sits on the appliances subnet, its address configured statically on the host. Both the host
NIC and the BMC reach the CCR2004 through the FS S3600-48T4S switch (router C) on sfp+12; neither
is on the nodes switch (CRS326). The BMC has a DHCP reservation and answers to epyc1-bmc — see the
network page.
epyc1 identifies itself as epyc1.isc-vs.ch, while the rest of the rack uses the .isc3 DHCP
domain.
The data path is one 1 GbE link: nic0 of the onboard Intel I350, at 1000 Mb/s, with the second port
nic1 down. Local storage measures 4.7–7.8 GB/s, so the link, not the disks, bounds anything moving
data in or out of this node. Worth settling before guests land here
(todo).
An unused ConnectX-6
epyc1 carries a Mellanox InfiniBand card that nothing uses, found during the
2026-08-13 checkup:
| Device | a1:00.0 — MT4123 / MT28908, ConnectX-6 VPI, single port, firmware 20.31.2354 |
| Link | PCIe Gen4 x16 on CPU1, 252 Gb/s available — headroom for HDR200 |
| Mode | InfiniBand, not Ethernet |
| State | port DOWN, physical state Disabled, "Cable unplugged" — no transceiver fitted |
Because it is in InfiniBand mode the card appears as mlx5_0 under /sys/class/infiniband/ and
creates no network interface, which is why ip link lists only the two I350 ports. Do not read that
as a driver problem. Turning it into Ethernet would need Mellanox MFT (mlxconfig LINK_TYPE_P1=2),
which is not installed, and there is no 100 GbE port in the rack to reach either way. It is an asset
for the future fabric rather than usable networking
today, and it bears on the open HDR100-versus-HDR200 question in the
todo.
Power
Two 1600 W supplies, both PS-2162-6L2, of which only PSU2 is fed — PSU1's cord is not plugged in.
The BMC consequently reports PS1_Status as Health: Critical, State: Disabled with
LineInputVoltage: 0 while its FRU still reads normally, which is the signature of a supply with no
AC rather than a failed one. The node runs without power redundancy by choice. Draw is 153 W of the
1600 W available (August 2026, idle, no guests).
BIOS and firmware
| Component | Version (August 2026) |
|---|---|
| BIOS | M30, dated 2026-06-23 |
| BMC | 12.61.39 (AMI MegaRAC) |
MB_CPLD1 | 15 |
BPB_CPLD1 / BPB_CPLD2 | 13 / 10 |
The spare R282-Z92 came with EPYC 7302 (7002 series). Moving a 7443 (7003) into it needs a BIOS
revision that supports Milan, and that flash has to happen while a Rome chip is still installed —
check before swapping CPUs, not after. Guests are unaffected either way: they run
cpu: x86-64-v3, which Zen 2 and Zen 3 both provide.
The BMC exposes all 562 BIOS attributes read-only. PATCH /redfish/v1/Systems/Self/Bios returns 405
with Allow: GET, the standard Bios/Settings pending-object 404s, Bios.ChangeAttributes answers
ActionNotSupported, and the only BIOS actions are ChangePassword and ResetBios. The OEM
sections hold nothing relevant (Gbt → GPUManager, Ami → VirtualMedia). AMI's SCELNX_64, which
writes setup variables from the OS, is not in Gigabyte's package — Tool/LNX/ carries only the
afulnx wrappers, and afulnx flashes whole images rather than single variables. Any setup change
therefore needs the HTML5 KVM console.
Systems/Self does accept PATCH, and BiosSetup is an allowable BootSourceOverrideTarget, so
arming a one-shot override and rebooting drops the machine into setup without anyone having to catch
the POST keypress.
Attribute names are opaque codes (Milan####, GBT####, FBO###) — translate them through
/redfish/v1/Registries/BiosAttributeRegistry.json, which carries the display names, defaults and
help text. Milan0134 DRAM Soft Post Package Repair reads Enabled since 2026-08-13, against a
default of Disabled; Milan0137 DRAM ECC Enable is Auto, which means enabled.
PCIe AER is off, and can be turned on
The kernel reports _OSC: platform does not support [SHPCHotplug AER LTR DPC] on all eight root
complexes, so the OS receives no PCIe error reporting. Two things explain it, and neither is a
malfunction:
Milan0168 Enable AER CapisDisabled, which is Gigabyte's shipped default.- The platform runs APEI firmware-first error handling —
GHES: APEI firmware first mode is enabled by APEI bit, with a 346 KB HEST — so firmware keeps errors and reports them through APEI. That is the same path the channel-Pmap-out record arrives on.
The capability itself is present on 85 devices, whose aer_dev_correctable counters exist and read
zero. Enabling it is one change, Milan0168 → Enabled, optionally with Milan0202 PCIe Aer Reporting Mechanism → OS First so Linux logs the errors rather than firmware.
What that buys is visibility of degradation on links that are up: a marginal riser, a failing
transceiver, an NVMe link retraining under heat. What it does not buy is anything for the
dead bays — AER reports on trained links, and a drive that never enumerates never
trains one. Two things not to chase: Milan0164 ACS Enable notes that AER must be on for ACS to
work, but isolation here is already per-device (144 IOMMU groups, with the ConnectX-6 and each NVMe
alone in theirs), so PCI passthrough is not a reason to enable it; and the pcie_ports=native kernel
parameter is the wrong lever, because it overrides the firmware's _OSC refusal and leaves both
firmware and OS handling the same error.
Access
root SSH accepts pmudry's key, held in PVE's /etc/pve/priv/authorized_keys; password
authentication is also enabled. Credentials for both machines and their BMCs are in
secretzone/epyc.md.
Open items for these machines are on the ops todo.