Proxmox Fleet Reference

ChiefOperationsOfficer VP TechOps
,
1

Canonical source: This wiki topic is the living reference. Updated in place; edit history preserves all prior states.

Covers: per-host hardware/network/storage/tuning reference, VM inventory, physical systems, capacity goals, naming policy, project report.

Contents

Each section below is posted as a separate reply for independent editing.
Use the reply index to navigate.

2

Host Reference

One-page-per-host quick reference for every Proxmox node. Hardware,
network, storage, role, and running VMs. Data gathered 2026-07-30.

Companion docs: NETWORK-TOPOLOGY.md,
STORAGE-ARCHITECTURE.md,
VM-INVENTORY.md.

Fleet summary

graph TB
    subgraph "RACK 5 -- Core / Data / Infra"
        T1["tsys1 -- Infra<br>i7-4770 | 32GB | 9 VMs<br>57% RAM"]
        T3["tsys3 -- Compute<br>Xeon E3-1535M v5 | 32GB | 1 VM<br>10% RAM"]
        T4["tsys4 -- Storage+PBS<br>Xeon E3-1246 v3 | 16GB | 1 VM<br>15% RAM"]
        T5["tsys5 -- Storage+Sandbox<br>Xeon E5620 | 96GB | 4 VMs<br>15% RAM"]
        T9["tsys9 -- Infra+Compute<br>i5-10500 | 24GB | 6 VMs<br>69% RAM"]
    end

    subgraph "RACK 3 -- Compute / RackRental"
        T6["tsys6 -- RackRental<br>2x Xeon E5530 | 126GB | 2 VMs<br>12% RAM"]
        T7["tsys7 -- RackRental<br>2x Xeon E5-2630 v2 | 189GB | 5 VMs<br>7% RAM"]
    end
Host CPU (era) Threads RAM PVE Kernel Role Running VMs RAM actual%
tsys1 i7-4770 Haswell (2013) 8 32 GB 9.2.5 7.0.14 Infra 9 57%
tsys3 Xeon E3-1535M v5 Skylake (2015) 8 32 GB 9.2.5 7.0.14 Compute 1 10%
tsys4 Xeon E3-1246 v3 Haswell (2013) 8 16 GB 9.1.5 6.17.9 Storage + PBS 1 15%
tsys5 Xeon E5620 Westmere (2010) 8 96 GB 9.2.5 7.0.14 Storage + Sandbox 4 15%
tsys6 2x Xeon E5530 Nehalem (2009) 16 126 GB 9.2.5 7.0.14 RackRental 2 12%
tsys7 2x Xeon E5-2630 v2 Ivy Bridge (2013) 24 189 GB 9.2.5 7.0.14 RackRental 5 7%
tsys9 i5-10500 Comet Lake (2020) 12 24 GB 9.2.5 7.0.14 Infra + Compute 6 69%

Retired: tsys2 (Win10, pending rebuild), tsys8 (permanently retired)

pfv-tsys1 – Infrastructure host

Attribute Value
Role Infrastructure: Home Assistant, CA, netinfra-01, LibreNMS, PDM, k8s cnode3, UCS-01, kali, workbench
Hardware Dell, i7-4770 Haswell (2013), 8 threads, 32 GB RAM (maxed)
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.3.11 (vmbr0, DHCP)
Tailscale 100.121.189.98
Storage IP 10.100.100.1 (datanet via nic1, USB dongle, no bond)
NFS mounts S1, S2, S3, S4, D2 (all nconnect=4)
Local storage local-lvm (~90 GB, HDD, marginal)
Tuning governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host
Bond None (single USB NIC for storage)
Special services NUT (UPS), powerman (PDU)
Running VMs 100, 101, 103, 104, 105, 106, 108, 114, 117

pfv-tsys3 – Compute host

Attribute Value
Role Compute: k8s wnode-tsys3 (28 GB RAM, 8 vCPU)
Hardware Xeon E3-1535M v5 Skylake (2015), 8 threads, 32 GB RAM
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.2.5 (vmbr0, DHCP)
Storage IP 10.100.100.3 (datanet via enx USB dongle, no bond)
NFS mounts D2, D5, S2, S4, T5-SSD (all nconnect=4)
Local storage local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) – best existing local storage
Tuning governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host
Bond None (single USB NIC for storage)
Running VMs 313

pfv-tsys4 – Storage server + PBS

Attribute Value
Role NFS storage server (D1/D2/D3/D5) + Proxmox Backup Server + console management (conman/ser2net)
Hardware Xeon E3-1246 v3 Haswell (2013), 8 threads, 16 GB RAM
PVE 9.1.5 / kernel 6.17.9-1-pve (BEHIND fleet)
mgmt IP 192.168.3.251 (vmbr0, DHCP)
Tailscale 100.70.77.93
Storage IP 10.100.100.4 (datanet via enx8cae4ccda926 USB dongle, bond0 with 1 slave)
NFS mounts None (NFS server only)
Local storage local-lvm (WDC WD10EZRX 1TB, ~62 GB free)
NFS exports D1 (WD 1TB USB), D2 (WDC Red 3TB), D3 (Seagate 500GB USB), D5 (Hitachi 2TB)
Other disks /mnt/albert (Hitachi 2TB, not exported), /mnt/backup (WD 5TB USB)
Tuning governor=performance, swappiness=1, bbr, rmem/wmem=16MB (should be 128MB), tuned=throughput-performance (should be network-throughput)
Bond bond0 (1 USB slave), hash=layer2+3
Special services ser2net + conman (7 switch consoles on :7890)
Running VMs 400 (PBS)
Notes USB NIC bottleneck. Needs PVE upgrade + tuning alignment.

pfv-tsys5 – Storage server + sandboxes

Attribute Value
Role NFS storage server (S1/S2) + sectestbed sandboxes + ultix streaming
Hardware Dell T7500, Xeon E5620 Westmere (2010) single CPU (CPU2 socket empty), 8 threads, 96 GB RAM
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.3.179 (vmbr0, DHCP)
Storage IP 10.100.100.5 (storagenet via bond0, 1 of 2 slaves active – nic2 DOWN)
NFS mounts D2, D3, D5 from tsys4 (missing nconnect)
NFS exports S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO)
Local storage local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB)
Unconfigured disks NVMe CT500P1 500GB (not mounted), SK hynix SC300B 512GB SSD (not mounted)
Tuning governor=N/A (no cpufreq driver on Westmere), swappiness=1, bbr, rmem/wmem=128MB, tuned=network-throughput
Bond bond0 (1 of 2 slaves), hash=layer2+3
Running VMs 509, 515, 5111, 5112 (+ 22 stopped sandboxes)
Notes Bond degraded (needs 2nd cable). NVMe + SSD need mounting/config. SSD 93% full. Single CPU.

pfv-tsys6 – RackRental host

Attribute Value
Role RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX
Hardware 2x Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.3.169 (vmbr0, DHCP)
Tailscale 100.73.35.111
Storage IP 10.100.100.6 (storagenet via bond0, 2 of 2 slaves active, 2G LACP)
NFS mounts D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4)
Local storage local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, 30 MB/s – do not use for VMs)
Tuning governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host
Bond bond0 (2 of 2 slaves), hash=layer3+4 (correct)
Running VMs 100 (wnode-tsys6, 64GB), 600 (tsys-awx)
Notes Best storage bond in fleet (2G LACP). Local storage is USB 2.0 – unusable for scratch.

pfv-tsys7 – RackRental host

Attribute Value
Role RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2
Hardware 2x Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.3.55 (vmbr0, DHCP)
Storage IP 10.100.100.7 (datanet via bond0, 2 of 2 slaves active, 2G LACP)
NFS mounts D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4)
Local storage local-lvm (WD portable USB 2.0 HDD, 1.7 TB, 30 MB/s – do not use for VMs)
Tuning governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host
Bond bond0 (2 of 2 slaves), hash=layer3+4 (correct)
Running VMs 701, 702, 703, 704, 705
Notes Best storage bond in fleet (2G LACP). Most RAM in fleet (189 GB). Local storage USB 2.0.

pfv-tsys9 – Infra + Compute

Attribute Value
Role Infra + Compute: k8s cnode1 + wnode-tsys9 + siem + UCS-02 + netinfra-02 + kali
Hardware i5-10500 Comet Lake (2020), 12 threads, 24 GB RAM
PVE 9.2.5 / kernel 7.0.14-6-pve
mgmt IP 192.168.3.58 (vmbr0, DHCP)
Storage IP 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond)
NFS mounts D2, D5, S2, S3, S4, T5-SSD (all nconnect=4)
Local storage local-lvm (PNY CS900 SSD, 136 GB, 0% used) – excellent for scratch
Tuning governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host
Bond None (single USB NIC for storage)
Running VMs 901, 902, 903, 904, 905, 906
Notes Newest CPU (best IPC in fleet). 69% RAM – approaching 80% goal. Target for compute growth (after siem migrates off).
3

VM Inventory

Every VM in the cluster, with host placement, disk location, CPU/RAM
allocation, and optimization status. Data gathered 2026-07-30.

Companion docs: NETWORK-TOPOLOGY.md,
STORAGE-ARCHITECTURE.md,
HOST-REFERENCE.md.

1. VM placement diagram

graph TB
    subgraph tsys1["tsys1 -- Infra (57% RAM)"]
        VM100["100 pfv-bms<br>2c/4G D2"]
        VM101["101 tsys-ca<br>2c/2G D2"]
        VM103["103 netinfra-01<br>2c/4G S2"]
        VM104["104 librenms<br>2c/2G D2"]
        VM105["105 proxmox-dc<br>2c/2G D2"]
        VM106["106 cnode3<br>4c/4G D5"]
        VM108["108 ucs-01<br>2c/8G D2"]
        VM114["114 kali-tsys<br>2c/2G D2"]
        VM117["117 workbench<br>2c/4G D2"]
    end

    subgraph tsys3["tsys3 -- Compute (10% RAM)"]
        VM313["313 wnode-tsys3<br>8c/28G D5"]
    end

    subgraph tsys4["tsys4 -- Storage (15% RAM)"]
        VM400["400 PBS<br>2c/2G local-lvm"]
    end

    subgraph tsys5["tsys5 -- Storage+Sandbox (15% RAM)"]
        VM509["509 wnode-tsys5<br>2c/32G D2"]
        VM515["515 hfnoc-uisp-preprod<br>2c/2G local-nonprod"]
        VM5111["5111 ultix-streaming<br>4c/9G SSD"]
        VM5112["5112 ultix-offstage<br>4c/6G local-lvm"]
    end

    subgraph tsys6["tsys6 -- RackRental (12% RAM)"]
        VM100b["100 wnode-tsys6<br>2c/64G S4"]
        VM600["600 tsys-awx<br>2c/12G D5"]
    end

    subgraph tsys7["tsys7 -- RackRental (7% RAM)"]
        VM701["701 wnode-tsys7<br>4c/32G S3"]
        VM702["702 hfnoc-uisp<br>2c/8G S1"]
        VM703["703 rr-middleware<br>2c/2G S2"]
        VM704["704 TCTC<br>4c/6G D2"]
        VM705["705 cnode2<br>4c/4G S2"]
    end

    subgraph tsys9["tsys9 -- Infra+Compute (69% RAM)"]
        VM901["901 tsys-siem<br>2c/8G D2"]
        VM902["902 ucs-02<br>2c/8G S2"]
        VM903["903 kali-rd<br>2c/2G D5"]
        VM904["904 netinfra-02<br>2c/4G D2"]
        VM905["905 wnode-tsys9<br>4c/4G S2"]
        VM906["906 cnode1<br>2c/4G S3"]
    end

2. Complete running VM inventory (28 running VMs)

Infrastructure (netinfra, UCS, CA, LibreNMS, PDM)

VMID Name Host vCPU CPU type RAM Disk store Spindle Disk type Cache iothread qga Notes
100 pfv-bms (HomeAssistant) tsys1 2 host 4G D2 (raw) D2 raw – – ✓ discard=on, ssd=1. HA for power/temp monitoring.
101 tsys-ca tsys1 2 v2-AES 2G D2 (virtio) D2 qcow2 – ✓ – no nested-virt flag. Certificate Authority.
103 pfv-netinfra-01 tsys1 2 v2-AES 4G S2 S2 qcow2 – ✓ – DNS primary (Technitium + Pi-hole).
104 tsys-librenms tsys1 2 host 2G D2 D2 qcow2 – ✓ ✓ SNMP monitoring for switches.
105 tsys-proxmox-datacenter tsys1 2 host 2G D2 D2 qcow2 – ✓ ✓ PDM (Proxmox Datacenter Manager).
108 tsys-ucs-01 tsys1 2 host 8G D2 D2 qcow2 writethrough ✓ ✓ Univention Corporate Server half 1.
902 tsys-ucs-02 tsys9 2 host 8G S2 S2 qcow2 – ✓ ✓ UCS half 2 (HA pair with 108).
904 pfv-netinfra-02 tsys9 2 v2-AES 4G D2 D2 qcow2 – ✓ – DNS secondary. HA pair with 103.

k8s control plane (cnode1/2/3)

VMID Name Host vCPU CPU type RAM Disk store Spindle Disk type Cache iothread qga Notes
106 pfv-k8s-cnode3 tsys1 4 host 4G D5 D5 qcow2 writethrough ✓ ✓ etcd member.
705 pfv-k8s-cnode2 tsys7 4 host 4G S2 S2 qcow2 writethrough ✓ ✓ etcd member.
906 pfv-k8s-cnode1 tsys9 2 host 4G S3 S3 qcow2 – ✓ – etcd member.

k8s workers (wnodes)

VMID Name Host vCPU CPU type RAM Disk store Spindle Disk type Cache iothread qga Notes
313 pfv-k8s-wnode-tsys3 tsys3 8 host 28G D5 D5 qcow2 writethrough ✓ ✓ Compute worker.
509 pfv-k8s-wnode-tsys5 tsys5 2 host 32G D2 (virtio) D2 qcow2 – ✓ – Undersized vCPU (2).
100 pfv-k8s-wnode-tsys6 tsys6 2 host 64G S4 S4 qcow2 – ✓ – Undersized vCPU (2), high RAM.
701 pfv-k8s-wnode-tsys7 tsys7 4 host 32G S3 (virtio) S3 qcow2 – ✓ – RackRental worker.
905 pfv-k8s-wnode-tsys9 tsys9 4 v2-AES 4G S2 S2 qcow2 – ✓ ✓ Target: grow to 6c/8G.

Application VMs

VMID Name Host vCPU CPU type RAM Disk store Spindle Disk type Cache iothread qga Notes
114 kali-tsys tsys1 2 host 2G D2 D2 qcow2 writethrough ✓ ✓ Kali Linux pentest.
117 tsys-secure-workbench tsys1 2 host 4G D2 D2 qcow2 writethrough ✓ ✓ Secure workbench.
400 pfv-proxmox-backup-server tsys4 2 – 2G local-lvm (local) raw – – – PBS – all VM backups.
515 hfnoc-uisp-preprod tsys5 2 host 2G local-nonprod S1 qcow2 – ✓ – Preprod. Locked to tsys5.
5111 ultix-streaming tsys5 4 – 9G SSD dir SSD qcow2 – – – 288GB on Samsung 860 PRO (93% full).
5112 ultix-offstage tsys5 4 – 6G local-lvm (local) raw – – – 288GB. Locked to tsys5.
600 tsys-awx tsys6 2 host 12G D5 D5 qcow2 writethrough ✓ ✓ Ansible AWX.
702 hfnoc-uisp tsys7 2 host 8G S1 S1 qcow2 – ✓ ✓ 100GB disk. UISP network monitoring.
703 rr-middleware tsys7 2 host 2G S2 S2 qcow2 writethrough ✓ ✓ RackRental middleware.
704 TCTC tsys7 4 host 6G D2 (virtio) D2 qcow2 – ✓ –
901 tsys-siem tsys9 2 host 8G D2 D2 qcow2 writethrough ✓ ✓ 132GB disk. SIEM.
903 kali-rd tsys9 2 host 2G D5 D5 qcow2 writethrough ✓ ✓ Kali R&D.

3. Optimization findings

3.1 Cache mode audit

cache=writethrough forces synchronous writes to the NFS server on every write
operation. For NFS-backed qcow2, this is unnecessarily slow – writeback or
none is recommended (qcow2 already provides journaling integrity).

Cache mode Count VMs
writethrough (slow) 10 106, 108, 114, 117, 313, 600, 703, 705, 901, 903
none/default (optimal) 18 all others

Fix: Change cache=writethrough → cache=writeback or remove (defaults to
writeback for qcow2). Requires VM stop/start.

3.2 CPU type audit

cpu: host passes the host’s full CPU instruction set to the VM (best
performance). x86-64-v2-AES is a conservative baseline (safe for migration
but lacks some host instructions).

CPU type Count VMs
host 21 most production VMs
x86-64-v2-AES 5 101, 103, 509, 904, 905
not set 2 400, 5111, 5112

Note: Mixed CPU types prevent live migration between hosts with different
CPU generations. K8s cnodes/wnodes should ideally match within each tier.

3.3 Disk controller audit

Controller Count Notes
scsi0 (virtio-scsi) 22 Optimal for Proxmox
virtio0 4 101, 509, 701, 704 – also good
raw (no virtio) 2 100 (pfv-bms), 500 (stopped)

All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks.

3.4 Network audit

All running VMs use virtio-net. Firewall enabled on most (firewall=1).
No legacy e1000/rtl8139 adapters.

3.5 Guest agent (qga) audit

Status Count VMs without qga
enabled 20 –
not set 8 101, 103, 509, 100(tsys6), 701, 704, 904, 906

Fix: Enable qm set <vmid> --agent 1 on the 8 VMs missing it. No VM
stop/start required (takes effect on next guest reboot, but the setting itself
is immediate).

3.6 k8s wnode sizing

VMID Name vCPU RAM Tier target Issue
509 wnode-tsys5 2 32G compute Undersized vCPU (2 for a compute worker)
100 wnode-tsys6 2 64G rackrental Undersized vCPU (2 for 64G RAM)
701 wnode-tsys7 4 32G rackrental OK for idle labs
905 wnode-tsys9 4 4G compute Target: grow to 6c/8G after siem migrates off
313 wnode-tsys3 8 28G compute OK (best compute worker)

4. Stopped VMs (not running, for reference)

Sectestbed / sandboxes (all on tsys5, all stopped)

VMID Name RAM Disk Notes
5100 sectestbed-template 4G local-lvm Base template
5101 sectestbed-siem 10G local-nonprod (S1) 132G disk
5102 sectestbed-proxmox-pve 4G local-lvm
5103 sectestbed-proxmox-datacenter 4G local-lvm
5104 sectestbed-proxmox-pbs 4G local-lvm
5105 sectestbed-awx 4G local-nonprod (S1) 288G disk
5106 sectestbed-k8s-cnode 4G local-nonprod (S1)
5107 sectestbed-k8s-wnode 4G local-nonprod (S1)
5108 sectestbed-librenms 4G local-nonprod (S1)
5109 sectestbed-netinfra 4G local-nonprod (S1)
51010 sectestbed-tctc 4G local-nonprod (S1)
51011 sectestbed-cloudron 4G local-nonprod (S1)
51012 sectestbed-hfnoc 4G local-nonprod (S1)
51013 sectestbed-rancherplatform 4G local-nonprod (S1)
51014 sectestbed-proxmox-mailgw 4G local-nonprod (S1)
53100 tsys-preprod-awx 9G local-nonprod (S1) 160G disk
53101 tsys-preprod-siem 12G local-nonprod (S1)
53102 tsys-preprod-rancherplatform 8G local-nonprod (S1)
53103 tsys-preprod-proxmoxmailgw 4G local-nonprod (S1)
6000 sectestbed-sandbox 4G local-nonprod (S1) Has 3 state snapshots

Other stopped VMs

VMID Name Host RAM Disk Notes
102 pfv-k8s-wnode-tsys1 tsys1 4G (no disk) Stopped, no boot disk
500 DellOpenManageEnterprise tsys5 20G D7 (gone!) Would fail – disk export removed
5500 RestoreTemplate tsys5 2G D2 Template
1000 KNELTemplate tsys1 2G D2 Base template
3000 KNELTemplate tsys3 2G D2 Base template copy
4000 KNELTemplate tsys4 2G (local) Base template copy
4

Physical Systems

Access patterns and locations for physical infrastructure devices.
Updated: 2026-07-31

Raspberry Pi systems

All accessible via Tailscale. Use localuser (passwordless sudo to root)
except where noted.

Name Tailscale IP LAN IP Access Purpose
pfvsvrpi 100.91.151.113 192.168.1.230 localuser Server Pi (general purpose)
subopi-dev-3 100.64.231.65 192.168.1.232 via subodev jump Suborbital dev Pi 3
subopi-dev-4 100.65.224.85 192.168.1.231 via subodev jump Suborbital dev Pi 4

Note from operator: “use subodev to get into subopi-*” — subodev is
a jump host for the suborbital Pi’s.

Network devices discovered (non-Proxmox)

IP Device Type Identified via
192.168.0.12 pfv-core-sw01 Dell PowerConnect 5448 SNMP
192.168.0.7 pfv-tor3-mgmt Neyland 24T (PC5324) SNMP
192.168.0.8 (unidentified) Neyland 24T (PC5324) SNMP
192.168.0.9 pfv-tor3-stor Neyland 24T (PC5324) SNMP
192.168.1.166 pfv-stor1 Linux 4.14.22 (Synology?) SNMP
192.168.1.182 UAP-AC-LR UniFi AP (FW 6.8.2.15592) SNMP
192.168.1.230 pfvsvrpi Linux 6.12.62 (RPi) SNMP
192.168.1.231 subopi-dev-4 Linux 6.12.25 (RPi) SNMP
192.168.1.232 subopi-dev-3 Linux 6.12.9 (RPi) SNMP
192.168.1.217 subopi3 Linux 6.12.25 (RPi) SNMP

Other identified hosts (SSH open, unidentified)

IP Notes
192.168.0.72 SSH open, no SNMP — investigate
192.168.0.250 SSH open, no SNMP — investigate
192.168.1.138 SSH open — investigate
192.168.1.181 SSH open — investigate
192.168.1.192 SSH open — investigate
192.168.1.201 SSH open — investigate
192.168.1.205 SSH open — investigate
192.168.1.214 SSH open — investigate
192.168.1.223 SSH open — investigate
192.168.1.226 SSH open — investigate
192.168.1.228 SSH open — investigate
192.168.2.5 SSH open — investigate (tsys3 vmbr0)
192.168.3.1 SSH open — investigate (gateway/router?)
192.168.3.78 SSH open — investigate

192.168.3.12-24 range

These 13 hosts respond to ping but have no SSH/SNMP. Likely:

  • IoT devices
  • Cameras
  • Phones (VoIP)
  • Other DHCP clients

UniFi Access Points

AP Port (core-sw01) Model FW
UAP-AC-LR g9 (2902 inErrors!) UAP-AC-LR 6.8.2.15592
UAP-AC-Lite g38 (590 inErrors!) UAP-AC-Lite (unknown)

Both managed by UniFi controller (likely on Home Assistant VM 100 or
pfv-stor1). UniPoller exports metrics to Prometheus.

5

Capacity Goals

Date: 2026-07-29
Status: Authoritative. These are the operator’s standing targets. Every
fleet audit and VM-placement decision is evaluated against them.

Companion to AUDIT-2026-07-29.md (the current
placement analysis) and PROJECT.md (the fleet hardware report).

1. Memory target: 80% steady-state

Rule: Keep each node at or below 80% physical RAM utilization at steady
state. Occasional bursts above 80% are acceptable (that is what swap is for);
the sustained idle-to-average load must sit at ~80% or below.

Two metrics must both be tracked:

Metric Definition Target
Actual usage MemTotal - MemAvailable from /proc/meminfo ≤ 80% steady state (the real rule)
Allocation commitment sum of running-VM memory: vs host RAM planning ceiling — keep < 100% where possible

Why both: KSM (same-page merging) and virtio-balloon mean actual usage is
often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
72% actual). Actual usage is the steady-state truth; allocation is the
worst-case ceiling if every VM suddenly fills its reservation. A host is
“healthy” when actual ≤ 80% and allocation isn’t so far over 100% that a
balloon-deflate storm could OOM it.

Tolerance: brief actual spikes to 90-100% are fine. Persistent swap use
(>~1 GB on a steady basis) means the host is over-committed and VMs should
migrate off.

2. CPU target: ~50% utilization at idle

Rule: Ideally, nodes sit around 50% CPU utilization at idle.

This is a utilization goal, not a headroom goal. The fleet is
CPU-underutilized (most hosts idle at 0-6%). On a shoestring budget, idle
silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
The intent is to pack workloads so the baseline load consumes roughly half
the compute, reserving the other half for bursts.

Practical implications:

  • The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
    doing useful work at ~50% or be candidates to power down / solar-gate when
    idle. Running them at 0-6% wastes power per core.
  • The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
    idle and are the natural home for always-on control-plane + infra.
  • vCPU overcommit is fine for idle workloads; the 50% target is about actual
    utilization, not vCPU ratios.

3. Workload placement model

The fleet serves three distinct workload families. Placement should follow CPU
quality + storage speed + RAM size, in that order of relevance per family.

3.1 ETL / HPC / cross-compile (CPU + storage bound)

Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
Needs high IPC (modern cores) + fast storage (SSD/NVMe). RAM is moderate
(8-32 GB per job).

Host CPU Storage Fit
tsys9 i5-10500 Comet Lake (2020) — best IPC in fleet local SSD (PNY CS900) Primary compute node
tsys3 Xeon E3-1535M v5 Skylake (2015) local NVMe (Samsung PM961) Fast compute node
tsys2 (incoming) i7-7820HQ Kaby Lake (2017) NVMe 960 PRO + SSD 850 EVO Fast compute node (post-rebuild)

These three hosts have the newest CPUs and the only fast local storage. Their
wnodes should be the compute-tier workers, sized larger (more cores/RAM) and
labeled workload=compute.

3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)

Business idea: rent out ephemeral network labs (containerlab topologies) on
demand. Many containers / network namespaces, short-lived, mostly idle once up.
Needs massive RAM + fast container-image pulls. CPU quality is secondary
(labs are idle between keystrokes).

Host CPU RAM Fit
tsys7 2× E5-2630 v2 Ivy Bridge (2013), 24 threads 189 GB Primary RackRental host
tsys6 2× E5530 Nehalem (2009), 16 threads 126 GB Secondary RackRental host

Verdict: tsys6/7 are the correct RackRental hosts. Their weak CPUs are
irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
labeled workload=rackrental.

Caveat (deferred): their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
is terrible for container image pulls. This is a disk problem, not a RAM/CPU
problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
ready today.

3.3 Infrastructure + k8s control plane (always-on, low resource)

netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
etcd diversity).

4. Host role summary (target)

Host CPU era RAM Primary role Wnode class
tsys1 Haswell 2013 32 GB (maxed) Infra + control plane light (4c/4G)
tsys2 Kaby Lake 2017 32 GB Compute (incoming) compute
tsys3 Skylake 2015 32 GB Compute compute
tsys4 Haswell 2013 16 GB Storage (NFS + PBS) —
tsys5 Westmere 2010 96 GB Storage + fast-tier hub compute (NVMe, post-Friday)
tsys6 Nehalem 2009 126 GB RackRental rackrental
tsys7 Ivy Bridge 2013 189 GB RackRental rackrental
tsys9 Comet Lake 2020 24 GB Infra + control plane + compute compute (growable)

5. DDR3 memory policy

Do not blindly add RAM to tsys6/7. They idle at 6%/9% actual usage with
117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
not starved). The fleet’s RAM pressure is on the infra hosts (tsys1/5/9
over-allocated), and that is solved by migrating VMs to tsys6/7’s existing
free RAM, not by installing sticks.

Compatibility is decisive and is documented per-host in
AUDIT-2026-07-29.md §5. Short version: tsys5/6/7 require
Registered ECC (RDIMM); only RDIMM sticks will POST. Consumer UDIMM
(Patriot / most Kingston) fits essentially nowhere useful here.

6. Local storage policy: ephemeral scratch, not OS drives

Principle: Local disk storage is a scratch tier, not a boot/OS tier.
VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
workloads where data loss on node failure is acceptable by definition.

6.1 Why OS drives stay on NFS

Individual Proxmox nodes have no local disk redundancy (single disk, no
RAID, no ZFS mirror). If a host’s local disk dies, every VM on it is gone.
Redundancy is instead achieved at the storage-server + application layer:

  • -01/-02 HA pairs (netinfra, UCS): each half on a different NFS server
  • k8s spreading: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
    single disk or server failure takes down the whole cluster
  • PBS backups: all VMs are backed up to Proxmox Backup Server regardless of
    disk tier

This is deliberate. Moving OS drives to local storage would trade NFS network
latency for single-disk-failure risk. Not worth it for persistent workloads.

6.2 What local storage IS good for: ephemeral scratch

Workloads that produce or consume temporary data that can be discarded on node
failure. These are exactly the workloads where NFS latency hurts most and local
IOPS help most:

Use case Why local scratch helps Examples
Container image pulls Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow RackRental labs, any pod startup
Build artifacts Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD ETL, cross-compile, CI pipelines
Temporary processing Intermediate data that doesn’t need to survive a reboot Weather/GIS transform scratch, log rotation
kubelet ephemeral storage emptyDir volumes, container writable layers All k8s pods

6.3 Per-host local storage inventory and suitability

Host Local storage Type Available Scratch-suitable? Notes
tsys2 NVMe 960 PRO + SSD 850 EVO NVMe + SSD 512 GB + 1 TB Yes — best in fleet Incoming (post-rebuild). No NFS needed at all
tsys3 local-lvm (Samsung PM961) NVMe 349 GB Yes — excellent Fastest existing local storage. 0% used
tsys9 local-lvm (PNY CS900) SSD 136 GB Yes — excellent Low latency. 0% used
tsys5 local-lvm (Hitachi) HDD ~1.7 TB Yes (bulk) Large capacity, HDD speed. Storage host — scratch is secondary
tsys1 local-lvm HDD ~90 GB Marginal (small) Barely worth it; tsys1 is infra-loaded anyway
tsys4 local-lvm HDD ~62 GB No PBS host; minimal free space
tsys6 local-lvm (WD My Passport) USB 2.0 HDD 1.7 TB NO — 30 MB/s Slower than NFS. Do not use for VMs
tsys7 local-lvm (WD portable) USB 2.0 HDD 1.7 TB NO — 30 MB/s Slower than NFS. Do not use for VMs

tsys6/7 are the hard problem. They are the RackRental hosts — exactly the
workload that benefits most from local scratch (fast container image pulls) —
but their local storage is USB 2.0 portable HDD, which is slower than
NFS-over-HDD. Options for the future:

  1. Install internal drives (tsys6 has a disabled SAS controller; tsys7 has
    5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
    local scratch. Deferred to a physical-hardware session.
  2. Accept NFS for image pulls and use local-lvm only for swap/temp that
    tolerates 30 MB/s. Functional but sluggish for lab spin-up.

6.4 How this maps to k8s StorageClasses (future, post-Friday)

The scratch tier maps to two k8s concepts:

StorageClass: local-fast
  provisioner: local-path
  where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
  use: emptyDir replacement, build artifacts, container runtime cache
  redundancy: none — ephemeral, dies with the pod

StorageClass: nfs-hdd  (existing concept)
  provisioner: nfs-subdir-external
  where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
  use: persistent bulk data, VM OS drives, anything that must survive restarts
  redundancy: cross-server spreading + PBS backup

A wnode with local scratch gets a second disk (or a partition on local-lvm)
mounted as /scratch. ETL jobs write intermediate data to /scratch
(local-fast) and final output to /data (nfs-hdd). Pods that need ephemeral
fast storage request local-fast PVCs; pods that need persistence request
nfs-hdd.

This is forward-looking. The concrete implementation (formatting local-lvm,
adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
happens after the Friday maintenance window when the SSD/NVMe tiers are
physically installed and the storage architecture is finalized.

6

Naming Policy

Canonical naming policy for the PFVCluster fleet.
Established: 2026-07-31

Rule

The Proxmox VM name is the canonical identifier. Everything else
derives from it:

  1. VM name (set via qm set <vmid> --name <name>) is the source of truth
  2. DNS records (Technitium, forward + reverse) must match the VM name exactly
  3. Tailscale node name must match the VM name
  4. Monitoring/alerting (Beszel, Uptime Kuma, LibreNMS) uses the VM name
  5. Inventory/k8s references use the VM name

Any mismatch between VM name and DNS/Tailscale name is a critical bug
that must be fixed. The Technitium DNS API key is stored at
~/.secrets/api-keys.env for programmatic DNS management.

Tools

  • Technitium DNS API key: ~/.secrets/api-keys.env
  • Uptime Kuma API key: ~/.secrets/api-keys.env
  • DNS audit: perf/scripts/probe-vm-dns.sh
7

Project Report

Date: 2026-07-27 (re-audited)
Prepared by: Performance Optimization Engagement
Status: Comprehensive fleet assessment with VM placement and redundancy analysis

Data freshness: All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
deploy-check.sh. VM placements reflect live state after user’s PDM
migrations. This is ground truth.

Table of Contents

  1. Executive Summary
  2. Host Fleet
  3. Storage Architecture
  4. VM Fleet Inventory
  5. Kubernetes Node Distribution
  6. Storage Redundancy Analysis
  7. Local SSD/NVMe Opportunity
  8. Role Alignment Audit
  9. Network Findings
  10. Recommendations
  11. Hardware End-of-Support Exposure
  12. Open Items

1. Executive Summary

The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
are blocked on physical hardware work scheduled for Friday.

Progress since initial audit: The user has been actively rebalancing k8s
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
needed for etcd quorum survival.

The VM-layer assessment reveals:

# Finding Severity Status
1 4 of 5 cnodes still store disks on tsys4 NFS. cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. CRITICAL Improving
2 Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only. HIGH TODO today
3 No k8s node uses SSD or NVMe yet. tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. HIGH Deferred to k8s session
4 D3 SSD (tsys4, USB) is 99% empty (445 GB free). Moving to tsys5 SAS Friday. MEDIUM Friday

2. Host Fleet

2.1 Inventory

Host Model CPU (year) Cores RAM Local Disk Role (intended) Tuning
pfv-tsys1 OptiPlex 9020 i7-4770 Haswell (2013) 4c/8t 32 GB DDR3 HDD (LVM-thin) Infrastructure Done
pfv-tsys2 Precision 5520 i7-7820HQ Kaby Lake (2017) 4c/8t 32 GB (max) NVMe 512GB + SATA SSD 1TB Kubernetes Incoming (Win10)
pfv-tsys3 Precision 7510 Xeon E3-1535M v5 Skylake (2015) 4c/8t 32 GB DDR4 HDD (LVM-thin) Kubernetes Done
pfv-tsys4 Precision T1700 Xeon E3-1246 v3 Haswell (2013) 4c/8t 16 GB DDR3 6 disks (HDD+SSD+SMR) Storage (NFS+PBS) Blocked (NIC+RAM)
pfv-tsys5 Precision T7500 Xeon E5620 Westmere (2010) 4c/8t 96 GB DDR3 6 disks (HDD+SSD) Storage (NFS+VMs) Blocked (cable)
pfv-tsys6 PowerEdge R610 2x Xeon E5530 Nehalem (2009) 8c/16t 128 GB DDR3 HDD (LVM-thin) Kubernetes Done
pfv-tsys7 PowerEdge R620 2x Xeon E5-2630 v2 Ivy Bridge (2013) 12c/24t 192 GB DDR3 HDD (LVM-thin) Kubernetes Done
pfv-tsys9 OptiPlex 7080 i5-10500 Comet Lake (2020) 6c/12t 24 GB DDR4 250 GB SSD (PNY CS900) Infrastructure Done

2.2 Role taxonomy (per user directive)

Role Hosts Workload
Infrastructure + k8s control tsys1, tsys9 Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes
Kubernetes workers tsys2, tsys3, tsys6, tsys7 pfv-k8s wnodes (heavy workers) – these hosts have the RAM (32-192 GB) for ETL/HPC
Storage tsys4, tsys5 NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs

Design rationale: cnodes (control plane) are lightweight (4 cores, 4 GB
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
run one per hypervisor host across the fleet to maximize total cluster capacity.

pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
running Cloudron – not in this cluster.

3. Storage Architecture

3.1 NFS exports from tsys4 (primary storage server)

Export Disk model Type Bus Total Used Free Use%
D2 WDC WD30EFRX Red HDD (7200rpm) SATA 2.7 TB 187 GB 2.4 TB 8%
D3 SK hynix SC300 SSD USB — — — moving to tsys5 Friday
D5 Hitachi HDS72302 HDD (7200rpm) SATA 1.8 TB 236 GB 1.5 TB 14%

Non-exported disks on tsys4:

  • sda (Hitachi 1.8T) at /mnt/albert – not NFS shared, 1.7 TB free
  • sdd (WDC 1T) – idle, unmounted, removable (free up for other use)
  • sdf (WDC 4.5T SMR) at /mnt/backup – PBS backup target, 4.3 TB free

D3 migration (Friday): The SK hynix SC300 SSD is currently USB-attached on
tsys4 (via a “ThinkPad SSD” USB adapter). It is moving to a tsys5 SAS port,
eliminating the USB bottleneck. tsys4’s 4 SATA ports are all occupied (sda/sdb
/sdc/sdd), so tsys5 is the better target. See section 3.2.

3.2 NFS exports from tsys5 (secondary storage – becoming the fast-tier hub)

Export Disk model Type Bus Total Used Free Use%
S1 Seagate ST1000VN HDD SAS 916 GB 60 GB 810 GB 7%
S2 Seagate ST1000VN HDD SAS 916 GB 6.9 GB 863 GB 1%
S3 Seagate ST1000VN HDD SAS 916 GB 7.0 GB 863 GB 1%
S4 Toshiba DT01ACA050 HDD SAS 458 GB 2 MB 435 GB 0%
T5-SSD Samsung 860 PRO SSD SAS 234 GB 122 GB 101 GB 55%
D3 (Friday) SK hynix SC300 SSD SAS 469 GB 2 MB 445 GB 0%

tsys5 storage controllers (plenty of free ports):

  • LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
    5 free
  • Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), 2 free
  • Intel ICH10 SATA #2 (2-port): status unknown, likely free
  • 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)

Key finding: S2 and S3 now have k8s node disks. S2 holds cnode5 +
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9’s old disk
(unused, 7 GB). S4 still 99% empty (435 GB free).

T5-SSD grew to 55% used (122 GB) – ultix-streaming is the primary consumer.

Friday additions:

  1. D3 (SK hynix SSD) moves from tsys4 USB to tsys5 SAS port. Eliminates
    USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
  2. PCI NVMe drive (local-only, not NFS-exported). Used for wnode-tsys5
    boot disk and HPC scratch. The fastest tier in the fleet.

After Friday, tsys5 consolidates all fast storage: NVMe (local) + 2 SSD
NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
natural home for latency-sensitive workloads and the k8s StorageClass design
center.

3.3 Local storage tiers (per host, with utilization)

Host Storage ID Disk type Bus Total Used Free Used by VMs?
tsys1 local-lvm HDD SATA ~90 GB low ~90 GB No (all VMs on NFS)
tsys3 local-lvm NVMe (Samsung PM961) NVMe 349 GB 0 GB 349 GB No (all VMs on NFS)
tsys4 local-lvm HDD SATA ~94 GB PBS VM ~62 GB Yes (PBS VM 400)
tsys5 local-lvm HDD (Hitachi 1.8T) SATA 1.7 TB 40 MB 1.7 TB No
tsys5 local-nonprod HDD (Seagate 1T, =S1) SATA 916 GB 53 GB 856 GB Yes (sectestbed suite)
tsys6 local-lvm HDD (WD My Passport) USB 2.0 1.7 TB 0 GB 1.7 TB No – DO NOT USE for VM storage
tsys7 local-lvm HDD (WD portable) USB 2.0 1.7 TB 0 GB 1.7 TB No – DO NOT USE for VM storage
tsys9 local-lvm SSD (PNY CS900) SATA 136 GB 0 GB 136 GB No (all VMs on NFS)
tsys2 NVMe (Samsung 960 PRO) NVMe NVMe 512 GB (Win10) 512 GB Incoming – fastest boot tier after tsys5 NVMe
tsys2 SATA SSD (Samsung 850 EVO) SSD SATA 1 TB (Win10) 1 TB Incoming – bulk data on SSD, not rust

CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.

The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
single USB 2.0-attached WD My Passport portable HDD (tsys6: “My Passport
260D”; tsys7: “Drive 2657”). Both servers’ only USB controllers are EHCI
(USB 2.0, ~480 Mbps). There is no USB 3.0/xHCI on either host.

USB 2.0 practical throughput is ~30-35 MB/s. This is 3-4x SLOWER than
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
hosts would decrease performance. local-lvm on tsys6/7 must NOT be used
for VM storage.

Additionally, both servers have completely empty internal drive bays:

  • tsys6: SAS controller present but DISABLED in BIOS. No internal
    drives.
  • tsys7: 6-port SATA AHCI controller present, 5 ports EMPTY (only
    DVD-ROM on port 5). No internal drives.

This is a reliability risk beyond performance: the entire host OS boots
from a consumer-grade portable USB drive not designed for 24/7 server use.

tsys3 correction: Previously documented as HDD. Actually boots from a
Samsung PM961 NVMe 512GB SSD – the fastest existing local storage in the
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.

Critical observation: every k8s host has 0% used local-lvm. tsys3
(Samsung PM961 NVMe, 349 GB), tsys6 (WD My Passport USB 2.0 HDD,
1.7 TB), tsys7 (WD portable USB 2.0 HDD, 1.7 TB) all have unused local
storage.

However, only tsys3’s local-lvm is suitable for VM storage. tsys6 and
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) – slower than
NFS-over-HDD and unsuitable for wnode boot disks.

3.4 Disk speed tiers summary

Tier Where Speed class Best for
NVMe tsys3 (Samsung PM961), tsys5 (Friday addition), tsys2 (Samsung 960 PRO 512GB) 2000-3500 MB/s HPC scratch, ETL staging, container runtime, wnode boot, etcd
Local SSD tsys9 (PNY CS900, 136 GB), tsys2 (Samsung 850 EVO 1TB) 500 MB/s wnode boot disk, etcd
NFS-SSD tsys5 D3 (SK hynix, SAS post-Friday), tsys5 T5-SSD (Samsung) 200-400 MB/s over NFS k8s worker scratch only (etcd, container cache, ephemeral volumes)
NFS-HDD tsys4 D2/D5, tsys5 S1-S4 80-120 MB/s over NFS Bulk data, large disks, non-critical VMs, wnode boot on tsys6/7
Local SATA HDD tsys1 local-lvm 100-150 MB/s Host OS only
USB 2.0 HDD tsys6/7 local-lvm (WD My Passport) ~30-35 MB/s NOTHING – slower than NFS, do not use for VMs

3.5 Storage tier characterization per host

Host Storage profile Detail
tsys2 SSD/NVMe only 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk.
tsys3 NVMe only Samsung PM961 NVMe 512GB. No spinning disk.
tsys5 Hybrid (fast-tier hub) NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD
tsys4 Bulk/spinning disk only D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD.
tsys9 Local SSD + NFS PNY CS900 SSD 136GB local + NFS client
tsys1 Local HDD + NFS Small local-lvm + NFS client
tsys6/7 NFS only local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS.

3.6 All exports are single-disk with no redundancy

Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
on that export. This applies to both storage servers and to the PBS
backup target (a single 4.5T SMR drive).

4. VM Fleet Inventory

4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)

tsys1 (Infrastructure) – 11 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
100 pfv-bms (HomeAssistant) 2 4096 32 GB D2 (tsys4 HDD) NFS
101 tsys-ca 2 2048 32 GB D2 (tsys4 HDD) NFS
103 pfv-netinfra-01 2 2048 32 GB D5 (tsys4 HDD) NFS
104 tsys-librenms 2 2048 50 GB D2 (tsys4 HDD) NFS
105 tsys-proxmox-datacenter 2 2048 32 GB D2 (tsys4 HDD) NFS
106 pfv-k8s-cnode3 2 4096 32 GB D2 (tsys4 HDD) NFS
107 pfv-k8s-cnode1 2 4096 32 GB D5 (tsys4 HDD) NFS
108 tsys-ucs-01 2x2 8000 32 GB D2 (tsys4 HDD) NFS
109 tailscale-router 2 2048 25 GB D2 (tsys4 HDD) NFS
114 kali-tsys 2 2048 32 GB D2 (tsys4 HDD) NFS
117 tsys-secure-workbench 2 4000 32 GB D2 (tsys4 HDD) NFS

tsys3 (Kubernetes) – 1 running VM

VMID Name Cores RAM (MB) Disk Storage Tier
313 pfv-k8s-wnode-tsys3 4x2 20000 32 GB D5 (tsys4 HDD) NFS

tsys4 (Storage) – 1 running VM

VMID Name Cores RAM (MB) Disk Storage Tier
400 pfv-proxmox-backup-server 2 2048 32 GB local-lvm LOCAL

tsys5 (Storage) – 15 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
509 pfv-k8s-wnode-tsys5 2x4 32000 32 GB D2 (tsys4 HDD) NFS
5101 sectestbed-siem 2x2 10000 132 GB local-nonprod LOCAL
5105 sectestbed-awx 2x2 4096 288 GB local-nonprod LOCAL
5106 sectestbed-k8s-cnode 2x2 4096 32 GB local-nonprod LOCAL
5107 sectestbed-k8s-wnode 2x2 4096 32 GB local-nonprod LOCAL
5108 sectestbed-librenms 2x2 4096 32 GB local-nonprod LOCAL
5109 sectestbed-netinfra 2x2 4096 32 GB local-nonprod LOCAL
5111 ultix-streaming 2x2 9000 288 GB T5-SSD (tsys5 SSD) NFS-SSD
5112 ultix-offstage 2x2 6000 288 GB local-lvm LOCAL
6000 sectestbed-sandbox 2x2 4096 32 GB local-nonprod LOCAL
51010 sectestbed-tctc 2x2 4096 32 GB local-nonprod LOCAL
51011 sectestbed-cloudron 2x2 4096 32 GB local-nonprod LOCAL
51012 sectestbed-hfnoc 2x2 4096 32 GB local-nonprod LOCAL
51013 sectestbed-rancherplatform 2x2 4096 32 GB local-nonprod LOCAL
53100 tsys-preprod-awx 2x2 9000 160 GB local-nonprod LOCAL
53101 tsys-preprod-siem 2x2 12000 32 GB local-nonprod LOCAL
53102 tsys-preprod-rancherplatform 2x2 8000 32 GB local-nonprod LOCAL

tsys6 (Kubernetes) – 3 running VMs (1 wnode stopped)

VMID Name Cores RAM (MB) Disk Storage Tier
100 pfv-k8s-wnode-tsys6 2x2 32000 32 GB D5 (tsys4 HDD) NFS – STOPPED
600 tsys-awx 2x2 12000 32 GB D2 (tsys4 HDD) NFS – STOPPED
601 pfv-k8s-cnode4 4 4096 32 GB D2 (tsys4 HDD) NFS
602 pfv-k8s-cnode5 4 4096 32 GB D5 (tsys4 HDD) NFS
603 pfv-k8s-cnode2 4 4096 32 GB D2 (tsys4 HDD) NFS

tsys7 (Kubernetes) – 4 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
701 pfv-k8s-wnode-tsys7 4 32000 32 GB D5 (tsys4 HDD) NFS
702 hfnoc-uisp 2x2 8000 100 GB D2 (tsys4 HDD) NFS
703 rr-middleware 2 2048 32 GB D2 (tsys4 HDD) NFS
704 TCTC 4 6000 32 GB D2 (tsys4 HDD) NFS

tsys9 (Infrastructure) – 5 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
901 tsys-siem 2 8000 132 GB D2 (tsys4 HDD) NFS
902 tsys-ucs-02 2x2 8000 50 GB D5 (tsys4 HDD) NFS
903 kali-rd 2 2048 32 GB D5 (tsys4 HDD) NFS
904 pfv-netinfra-02 2 4000 32 GB D2 (tsys4 HDD) NFS
905 pfv-k8s-wnode-tsys9 4 4096 32 GB S3 (tsys5 HDD) NFS

4.2 Storage concentration summary

Storage target # of running VMs % of fleet
D2 (tsys4 WDC Red 3TB HDD) 18 45%
D5 (tsys4 Hitachi 2TB HDD) 9 23%
local-nonprod (tsys5 local HDD) 10 25%
S3 (tsys5 Seagate 1TB HDD) 1 3%
T5-SSD (tsys5 Samsung SSD) 1 3%
local-lvm (tsys4 local) 1 3%

27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.
If tsys4 goes offline, two-thirds of the fleet loses its storage.

5. Kubernetes Node Distribution (re-audited 21:50 CDT)

5.1 pfv-k8s cnode (control plane) placement – CURRENT

VMID Name Hypervisor Storage NFS Server Changed?
906 cnode1 tsys9 D5 tsys4 MOVED from tsys1
705 cnode2 tsys7 D2 tsys4 MOVED from tsys6
106 cnode3 tsys1 D2 tsys4 no change
601 cnode4 tsys6 D2 tsys4 no change
706 cnode5 tsys7 S2 tsys5 MOVED from tsys6, storage moved D5→S2

Storage distribution:

Storage server cnodes Quorum impact if it fails
tsys4 (D2+D5) 4 (cnode1,2,3,4) Only cnode5 survives = QUORUM LOST
tsys5 (S2) 1 (cnode5) 4 survive = quorum OK

Progress: cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
still means a tsys4 failure loses quorum. Need 2 more cnodes on tsys5.

Host distribution: cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
tsys9) – good host diversity.

5.2 pfv-k8s wnode (worker) placement – CURRENT

VMID Name Hypervisor Storage NFS Server RAM Status Changed?
102 wnode-tsys1 tsys1 S2 tsys5 4 GB STOPPED NEW
313 wnode-tsys3 tsys3 D5 tsys4 28 GB running RAM bumped 20→28
509 wnode-tsys5 tsys5 D2 tsys4 32 GB running no change
100 wnode-tsys6 tsys6 D5 tsys4 32 GB running NOW RUNNING
701 wnode-tsys7 tsys7 D5 tsys4 32 GB running no change
905 wnode-tsys9 tsys9 S2 tsys5 4 GB running Storage moved S3→S2

One wnode per host achieved (tsys1,3,5,6,7,9). wnode-tsys1 is created but
stopped. wnode-tsys6 restarted.

Storage distribution:

Storage server wnodes Notes
tsys4 (D2+D5) 4 (tsys3,5,6,7) Still concentrated
tsys5 (S2) 2 (tsys1,tsys9) Improving

5.3 Summary: k8s node storage distribution

Storage server cnodes wnodes Total k8s nodes
tsys4 NFS 4 (80%) 4 (67%) 8 (73%)
tsys5 NFS 1 (20%) 2 (33%) 3 (27%)
Local SSD/NVMe 0 0 0 (0%)

Was 90%/10%. Now 73%/27%. Improving but still tsys4-heavy. Target: 3
cnodes on each storage server (60/40 or better) so either server failing
leaves quorum intact.

5.4 Remaining cnode migration needed for etcd quorum survival

To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
be on tsys5:

Action Effect
Move cnode3 (D2→S3 on tsys5) 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive
Move cnode4 (D2→S2 on tsys5) Same result, different disk
Leave cnode1 and one other on tsys4 tsys5 fail = 4 survive (OK)

Simplest path: migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.

5.5 Future k8s architecture (next session – see K8S.md)

The k8s layer will be tackled soon. Key requirements from user:

  • Platform: vcluster + Rancher for multi-tenant management
  • Auth: OIDC to Keycloak (running on Cloudron in Reston, VA production)
  • Workload isolation (vcluster per tenant):
    • RackRental workloads (containerlab network labs)
    • Suborbital ITAR (compliance-restricted)
    • Suborbital non-ITAR
    • Starting Line Productions customer workloads
  • Solar-aware scale-out: PowerEdge 19xx and 2950 systems (older hardware)
    will be brought online during peak solar production for burst capacity.
    These older cores/ram supplement the main fleet when power is abundant.
  • WNode sizing: every Proxmox node will have a wnode. Some nodes will host
    both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
    leftover host capacity) to large (28-32 GB, consuming most of a host).
  • Friday final audit: tsys2 will be loaded with Proxmox on Friday, and a
    full final audit will be performed at that time (post-NVMe install on tsys5,
    post-D3 SSD relocation, post-tsys4 NIC+RAM).

5.4 sectestbed k8s nodes (separate from pfv-k8s)

tsys5 also hosts a separate sectestbed kubernetes stack using local storage:

VMID Name Storage
5106 sectestbed-k8s-cnode local-nonprod (local HDD)
5107 sectestbed-k8s-wnode local-nonprod (local HDD)

These are on local storage (good – no NFS dependency) but on a single host’s
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.

6. Storage Redundancy Analysis

6.1 -01/-02 infrastructure pair audit

Two -01/-02 pairs exist in the fleet:

Pair 1: pfv-netinfra (network infrastructure)

Role VMID Host Storage NFS Server
-01 103 tsys1 D5 tsys4
-02 904 tsys9 D2 tsys4

Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
disks. The -02 half should be on an S2/S3/S4 export from tsys5.

Pair 2: tsys-ucs (Univention Corporate Server)

Role VMID Host Storage NFS Server
-01 108 tsys1 D2 tsys4
-02 902 tsys9 D5 tsys4

Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.

6.2 Redundancy principle for paired VMs

For any -01/-02 pair to survive a single storage server failure:

-01 VM disk -> tsys4 NFS export (D2/D3/D5)
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)

This ensures that losing either tsys4 or tsys5 takes down only one half of
the pair. Currently, both pairs fail this test because both halves are on
tsys4.

6.3 NFS server failure blast radius

If tsys4 goes offline (USB NIC failure, disk failure, reboot):

Impact Count
k8s cnodes that lose storage 5 of 5 (etcd quorum lost)
k8s wnodes that lose storage 4 of 5
Infrastructure VMs that lose storage 11 of 12 on tsys1 (all on D2/D5)
Total VMs that lose storage 27 of 40 (68%)

If tsys5 goes offline:

Impact Count
k8s cnodes that lose storage 0 of 5
k8s wnodes that lose storage 1 of 5
Total VMs that lose storage 1 of 40 (3%)

tsys4 is a massive blast-radius liability. tsys5 is barely used.
Rebalancing VM storage across both servers dramatically reduces risk.

7. Local SSD/NVMe Opportunity

7.1 Available fast tiers (currently unused by k8s)

Host Device Type Size Available for VMs? Currently used by k8s?
tsys4 D3 (SK hynix SC300, USB) SSD 512 GB Yes (via NFS) No
tsys5 T5-SSD (Samsung 860 PRO) SSD 256 GB Yes (via NFS) No (used by ultix-streaming)
tsys5 New NVMe (Friday) NVMe TBD Yes (local or NFS) No
tsys9 local-lvm (PNY CS900) SSD 137 GB free Yes (local) No

7.2 Why local storage matters for k8s nodes

Kubernetes nodes are latency-sensitive in two specific areas:

  1. etcd (control plane): etcd writes are synchronous and latency-critical.
    On NFS over HDD, every etcd write traverses: VM → virtio-scsi → NFS
    client → TCP → USB dongle (on tsys4) → ext4 → spinning disk. Typical
    latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
    This directly affects k8s API responsiveness and pod scheduling speed.

  2. Container image pulls: Worker nodes pull container images frequently.
    On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
    eliminates the network hop and reduces seek time. This matters most for
    RackRental/containerlab workloads that spin up containers rapidly.

7.3 Current waste: tsys9 local SSD

tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
Zero VMs use it. All 5 VMs on tsys9 boot from NFS. The local SSD sits
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD –
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).

7.4 Upcoming opportunity: tsys5 NVMe (Friday)

The PCI NVMe being added to tsys5 will be the fastest storage tier in the
fleet. Two placement options:

Option A: NFS-export the NVMe (shared). All hosts can use it. Good for
VMs that might need migration. Adds the NFS/network overhead back.

Option B: Local-only on tsys5. VMs on tsys5 get full NVMe speed with no
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
accessed from other hosts.

Recommendation: Option B (local-only). k8s worker nodes do not need
shared storage – pods are ephemeral and reschedule on failure. The NVMe
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
and used for local VM images.

8. Role Alignment Audit

Per the user’s intended role taxonomy: tsys1/9 = infrastructure + k8s control
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
correct by design (keeps heavy hosts free for workers).

8.1 VMs that need to move

VMID Name Current host Issue Target
509 pfv-k8s-wnode-tsys5 tsys5 (storage) Worker on storage host tsys7 or tsys2 (when online)
905 pfv-k8s-wnode-tsys9 tsys9 (infra) Can stay if small; user decides tsys9 OK if small wnode

8.2 Host capacity for k8s nodes

Host Role Current k8s nodes k8s RAM used RAM total Headroom
tsys1 Infra+k8s ctrl 2 cnodes 8 GB 32 GB ~12 GB (after 11 infra VMs)
tsys3 K8s worker 1 wnode 20 GB 32 GB ~12 GB
tsys6 K8s worker 3 cnodes + 1 wnode (stopped) 12 GB 128 GB ~116 GB
tsys7 K8s worker 1 wnode 32 GB 192 GB ~160 GB
tsys9 Infra+k8s ctrl 1 wnode 4 GB 24 GB ~12 GB (after 4 infra VMs)
tsys2 K8s worker 0 (incoming) 0 32 GB ~32 GB

tsys6 and tsys7 are dramatically underutilized – 116 GB and 160 GB of
free RAM respectively. They should be the primary targets for heavy worker
nodes and ETL/HPC workloads.

8.3 tsys6 wnode-tsys6 is stopped

VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
cluster capacity.

9. Network Findings

9.1 tsys9 storage NIC is a USB dongle (new finding)

Validating tsys9 revealed that its storage network interface
(enx9c69d36a5b6c) is USB-attached (parentbus usb). This is the same
anti-pattern as tsys4. The onboard Intel NIC (enp0s31f6) is used for
management; storage uses the USB adapter.

Impact: Same as tsys4 – achieves line rate but is susceptible to cable
wobble, ESD, and USB controller resets. For an infrastructure host with 5
VMs, this is a reliability risk.

Mitigation: tsys9 is an OptiPlex 7080 SFF – it has PCIe slots. A
PCIe NIC would eliminate this risk (same recommendation as tsys4).

9.2 tsys4 and tsys5 still blocked (Friday hardware work)

Host Blocker Staged fix
tsys4 USB cdc_ncm storage NIC PCIe NIC install + RAM upgrade (16 to 64 GB)
tsys5 bond0 broken (1 of 2 slaves) Plug 2nd ethernet cable + apply layer3+4 hash

9.3 LACP resolved on tsys6/tsys7

tsys6 to tsys7 storage path now measures 1.83 Gbps (was 943 Mbps).
The switch LACP hash change took effect after renegotiation. The 56-106K
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
LACP overhead. See RESULTS.md (not yet created) for the full analysis.

9.4 NFS nconnect=4 + noatime confirmed active

All hosts (including tsys9) show nconnect=4,noatime in their NFS mount
options. Each host maintains 4 TCP connections per NFS mount to each storage
server. This was the Tier 0 tuning item from the performance optimization
engagement and is confirmed working cluster-wide.

10. Recommendations

**No changes have been made. These are asses

8

(continued) (part 2/2)

sment-only recommendations.**
See K8S.md for the detailed kubernetes architecture deep-dive.

10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)

These are the most critical production VMs in the fleet. They must survive
any single-point failure (host, storage server, or disk).

Design principle for -01/-02 HA pairs:

  • Different hypervisors (already satisfied: tsys1 vs tsys9)
  • Different storage servers (currently FAILED: all on tsys4)
  • Prefer SSD for latency-sensitive services

Recommended placement:

VM Host Storage Tier Free space Rationale
netinfra-01 (103) tsys1 D5 (tsys4 HDD) NFS-HDD 1.5 TB DNS/DHCP/NTP = minimal I/O. Stays put.
netinfra-02 (904) tsys9 S3 (tsys5 HDD) NFS-HDD 870 GB Move from D2. Cross-server redundancy. Minimal I/O.
ucs-01 (108) tsys1 D2 (tsys4 HDD) NFS-HDD 2.4 TB Stays put. LDAP/AD does not need SSD.
ucs-02 (902) tsys9 S2 (tsys5 HDD) NFS-HDD 870 GB Move from D5. Cross-server redundancy. No SSD needed.

Failure survival matrix (all single-point failures):

Failure netinfra-01 netinfra-02 ucs-01 ucs-02 Result
tsys4 dies dies (D5) alive (S3) dies (D2) alive (S2) netinfra-02 + ucs-02 alive
tsys5 dies alive (D5) dies (S3) alive (D2) dies (S2) netinfra-01 + ucs-01 alive
tsys1 dies dies alive dies alive -02 pair survives
tsys9 dies alive dies alive dies -01 pair survives
Any single disk all 4 on different disks/servers all 4 alive

Why this works: Every row has at least one netinfra and one UCS alive.
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.

D3 SSD repurposed: With UCS staying on HDD, the D3 SSD (moving to tsys5
Friday) is freed for latency-sensitive workloads that actually benefit from
SSD – sectestbed k8s nodes, CI/CD artifact cache, or a container image
registry mirror. Not infrastructure VMs.

10.2 CRITICAL: Cnode (control plane) storage split

Problem: All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
quorum (3 of 5 cnodes share D2).

Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.

cnode Current host Target host Current storage Target storage Rationale
cnode1 (107) tsys1 tsys1 (stays) D5 (tsys4) D5 (tsys4) – no change Already correct
cnode3 (106) tsys1 tsys1 (stays) D2 (tsys4) S2 (tsys5) – MOVE disk Split storage to tsys5
cnode2 (603) tsys6 tsys9 D2 (tsys4) D2 (tsys4) – no disk change Free tsys6 for heavy workers
cnode4 (601) tsys6 tsys9 D2 (tsys4) D5 (tsys4) – spread disk Free tsys6; spread off D2
cnode5 (602) tsys6 tsys3 D5 (tsys4) S3 (tsys5) – MOVE disk Free tsys6; split storage to tsys5

Result after changes:

Host cnodes Storage server
tsys1 cnode1 (D5), cnode3 (S2) tsys4 + tsys5
tsys9 cnode2 (D2), cnode4 (D5) tsys4
tsys3 cnode5 (S3) tsys5
  • 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
    etcd keeps quorum (3 of 5 survive).
  • D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
    holds more than 2 cnodes.
  • Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
    at least 3 cnodes alive.
  • tsys6 and tsys7 are fully freed for heavy worker nodes.

Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.

10.3 HIGH: Wnode distribution – one per host, tiered storage

Target: one wnode per hypervisor host, using local storage where possible
and NFS-HDD for bulk data.

wnode Host Boot disk (OS+containers) Data disk (bulk/ETL) Rationale
wnode-tsys1 tsys1 D5 (tsys4 NFS) – Small wnode on infra host. Minimal capacity.
wnode-tsys3 tsys3 local-lvm (349 GB NVMe) S3 (NFS) Move from NFS to NVMe (Samsung PM961). Fastest boot disk after tsys5/2.
wnode-tsys6 tsys6 D2 (tsys4 NFS) – Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) – slower than NFS.
wnode-tsys7 tsys7 D5 (tsys4 NFS) – Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) – slower than NFS.
wnode-tsys9 tsys9 local-lvm (136 GB SSD) S2 (NFS) Move from NFS to local SSD. Fast boot, NFS for bulk.
wnode-tsys2 tsys2 NVMe (Samsung 960 PRO 512GB) SATA SSD (Samsung 850 EVO 1TB) Best storage of any wnode. No NFS needed – 1.5 TB local SSD.
wnode-tsys5 tsys5 new NVMe (local) local-nonprod (HDD) Fastest wnode in fleet. HPC/ETL workloads land here.

Storage tiering strategy per wnode:

Each wnode gets two disk tiers mapped to k8s StorageClasses:

  1. Boot + container runtime (local-lvm or NVMe): OS, kubelet, container
    images, ephemeral storage. This is where local SSD/NVMe shines – container
    image pulls and layer extraction are seek-bound and benefit enormously from
    low-latency storage.

  2. Bulk data (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
    areas, large files that do not fit on local storage. Mounted as a second
    disk in the VM and exposed to k8s as a StorageClass.

This maps to two k8s StorageClasses:

  • local-storage: bound to the wnode’s boot/local disk (fast, ephemeral)
  • nfs-bulk: bound to NFS exports (slow, persistent, large capacity)

10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch

Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
space and ultix-streaming (developer workstation running “cluster of 1”
pre-production jobs). Spinning rust hosts all other infrastructure VMs
(UCS, netinfra, LibreNMS, SIEM, etc.).

SSD allocation after Friday:

  • D3 (tsys5 SAS, 445 GB free, 0% used) – dedicated to k8s scratch via
    the nfs-ssd StorageClass (etcd, container cache, ephemeral volumes).
  • T5-SSD (tsys5 SAS, 140 GB free) – ultix-streaming (VM 5111) stays here
    (developer workstation, runs single-node test jobs before k8s). Remaining
    140 GB available for k8s.

Deep-dive on exact k8s scratch allocation is deferred to the next session
(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.

10.5 MEDIUM: Restart wnode-tsys6

VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
the boot disk to local storage simultaneously.

10.6 MEDIUM: tsys5 NVMe placement (Friday)

Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
directory storage.

Use for:

  • wnode-tsys5 boot disk (primary beneficiary – HPC/ETL workloads)
  • sectestbed VMs that need fast scratch space
  • Not NFS-exported (avoid adding network overhead to the fastest tier)

10.7 LOW: Add PCIe NIC to tsys9

tsys9’s storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
slots. A $150 PCIe NIC eliminates the USB reliability risk.

10.8 LOW: Standardize PVE/kernel versions

tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.

11. Hardware End-of-Support Exposure

Host EOS date Years past Form factor
pfv-tsys6 (R610) 2013-05 13.2 1U server
pfv-tsys5 (T7500) 2014-12 11.7 Workstation
pfv-tsys4 (T1700) 2018-03 8.4 Workstation
pfv-tsys7 (R620) 2019-03 7.4 1U server
pfv-tsys1 (9020) 2019-07 7.0 SFF desktop
pfv-tsys3 (7510) 2020-07 6.0 Laptop
pfv-tsys2 (5520) TBD – Laptop
pfv-tsys9 (7080) 2024-02 2.4 SFF desktop (only supported)

6 of 8 hosts are past vendor end-of-support. Only tsys9 is still covered.
The two actual rack servers (R610, R620) are the most overdue for replacement.
This belongs in the next budget cycle.

12. Open Items

12.1 Immediate (do today via PDM)

  1. Migrate ucs-02 (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
    redundancy. UCS stays on HDD.
  2. Migrate netinfra-02 (VM 904) from D2 (tsys4) to S3 (tsys5).
  3. Migrate cnode3 (VM 106) from D2 (tsys4) to S3 (tsys5) – etcd quorum.
  4. Migrate cnode4 (VM 601) from D2 (tsys4) to S2 (tsys5) – etcd quorum.
  5. Start wnode-tsys1 (VM 102) if the cluster needs the capacity.

12.2 Friday maintenance window (user action)

  1. tsys4: Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
    reconfigure /etc/network/interfaces, reboot.
  2. tsys5: Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
    install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
    format NVMe as local storage, reboot.
  3. tsys2: Load Proxmox (replacing Windows 10).
  4. Final audit: Re-run deploy-check.sh across all hosts including tsys2.

12.3 Post-Friday validation

  1. Re-run iperf matrix: ./iperf-full-matrix.sh
  2. Validate tsys4 and tsys5: ./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5
  3. Run scripts/check.sh on tsys2 once Proxmox is loaded.
  4. Update PROJECT.md with post-hardware numbers.

12.4 Future: Kubernetes deep-dive (see K8S.md)

Next major workstream. Requirements captured:

  • vcluster + Rancher for multi-tenant k8s management
  • OIDC auth to Keycloak (on Cloudron, Reston VA production)
  • Workload isolation via separate vclusters:
    • RackRental (containerlab)
    • Suborbital ITAR
    • Suborbital non-ITAR
    • Starting Line Productions customer workloads
  • Solar-aware scale-out: PowerEdge 19xx + 2950 systems brought online
    during peak solar production for burst capacity
  • Every host gets a wnode (variable sizing: small 4 GB to large 32 GB)
  • SSD/NVMe reserved for k8s scratch (plus ultix-streaming exception)
  • Spinning rust for all infrastructure VMs

12.5 Data gaps

Gap How to close
tsys5 SDR/parallel-port workload dependency Confirm what uses the SDR
tsys2 post-Proxmox baseline Run check.sh after Friday install
tsys5 NVMe size and model Confirm after Friday installation
tsys3 thermal state (laptop in rack) Check sensors on next maintenance
PowerEdge 19xx/2950 inventory When solar scale-out is planned

12.6 Scripts and tools available

Script Purpose
scripts/check.sh Read-only data collector (run on hosts)
scripts/apply-tunings.sh Apply all Tier 0 tunings (dry-run/apply/rollback)
scripts/fix-bond-nfs.sh Fix NFS options + bond hash
validate-fixes.sh Read-only validation of all applied changes
iperf-full-matrix.sh Full iperf test suite (mgmt + storage)
deploy-check.sh Deploy check.sh to all hosts via SSH (now includes tsys9)