Storage Architecture

Canonical source: This wiki topic is the living reference. Updated in place; edit history preserves all prior states.

Covers: Full storage reference — tier definitions, physical disk inventory, NFS export/mount map, per-spindle VM placement, capacity summary.


Storage Architecture

Complete storage reference for the PFVCluster. Covers physical disks,
NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
VM placement. Data gathered 2026-07-30 via live host probes.

Companion docs: NETWORK-TOPOLOGY.md,
HOST-REFERENCE.md, VM-INVENTORY.md.
Findings/analysis: AUDIT-2026-07-30.md.


1. Storage tier definitions

Tier Definition Typical speed Reliability Use case
Tier 1 NVMe or SSD on internal SATA/SAS/PCIe 200-3000 MB/s High OS drives, high-IOPS workloads, build scratch
Tier 2 HDD on internal SATA/SAS 80-150 MB/s Moderate Production VM OS drives, general-purpose storage
Tier 3 HDD/SSD on USB adapter / “toaster” 30-60 MB/s Low (USB) Bulk, backup, scratch only. Never for production OS drives.

Policy (from CAPACITY-GOALS.md §6): Local disk
storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
and HA pairs. Local storage is for ephemeral data where loss on node failure
is acceptable.


2. Physical disk inventory

tsys4 – NFS storage server + PBS (7 disks)

Device Model Size RPM Interface Mount NFS ID Tier SMART Used
sda Hitachi HDS723020BLA642 2.0 TB 7200 SATA /mnt/albert (not exported) Tier 2 PASSED, 0 reallocated 0%
sdb HITACHI HUA723020ALA640 2.0 TB 7200 SATA /mnt/tsys4/D5 D5 Tier 2 PASSED, 0 reallocated 13% (220 GB)
sdc WDC WD30EFRX (Red) 3.0 TB 5400 SATA /mnt/tsys4/D2 D2 Tier 2 PASSED, 0 reallocated 7% (170 GB)
sdd WDC WD10EZRX 1.0 TB SATA (pve boot/swap/lvm) Tier 2 PASSED, 1 reallocated (sdf has 1) OS disk
sde ST500LM021 (Seagate) 500 GB 7200 USB /mnt/tsys4/D3 D3 Tier 3 PASSED 0%
sdf WDC WD10EARS 1.0 TB USB /mnt/tsys4/D1 D1 Tier 3 PASSED, 1 reallocated sector 0%
sdg WDC WD50NDZW (easystore) 5.0 TB 5400 USB /mnt/backup (not exported) Tier 3 PASSED 0%

Changes from previous audit:

  • D3 changed: was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
  • /mnt/albert added: Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
  • /mnt/backup added: WD 5TB USB easystore. Bulk backup only.
  • D7 removed: Previously held DellOpenManageEnterprise. Now gone.

tsys5 – NFS storage server + sandboxes (7 disks)

Device Model Size RPM Interface Mount NFS/Local ID Tier SMART Used
sda ST1000VN002 (IronWolf) 1.0 TB 5900 SATA (not mounted) was S3 Tier 2 PASSED
sdb SK hynix SC300B 512 GB SSD SATA (not mounted) Tier 1 PASSED NEW, not configured
sdc HITACHI HUS724020ALA640 2.0 TB 7200 SAS (pve boot/swap/lvm) Tier 2 PASSED OS disk
sdd Samsung SSD 860 PRO 256 GB SSD SAS /mnt/pfv-tsys5/ssd ssd / SSD dir (local) Tier 1 PASSED 93% full!
sde ST1000VN002 (IronWolf) 1.0 TB 5900 SAS /mnt/pfv-tsys5/S1 S1 + local-nonprod Tier 2 PASSED 11% (96 GB)
sdf ST1000VN002 (IronWolf) 1.0 TB 5900 SAS /mnt/pfv-tsys5/S2 S2 Tier 2 PASSED 4% (32 GB)
nvme0n1 CT500P1SSD8 (Crucial) 500 GB NVMe PCIe (not mounted) Tier 1 PASSED, 0 integrity errors, 34K hrs NEW, not configured

Changes from previous audit:

  • NVMe added: Crucial CT500P1 500GB. True NVMe – highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
  • SK hynix SSD relocated: Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
  • S3 unmounted: sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
  • S4 gone: Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken dir: S4 entry.
  • S1/S2 drives changed: Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
  • SSD nearly full: Samsung 860 PRO at 93% (206/234 GB) – needs attention.
  • Boot disk changed: HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).

3. NFS export → Proxmox storage ID map

graph LR
    subgraph "tsys4 -- NFS Server"
        D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
        D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
        D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
        D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
    end

    subgraph "tsys5 -- NFS Server"
        S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
        S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
        S3stale["S3: (stale export)<br>disk not mounted"]
    end

    subgraph "All hosts -- NFS Clients"
        MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
    end

    D2 -->|NFS v4.2| MNT
    D5 -->|NFS v4.2| MNT
    D1 -->|NFS v4.2| MNT
    D3 -->|NFS v4.2| MNT
    S1 -->|NFS v4.2| MNT
    S2 -->|NFS v4.2| MNT
    S3stale -.->|broken| MNT

NFS export details

Server Export path NFS ID Protocol Options (client-side)
tsys4 /mnt/tsys4/D1 D1 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D2 D2 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D3 D3 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D5 D5 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S1 S1 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S2 S2 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S3 S3 NFSv4.2 stale (disk not mounted on server)
tsys5 /mnt/pfv-tsys5/ssd T5-SSD NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S4 S4 broken (disk gone, storage.cfg entry stale)

Note: tsys5 is the only host where NFS client mounts (D2/D3/D5 from
tsys4) are missing nconnect=4. All other client hosts have nconnect on
all mounts.


4. Per-spindle VM placement (running VMs only)

Shows every running VM’s primary disk and the physical spindle it lives on.
This is the IOPS distribution map – the key to storage balancing.

graph TB
    subgraph "tsys4 spindles"
        D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
        D5["D5 -- Hitachi 2TB<br>4 VMs"]
        D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
        D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
        ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
    end

    subgraph "tsys5 spindles"
        S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
        S2["S2 -- IronWolf 1TB<br>5 VMs"]
        SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
        NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
        HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
    end

    D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
    D5 --- D5vms["106 313 600(d0) 903"]
    S1 --- S1vms["702 hfnoc-uisp"]
    S2 --- S2vms["103 703 705 902 905"]
    SSD --- SSDvms["5111 ultix-streaming"]

Detailed per-spindle load

Spindle Server NFS ID Tier Running VMs VMIDs Notes
D2 (WDC Red 3TB) tsys4 D2 Tier 2 13 100,101,104,105,108,114,117,509,600,702,704,901,904 Worst IOPS hotspot
D5 (Hitachi 2TB) tsys4 D5 Tier 2 4 106,313,600(disk0),903 Moderate
S1 (IronWolf 1TB) tsys5 S1 + local-nonprod Tier 2 1 (NFS) + 0 (local, 12 stopped) 702 Low; 12 stopped sandboxes on local-nonprod
S2 (IronWolf 1TB) tsys5 S2 Tier 2 5 103,703,705,902,905 Moderate
ssd (Samsung 860 PRO) tsys5 SSD dir Tier 1 1 5111 93% full
D1 (WD 1TB USB) tsys4 D1 Tier 3 0 Empty
D3 (Seagate 500GB USB) tsys4 D3 Tier 3 0 Empty
/mnt/albert (Hitachi 2TB) tsys4 (none) Tier 2 0 Not in Proxmox
NVMe (CT500P1 500GB) tsys5 (none) Tier 1 0 Not configured
SK hynix SSD (512GB) tsys5 (none) Tier 1 0 Not configured

Local-only VM disks (not on NFS)

Host Storage ID Disk VMs Notes
tsys4 local-lvm sdd (1TB internal) 400 (PBS) Proxmox boot/OS disk
tsys5 local-lvm sdc (2TB internal) 5102,5103,5104,5112 Mixed: stopped sandboxes + ultix-offstage
tsys5 local-nonprod (=S1) sde (IronWolf 1TB) 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 All STOPPED sandboxes. Same physical disk as S1 NFS.
tsys5 SSD dir sdd (Samsung 860 PRO) 5111 ultix-streaming. 93% full.

5. Storage capacity summary

Spindle Size Used Available % Used
D2 (WDC Red 3TB) 2.7 TB 170 GB 2.4 TB 7%
D5 (Hitachi 2TB) 1.8 TB 220 GB 1.5 TB 13%
D1 (WD 1TB USB) 916 GB 2 MB 870 GB 0%
D3 (Seagate 500GB USB) 458 GB 2 MB 435 GB 0%
/mnt/albert (Hitachi 2TB) 1.8 TB 2 MB 1.7 TB 0%
/mnt/backup (WD 5TB USB) 4.6 TB 2 MB 4.3 TB 0%
S1 (IronWolf 1TB) 916 GB 96 GB 774 GB 11%
S2 (IronWolf 1TB) 916 GB 32 GB 838 GB 4%
SSD (Samsung 860 PRO) 234 GB 206 GB 16 GB 93%
NVMe (CT500P1) 466 GB Not configured
SK hynix SSD 477 GB Not configured

Total Tier 2 available (NFS production): ~5.5 TB unused across D2/D5/S1/S2
Total Tier 3 available (USB): ~6 TB unused across D1/D3/backup
Total Tier 1 available (unconfigured): ~940 GB across NVMe + SK hynix SSD

Advisory: etcd VMs require writeback cache (2026-08-06)

All k8s control-plane VMs (cnode1/2/3) MUST use cache=writeback on their Proxmox disk config. This is mandatory, not optional.

Why

etcd (the k8s backing store) requires fsync latency under 10ms. The cnode VMs use qcow2 disk images on NFS-backed spinning storage (Tier 2/3 HDD). Without writeback cache:

  • Guest fsync latency: 8,900ms (890x over budget)
  • Result: etcd leader election fails on all nodes → k3s crash-loop → API flaps → Uptime Kuma alerts

With cache=writeback:

  • Guest fsync latency: 2-3ms (within budget)
  • Write cache confirmed write back inside guest via /sys/block/sda/queue/write_cache

Current configuration

VM VMID Host Storage Cache
pfv-k8s-cnode1 102 tsys1 S2 (NFS→tsys5) writeback (was: none)
pfv-k8s-cnode2 705 tsys7 S1 (NFS→tsys5) writeback (was: writethrough)
pfv-k8s-cnode3 603 tsys6 D1 (NFS→tsys4) writeback (was: writethrough)

Tradeoff

cache=writeback acknowledges writes in host RAM before flushing to physical disk. If a Proxmox host crashes (power loss), unflushed writes are lost. For this R&D cluster, etcd raft re-syncs from surviving nodes, making this acceptable. Do NOT change back to writethrough or none — the cluster will destabilize within minutes.

Better long-term fix

The fleet has unconfigured Tier 1 storage that would eliminate this concern entirely:

  • NVMe CT500P1 466GB (tsys5, not mounted/configured)
  • SK hynix SSD 477GB (tsys5, not mounted/configured)

Migrating cnode disk images to SSD-backed storage (even over NFS) would bring fsync latency to <1ms without relying on writeback cache. This is the recommended path for production stability.

Refs: Redmine [#391], [#367]; Audit log entry 2026-08-06 in [#298]