Canonical source: This wiki topic is the living reference. Updated in place; edit history preserves all prior states.
Covers: Full storage reference — tier definitions, physical disk inventory, NFS export/mount map, per-spindle VM placement, capacity summary.
Storage Architecture
Complete storage reference for the PFVCluster. Covers physical disks,
NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
VM placement. Data gathered 2026-07-30 via live host probes.Companion docs:
NETWORK-TOPOLOGY.md,
HOST-REFERENCE.md,VM-INVENTORY.md.
Findings/analysis:AUDIT-2026-07-30.md.
1. Storage tier definitions
| Tier | Definition | Typical speed | Reliability | Use case |
|---|---|---|---|---|
| Tier 1 | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch |
| Tier 2 | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage |
| Tier 3 | HDD/SSD on USB adapter / “toaster” | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. |
Policy (from CAPACITY-GOALS.md §6): Local disk
storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
and HA pairs. Local storage is for ephemeral data where loss on node failure
is acceptable.
2. Physical disk inventory
tsys4 – NFS storage server + PBS (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
|---|---|---|---|---|---|---|---|---|---|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | Tier 2 | PASSED, 0 reallocated | 0% |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | D5 | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | D2 | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
| sdd | WDC WD10EZRX | 1.0 TB | – | SATA | (pve boot/swap/lvm) | – | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sde | ST500LM021 (Seagate) | 500 GB | 7200 | USB | /mnt/tsys4/D3 | D3 | Tier 3 | PASSED | 0% |
| sdf | WDC WD10EARS | 1.0 TB | – | USB | /mnt/tsys4/D1 | D1 | Tier 3 | PASSED, 1 reallocated sector | 0% |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | USB | /mnt/backup | (not exported) | Tier 3 | PASSED | 0% |
Changes from previous audit:
- D3 changed: was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
- /mnt/albert added: Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
- /mnt/backup added: WD 5TB USB easystore. Bulk backup only.
- D7 removed: Previously held DellOpenManageEnterprise. Now gone.
tsys5 – NFS storage server + sandboxes (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
|---|---|---|---|---|---|---|---|---|---|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | (not mounted) | was S3 | Tier 2 | PASSED | – |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | (not mounted) | – | Tier 1 | PASSED | NEW, not configured |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | – | Tier 2 | PASSED | OS disk |
| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | ssd / SSD dir (local) |
Tier 1 | PASSED | 93% full! |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | S1 + local-nonprod |
Tier 2 | PASSED | 11% (96 GB) |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | S2 | Tier 2 | PASSED | 4% (32 GB) |
| nvme0n1 | CT500P1SSD8 (Crucial) | 500 GB | NVMe | PCIe | (not mounted) | – | Tier 1 | PASSED, 0 integrity errors, 34K hrs | NEW, not configured |
Changes from previous audit:
- NVMe added: Crucial CT500P1 500GB. True NVMe – highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- SK hynix SSD relocated: Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
- S3 unmounted: sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
- S4 gone: Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken
dir: S4entry. - S1/S2 drives changed: Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
- SSD nearly full: Samsung 860 PRO at 93% (206/234 GB) – needs attention.
- Boot disk changed: HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).
3. NFS export → Proxmox storage ID map
graph LR
subgraph "tsys4 -- NFS Server"
D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
end
subgraph "tsys5 -- NFS Server"
S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
S3stale["S3: (stale export)<br>disk not mounted"]
end
subgraph "All hosts -- NFS Clients"
MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
end
D2 -->|NFS v4.2| MNT
D5 -->|NFS v4.2| MNT
D1 -->|NFS v4.2| MNT
D3 -->|NFS v4.2| MNT
S1 -->|NFS v4.2| MNT
S2 -->|NFS v4.2| MNT
S3stale -.->|broken| MNT
NFS export details
| Server | Export path | NFS ID | Protocol | Options (client-side) |
|---|---|---|---|---|
| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | stale (disk not mounted on server) |
| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | – | broken (disk gone, storage.cfg entry stale) |
Note: tsys5 is the only host where NFS client mounts (D2/D3/D5 from
tsys4) are missing nconnect=4. All other client hosts have nconnect on
all mounts.
4. Per-spindle VM placement (running VMs only)
Shows every running VM’s primary disk and the physical spindle it lives on.
This is the IOPS distribution map – the key to storage balancing.
graph TB
subgraph "tsys4 spindles"
D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
D5["D5 -- Hitachi 2TB<br>4 VMs"]
D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
end
subgraph "tsys5 spindles"
S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
S2["S2 -- IronWolf 1TB<br>5 VMs"]
SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
end
D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
D5 --- D5vms["106 313 600(d0) 903"]
S1 --- S1vms["702 hfnoc-uisp"]
S2 --- S2vms["103 703 705 902 905"]
SSD --- SSDvms["5111 ultix-streaming"]
Detailed per-spindle load
| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes |
|---|---|---|---|---|---|---|
| D2 (WDC Red 3TB) | tsys4 | D2 | Tier 2 | 13 | 100,101,104,105,108,114,117,509,600,702,704,901,904 | Worst IOPS hotspot |
| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate |
| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | 93% full |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | – | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | – | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | – | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | – | Not configured |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | – | Not configured |
Local-only VM disks (not on NFS)
| Host | Storage ID | Disk | VMs | Notes |
|---|---|---|---|---|
| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk |
| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage |
| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. |
| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. |
5. Storage capacity summary
| Spindle | Size | Used | Available | % Used |
|---|---|---|---|---|
| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% |
| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% |
| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% |
| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% |
| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% |
| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% |
| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
| SSD (Samsung 860 PRO) | 234 GB | 206 GB | 16 GB | 93% |
| NVMe (CT500P1) | 466 GB | – | – | Not configured |
| SK hynix SSD | 477 GB | – | – | Not configured |
Total Tier 2 available (NFS production): ~5.5 TB unused across D2/D5/S1/S2
Total Tier 3 available (USB): ~6 TB unused across D1/D3/backup
Total Tier 1 available (unconfigured): ~940 GB across NVMe + SK hynix SSD