Kubernetes Platform

ChiefOperationsOfficer VP TechOps
,
1

Canonical source: This wiki topic is the living reference. Updated in place; edit history preserves all prior states.

Covers: k8s distro decision, target architecture, control-plane design, bootstrap/DR procedures for the pfv-k8s cluster.

Contents

Each section below is posted as a separate reply for independent editing.
Use the reply index to navigate.

2

Architecture

Companion to: DISTRO-DECISION.md (why Talos),
../proxmox/K8S.md (storage/host analysis from the
Proxmox audit).

Last updated: 2026-07-28

Table of Contents

  1. High-Level Architecture
  2. Control Plane
  3. Network Topology
  4. CNI: Cilium
  5. Identity and Trust
  6. Tenant Isolation (vcluster)
  7. Storage Integration
  8. Local Image Registry
  9. Bootstrap Procedure
  10. Disaster Recovery
  11. Migration from Current State

1. High-Level Architecture

flowchart TB
    subgraph RESIDENCE["Residence — Proxmox LAN"]
        subgraph CP["Talos Control Plane (3 cnodes)"]
            C1[cnode1<br/>tsys9 · local-SSD]
            C2[cnode2<br/>tsys9 · local-SSD]
            C3[cnode3<br/>tsys1 · local-HDD]
        end
        subgraph WP["Talos Worker Plane"]
            W3[wnode-tsys3<br/>NVMe · 28GB]
            W5[wnode-tsys5<br/>NVMe · 32-64GB]
            W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
            W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
            W9[wnode-tsys9<br/>local-SSD · 4-8GB]
        end
        ETCD[(etcd<br/>raft, mTLS)]
        REG[(Harbor registry<br/>on D3 SSD · tsys5)]
        BASTION[tailscale-router VM<br/>subnet router]
    end

    subgraph TAILNET["Tailscale overlay"]
        OP[Operator devices]
    end

    subgraph CLOUDRON["Cloudron production — Reston VA"]
        KC[Keycloak OIDC IdP]
    end

    C1 ---|mTLS LAN| ETCD
    C2 ---|mTLS LAN| ETCD
    C3 ---|mTLS LAN| ETCD
    CP -->|pull images| REG
    WP -->|pull images| REG

    OP -->|Talos API :50000<br/>via subnet route| BASTION
    BASTION -.->|LAN| CP
    CP -->|OIDC| KC
    WP -->|OIDC| KC

    classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
    classDef infra fill:#0f3460,stroke:#e94560,color:#fff
    classDef external fill:#16213e,stroke:#533483,color:#fff
    class CP,WP,ETCD talos
    class REG,BASTION infra
    class OP,KC,EXTERNAL external

Design principles

  1. LAN-only cluster nodes. Zero internet egress from cnodes/wnodes.
    Strongest posture for ITAR/classified.
  2. Admin via Tailscale subnet router. Existing tailscale-router VM
    advertises the cluster LAN subnet. Operator reaches Talos API from
    anywhere.
  3. Local-first storage. Cnodes boot from local disk (no NFS dependency
    for etcd). Workers boot from local disk where available; NFS for bulk
    data only.
  4. Per-tenant vcluster. Workload isolation via virtual clusters on top
    of the Talos host cluster.
  5. OIDC everywhere. Talos API and Kubernetes API both trust Keycloak
    tokens. No long-lived static credentials for humans.

2. Control Plane

2.1 Recommendation: 3 cnodes (down from 5)

Option Quorum Failure tolerance etcd write cost Resource cost
3 cnodes (recommended) 2 of 3 Tolerates 1 failure Lower (faster commits) 3 × (2c/4GB/32GB) = 6c / 12GB
5 cnodes (current plan) 3 of 5 Tolerates 2 failures Higher 5 × (2c/4GB/32GB) = 10c / 20GB

For a solo-operated R&D cluster, 3 cnodes is the HA standard. The
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
host capacity) are better spent on tenant worker allocations.

Caveat: if your ITAR/classified accreditation counsel mandates 2-failure
tolerance on the control plane, keep 5. Otherwise 3.

2.2 Cnode placement

Per ../proxmox/K8S.md §4.3, cnodes should use
local-lvm boot disks so etcd has no NFS dependency. Concrete placement:

cnode Host Boot disk Type Why
cnode1 tsys9 local-lvm (PNY CS900 SSD) LOCAL-SSD Fastest available for etcd.
cnode2 tsys9 local-lvm (PNY CS900 SSD) LOCAL-SSD Same host, different disk OK (host failure is the failure domain, not disk).
cnode3 tsys1 local-lvm (HDD) LOCAL-HDD Host diversity. Slower than SSD but no NFS hop.

Quorum survival:

Failure cnodes lost Quorum OK?
tsys9 host dies cnode1 + cnode2 NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk.
tsys1 host dies cnode3 YES (2 of 3)
Any storage server dies 0 YES (3 of 3) — local disks unaffected

Refinement: putting both SSD cnodes on tsys9 means tsys9 host failure
loses quorum. Alternative: spread cnodes across 3 different hosts. See
“open question” at end of this section.

2.3 Machine config strategy

Talos nodes are configured by machine configs (YAML). Two flavors:

  • controlplane.yaml — for cnodes. Enables etcd, scheduler,
    controller-manager, API server.
  • worker.yaml — for wnodes. Joins cluster, runs kubelet + containerd.

Strategy for this cluster:

  1. One shared talosconfig (client identity) — stored in 1Password
    and in the Proxmox Backup Server (PBS) encrypted backup target.
  2. Per-node machine config patches — small patches on top of the base
    controlplane.yaml / worker.yaml for node-specific settings:
    • Hostname
    • Network interface + IP (DHCP or static — recommend static for cnodes)
    • Schematic image digest (pinned Talos version)
    • System extensions (e.g., tailscale — only if running Pattern A
      instead of recommended Pattern C)
  3. All machine configs in Git under a future k8s/talos-configs/
    directory. Secrets are templated in at apply-time from 1Password / sops.
flowchart LR
    BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
    BASE --> PATCH2[patch: cnode2]
    BASE --> PATCH3[patch: cnode3]
    BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
    PATCH1 --> APPLY1[talosctl apply]
    PATCH2 --> APPLY2[talosctl apply]
    PATCH3 --> APPLY3[talosctl apply]
    PATCHW --> APPLYW[talosctl apply]

2.4 Open question: cnode host spread

If you accept “tsys9 failure = quorum loss” as a tolerable risk (solo R&D
cluster, tsys9 is brand-new hardware, single digit annual failure
probability), the layout in §2.2 is fine.

If not, alternative spread across 3 hosts:

cnode Host Boot disk
cnode1 tsys9 local-lvm SSD
cnode2 tsys1 local-lvm HDD
cnode3 tsys3 local-lvm NVMe

tsys3’s local-lvm is 349 GB Samsung PM961 NVMe (per
../proxmox/PROJECT.md §3.3) — currently unused,
would make an excellent etcd disk.

This 3-host spread survives any single host failure with quorum intact.
Recommended.

3. Network Topology

3.1 Zones

flowchart TB
    subgraph INTERNET["Internet"]
        FIBER[Gigabit symmetric fiber]
    end

    subgraph RESLAN["Residence LAN 192.168.x.x/24"]
        subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
            CNODES[Cnodes 192.168.3.x]
            WNODES[Wnodes 192.168.3.x]
        end
        BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
        REG[Harbor registry<br/>192.168.3.x]
        STORAGE[NFS servers<br/>tsys4, tsys5]
    end

    subgraph TSNET["Tailscale 100.x.x.x/8"]
        OPS[Operator devices]
        KC[Keycloak<br/>via Cloudron prod]
    end

    FIBER --> BASTION
    BASTION <-. subnet route .-> CLUSTERNET
    OPS -->|TCP 50000 talos API| BASTION
    BASTION -->|LAN forward| CNODES
    CNODES -->|LAN mTLS| WNODES
    CNODES -->|OIDC HTTPS| KC
    WNODES -->|pull images| REG
    WNODES -->|bulk data IO| STORAGE
    CNODES -->|pull images| REG

3.2 Address plan (suggested)

Reserve a small contiguous block in the residence LAN for cluster nodes:

Role Range Count
Cnodes 192.168.3.31-33 3
Wnodes 192.168.3.41-49 up to 9 (1 per Proxmox host + spare)
Bastion existing tailscale-router 1
Registry 192.168.3.50 1 (Harbor)

Static IPs are strongly recommended for cnodes (etcd cluster membership
is hostname-based; stable IPs make talosctl targeting simple). Workers
can DHCP.

3.3 Firewall posture

Each cnode/wnode has:

  • Ingress from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
    cnodes only), plus CNI ports (varies by CNI — see §4).
  • Ingress from Tailscale: none (cluster nodes are not on Tailscale).
  • Egress: LAN-only. Block all RFC1918-external traffic at the perimeter
    firewall for these IPs. ITAR workloads must not be able to phone home.

The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
via the subnet route.

4. CNI: Cilium

Recommendation: Cilium (eBPF-based CNI).

Property Why it matters here
NetworkPolicy (incl. L7) Per-tenant isolation rules in vclusters.
Node-to-node encryption WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. Important for ITAR tenants.
Hubble Observable flows — forensic record of which pod talked to which. Useful for compliance evidence.
No kube-proxy Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node.
Talos integration First-class. Talos docs document the install path.

Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
enabled. Default-deny NetworkPolicy applied per namespace.

5. Identity and Trust

5.1 Trust flow

sequenceDiagram
    autonumber
    participant Human as Operator
    participant TAIL as Tailscale
    participant BAST as Bastion
    participant TALOS as Talos API :50000
    participant KC as Keycloak (Cloudron)
    participant K8S as Kubernetes API :6443

    Human->>TAIL: Authenticate (device + SSO)
    TAIL-->>Human: Tailnet IP
    Human->>BAST: Reach bastion via tailnet
    BAST->>TALOS: Forward to LAN node :50000
    Human->>KC: OIDC login (browser)
    KC-->>Human: Bearer token (short-lived)
    Human->>TALOS: talosctl (mTLS with client cert)
    Human->>K8S: kubectl --oidc (Keycloak token)
    K8S->>KC: Validate token (introspection)
    KC-->>K8S: Valid + claims
    K8S-->>Human: Authorized response

5.2 Two distinct identity layers

Layer Mechanism Audience
Talos API (node ops) Mutual TLS with client certificate generated from the Talos secrets bundle. Operators (automation + humans).
Kubernetes API (kubectl) OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. Humans. Service accounts use projected tokens (no OIDC).

The Talos secrets bundle is the root of trust for the cluster. Lose it
and you cannot operate the cluster; an attacker with it owns the cluster.
Storage:

  1. Primary: 1Password (or equivalent) — operator-accessible.
  2. Backup: PBS encrypted backup target on tsys4 (existing infra).
  3. NOT in Git. Machine configs go in Git; secrets stay out.

5.3 Keycloak client configuration

On Cloudron-hosted Keycloak, register a client pfv-k8s-talos:

  • Authorization Code + PKCE flow (no implicit, no password).
  • Redirect URIs: http://localhost:8000 (kubectl oidc-login) + Sidero
    Omni/Rancher URLs if/when those are added.
  • Group claims: k8s-admin, k8s-readonly, k8s-tenant-itar,
    k8s-tenant-rackrental, etc. These map to Kubernetes RBAC ClusterRoleBinding.

6. Tenant Isolation (vcluster)

6.1 Why vcluster

vcluster runs a virtual Kubernetes control
plane (API server, scheduler, controller-manager, etcd) inside a namespace
of the host cluster. Tenant workloads run on the host’s worker nodes but
are isolated by:

  • Separate API server (tenant cannot see host cluster objects).
  • Separate RBAC and admission control.
  • Separate network policies (per-namespace).
  • Separate resource quotas.

This aligns with the user’s per-tenant plan from
../proxmox/K8S.md §1.

6.2 Tenant registry

Tenant Compliance Workload example vcluster name
RackRental None (internal R&D) containerlab topology tests vc-rackrental
Suborbital non-ITAR EAR/ITAR-aware but unclassified Payload telemetry processing vc-suborbital-open
Suborbital ITAR ITAR-controlled Firmware build for USML items vc-suborbital-itar
Starting Line Productions Commercial Customer media pipeline vc-slp 
flowchart TB
    subgraph HOST["Talos host cluster"]
        CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
        subgraph NS["Host cluster namespaces"]
            NS_RR[ns: vc-rackrental]
            NS_SO[ns: vc-suborbital-open]
            NS_SI[ns: vc-suborbital-itar]
            NS_SLP[ns: vc-slp]
        end
    end

    subgraph VRR["vcluster: vc-rackrental"]
        API_RR[k8s API + etcd]
    end
    subgraph VSO["vcluster: vc-suborbital-open"]
        API_SO[k8s API + etcd]
    end
    subgraph VSI["vcluster: vc-suborbital-itar"]
        API_SI[k8s API + etcd]
    end
    subgraph VSLP["vcluster: vc-slp"]
        API_SLP[k8s API + etcd]
    end

    CP --> NS_RR & NS_SO & NS_SI & NS_SLP
    NS_RR --> API_RR
    NS_SO --> API_SO
    NS_SI --> API_SI
    NS_SLP --> API_SLP

    classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
    class NS_SI,API_SI itar

6.3 ITAR enforcement at host layer

For the ITAR tenant (vc-suborbital-itar), enforce additional host-layer
controls:

  • Node taint workload=itar:NoSchedule on worker nodes dedicated to
    ITAR workloads (subset of wnodes, marked in node labels).
  • NetworkPolicy default-deny egress for the vc-suborbital-itar
    namespace. Allow only explicit destinations (registry, NFS for ITAR
    data tier, Keycloak).
  • Storage isolation: ITAR PVCs target a dedicated NFS export (e.g.,
    D3-itar on tsys5) that no other tenant can mount.
  • Audit: Hubble flows + auditd on the host worker nodes capture all
    access to ITAR data.

Rancher (or Sidero Omni) sits above this, presenting each tenant’s
vcluster as a separate “cluster” in its UI, with Keycloak SSO gating
access per tenant group claim.

7. Storage Integration

Per ../proxmox/K8S.md §6. Three StorageClasses:

StorageClass Provisioner Backing Speed Use
local-fast local-path wnode local disk (NVMe/SSD/HDD depending on host) 100-3500 MB/s Container runtime, scratch, ephemeral
nfs-hdd nfs.csi.k8s.io tsys4 D2/D5, tsys5 S1-S4 80-120 MB/s Bulk data, weather/GIS datasets
nfs-ssd nfs.csi.k8s.io tsys5 D3, tsys5 T5-SSD 200-400 MB/s Latency-sensitive persistent data

7.1 CSI driver notes

  • NFS CSI: csi-driver-nfs
    (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
    NFS server + base export path.
  • local-path: Rancher Local Path Provisioner. Single-binary, deploys
    with one manifest. Uses wnode’s kubelet root dir.

7.2 ITAR data isolation

The ITAR tenant should target a dedicated NFS export, not shared
nfs-hdd. Recommended:

  • Allocate S4 on tsys5 (currently 99% empty, 435 GB free) as
    nfs-itar StorageClass. Mountable only from vc-suborbital-itar
    namespace via RBAC + NetworkPolicy.

8. Local Image Registry

8.1 Recommendation: Harbor on D3 SSD

D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
for a Harbor instance:

Property Value
Storage D3 SSD on tsys5 (NFS export, fast tier)
VM New VM pfv-registry on tsys5, local-nonprod boot, D3 data
Function (a) Pull-through cache for Docker Hub / Quay / gcr.io
(b) Host private images
(c) Cosign image signing verification
Exposure LAN-only. 192.168.3.50:443. Not exposed to internet.

8.2 Pull-through cache benefit

Cluster nodes have zero internet egress (per §3.3). Without a local cache,
image pulls fail. With Harbor as a pull-through cache:

sequenceDiagram
    WNODE->>HARBOR: docker pull nginx:1.25
    alt cache hit
        HARBOR-->>WNODE: layer bytes (LAN-speed)
    else cache miss
        HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
        DOCKERHUB-->>HARBOR: layer bytes
        HARBOR-->>WNODE: layer bytes (cached for next time)
    end

Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
machine in the cluster with container-registry internet egress, and that
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
ghcr.io).

8.3 Supply-chain integrity (future)

Harbor + Cosign lets you require that all images deployed to the ITAR
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
control. Implementation deferred to a later session.

9. Bootstrap Procedure

9.1 One-time setup

sequenceDiagram
    autonumber
    participant OP as Operator
    participant GIT as Git repo
    participant ONEPW as 1Password
    participant PBS as PBS (tsys4)
    OP->>GIT: Clone PFVCluster repo
    OP->>ONEPW: Generate Talos secrets bundle (offline)
    ONEPW-->>OP: secrets.yaml
    OP->>PBS: Backup secrets.yaml (encrypted)
    OP->>GIT: Write machine configs (no secrets)

9.2 Provision first cnode (bootstrap)

sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C1 as cnode1
    participant ETCD as etcd (new)

    OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
    OP->>PX: qm start VMID
    C1->>C1: Boots Talos (no config yet, "maintenance mode")
    OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
    C1->>C1: Applies config, restarts services
    OP->>C1: talosctl bootstrap
    C1->>ETCD: Initialize single-node raft
    ETCD-->>C1: ready
    OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
    OP->>C1: talosctl etcd snapshot (initial backup → PBS)

9.3 Add second and third cnodes

sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C2 as cnode2
    participant C3 as cnode3
    participant C1 as cnode1 (existing)

    OP->>PX: qm create + start cnode2 VM
    C2->>C2: Boots Talos maintenance mode
    OP->>C2: talosctl apply --patch cnode2.yaml
    C2->>C1: Join etcd cluster
    OP->>PX: qm create + start cnode3 VM
    C3->>C3: Boots Talos maintenance mode
    OP->>C3: talosctl apply --patch cnode3.yaml
    C3->>C1: Join etcd cluster
    Note over C1,C3: etcd now has 3/3 members → HA quorum

9.4 Post-bootstrap cluster configuration

Once 3 cnodes are up and joined:

  1. Install Cilium (CNI) via Helm. Enable node-to-node encryption.
  2. Install CSI drivers — nfs-csi + local-path provisioner.
  3. Create StorageClasses — local-fast, nfs-hdd, nfs-ssd.
  4. Deploy Harbor on the pfv-registry VM, exposed at 192.168.3.50.
  5. Configure Kubernetes API OIDC — Keycloak client (§5.3).
  6. Apply default-deny NetworkPolicy in all namespaces.
  7. Install vcluster CLI + create 4 tenant vclusters (§6).
  8. First etcd snapshot + automated daily snapshot cron → PBS.

9.5 Add workers

Workers are simpler (no etcd):

sequenceDiagram
    OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
    WNODE->>WNODE: Boots maintenance mode
    OP->>WNODE: talosctl apply --patch worker-X.yaml
    WNODE->>C1: Kubelet registers with API server
    C1-->>WNODE: Approved (auto via bootstrap token)
    Note over WNODE: Joins cluster, becomes Ready

10. Disaster Recovery

10.1 Backup strategy

Artifact Frequency Storage Tool
Talos secrets bundle Once (regen only on rotation) 1Password + PBS (encrypted) Manual
Machine configs Continuous (Git) Git remote + PBS Git
etcd snapshot Daily + before each change PBS (tsys4 SMR target, 4.3 TB free) talosctl etcd snapshot
vcluster etcd Daily per vcluster PBS kubectl exec ... etcdctl snapshot
Harbor metadata Daily PBS Harbor built-in backup

10.2 Restore scenarios

Lost 1 cnode (e.g., tsys9 disk failure):

  1. Provision new VM on tsys9 (or other host with local SSD).
  2. Apply cnode2 machine config patch.
  3. New cnode joins etcd, syncs state from survivors.
  4. Quorum was never lost (2 of 3 alive throughout).

Lost 2 cnodes simultaneously (quorum lost):

  1. Use surviving cnode’s etcd snapshot.
  2. Provision 3 new cnode VMs.
  3. On first: talosctl bootstrap --recover-from=snapshot.db.
  4. Join other 2 cnodes.
  5. Workers reconnect automatically once API server is back.

Total cluster loss (all 3 cnodes):

  1. Restore from latest PBS etcd snapshot.
  2. Provision new cnode VMs.
  3. talosctl bootstrap --recover-from=snapshot.db.
  4. Re-join workers.
  5. Verify tenant vclusters restored.

10.3 Recovery time objectives

Scenario RTO RPO
Single cnode failure < 30 min 0 (no data loss)
Quorum loss (2 cnodes) < 2 hours ≤ 24 hours (last snapshot)
Total cluster loss < 4 hours ≤ 24 hours

11. Migration from Current State

11.1 Current state

  • 5 cnode VMs exist (Debian stock + Tailscale).
  • No k3s deployed yet. Cluster was never bootstrapped.
  • 6 wnode VMs exist (some stopped).
  • No workloads running in k8s.

11.2 Migration: clean cutover (not a migration)

Since there is no etcd data and no workloads to preserve, the path is a
clean rebuild:

Phase Action Risk
0. Prep Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. Low.
1. Bootstrap 3 new cnodes Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). Low. Existing Debian cnodes can keep running idle.
2. Configure cluster Install Cilium, CSI, StorageClasses, OIDC, Harbor. Low.
3. Add workers Re-image existing wnode VMs as Talos, or build new ones. Low. No workloads to drain.
4. Decommission old Debian cnodes Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. Low.
5. Tenant vclusters Stand up per-tenant vclusters. Medium (policy tuning).

11.3 Open dependency: Friday hardware work

Phases 1-2 require:

  • tsys3 local-lvm available. Per
    ../proxmox/PROJECT.md §3.3, tsys3 has 349 GB
    free NVMe local-lvm. Currently unused. Ready.
  • tsys9 local-lvm available. 136 GB PNY CS900 SSD. Ready.
  • D3 SSD relocated to tsys5 SAS. Currently USB on tsys4. Per
    ../proxmox/TODO.md §2, scheduled for Friday.
    Harbor depends on D3 being available on tsys5.

Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
Harbor registry does.

Appendix: Open questions for next session

  1. Confirm 3 vs 5 cnodes (§2.1). Recommendation: 3.
  2. Confirm cnode host spread (§2.4). Recommendation: 3-host spread
    (tsys9, tsys1, tsys3).
  3. Static IPs for cnodes (§3.2). Recommendation: yes, 192.168.3.31-33.
  4. Rancher vs Sidero Omni for cluster management UI. Both viable.
    Defer until cluster is up.
  5. Subnet router ACL approval on Tailscale admin console (§3). Needs
    approval of 192.168.3.0/24 route advertisement.
  6. ITAR worker node subset (§6.3). Which wnodes are tainted for ITAR?
    Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
    general capacity; tsys3 + tsys5 (local fast storage) reserved for
    non-ITAR HPC.
3

Distro Decision

Recommendation: Talos Linux.
The k3s-on-Debian plan was sound before the ITAR/classified requirement
entered scope. Once classified workloads are on the table, Talos’s
immutable, API-only, measured-boot-capable posture is materially easier
to certify and defend.

Last updated: 2026-07-28

1. Decision context

Factor Constraint
Workload class R&D + RackRental (containerlab) + ITAR / classified suborbital workloads + commercial (Starting Line Productions)
Compliance drivers ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253)
Hardware 7 standalone Proxmox hosts (no pvecm), managed via PDM. Live migration NOT available — disk moves via Proxmox “Storage Migrate” UI.
Network Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access).
Current cnode state Stock Debian VMs joined to Tailscale. No k8s distribution has been deployed yet. Clean cutover possible.
Operations Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead.

2. Head-to-head comparison

2.1 ITAR / classified posture

Property Talos Linux k3s on Debian
Node OS mutability Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. Mutable. apt install, file edits persist.
Shell / SSH access None. No SSH daemon, no shell, no kubectl debug node shell. Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface.
Operational surface Single gRPC API (mTLS, signed certs, audit log) on port 50000. SSH + kubelet API + etcd API + package manager + cron + systemd + userland.
Measured boot Supported. TPM attestation can prove the node booted the signed Talos image you pinned. Possible but bolt-on; auditors will ask why you didn’t disable the bootloader first.
Configuration provenance Entire node state is a YAML machine config in Git. talosctl apply is the only mutation path. Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply.
Supply chain Every Talos release is a signed artifact (cosign). Pin by image digest. Debian package provenance is good but the surface is enormous (~30K packages in a base install).
Forensic readiness API log + kernel log + Talos event log = sufficient for “what ran, when, with what config.” Same possible but requires explicit configuration to be trustworthy.
STIG / CIS conformance Intrinsically close. Talos publishes CIS benchmark results per release. Requires running SCAP-STIG (already in this repo) and remediating findings continuously.

Bottom line: For classified workloads, an auditor’s first question is
“how do you prevent unauthorized changes to a node?” Talos’s answer is
“the OS is immutable and the only path is a signed API call.” k3s’s answer
is"SSH is locked down and we scan with STIG." The first is structurally
stronger; the second is operationally maintained.

2.2 Operational considerations

Property Talos k3s
Familiarity New model (talosctl apply, no SSH). Learning curve. Stock Debian + k3s binary. Familiar.
Debugging talosctl logs, talosctl dmesg, talosctl dashboard. No shell. ssh, journalctl, crictl. Full shell.
Tailscale integration System extension (siderolabs/tailscale). Stable since Talos 1.3. Native — apt install tailscale. Zero friction.
Backup / DR talosctl etcd snapshot (one command). Cluster can be restored from snapshot + machine configs. DIY (etcdctl snapshot + manual cert management).
Upgrades talosctl upgrade — atomic, automated rollback on health-check failure. Manual: drain, k3s package update, reboot, uncordon.
Proxmox compatibility QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. Same.
Ecosystem maturity Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). Production-grade. Rancher (SUSE) backs it.

2.3 Cost of choosing Talos over the existing k3s plan

The cnodes are currently stock Debian VMs joined to Tailscale. Critically,
no k3s cluster has been deployed yet — k3s was only the plan. Therefore:

  • No etcd data to migrate. Clean cutover, not a migration.
  • No workloads to drain. The cluster is empty.
  • Cnode VMs get re-imaged with Talos QCOW2 (or rebuilt from scratch —
    either way it’s a qm script, not a stateful migration).
  • Tailscale config shifts from “installed via apt” to “Talos system
    extension.” (Or, per our recommendation in
    ARCHITECTURE.md §3, Tailscale moves off the cluster
    nodes entirely     and onto the existing tailscale-router bastion as a
    subnet router. Cluster nodes become LAN-only.)

Net cost: rebuilding 3 cnode VMs as Talos + writing ~200 lines of
machine config YAML. The hardening investment already encoded in
provisioning/Modules/Security/ is not wasted — it still applies to
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
the cnodes/wnodes move to Talos.

3. Tailscale compatibility (deep-dive)

Tailscale on Talos is well-supported but introduces a configuration dimension
worth being explicit about. Three patterns exist:

Pattern A — Tailscale on every cluster node (what you have now, on Debian)

Each cnode/wnode runs tailscaled and joins the tailnet. Cluster nodes have
internet egress (to Tailscale DERP servers and for coordinate).

  • Talos implementation: add siderolabs/tailscale system extension to
    each machine config, configure machine.network.interfaces.
  • Pros: Operator can hit any node’s Talos API from any Tailscale device.
  • Cons: Cluster nodes have internet egress. For ITAR workloads, this is a
    finding (data exfiltration path).

Pattern B — Tailscale on bastion only, SSH/API jump

Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
tailscale-router VM), then runs talosctl from the bastion.

  • Pros: Zero internet egress from cluster nodes.
  • Cons: Two-step access. Bastion must run recent talosctl. Each operator
    action originates from the bastion (auditable but clunky).

Pattern C — Tailscale subnet router on bastion (recommended)

The existing tailscale-router VM advertises the cluster LAN subnet
(e.g. 192.168.3.0/24) into the tailnet as a subnet route. Operator’s
Tailscale client transparently routes cluster-bound traffic through the
bastion. From the operator’s workstation, talosctl --nodes 192.168.3.x
“just works.”

  • Pros:
    • Cluster nodes have zero internet egress (strongest ITAR posture).
    • Operator UX is unchanged from direct LAN access.
    • All access is mediated by Tailscale’s identity + ACLs (already integrated
      with your env).
    • Audit trail lives in Tailscale + bastion logs.
  • Cons:
    • Bastion becomes a dependency for remote admin (LAN-local admin still
      works without it).
    • Must enable IP forwarding + subnet route approval in Tailscale ACLs.

Recommendation: Pattern C. Documented in
ARCHITECTURE.md §3.

4. Recommendation

Deploy Talos Linux as the k8s distribution for pfv-k8s.

Justification

  1. Compliance posture is structural, not operational. “Immutable,
    API-only, measured-boot” is a property of Talos itself; “STIG-hardened”
    is a property of how Debian is operated. The first is dramatically
    easier to argue to an ITAR counsel or classified accreditation officer
    (DSS, DCSA) than the second.
  2. Zero migration cost. The k3s cluster was never deployed. Reimaging
    3 cnodes with Talos is a qm script invocation, not a stateful
    migration. The sunk cost of “we planned k3s” is zero deployed state.
  3. Operational headroom. Talos’s etcd snapshot + upgrade --stage +
    apply-mode auto reduce solo-founder ops burden. k3s is simpler to
    learn but more error-prone to operate at HA.

Acknowledged tradeoffs

  • Learning curve. The Talos mental model (machine config + talosctl)
    replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
  • No shell debugging. When something breaks on a node, you cannot ssh
    in. Mitigation: talosctl logs/support produces a support bundle
    equivalent to a sosreport.
  • Hardware/module surprises. Talos ships a curated kernel. Anything
    beyond virtio + common NIC drivers needs a system extension. On Proxmox
    VMs this is not expected to be a problem — virtio is the path.
  • Tailscale via system extension. Adds one config dimension per node.
    Mitigated by Pattern C (above), which removes Tailscale from cluster
    nodes entirely.

What we keep from the k3s mental model

  • Single binary on each node semantics (Talos is conceptually similar).
  • kubectl workflow unchanged. Talos exposes a standard Kubernetes
    API. kubectl, helm, kustomize all work as-is.
  • Storage CSI choices (local-fast, nfs-hdd, nfs-ssd) are
    distro-independent.

5. What we are NOT deciding here

Topic Deferred to
ETL tooling (GDAL/PostGIS/xarray/Dask) Future session — affects StorageClass RWX/RWO design
HPC scheduler (Jobs/Argo/Volcano) Future session — affects taint/label strategy
Per-tenant vcluster policy templates Future session, post-bootstrap
Solar-aware scale-out hosts Future capacity planning session
Container network plugin (CNI) details Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants)

6. Next step

Proceed to ARCHITECTURE.md for the control-plane
design, network topology, identity flow, and bootstrap procedure.

4

k8s Setup README

Redmine: #362 (initial build, closed) · #367 (rebuild, open) · #368 (worker join, open)

Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
All cluster communication goes over Tailscale IPs — no LAN traffic.

Current State

3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):

Node Tailscale IP Role Tainted
pfv-k8s-cnode1 100.97.178.106 control-plane, etcd NoSchedule
pfv-k8s-cnode2 100.109.34.72 control-plane, etcd NoSchedule
pfv-k8s-cnode3 100.106.222.18 control-plane, etcd NoSchedule

Scripts

Script Purpose
env.sh Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts.
wipe.sh Remove existing k3s from all cnodes (clean slate).
install-cp.sh Full bootstrap: cnode1 (–cluster-init) then cnode2/3 join.
join-servers.sh Re-join cnode2/3 only (if cnode1 is already up).
post-setup.sh Apply NoSchedule taints, fetch kubeconfig, verify.
verify.sh Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS.
probe-nodes.sh Verify SSH + Tailscale reachability.

Usage

# Full bootstrap from scratch:
bash k8s/wipe.sh
bash k8s/install-cp.sh
bash k8s/post-setup.sh
bash k8s/verify.sh

# Access the cluster:
export KUBECONFIG=~/.kube/config.pfv-k8s
kubectl get nodes

Design Decisions

  • k3s (not Talos): This is a regular R&D cluster, not ITAR/classified.
    Talos architecture is documented in docs/ for when
    that requirement comes online. k3s on stock Debian is simpler to operate.
  • Tailscale-only transport: --node-ip, --advertise-address, and
    --tls-san are all set to Tailscale IPs. No LAN IP appears in any node
    status or certificate.
  • VXLAN flannel: Pods communicate via flannel VXLAN overlay on top of
    Tailscale’s WireGuard. Double-encrypted, but functional and reliable.
  • NoSchedule taint: All 3 cnodes are tainted so no user workloads
    schedule on the control plane. Only system components (CoreDNS,
    metrics-server, flannel, kube-proxy) with built-in tolerations run here.
  • Embedded etcd: 3-node HA etcd quorum. Tolerates 1 node failure.
5

k8s Docs Index

Status: Draft for review. Companion to ../proxmox/K8S.md
(which captured the storage/host analysis from the Proxmox audit).
This directory takes the next step: which distro, how to build it,
how to operate it.

Last updated: 2026-07-28

Documents in this directory

Document Purpose
DISTRO-DECISION.md Talos vs k3s analysis. Recommendation: Talos, with rationale grounded in the ITAR/classified requirement.
ARCHITECTURE.md Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included.

TL;DR

Decision Recommendation Why
Distro k3s (deployed) / Talos (for future ITAR) k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online.
Runtime containerd Talos/k3s default.
Cnode count 3 (deployed) Standard HA. Tolerates 1 failure.
Admin access Tailscale (all nodes joined) Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern.
Cluster network Tailscale-only IPs All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state.
Identity OIDC to Keycloak on Cloudron (production) Future work.
Multi-tenancy vcluster (per tenant) Future work.
Local registry Harbor on D3 SSD (tsys5, 445 GB free) Future work.
Storage classes local-fast, nfs-hdd, nfs-ssd Per ../proxmox/K8S.md §6. Future work.

What this directory does NOT cover (deferred)

  • ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
  • HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
  • vcluster per-tenant policy templates.
  • Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.

These are tracked as future session work in ../../STATUS.md.

Open question for the user

  1. Cnode count: confirm 3 vs 5. Recommendation is 3 (rationale in
    ARCHITECTURE.md §2). If your ITAR counsel requires
    2-failure tolerance on the control plane, keep 5.
6

Legacy k8s Storage/Host Analysis

Date: 2026-07-27
Purpose: Detailed kubernetes architecture plan for the pfv-k8s cluster.
Companion to PROJECT.md (which has the fleet-wide assessment).
Status: For discussion in a future session. No changes made.

Table of Contents

  1. Workload Profile
  2. Current State
  3. Target Architecture
  4. Control Plane (Cnodes)
  5. Worker Nodes (Wnodes)
  6. Storage Class Design
  7. ETL/HPC Considerations
  8. Migration Plan

1. Workload Profile

This cluster runs R&D and RackRental (containerlab) workloads via
Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
Reston, VA running Cloudron.

Workload types expected:

Type Description Storage need RAM need Examples
ETL (weather/GIS) Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. High capacity (100s of GB), moderate IOPS Medium (8-32 GB per job) GRIB/NetCDF processing, raster reprojection
HPC (hardware startup) Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. Low capacity, moderate IOPS High (32-128 GB per job) RTL simulation, PCB thermal analysis
RackRental (containerlab) Rapid deployment/teardown of network lab topologies. Many containers, short-lived. Low capacity, high IOPS (container image pulls) Low-Medium (4-16 GB) Network topology testing, protocol validation

Key storage insight: ETL workloads need bulk capacity (NFS-HDD is fine –
sequential I/O). HPC and containerlab need low-latency random I/O (local
SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.

2. Current State

2.1 pfv-k8s nodes and their storage

Node Type Host Storage Disk type Status
cnode1 (107) control tsys1 D5 (tsys4) NFS-HDD running
cnode2 (603) control tsys6 D2 (tsys4) NFS-HDD running
cnode3 (106) control tsys1 D2 (tsys4) NFS-HDD running
cnode4 (601) control tsys6 D2 (tsys4) NFS-HDD running
cnode5 (602) control tsys6 D5 (tsys4) NFS-HDD running
wnode-tsys3 (313) worker tsys3 D5 (tsys4) NFS-HDD running
wnode-tsys5 (509) worker tsys5 D2 (tsys4) NFS-HDD running
wnode-tsys6 (100) worker tsys6 D5 (tsys4) NFS-HDD STOPPED
wnode-tsys7 (701) worker tsys7 D5 (tsys4) NFS-HDD running
wnode-tsys9 (905) worker tsys9 S3 (tsys5) NFS-HDD running

2.2 Problems

  1. 100% of cnodes on tsys4 NFS. D2 disk failure loses 3 of 5 cnodes =
    etcd quorum lost.
  2. 90% of all k8s nodes on tsys4 NFS. tsys4 failure kills the cluster.
  3. Zero nodes use SSD or NVMe. All on NFS-over-HDD.
  4. Zero nodes use local-lvm. tsys3/6/7/9 all have empty local storage
    (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
  5. wnode-tsys6 is stopped. Reduces cluster capacity.
  6. 3 cnodes on tsys6 – should be on lighter hosts to free tsys6 for workers.

3. Target Architecture

3.1 Design principles

  1. Cnodes on lightweight hosts (tsys1, tsys9, tsys3) – frees tsys6/7 for
    heavy workers.
  2. Cnode storage split across tsys4 and tsys5 – etcd survives either
    storage server failing.
  3. Wnode boot disks on local storage – eliminates NFS latency for
    container runtime and kubelet.
  4. Wnode data disks on NFS-HDD – bulk capacity for ETL/weather/GIS.
  5. tsys5 NVMe dedicated to wnode-tsys5 – fastest tier for HPC jobs.
  6. One wnode per hypervisor host – maximize total cluster capacity.

3.2 Target node-host-storage matrix

Node Type Host Boot disk Data disk Disk type
cnode1 control tsys1 D5 (tsys4) – NFS-HDD
cnode2 control tsys9 D2 (tsys4) – NFS-HDD
cnode3 control tsys1 S2 (tsys5) – NFS-HDD
cnode4 control tsys9 D5 (tsys4) – NFS-HDD
cnode5 control tsys3 S3 (tsys5) – NFS-HDD
wnode-tsys1 worker tsys1 D5 (tsys4) – NFS-HDD (small)
wnode-tsys2 worker tsys2 NVMe (960 PRO 512GB) SATA SSD (850 EVO 1TB) NVMe + SSD – no NFS needed
wnode-tsys3 worker tsys3 local-lvm (NVMe PM961) S3 (NFS) LOCAL-NVMe
wnode-tsys5 worker tsys5 NVMe (local, Friday) local-nonprod (HDD) NVMe
wnode-tsys6 worker tsys6 D2 (tsys4 NFS) – NFS-HDD (local-lvm is USB 2.0 – do not use)
wnode-tsys7 worker tsys7 D5 (tsys4 NFS) – NFS-HDD (local-lvm is USB 2.0 – do not use)
wnode-tsys9 worker tsys9 local-lvm (SSD) S2 (NFS) LOCAL-SSD

3.3 Storage server distribution after changes

Storage server cnodes wnodes (boot) wnodes (data)
tsys4 (D2) cnode2 wnode-tsys6 wnode-tsys7
tsys4 (D5) cnode1, cnode4 wnode-tsys1 –
tsys5 (S2) cnode3 wnode-tsys9 –
tsys5 (S3) cnode5 – wnode-tsys3

Note: wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
NFS by design – their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
than NFS-HDD, and the user has chosen not to install internal drives.

No single disk or server is a quorum-losing failure point.

4. Control Plane (Cnodes)

4.1 Cnode sizing

Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
kubernetes control plane components (API server, scheduler, controller-manager).

4.2 Cnode host placement rationale

Host cnodes RAM for cnodes Total host RAM Remaining for other VMs
tsys1 2 (cnode1, cnode3) 8 GB 32 GB ~24 GB (but 11 infra VMs consume most)
tsys9 2 (cnode2, cnode4) 8 GB 24 GB ~16 GB (4 infra VMs + 1 wnode)
tsys3 1 (cnode5) 4 GB 32 GB ~28 GB (1 wnode at 20 GB = 8 GB headroom)

tsys6 and tsys7 have ZERO cnodes – fully dedicated to heavy worker nodes.

4.3 Cnode storage placement rationale

The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:

Disk cnodes Rationale
D5 (tsys4 HDD) cnode1, cnode4 Spread load across 2 disks on tsys4
D2 (tsys4 HDD) cnode2 Only 1 cnode on D2 (was 3 – reduces blast radius)
S2 (tsys5 HDD) cnode3 tsys5 storage for quorum diversity
S3 (tsys5 HDD) cnode5 tsys5 storage, different disk than S2

If D2 fails: cnode2 dies. 4 of 5 survive. Quorum OK.
If D5 fails: cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
If tsys4 fails: cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
tsys5. Only 2 of 5 – QUORUM LOST.

Wait – that is a problem. If tsys4 goes completely offline, we lose 3
cnodes and only have 2 on tsys5. That loses quorum (need 3).

Revision needed: Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
tsys4. Same problem inverted.

The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
holding 2 cnodes is safe (3 survive).

Proper solution: 3 cnodes on the “less likely to fail” server, 2 on the
other. After Friday’s hardware work:

  • tsys4 will have a new PCIe NIC + 64 GB RAM – more reliable
  • tsys5 will have bond0 fixed + NVMe – more reliable

Either way, 3/2 split means one server failure could lose quorum. To truly
solve this, use a 3rd storage target. Options:

  • Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
    independent failure domain)
  • Use local-lvm on the cnode’s own host (etcd data is local to the VM’s host,
    no NFS dependency at all)

Best option: put cnode boot disks on local-lvm where available. This
eliminates NFS entirely for the control plane. Each cnode’s etcd data lives on
its own host’s local disk – no shared dependency.

cnode Host Recommended storage Type
cnode1 tsys1 local-lvm (if space) or D5 (tsys4) LOCAL-HDD or NFS-HDD
cnode2 tsys9 local-lvm (SSD) LOCAL-SSD
cnode3 tsys1 S2 (tsys5) NFS-HDD
cnode4 tsys9 local-lvm (SSD) LOCAL-SSD
cnode5 tsys3 local-lvm LOCAL-HDD

With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
survives.

This is the recommended approach. Local storage for cnodes wherever
possible. NFS only as fallback.

4.4 etcd performance on local vs NFS

Storage Typical fsync latency etcd commit latency Impact
NFS-HDD (via USB dongle on tsys4) 5-15 ms 10-30 ms Slow API responses, sluggish pod scheduling
NFS-HDD (via PCIe NIC, post-Friday) 2-8 ms 5-15 ms Better but still network-bound
Local HDD (tsys1/3/6/7 local-lvm) 1-5 ms 3-10 ms No network hop, moderate improvement
Local SSD (tsys9 PNY CS900) 0.1-0.5 ms 0.5-2 ms 10-30x faster than NFS-HDD
NVMe (tsys5, Friday) 0.02-0.1 ms 0.1-0.5 ms 100x faster than NFS-HDD

etcd is the heartbeat of the kubernetes control plane. Every API call, every
pod schedule, every controller reconciliation involves an etcd write. Cutting
etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
responsive. This is the single highest-impact change for k8s performance.

5. Worker Nodes (Wnodes)

5.1 One wnode per hypervisor host

Host wnode Boot disk Data disk Total RAM wnode RAM Role
tsys1 wnode-tsys1 D5 (tsys4 NFS) – 32 GB 4-8 GB Small worker, infra co-tenant
tsys2 wnode-tsys2 NVMe (960 PRO 512GB) SATA SSD (850 EVO 1TB) 32 GB 16-24 GB Best storage of any worker – 1.5TB local SSD, no NFS needed
tsys3 wnode-tsys3 local-lvm (349 GB) S3 (NFS) 32 GB 20 GB General worker
tsys5 wnode-tsys5 NVMe (local) local-nonprod (HDD) 96 GB 32-64 GB HPC/ETL powerhouse
tsys6 wnode-tsys6 D2 (tsys4 NFS) – 128 GB 64-96 GB Heavy worker, max RAM. local-lvm is USB 2.0 – stays on NFS
tsys7 wnode-tsys7 D5 (tsys4 NFS) – 192 GB 96-128 GB Heavy worker, max RAM. local-lvm is USB 2.0 – stays on NFS
tsys9 wnode-tsys9 local-lvm SSD (136 GB) S2 (NFS) 24 GB 4-8 GB Small worker, SSD boot

5.2 Why boot disks on local-lvm

Current: all wnodes boot from NFS. Every container image pull, every kubelet
log write, every ephemeral volume traverses the NFS network path.

With local-lvm boot disks:

  • Container image pulls write to local disk (100-150 MB/s HDD, no network
    hop) instead of NFS-HDD (80-120 MB/s with network latency)
  • kubelet logs stay local (no NFS writes for log rotation)
  • ephemeral storage (emptyDir volumes) uses local disk by default
  • NFS server failure does not kill the wnode – the VM stays running, only
    the data disk (if mounted) goes away

5.3 Wnode sizing guidance

Host Recommended wnode config Rationale
tsys7 (192 GB) 8-12 cores, 96-128 GB RAM, NFS boot Largest host – run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0
tsys6 (128 GB) 8 cores, 64-96 GB RAM, NFS boot Second-largest – parallel heavy jobs. local-lvm is USB 2.0
tsys5 (96 GB + NVMe) 4 cores, 32-64 GB RAM, NVMe boot + HDD data NVMe makes this fastest for I/O-bound HPC
tsys3 (32 GB) 4 cores, 20 GB RAM, local-lvm boot General-purpose worker
tsys2 (32 GB, NVMe+SSD, incoming) 4 cores, 16-24 GB RAM, NVMe boot + SSD data Fastest storage worker – HPC with I/O bounds
tsys1 (32 GB) 2 cores, 4-8 GB RAM Small worker, don’t starve infra VMs
tsys9 (24 GB) 2-4 cores, 4-8 GB RAM Small worker, SSD boot is the advantage

5.4 Tainting and labeling strategy

Label wnodes by capability so the k8s scheduler can target them:

# Heavy RAM hosts (ETL/HPC)
wnode-tsys6: workload=heavy, ram=128g
wnode-tsys7: workload=heavy, ram=192g

# NVMe host (I/O-intensive HPC)
wnode-tsys5: workload=hpc, storage=nvme

# SSD boot host (low-latency)
wnode-tsys9: workload=light, storage=ssd

# General workers
wnode-tsys3: workload=general
wnode-tsys2: workload=storage-fast, storage=nvme
wnode-tsys1: workload=light

Then use nodeSelector or nodeAffinity in job specs:

# Weather/GIS ETL job -- needs lots of RAM
spec:
  nodeSelector:
    workload: heavy

# Firmware build -- needs fast storage
spec:
  nodeSelector:
    storage: nvme

6. Storage Class Design

6.1 Proposed StorageClasses

StorageClass Provisioner Where Speed Use case
local-fast local-path (k8s) wnode local-lvm / NVMe 100-3500 MB/s Container runtime, scratch, databases
nfs-hdd nfs-subdir-external-provisioner tsys4 D2/D5, tsys5 S1-S4 80-120 MB/s Bulk data, weather/GIS datasets
nfs-ssd nfs-subdir-external-provisioner tsys4 D3, tsys5 T5-SSD 200-400 MB/s Latency-sensitive persistent data

6.2 How this maps to wnode disk topology

Each wnode has:

  • Disk 1 (boot/OS): local-lvm or NVMe. Contains the OS, kubelet, container
    runtime. k8s local-fast StorageClass provisioner points here.
  • Disk 2 (bulk data, optional): NFS mount. Mounted inside the VM as a
    second block device or filesystem. k8s nfs-hdd provisioner points here.

Inside k8s, pods request storage via PVC:

# ETL job: needs bulk storage for weather data
apiVersion: v1
kind: PersistentVolumeClaim
spec:
  storageClassName: nfs-hdd
  accessModes: [ReadWriteMany]  # NFS allows RWX
  resources:
    requests:
      storage: 500Gi

# HPC job: needs fast scratch
spec:
  storageClassName: local-fast
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi

6.3 NFS-SSD tier (D3 and T5-SSD – both on tsys5 after Friday)

Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
scratch space, with the exception of ultix-streaming which stays on T5-SSD.
Spinning rust hosts all other infrastructure VMs (UCS, netinfra, LibreNMS,
SIEM, etc.).

The SSD NFS exports:

  • D3 (tsys5 SAS, 445 GB free): k8s scratch exclusively (etcd, container
    cache, ephemeral volumes). Currently 99% empty.
  • T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming): ultix-streaming
    occupies 83 GB. Remaining 140 GB available for k8s use.

tsys5 is the fast-tier hub: NVMe (local) + D3 SSD + T5-SSD all on one host.
This simplifies the StorageClass design – latency-sensitive k8s PVCs target
tsys5 SSD exports, bulk PVCs target either server.

6.4 NFS data distribution across storage servers

To avoid re-creating the “everything on tsys4” problem, distribute NFS data
disks across both servers:

wnode Boot (local) Bulk data (NFS) NFS server
wnode-tsys3 local-lvm S3 tsys5
wnode-tsys5 NVMe local-nonprod local (no NFS)
wnode-tsys6 D2 (tsys4 NFS) – tsys4
wnode-tsys7 D5 (tsys4 NFS) – tsys4
wnode-tsys9 local-lvm (SSD) S2 tsys5

This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.

7. ETL/HPC Considerations

7.1 Weather/GIS ETL pipeline

Typical flow: download GRIB/NetCDF files → process (reproject, aggregate) →
store results.

Stage Storage class Why
Download raw data nfs-hdd Large sequential writes. NFS-HDD handles this well.
Processing scratch local-fast Random access during transform. Local disk avoids NFS latency.
Store results nfs-hdd Large sequential writes. Persistent.

Recommendation: Deploy a local-fast PV mount as /scratch on every
wnode. ETL jobs use /scratch for intermediate processing and write final
output to the NFS-mounted /data.

7.2 HPC workloads (hardware startup)

Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.

Workload Best wnode Why
RTL simulation (CPU-bound, high RAM) tsys7 (192 GB) Most RAM, most cores (24t)
Firmware builds (I/O-bound, moderate RAM) tsys5 (NVMe) Fastest storage for compile I/O
Hardware-in-the-loop (latency-sensitive) tsys9 (local SSD) Lowest latency storage
Parallel batch jobs tsys6 + tsys7 Distribute across both heavy hosts

7.3 RackRental/containerlab

Rapid container deployment. Key need: fast container image pulls.

This is where local-lvm boot disks shine. Currently, every container image
pull writes through NFS to a spinning disk – slow. With local-lvm, images
cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.

7.4 Data locality for ETL

For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
it on local-lvm of the heavy hosts:

tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/  -- fastest cache tier
tsys5 NVMe (local): /data/cache/gis/                 -- fastest cache tier

Note: tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) – cannot
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.

This avoids re-reading the same data from the same NFS export on every job
if the data is already cached in the page cache.

8. Migration Plan

Key enabler: The hosts are standalone Proxmox installs, but Proxmox
Datacenter Manager (PDM) manages them collectively and supports VM migration
between nodes. Storage migration can be done via the PDM/Proxmox UI rather
than manual disk copies – the destination node just needs access to the target
storage (which all nodes have for NFS exports, and local storage can be
migrated through the UI’s “Storage Migrate” function).

8.1 Phase 1: Friday (after hardware work)

After tsys5 cable + NVMe and tsys4 NIC + RAM:

  1. Format tsys5 NVMe as local directory storage (e.g., nvme-local)
  2. Restart wnode-tsys6 (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 –
    do not use for VM storage. Recreate on D2 or D5 NFS.
  3. Move wnode-tsys9 (VM 905) disk from S3 (NFS) to local-lvm (SSD).

8.2 Phase 2: Cnode rebalance (maintenance window)

These changes require creating new VMs on target hosts and migrating disks.
Plan for a maintenance window with the k8s cluster briefly down.

  1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
  2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
  3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
  4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
  5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.

8.3 Phase 3: Wnode local storage migration (maintenance window)

  1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
  2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD – not suitable).
  3. wnode-tsys7 stays on NFS (same reason).
  4. Recreate wnode-tsys5 with boot disk on NVMe.
  5. Add data disks (NFS) as second SCSI devices where applicable.

8.4 Phase 4: tsys2 integration (when rebuilt)

  1. Install Proxmox on tsys2.
  2. Run scripts/check.sh to inventory.
  3. Run scripts/apply-tunings.sh --apply.
  4. Create wnode-tsys2 with boot disk on NVMe (960 PRO) and data disk on SATA SSD (850 EVO). No NFS needed – 1.5 TB local SSD is the most local storage of any worker.
  5. Join to k8s cluster.

8.5 Phase 5: Critical VM relocation

  1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
  2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
  3. (No change to T5-SSD – ultix-streaming stays.)

Open questions for next session

  1. Are the hosts a Proxmox cluster (pvecm) or standalone? This determines
    whether live migration is available (huge simplification) or we need manual
    disk migration. Check pvecm status on each host.
  2. What k8s distribution is in use? (k3s, kubeadm, RKE2?) This affects how
    nodes are joined/drain and how StorageClasses are configured.
  3. Container runtime? (containerd, cri-o?) Affects local storage layout.
  4. Is there a container image registry mirror in the cluster? Or do all
    pulls go to Docker Hub / external? A local registry on D3 SSD would speed
    up all pulls.
  5. What specific ETL tools? (GDAL, PostGIS, xarray, Dask?) This affects
    whether jobs need shared (RWX) or exclusive (RWO) storage.
  6. HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?) Affects
    how we label and taint nodes.
7

Storage dependency: cnodes require writeback cache or SSD (2026-08-06)

The k8s control-plane VMs (cnode1/2/3) depend on fast disk fsync for etcd stability. Current storage is qcow2-on-NFS-on-spinning-HDD. Disk cache MUST be writeback or the cluster will crash-loop. See Storage Architecture advisory for full details.

Status: Writeback cache applied to all 3 cnodes. Cluster boots but etcd heartbeat tuning may be needed for full stability. SSD migration recommended for production reliability.

Refs: Redmine [#391], [#367]