Complete Linux System Inventory (2026-08-06)

Canonical source for the complete Linux system inventory. Updated periodically; edit history preserves prior states. Reference this topic for monitoring coverage, access management, and hostname consistency.

Data gathered 2026-08-06 via: Proxmox qm list, tailscale status, DNS zone transfer (AXFR knel.net), SSH access audit.

1. Proxmox Hypervisor Hosts (7 physical)

Host Model CPU RAM Tailscale IP Mgmt IP Datanet IP tuned Access
pfv-tsys1 OptiPlex 9020 8c 31Gi 100.121.189.98 192.168.3.11 10.100.100.1 virtual-host root
pfv-tsys3 Precision 7510 8c 31Gi 100.66.186.10 192.168.2.5 10.100.100.3 virtual-host root
pfv-tsys4 Precision T1700 8c 15Gi 100.70.77.93 192.168.3.251 10.100.100.4 virtual-host root
pfv-tsys5 Precision T7500 8c 94Gi 100.87.135.12 192.168.3.250 10.100.100.5 virtual-host root
pfv-tsys6 PowerEdge R610 16c 125Gi 100.73.35.111 192.168.3.169 (bond) virtual-host root
pfv-tsys7 PowerEdge R620 24c 188Gi 100.110.146.27 192.168.0.250 10.100.100.7 virtual-host root
pfv-tsys9 OptiPlex 7080 12c 23Gi 100.101.158.76 192.168.3.58 10.100.100.9 virtual-host root

2. Production VMs

VMID VM Name Host CPU RAM Tailscale? TS IP Access tuned
100 pfv-bms tsys1 4c 6GB Yes 100.67.108.125 SSH 22222 (LAN: pfv-bms-lan.knel.net) -
101 tsys-ca tsys1 2c 4GB No (DNS: 100.102.96.24) - NO -
102 pfv-k8s-cnode1 tsys1 2c 4GB Yes 100.125.134.53 localuser network-latency
103 pfv-netinfra-01 tsys1 2c 4GB Yes 100.70.181.72 localuser network-latency
104 tsys-librenms tsys1 2c 4GB No (DNS: 100.86.204.77) - NO -
105 tsys-proxmox-datacenter tsys1 2c 2GB No (DNS: 100.125.183.68) - NO -
108 tsys-ucs-01 tsys1 2c 6GB No (DNS: 100.109.13.110) - NO -
313 pfv-k8s-wnode-tsys3 tsys3 8c 28GB Yes 100.126.9.112 localuser -
500 pfv-k8s-wnode-tsys5 tsys5 2c 12GB Yes 100.122.252.116 localuser -
501 devbox-cloudron tsys5 2c 4GB Yes 100.119.72.25 NO -
515 hfnoc-uisp-preprod tsys5 2c 2GB No - NO -
5111 ultix-streaming tsys5 4c 25GB Yes 100.101.187.119 NO (do-not-reboot) -
5112 ultix-offstage tsys5 4c 22GB Yes 100.70.119.59 localuser (do-not-reboot) -
600 tsys-awx tsys6 2c 12GB No (DNS: 100.91.39.53) - NO -
601 pfv-k8s-wnode-tsys6 tsys6 2c 98GB Yes 100.83.49.75 localuser -
602 pfv-rr-middleware-02 tsys6 2c 6GB Yes 100.93.47.4 NO -
603 pfv-k8s-cnode3 tsys6 4c 4GB Yes 100.106.222.18 localuser network-latency
604 tsys-proxmox-mailgw-01 tsys6 2c 4GB No (DNS: 100.68.129.71) - NO -
701 pfv-k8s-wnode-tsys7 tsys7 2c 98GB Yes 100.119.240.11 localuser -
702 hfnoc-uisp tsys7 2c 8GB Yes 100.94.188.89 NO -
703 rr-middleware-01 tsys7 2c 6GB Yes 100.106.54.59 NO -
705 pfv-k8s-cnode2 tsys7 4c 4GB Yes 100.109.34.72 localuser network-latency
706 kali-rd tsys7 2c 6GB Yes 100.105.136.23 NO -
707 tsys-siem tsys7 2c 8GB No (DNS: 100.72.35.113) - NO -
708 kali-tsys tsys7 4c 6GB Yes 100.82.30.115 NO -
709 tsys-voip tsys7 4c 4GB No (DNS: 100.83.126.67) - NO -
710 tsys-umbrel tsys7 2c 26GB No (DNS: 100.66.182.14) - NO -
711 tsys-proxmox-mailgw-02 tsys7 2c 4GB No (DNS: 100.126.29.88) - NO -
902 tsys-ucs-02 tsys9 2c 4GB No (DNS: 100.68.10.17) - NO -
904 pfv-netinfra-02 tsys9 2c 4GB Yes 100.71.171.20 localuser network-latency
905 pfv-k8s-wnode-tsys9 tsys9 4c 14GB Yes 100.95.201.66 localuser -

3. Sectestbed VMs (all on tsys5)

VMID VM Name Tailscale? TS IP Access tuned
5000 sectestbed-sandbox Yes 100.64.20.60 NO -
5101 sectestbed-siem Yes 100.108.121.18 NO -
5102 sectestbed-proxmox-pve Yes 100.80.72.71 NO -
5103 sectestbed-proxmox-datacenter Yes 100.94.1.34 NO -
5104 sectestbed-proxmox-pbs Yes 100.127.238.29 NO -
5105 sectestbed-awx Yes 100.64.56.24 NO -
5106 sectestbed-k8s-cnode Yes 100.68.155.111 NO -
5107 sectestbed-k8s-wnode Yes 100.107.110.89 NO -
5108 sectestbed-librenms Yes 100.92.94.87 NO -
5109 sectestbed-netinfra Yes 100.100.220.117 NO -
51011 sectestbed-cloudron Yes 100.97.140.105 NO -
51012 sectestbed-hfnoc-uisp Yes 100.101.168.80 NO -
51013 sectestbed-rancherplatform Yes 100.88.171.10 NO -
51014 sectestbed-proxmox-mailgw Yes 100.117.24.21 NO -
51015 sectestbed-ca Yes 100.113.245.124 NO -
51016 sectestbed-voip Yes 100.86.176.105 NO -

4. Preprod VMs (all on tsys5)

VMID VM Name Tailscale? TS IP Access tuned
53100 preprod-awx Yes 100.77.216.36 NO -
53101 preprod-siem Yes 100.98.162.14 NO -
53102 preprod-rancherplatform Yes 100.126.231.121 NO -
53103 preprod-proxmoxmailgw Yes 100.114.9.49 NO -
53104 preprod-ca Yes 100.94.119.5 NO -
53105 preprod-proxmox-datacenter Yes 100.101.250.10 NO -
53106 preprod-librenms Yes 100.79.52.34 NO -
53107 preprod-voip Yes 100.109.99.109 NO -
53108 preprod-cloudron Yes 100.95.69.89 NO -

5. Other Physical/SBC Linux Systems (not VMs)

Name Tailscale IP DNS Record Notes
pfvsvrpi 100.91.151.113 pfvsvrpi.knel.net Raspberry Pi
pfv-jetson-nano-1 100.82.230.119 pfv-jetson-nano-1.knel.net NVIDIA Jetson
subopi-dev-3 100.64.231.65 subopi-dev-3.knel.net SBC (Pi?)
subopi-dev-4 - subopi-dev-4.knel.net (100.65.224.85) SBC
subopi3 - subopi3.knel.net (100.93.17.77) SBC
stlpc-artroom 100.120.77.113 stlpc-artroom.knel.net Linux PC
stlpc-garage 100.72.192.22 stlpc-garage.knel.net Linux PC
stlp-3dscanner 100.125.14.37 stlp-3dscanner.knel.net OFFLINE 184d
tailscale-router - tailscale-router.knel.net (100.103.48.57) RETIRED 2026-09-02 — subnet-router role replaced by pfv-netinfra-01/02; host removed from tailnet; no DNS records remain
netbird 100.123.45.23 - NetBird VPN
pfv-proxmox-backup-server 100.114.81.107 pfv-proxmox-backup-server.knel.net PBS
stlpc-bizoffice 100.96.130.53 stlpc-bizoffice.knel.net WINDOWS (not Linux)

6. DNS Records Without Running Systems (stale/orphaned)

CLEANED 2026-09-03 (CMDB reconcile, #705). Every row previously listed
here was re-verified against live DNS (Technitium via dns-cli),
tailscale status, and qm/pct lists on all 7 PVE hosts. Result: no
stale DNS records remain
— the zone was already clean; this section
itself was the stale artifact. Findings:

  • pfv-tsys2, knelai-highassurance, knelai-prod, cnw-worker, fastdata,
    mpi-01..04, peertube-01..04, bare ultix, tailscale-router: no DNS
    records exist anymore
    (tailscale-router was removed with its
    2026-09-02 retirement).
  • tsys-cloudron (100.107.35.78): ALIVE — live tailnet node, NOT a
    PVE guest on tsys1-9 (off-cluster, likely the Reston VPS; owner:
    Cloudron lane). Record correct; the old row was wrong to list it.
  • ultix-field (100.115.233.124) and ultix-highside (100.109.70.137):
    ALIVE on the tailnet. Records correct; rows removed.
  • All 5 ultix-* records (field/highside/mini/sidecar/streaming) match
    live tailnet nodes.

7. Hostname Discrepancies (VM name vs Tailscale name vs DNS name)

VM Name Tailscale Name DNS Name Issue
rr-middleware-01 (703) pfv-rr-middleware-01 pfv-rr-middleware-01.knel.net VM missing pfv- prefix
rr-middleware-02 (602) pfv-rr-middleware-02 pfv-rr-middleware-02.knel.net VM missing pfv- prefix
preprod-proxmoxmailgw (53103) preprod-proxmox-mailgw preprod-proxmox-mailgw.knel.net VM missing hyphen in mailgw

8. SSH Access Summary

Category Access Count
Proxmox hosts root only 7
K8s nodes (cnodes+wnodes) localuser only 8
Netinfra VMs localuser only 2
ultix-offstage localuser only 1
Total accessible 18
Production VMs on Tailscale (no key) NO ACCESS 9
Production VMs NOT on Tailscale NO ACCESS 11
Sectestbed VMs NO ACCESS 16
Preprod VMs NO ACCESS 9
Other systems NO ACCESS 7+
Total inaccessible 52+

Systems on Tailscale that need SSH key pushed (8):
devbox-cloudron, hfnoc-uisp, kali-rd, kali-tsys, pfv-rr-middleware-01, pfv-rr-middleware-02, pfv-k8s-wnode-tsys3/5/6/7/9 (wnodes are accessible via localuser but could also use root)

Production VMs NOT on Tailscale (11 — need Tailscale installed):
tsys-ca, tsys-librenms, tsys-proxmox-datacenter, tsys-ucs-01, tsys-awx, tsys-proxmox-mailgw-01, tsys-proxmox-mailgw-02, tsys-siem, tsys-voip, tsys-umbrel, tsys-ucs-02

Update: Stale DNS records cleaned up (2026-08-06)

15 retired system DNS records deleted from knel.net zone via Technitium API:

  • pfv-tsys2, knelai-highassurance, knelai-prod, mpi-01 through mpi-04, peertube-01 through peertube-04, cnw-worker, fastdata, ultix, ultix-field

Verified via direct Technitium query — all return NXDOMAIN.

DNS CLI tool created at ~/daytoday/dns/bin/dns for future DNS lifecycle operations.

Note: tsys-cloudron is OUT OF SCOPE for PFVCluster project (Reston VA production). ultix-highside is a Windows system — DNS record kept. ultix-field is at SITES, not PFV — record deleted.

Redmine: [#398] (closed)

Update: Name + access reconciliation (2026-08-07, onsite)

Section 7 (hostname discrepancies) — RESOLVED / corrected:

  • VMID 703/602 (rr-middleware) were already correctly pfv-rr-middleware-01/02 — the prior entry was stale.
  • VMID 53103 renamed → preprod-proxmox-mailgw (was missing hyphen).
  • VMID 515 renamed → preprod-hfnoc-uisp (was hfnoc-uisp-preprod, word order swapped).
  • preprod-hfnoc-uisp (100.77.14.7) should be in the Preprod table (was missing).
  • VMID 712 DellOpenManageEnterprise on tsys7 — undocumented VM (related to #339). No Tailscale/DNS yet.

Section 5 correction: pfv-proxmox-backup-server is a VM (VMID 400) on tsys4, not a physical system.

PTR (#333, 85%): 10 PTRs fixed via Technitium. 63/79 Tailscale nodes fully consistent. Outstanding: ultix-highside (needs reverse zone), ultix-mini/sidecar/netbird forward records.

Access (section 8): workstation SSH key now on all 16 guest-agent-OK VMs (root via qm guest exec). 14 VMs remain GA-NO (need in-guest qemu-guest-agent install/start — console or reboot required).

DNS conflict flagged: subodev-torsw01.knel.net and pfv-sw-02.knel.net both resolve 192.168.0.8 (two distinct switches) — needs onsite resolution.

Correction (2026-08-26) [#446]: VM 51012’s Tailscale/OS hostname is now sectestbed-hfnoc-uisp (was drifted to sectestbed-hfnoc); guest was empty, rename aligned it with VM + DNS. These are two distinct planned systems — the HFNOC GIS/app sectestbed (sectestbed-hfnoc) does not exist yet (proposal: Redmine #448). Also: duplicate DNS record ultixfield.knel.net deleted (canonical ultix-field.knel.net); tsys-cloudron-new (Reston VPS) removed from audit scopes. Full audit: Fleet Audit: Linux Access + Naming — 2026-08-26 [#446]

Wiki delta 2026-08-27 (plant session) — promote into the tables above on next full inventory rewrite:

Proxmox fleet = 7 hosts (pfv-tsys1/3/4/5/6/7/9) — all now SNMP-instrumented (lmsensors-extend + snmpd, v2c kn3lmgmt, explicit LAN+TS binds, source-scoped ACLs; see dcinfra/sensors/temper/ + dcinfra/ha/). pfv-tsys8: retired/confirmed gone (stale refs only in returned-logs captures; its old OOB reservation MAC was core-switch OUI junk, corrected 2026-08-27).

OOB / iDRAC (new section needed):

Name IP Device Status (2026-08-27)
pfv-tsys6-oob.knel.net 192.168.3.196 R610 iDRAC6 fw1.41, MAC 00:21:9b:a2:7c:5b LIVE (DHCP lease, Kuma green)
pfv-tsys7-oob.knel.net 192.168.3.197 R620 iDRAC6 fw2.0f, MAC f8:bc:12:35:1e:c6 Dark — awaiting physical cable (#460)

DRAC SNMP agent on both needs racadm/OMSA enable (#462, priority raised); community kn3lmgmt already set in-band.

Not yet instrumented: subopi* Pi fleet (#350/#351 onboarding), pfv-jetson-nano-1 (DHCP reservation 192.168.3.186 exists; on-net status unverified).

Correction 2026-09-03: the worker list ‘pfv-k8s-wnode-tsys3/5/6/7/9’ is stale — there is NO wnode on pfv-tsys5 (verified: no such VM; no DNS record). Actual k8s workers: tsys3/6/7/9 (DNS records all Tailscale IPs). Also noted during tonight’s tsys3 incident: fleet iDRAC/BMC has no DNS records or addresses anywhere — the OOB section below needs real data (R610/R620 MACs already listed).

Addendum to my correction: per founder, retired pfv-k8s-wnode5’s replacement in the cluster is ULTIX-STREAMING (VM 5111, pfv-tsys5 — the workstation VM; k3s-agent verified active 2026-09-03). Worker fleet: ultix-streaming + pfv-k8s-wnode-tsys3/6/7/9.

OOB correction (2026-09-03, founder): pfv-tsys6-oob.knel.net (192.168.3.196) and pfv-tsys7-oob.knel.net (192.168.3.197) DO exist - verified resolving + pinging. My earlier ‘no OOB names’ statement was wrong for 6/7. Confirmed status: tsys3 = Precision laptop, NO BMC by design (no -oob ever). tsys1/4/5/9 iDRACs (R610/R620, MACs below) exist but are unnamed - candidates for a pfv-tsysN-oob DNS convention. Also fleet-fixed today: snmpd boot-race (systemd drop-in, network-online + Restart=always) on 10 hosts; subopi3/dev-3/dev-4 pending SSH access (#733).

Hardware-class correction (2026-09-03, founder): ONLY pfv-tsys6 and pfv-tsys7 are PowerEdge (hence their -oob iDRAC names, verified up). pfv-tsys1/9 are Optiplex, pfv-tsys3/4/5 are Precision - NONE of those five have a BMC/iDRAC, so the R610/R620 iDRAC6 MACs listed in this topic belong to OTHER machines (unmapped). OOB coverage is therefore COMPLETE as-built: 2/7 hosts have remote power (6/7); the other 5 are BMC-less by hardware class (tsys3 is the Precision 7510 laptop). If remote power recovery is ever wanted for the BMC-less five, the fleet pattern is smart-plug/PDU-relay on their power feeds - noted as an option, not proposed for action.