Fleet-wide mail audit 2026-09-03 + founder design rulings. Ticket: #696 (child of umbrella #694). Tooling: netinfra/mail/ in KNEL/PFVCluster (audit-mail-env.sh + run-mail-audit.sh + audit-targets.txt); raw per-host probe output lives in workspace scratch .crush/audit/mail/ (not committed).
FOUNDER RULING (2026-09-03): OUTBOUND ONLY. All PFV hosts go via this gateway; no inbound mail will ever flow through these systems. The original “MX cut-over / inbound DNAT” scope of #696 is DROPPED. Expectation: lots of mail from k8s etc. — the gateway must be very robust (VIP pair, queueing, monitoring).
Backend
- Production mail lives on Cloudron, VPS in Reston VA:
mail.knownelement.com=152.53.37.179(SMTP :25). - Public MX for the mail domains points there (founder DNS at OVH/Cloudron). The internal
knel.netzone has NO MX records — and with outbound-only, it never needs one.
Current outbound flow (audit-verified)
Every MTA-bearing system sends DIRECT to public MX — postfix relayhost is EMPTY fleet-wide (zero exceptions found). Remote-domain delivery resolves public DNS and connects straight to the Cloudron VPS:
pfv-tsys1 → to=<coo@turnsys.com> relay=mail.knownelement.com[152.53.37.179]:25 status=sent (2026-09-03)
- Postfix present (16 systems): all 7 PVE hosts, pfv-proxmox-backup-server, tsys-ucs-01/02, tsys-proxmox-datacenter (prod + sectestbed), sectestbed-proxmox-pve, sectestbed-proxmox-pbs, and the 3 PMG installs (below).
- Active senders (log evidence): pfv-tsys1 (root/cron mail → coo@turnsys.com); the three PMG boxes’ daily 00:01 reports → Cloudron. Other hosts hold postfix but show no sends in retained logs.
- No ssmtp/msmtp/nullmailer/dma anywhere; no stale relayhosts found.
- Local root mail: 13 PVE/PBS hosts chain via
/usr/libexec/proxmox-mail-forward; the PMG prod pair forwards to coo@turnsys.com; tsys-ucs-02’s alias targetssystemmail@unassigned-hostname...(blackhole — fix rides along with cutover). - App-level SMTP: Wazuh alerting still on stock example config (inactive); LibreNMS none configured; HA (pfv-bms) notify = verify in the pfv-bms lane. Cloudron-internal apps use Cloudron’s own MTA (out of PMG scope).
- k8s nodes have no postfix today — future k8s mail (apps, alerting) will target the gateway directly (see ports below).
PMG estate (already installed)
| System | VMID / host | LAN IP (DHCP-reserved) | TS name | PMG | State |
|---|---|---|---|---|---|
| tsys-proxmox-mailgw-01 | 604 / tsys6 | 192.168.1.11 | 100.68.129.71 | 9.1.0 | running, onboot, UNCLUSTERED, default config |
| tsys-proxmox-mailgw-02 | 711 / tsys7 | 192.168.1.10 | 100.126.29.88 | 9.1.0 | running, onboot, UNCLUSTERED, default config |
| sectestbed-proxmox-mailgw | 51014 / tsys5 | — | 100.117.24.21 | 9.x | dev lane |
| preprod-proxmox-mailgw | 53103 / tsys5 | — | 100.114.9.49 | 9.x | preprod lane |
Deploy note: both prod VMs currently have firewall=1 on vmbr0 — verify the PVE firewall ruleset for these VMs (or align with fleet convention) during bring-up so LAN SMTP is not blocked.
DHCP / VIP slot
- Single shared subnet 192.168.0.0/22, gateway .3.254; dynamic pool 192.168.0.1–192.168.3.200; ~60 static reservations (both PMG nodes already reserved at .1.11 / .1.10).
- Above pool end: .201–.248 unreserved; infra block taken at .250 (tsys5), .251 (tsys4), .252/.253 (netinfra-01/02 DNS+NTP), .254 (gateway).
- Proposal: VIP = 192.168.3.249, keepalived VRRP on the PMG pair (VRID 10, VRRP MAC 00:00:5e:00:01:0a), pinned with a dhcpd host declaration so nothing else can ever claim it. DNS: A record
smtp.knel.net→ 192.168.3.249 (LAN senders; the knel.net zone already carries LAN records like the-lanaliases).
Outbound-only architecture (target state)
[all PFV hosts, k8s nodes/pods] --smtp--> smtp.knel.net (VIP .3.249)
--> PMG-01/02 (clustered, filter+queue) --submission--> Cloudron (mail.knownelement.com) --> internet MXs
- Fleet → gateway: unauthenticated :25 from trusted LAN sources (PMG trusted-networks = LAN /22; k8s pod traffic SNATs to node IPs, so covered). Optionally :587 with auth later for stricter apps — open question.
- Gateway → Cloudron (recommended): authenticated submission to a dedicated Cloudron relay user, so Cloudron relays fleet mail with its proper rDNS/reputation. Requires founder/Cloudron-side: create relay user (or allowlist the PMG LAN IPs at Cloudron’s discretion). Alternative if refused: PMG delivers direct to recipient MXs (same egress as today, no deliverability gain).
- Robustness: two nodes on different physical hosts + VRRP VIP; postfix queues on PMG if Cloudron is unreachable (32G disk headroom); PMG native cluster keeps config + quarantine in sync; queue-length + service monitoring lands with the monitoring stack after cutover.
- No inbound: no DNAT, no MX changes, no internet-facing :25 anywhere at PFV.
Deploy plan (blue/green, serial, health-gated — #696)
- Prep (founder go): dhcpd VIP declaration + Technitium
smtp.knel.netA record (serial across the netinfra pair, drift-check + snapshot sync per DNS change discipline). - Cluster the pair: PMG native cluster (config + quarantine replication), managed strictly serial.
- Transport: outbound-only policy; smarthost = authenticated submission to Cloudron relay user (founder-side prerequisite); TLS via fleet CA (#697) for admin UIs; alerts → Charles’s address.
- Outbound cutover: set
relayhost = [smtp.knel.net]on the 16 MTA systems via AWX (codified per #454), staged serially with per-host verification in PMG logs + Cloudron receipt; ucs-02 alias fix rides along. - Verify + monitor: end-to-end sends from every MTA host and one k8s node; queue/service monitoring; ticket notes + this topic kept in lockstep.
Gaps
- 9 systems unreachable for the audit (no ssh path): rr-middleware-01, hfnoc-uisp-preprod, stlpc-artroom, stlpc-garage, subopi-dev-3/4, netbird, sectestbed-sandbox. None are known mail senders, but they are unaudited.
Cross-links: Redmine #696 · repo KNEL/PFVCluster (netinfra/mail/)