Data Center Infrastructure

Canonical source: This wiki topic is the living reference. Updated in place; edit history preserves all prior states.

Covers: PDU management (powerman), serial console (ser2net + conman), UPS management (NUT).


PDU Management (powerman)

Redmine: #359 (deployment, closed) · #374 (outlet renaming, open)

Centralized power management for the Cyclades AlterPath PM10i PDU via
Powerman, running on pfv-tsys1.

Hardware

Component Details
PDU Cyclades AlterPath PM10i (10 controllable AC outlets)
Firmware v1.9.0 (Aug 4, 2006)
Connection USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07)
Host pfv-tsys1 (OptiPlex 9020, Proxmox)
Serial 9600 baud, 8N1, raw mode
Credentials Factory defaults: admin / pm8 (in cyclades-pm10.dev)
Network access powermand listens on 127.0.0.1:10101 (local) + 100.121.189.98:10101 (Tailscale)

Device mapping

USB adapter (067b:23a3, serial BJAAb144J07)
  └─ pl2303 driver → /dev/ttyUSB1
       └─ udev symlink → /dev/cyclades-pm10 (stable across reboots)
            └─ powermand reads/writes serial → Cyclades PM10i
                 └─ 10 outlets (factory default names: 1-10)

The udev rule (/etc/udev/rules.d/99-cyclades-pdu.rules) pins the adapter
by its USB serial number, so the symlink survives replugs and reboots.

Scripts

All scripts run on the target host (pfv-tsys1) via tests/remote.sh:

# Setup (idempotent — safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh

# Validate PDU control (cycles outlet 10 off → on):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh

# Status check:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh

Customizing for other hosts/PDUs

The setup script accepts environment overrides:

PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \
  PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh

Usage (daily operations)

From pfv-tsys1 (or any host with network access to port 10101):

# List all outlets
powerman -l

# Query status (all outlets)
powerman -q

# Turn outlet off
powerman -0 outlet-10

# Turn outlet on
powerman -1 outlet-10

# Cycle outlet (off → 4s delay → on)
powerman -c outlet-10

# Query a specific outlet
powerman -q outlet-10

Remote access from other hosts

powermand listens on 0.0.0.0:10101. From another tailnet host:

powerman --server-host pfv-tsys1 --server-port 10101 -q

Or set POWERMAN_SERVER=pfv-tsys1:10101 in the environment.

Configuration files on pfv-tsys1

File Purpose
/etc/udev/rules.d/99-cyclades-pdu.rules Stable symlink for USB-DB9 adapter
/etc/powerman/powerman.conf Device definition + 10 outlet nodes
/etc/powerman/cyclades-pm10.dev Cyclades PM10 protocol spec (shipped with powerman)

Validation results

2026-07-28: All 8 checks passed.
Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled.

TODO (Friday onsite)

  • Rename outlets in /etc/powerman/powerman.conf to match the
    physical devices plugged into each outlet (e.g., node "tsys4-psu" "cyclades-pm10" "3"). Currently all outlets are generically named
    outlet-1 through outlet-10.
  • Change PDU admin password from factory default (pm8) if
    security-sensitive. Update /etc/powerman/cyclades-pm10.dev login
    script to match.
  • Verify all 10 outlets individually once device mapping is known.

Serial Console (ser2net + conman)

Redmine: #360 (deployment, closed) · #373 (pfv-r2-sw fix, open)

Network-accessible serial console management for all production network
switches and routers, running on pfv-tsys4 (storage server).

Architecture

USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)

ser2net owns the physical serial devices and exposes them on TCP ports
using the telnet(rfc2217) protocol bound to the Tailscale interface
only
(100.70.77.93:200X). conman connects to those TCP ports via
telnet for session logging, output capture, and multi-user console
sharing.

Why telnet(rfc2217)? The serial devices send `

(LF+CR) line endings instead of standard
`. Raw TCP transport caused conman’s
telnet NVT to strip bare CR characters, producing stair-stepped output.
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
translation is handled correctly by the telnet layer.

conman and ser2net do NOT share ports — only one process can open a
serial device at a time. ser2net owns the physical device; conman connects
over TCP.

The USB Enumeration Problem (SOLVED)

The 9 Prolific USB-to-DB9 adapters (067b:2303) on pfv-tsys4 have no
unique USB serial numbers
and get assigned /dev/ttyUSB0-8 based on
enumeration order, which shifts on every boot. This made the old
/root/conmap + manual screen workflow break after every reboot.

Fix: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/ that never changes.

The udev rules are generated from mapping.txt, which maps each adapter’s
ID_PATH to a console name and TCP port. To re-map after physically moving
an adapter, update mapping.txt and re-run setup.sh.

Fallback: if udev trigger doesn’t create symlinks for already-discovered
devices (common on first run), setup.sh creates them manually by matching
ID_PATH. On subsequent boots, udev creates them automatically.

Port Assignments

TCP Port Console Name ID_PATH Description
2001 pfv-core-sw01 usb-0:1.5.4.4 Dell PowerConnect 5448 (core switch)
2002 pfv-tor3-mgmt usb-0:1.6.3.1 Rack 3 management TOR switch
2003 pfv-tor3-stor usb-0:1.6.3.3.2 Rack 3 storage TOR switch
2004 pfv-rrinfra-rtr usb-0:1.6.3.3.1 Cisco router (rrinfra)
2005 pfv-r2-tor-top usb-0:1.6.3.3.3 Rack 2 top-of-rack switch
2006 subodev-torsw usb-0:1.5.4.1 Suborbital device TOR switch
2007 pfv-r2-sw usb-0:1.6.3.2 Rack 2 old Dell switch

All ports listen on the Tailscale IP (100.70.77.93) using telnet(rfc2217).

Scripts

Script Purpose
mapping.txt Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud
generate-config.sh Generates udev rules, ser2net.yaml, conman.conf from mapping.txt
setup.sh Full deploy: generate configs, create symlinks, restart services
discover.sh Read-only discovery of USB adapters, existing config, services

Usage

Connect to a console

Primary method — conman client (with logging + multiplexing):

# From any Tailscale-connected workstation:
conman -d pfv-tsys4:7890 -f pfv-core-sw01     # connect to console
conman -d pfv-tsys4:7890 -q                    # list all consoles

Escape sequence: &. to disconnect, &? for help.

Direct telnet (emergency only — conflicts with conman):

# Direct telnet to ser2net works ONLY when conmand is stopped, because
# conmand maintains persistent connections to all 7 TCP ports. Use:
ssh pfv-tsys4 'systemctl stop conmand'
telnet pfv-tsys4 2001                          # pfv-core-sw01
ssh pfv-tsys4 'systemctl start conmand'        # restart when done

Do NOT use telnet while conmand is running — conmand will reconnect
and kick your telnet session immediately (“Connection closed by foreign host”).
The correct workflow is conman client → conmand → ser2net → device.

Re-deploy after changing mapping.txt

PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'

Find the ID_PATH for a new adapter

PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh

Then match the new adapter’s ID_PATH to its physical location and add a line
to mapping.txt.

Files on pfv-tsys4

File Purpose
/etc/udev/rules.d/99-console-ports.rules Stable symlinks by ID_PATH
/etc/ser2net.yaml ser2net config (telnet rfc2217 TCP ports → serial symlinks)
/etc/conman.conf conman config (CONSOLE entries between markers)
/etc/systemd/system/conmand.service systemd unit for conmand
/root/console/mapping.txt Copy of the source-of-truth mapping
/root/console/setup.sh Setup script (re-runnable)
/root/console/generate-config.sh Config generator

Old workflow (replaced)

The old /root/conmap file and manual screen sessions are no longer
needed. The new setup is fully automated and survives reboots.


UPS Management (NUT)

Redmine: #340 (APC, closed) · #372 (Tripp Lite, open)

Centralized UPS monitoring for the server room via
NUT, running on pfv-tsys1. USB HID UPS
units feed one upsd network server; Home Assistant polls it over Tailscale for
real-time power/load/runtime tracking, and a local upsmon shuts the hypervisor
down gracefully when battery is low.

Why NUT (not apcupsd)? Two different UPS brands (APC + Tripp Lite) must be
covered. apcupsd only supports APC, so it would require a second daemon
stack. NUT’s usbhid-ups driver speaks to both via the USB HID Power
Device class, and Home Assistant ships a first-class NUT integration.

Hardware

UPS Model VID:PID USB Serial Status
APC Smart-UPS C 1500 (FW 02.2) 051d:0003 AS1213210423 LIVE
Tripp Lite UPS (HID PDC) 09ae:3016 2352CVLSM871900694 Blocked — see below

Current State (2026-07-30)

APC Smart-UPS C 1500 — OPERATIONAL

Fully reporting via usbhid-ups + APC HID 0.100 subdriver. Data validated:

battery.charge: 100         battery.runtime: 1800     battery.voltage: 27.4
ups.status: OL              ups.load: (via HA)         ups.model: Smart-UPS C 1500

Tripp Lite UPS — BLOCKED (hardware issue)

The driver finds the device, matches the TrippLite HID 0.85 subdriver, claims
the interface, and reads the HID descriptor — but fails reading the 878-byte
HID Report Descriptor
(Resource temporarily unavailable / EAGAIN after 5s).
The driver is masked to prevent restart-loop spam.

USB descriptors (manufacturer, product, serial) are readable via lsusb -v and
nut-scanner, but the bulk control transfer for the full report descriptor
times out. Likely causes:

  1. USB hub — the Tripp Lite is behind a Genesys Logic hub (05e3:0608).
    Try plugging directly into a motherboard USB port.
  2. USB cable — try a high-quality data cable (not charge-only).
  3. UPS firmware — the USB controller may not properly implement all HID
    endpoints.

To retry after physical reseat:

# On pfv-tsys1:
systemctl unmask nut-driver@tripp-lite-ups
systemctl start nut-driver@tripp-lite-ups
upsc tripp-lite-ups@localhost

Architecture

pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
 ├─ APC Smart-UPS C 1500      ──┐
 └─ Tripp Lite UPS (masked)  ──┤  USB HID
                                ▼
                       nut-driver@apc-smartups-c1500 (usbhid-ups)
                                ▼
          upsd  :3493  (LISTEN 127.0.0.1 + Tailscale + LAN)
                                ▼            ▼
                  upsmon (local)        Home Assistant (NUT integration)
                  graceful shutdown      via LAN 192.168.3.11 (HAOS can't
                                         route to Tailscale IPs)
  • Driver layerusbhid-ups process, pinned by USB serial. Debian uses
    templated nut-driver@<upsname>.service units managed by
    nut-driver-enumerator.
  • Server layerupsd exposes UPS data on TCP 3493 (localhost + Tailscale
    • LAN). Clients authenticate via upsd.users.
  • Monitor layerupsmon runs locally as master to trigger
    SHUTDOWNCMD (/sbin/shutdown -h now) when a UPS reports LOWBATT.
  • Home Assistant — native NUT integration connects to upsd over Tailscale
    and exposes ups.load, battery.runtime, ups.status, etc. as sensors.

Key deployment lesson: udev must cover raw USB devices

The usbhid-ups driver opens /dev/bus/usb/BBB/DDD (raw USB device files),
not /dev/hidraw*. After calling setuid(111) to drop to the nut user,
it needs write access to those raw USB files. The udev rule must match
SUBSYSTEM=="usb" by vendor/product ID to set GROUP="nut" — matching only
hidraw is insufficient. See /etc/udev/rules.d/99-nut-ups.rules.

Scripts

NUT host scripts run on pfv-tsys1 via tests/remote.sh:

# Idempotent install + configure (safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh

# Discover USB UPS + NUT state (read-only diagnostic):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh

# Query UPS data + service health:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh

The HA integration script runs from your workstation (needs HA API access):

# Add the NUT integration to Home Assistant (idempotent):
bash ups/setup-ha-nut.sh

setup.sh accepts environment overrides for serials/VIDs/PIDs/usernames, so it
can be repurposed for other hosts or UPS units. Passwords for monuser and
homeassistant are auto-generated on first run and reused on subsequent runs
(stored in /etc/nut/upsd.users).

Set TRIPP_ENABLED=0 to skip the Tripp Lite entirely (useful if it’s physically
unplugged).

Configuration files on pfv-tsys1

File Purpose
/etc/udev/rules.d/99-nut-ups.rules Grant nut group rw on raw USB + hidraw devices (both subsystems)
/etc/nut/ups.conf usbhid-ups device(s), pinned by serial + subdriver
/etc/nut/upsd.conf LISTEN 127.0.0.1 + LISTEN <tailscale> + LISTEN <lan> on port 3493
/etc/nut/upsd.users monuser (master) + homeassistant (read-only) credentials
/etc/nut/upsmon.conf Local master monitor + SHUTDOWNCMD
/etc/nut/nut.conf MODE=netserver

Home Assistant integration

The NUT integration is added automatically by setup-ha-nut.sh, which drives
HA’s REST config-flow API. It is idempotent (skips if the entry exists).

# Prerequisites: create token + password files (one-time):
mkdir -p ~/.config/pfvcluster
# HA → Profile → Long-Lived Access Tokens → Create Token:
echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token
# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user):
echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password
chmod 600 ~/.config/pfvcluster/{ha-token,nut-password}

# Run:
bash ups/setup-ha-nut.sh

Why LAN IP, not Tailscale

upsd listens on both the Tailscale IP (100.121.189.98) and the LAN IP
(192.168.3.11). The HA NUT integration uses the LAN IP because HAOS runs
Tailscale as an isolated add-on container — the HA core container cannot route
to Tailscale IPs. Since pfv-bms (HA, 192.168.3.12) and pfv-tsys1 (192.168.3.11)
share the same vmbr0 bridge, LAN connectivity is instant and reliable.

Manual UI alternative

In Home Assistant → Settings → Devices & Services → Add Integration → NUT:

Field Value
Host 192.168.3.11 (LAN — HAOS can’t reach Tailscale IPs from the HA container)
Port 3493
Username homeassistant
Password (stored in /etc/nut/upsd.users on pfv-tsys1)
UPS apc-smartups-c1500

Live sensors

HA exposes UPS data as sensors (prefix sensor.apc_smartups_c1500_):
battery_charge, status (Online/On Battery), status_data (OL/OB/DISCHRG).
Additional sensors (load, runtime, voltage) populate as the UPS reports them.

Daily operations

From pfv-tsys1 (or any tailnet host with NUT client installed):

# List UPS units served by upsd
upsc -l pfv-tsys1

# Full variable dump for one UPS
upsc apc-smartups-c1500@pfv-tsys1

# Battery runtime (the only runtime/charge data this UPS exposes)
upsc apc-smartups-c1500@pfv-tsys1 battery.runtime

Notes

  • No USB passthrough to the HA VM. Keeping the UPS on the host preserves
    hypervisor graceful-shutdown capability and matches the powerman/ pattern
    (PDU managed on the host where the adapter physically lives).
  • No ups.load / ups.realpower on this UPS (FW 02.2, mfg 2012): The
    APC Smart-UPS C 1500 does not expose load or power data over USB HID.
    Both NUT usbhid-ups and apcupsd (USB mode, tested 2026-07-30) read the
    same HID descriptor — the variable simply isn’t there. This means the HA
    NUT integration provides battery/runtime/status sensors only, not
    wattage for the Energy Dashboard.
    • apcupsd test note: Debian’s apcupsd package conflicts with
      nut-server (mutually exclusive). apcupsd USB mode returned COMMLOST
      even before we could check load. The APC Smart Serial protocol (serial
      cable, AP940-1524C, ~$30) DOES report load%, but this requires a serial
      port on the UPS and on the host.
    • Energy Dashboard path: A smart plug (Shelly Plug S / TP-Link Kasa,
      ~$15-25) on the UPS output reports real watts natively and feeds the
      Energy Dashboard with zero UPS-driver hacking. The NUT sensors remain
      valuable for outage detection and graceful-shutdown automations.