Support coordinated HA certificate rotation

[ankitgoswami]

🤖 Follow-up to #961, which changes the HA root CA and all generated service certificates to a shared 824-day validity so Apple TLS clients accept the Fleet-facing certificate.

Problem

HA installation currently generates one root CA plus the etcd, Patroni, PostgreSQL, and Fleet-facing leaf certificates, but it has no supported way to replace them before expiry. Because #961 gives the whole certificate set the same lifetime, an installation needs coordinated rotation before the 824-day deadline to avoid losing authenticated service connectivity.

The HA service certificates are not the complete trust boundary. Every existing Fleet Node and other external client that connects to the Fleet VIP also trusts the current private HA CA through its host OS trust store. Replacing the root and Fleet-facing leaf without first distributing the new CA causes those clients to fail TLS before sending an HTTP/RPC request; their enrollment remains valid but their heartbeat and ControlStream disappear.

Scope

• Add a supported operator-initiated rotation command or workflow for the HA service CA and every generated leaf certificate.
• Inventory and include external trust consumers, especially already-enrolled Fleet Nodes, in the rotation plan.
• Provide a trust-overlap phase that distributes and verifies the new CA on existing Fleet Node hosts before the VIP begins serving a leaf from it. Preserve the old CA until the new path is proven and rollback is no longer needed.
• Preserve Fleet Node identity, enrollment state, API key, and session state through the trust cutover; CA rotation must not require revocation or re-enrollment.
• Define how long-running Fleet Node processes reload host trust, including any required service restart after installing the new CA.
• Preserve service availability while trust and certificates move across both Fleet hosts, the etcd witness, and external Fleet Node hosts.
• Define the required ordering for distributing trust, replacing leaf certificates, and reloading or restarting etcd, Patroni, PostgreSQL, Fleet, and Fleet Nodes.
• Surface certificate expiry in HA status or another operator-visible preflight so rotation is not dependent on remembering the installation date.
• Document recovery if rotation is interrupted after only part of the cluster or external trust consumers are updated.
• Add automated coverage for the rotation ordering and a deployment-level validation procedure.

Acceptance criteria

• An operator can rotate a healthy HA installation without rebuilding it or manually editing generated secret files.
• The workflow rejects unsafe or incomplete inputs before changing a running node.
• Both Fleet hosts, the witness, the VIP endpoint, and every in-scope Fleet Node validate against the intended CA after rotation.
• Existing enrolled Fleet Nodes resume heartbeats and ControlStreams with their original Fleet Node IDs and without re-enrollment.
• Fleet Node trust is verified before the VIP certificate cutover, or the workflow explicitly stops when an enrolled node cannot be reached or qualified.
• Fleet remains reachable throughout the supported rotation sequence, apart from any explicitly documented bounded restart window.
• Status warns with enough lead time to complete rotation before certificate expiry.

Current manual fallback

Until the supported workflow above exists, rotation is an operator-run maintenance procedure with a planned full-cluster outage. Internal etcd, Patroni, PostgreSQL, and Fleet certificates cannot be rolled one host at a time because the current installation has no old/new CA overlap between HA members.

1. Preflight the current installation. Confirm all three HA members are healthy, the database has one leader and a synchronous standby, the Fleet active/passive roles agree, the VIP is owned by the active host, and all required units and containers are running. Record certificate fingerprints and expiry dates, verify free space, and inventory every external client that trusts the private CA.
2. Generate and validate one coherent certificate set. Use the release's HA certificate generator with the two Fleet-host addresses, witness address, and VIP. Before staging anything, verify every chain, SAN, EKU, key pair, validity period, and the Fleet-facing leaf with Apple TLS validation.
3. Stage external trust before cutover. Distribute the new root CA alongside the old CA to every enrolled Fleet Node and other external trust consumer. Verify the expected fingerprint and confirm each host can use its normal TLS stack. Prepare any service restart required for a long-running process to reload system roots. Stop if any required external client cannot be reached or qualified.
4. Stage and back up the HA hosts. Copy only the role-specific certificate and key files to each host. Back up the complete HA configuration directory and record hashes for passwords, fleet.env, and etcd JWT keys so their preservation can be verified.
5. Stop the cluster and preserve data. Stop HA services on all three hosts, confirm the VIP is withdrawn and containers are stopped, then take cold copies of each HA data directory. Leave the live data directories in place.
6. Install the certificates. Replace only the expected certificate and key files, with root ownership, 0644 for certificates, and 0600 for private keys. Do not replace passwords, fleet.env, etcd JWT keys, or anything under the live data directory. Recheck chains, key pairs, and the preserved-file hashes on each host.
7. Restart the cluster together. Start all three hosts concurrently so etcd can form quorum. Wait for etcd health, then Patroni leader/synchronous-standby health, Fleet active/passive convergence, and correct VIP ownership.
8. Restart external clients and verify identity continuity. Restart or reload Fleet Node services so they load the updated system roots. Verify the VIP's leaf and published CA, then confirm each original Fleet Node ID resumes heartbeats and opens its ControlStream without revocation or re-enrollment.
9. Observe before cleanup. Retain the configuration backups, cold data copies, and old CA in external trust stores through the rollback/observation window. Remove them only after the new certificate path and enrollment flow have been exercised successfully.

Manual rollback

If the HA cluster fails to converge, stop all three hosts, restore the backed-up HA configuration directory, and start all three together. Do not restore the cold database/etcd copies during a certificate-only rollback; use them only for a separately diagnosed data-recovery event. Keep both old and new CAs trusted by external Fleet Nodes during rollback so their existing identity and session can reconnect to either certificate set.

Lab evidence from 2026-08-21

A manual root/leaf rotation on the three-host HA lab preserved PostgreSQL/TimescaleDB, etcd, Fleet credentials, and the existing Fleet Node database/session rows. The previously enrolled stl-lab-proto-fleet node stopped heartbeating at the exact rotation outage (2026-08-21 21:14:56 UTC) and sent no HTTP/RPC requests after the VIP returned with the new private CA. Its enrollment remained CONFIRMED and its session remained valid. A separate macOS Fleet Node reached the same VIP only after the new CA was installed in its host trust store.

After installing the new CA alongside the old CA in the Debian trust store and restarting only fleetnode.service, stl-lab-proto-fleet reconnected as its original Fleet Node ID (1), sent a heartbeat, opened its ControlStream, and resumed receiving commands. No enrollment or state data was changed.

This demonstrates that rotating only the HA hosts is incomplete: external Fleet Node trust distribution, process reload/restart, original-ID recovery, and heartbeat/ControlStream verification must be part of the supported procedure.

Related: #961