cluster: encrypt and authenticate controller to replica connections

[jasonhernandez]

Motivation

PR 3 of 4. Turns #37995 (CTP mutual TLS support) and #37996 (internal secrets) into a working feature: controller↔replica traffic can be encrypted, with both ends authenticating each other.

Today CTP is plaintext bincode with no authentication. It carries user rows (peek results, SUBSCRIBE batches), and because a CTP server serves one client and cancels the incumbent on any new connect, anyone who can reach a replica's *ctl port can evict the controller and take over the command stream. The trust model is "the pod network is isolated," which stops holding once an operator can mirror ENI traffic.

Off by default everywhere, including in tests. Enabling it in the test and CI configurations is #37999, split out so that risky flip is independently revertable. The two are intended to land together.

Stacked on #37996 (which is stacked on #37995). Review those first; this branch merges both.

How it works

• environmentd, behind --cluster-transport-tls, bootstraps a per-environment CA into an internal secret at startup and mints its own controller certificate.
• The cluster controller mints a certificate per replica at provisioning time, writes it to the replica's internal secret, and passes --ctp-tls-secret=<name> to clusterd.
• clusterd reads the credentials through the SecretsReader it already links and requires mutual TLS on both controller listeners.
• Dropping a replica deletes its secret, and so does cleaning up replicas left behind by a past deploy generation.

Identity is the replica's service name in the certificate SAN. The controller verifies it reached the intended replica; the replica verifies the peer holds a controller certificate from the same environment. Distribution rides the machinery that already exists for user secrets: no cert-manager, no operator changes, no new RBAC.

No shell required, by construction

This is meant to hold up the other half of the distroless bargain, so the credential lifecycle never needs an operator inside a container:

• Nothing out of process. CA generation, issuance, and loading are all in-process (rcgen + the secrets reader). No init containers, no sidecars, no openssl CLI, no scripts — including for rotation.
• Rotation rides process lifecycle. Replica certificates are re-minted on every replica process creation, so there is no fleet-wide renewal operation. The 5-year validity is a backstop, not the rotation mechanism.
• Self-healing. A missing or corrupt credential secret is repaired by the controller re-ensuring it; clusterd retries the read at boot rather than crash-looping, since the write races process creation.
• Diagnosable from outside. A per-server mz_cluster_server_tls_handshake_failures_total counter, an mz_ctp_tls_cert_not_after gauge, and handshake-failure logs naming the peer and reason. kubectl logs plus scraping is enough — no openssl s_client needed.

Behavior changes

• Displacement now happens after authentication. A new client displaces the established one only once it has passed transport security, so with TLS on, an unauthenticated peer reaching the port cannot interrupt the live connection. The handshake runs in its own task bounded at 30s, so a stalled peer blocks neither the accept loop nor the live connection. Without TLS, displacement is unchanged (accept-time). A turmoil test covers the new property.
• CTP now flushes after each message. A raw socket write reaches the peer on its own, but a TLS writer buffers plaintext into records and need not emit them until flushed. Without this, messages (including the CTP handshake) strand indefinitely. Found by running the real process orchestrator; the turmoil socket masks it.
• Unmanaged replicas connect in plaintext. They run outside the orchestrator, so there is no channel for distributing credentials.

Credential cleanup

Past-generation replica cleanup (try_remove_past_generation_replicas) drops services directly rather than going through deprovision_replica, so an earlier revision of this branch leaked one credential secret per replica per deploy generation — unbounded growth across upgrades. It now deletes credentials too, and because that path is already async it awaits the deletion, so a transient failure propagates into the existing retry loop instead of being logged and dropped.

The DROP CLUSTER path (deprovision_replica) is synchronous and so still spawns the deletion and logs failures, matching the existing posture for user secrets, where orphaned-secret GC is already noted as future work. Worth a reviewer's opinion on whether that is good enough or should be made reliable too.

Testing

• New turmoil test asserting an unauthenticated peer cannot displace an authenticated connection, on top of the nine TLS tests from service: add mutual TLS support to the cluster transport protocol #37995 (23 tests in the CTP suite, run 15+ times with random seeds, no flakes).
• Verified end to end against the real process orchestrator: all 8 CTP connections (4 replicas × storage/compute) completed mutual TLS handshakes and builtin catalog dataflows were created over them. This is what surfaced the missing flush.
• Caveat, stated plainly: I could not get a full mz-environmentd integration test to pass locally — test_empty_subscribe_notice times out at 240s. A/B'ing it with cluster_tls forced to None times out identically, so the timeout is environmental (local debug build against a containerized CockroachDB), not caused by this change. Because TLS is off by default here, this PR's CI does not exercise the TLS path at all; cluster: enable cluster transport TLS in the test and CI configurations #37999 is where that gets tested.
• Performance is unmeasured — no feature benchmark was run. Called out on cluster: enable cluster transport TLS in the test and CI configurations #37999, which is where it would show up.

🤖 Generated with Claude Code