Dev multi-worker Trino benchmark lifecycle (fail-closed)

[bill-ph]

Stacked on #1048 (codex/trino-ducklake-smoke). Review/merge that first; this PR's diff is only the dev multi-worker lifecycle.

What

Compare one Duckgres worker over PGWire against a multi-worker Trino cluster reading the same per-run DuckLake snapshot, driven by the posthog_frozen_trino_perf scenario. #1048 established the pinned Trino + Brikk connector locally; this makes it a real dev-cluster comparison.

Control plane

• Lifecycle API on the existing admin-authenticated /api/v1 router: provision / status / deprovision. Responses carry only cluster ID, state, endpoint, requested/ready worker counts, and the pinned image. 202 create vs 200 idempotent repeat, 409 conflicting ownership/config, 503 disabled-or-unconfigured. Error bodies are fixed strings; infrastructure detail is logged, never returned.
• Kubernetes lifecycle manager: one coordinator, exactly the requested workers (default 4), a ClusterIP Service selecting only the coordinator, coordinator/worker/catalog ConfigMaps, and a short-lived Secret holding only the charts-created metadata reader password. requests == limits (Guaranteed QoS) so Trino does not contend with the Duckgres worker it is being compared against. Readiness requires the coordinator and every requested worker replica. Cleanup selects on ownership labels — safe after a partial provision, and structurally unable to reach a worker pod, another benchmark cluster, or the charts-created Secret.
• Reader identity resolver over the config-store warehouse row plus the Duckling CR's charts-published status.benchmarkReader block, read through a new credential-free status accessor so resolving a reader never pulls the tenant writer password into memory.

Security posture

• Fail-closed, no writer-credential fallback. Env-only config defaults the feature OFF; enabling it without a pinned image leaves the lifecycle unbuilt; a built lifecycle still refuses to provision until the charts reader identity resolves in full. The resolver additionally rejects a config whose reader S3 role or reader database user equals the warehouse writer identity.
• S3 access is an assumed read-only IAM role (renewable credentials), never static keys.
• The reader password exists in control-plane memory for exactly one hop — read by exact SecretReference, written into the cluster-owned Secret — and is never logged, returned, or stored on a struct. Tests assert no secret values in HTTP responses, logs, or artifacts.

Scenario and artifacts

• The lifecycle client now polls (interval / timeout / max attempts) instead of issuing a single status request; pending is a polling state, failed is terminal. Trino teardown is always_run and ordered before warehouse teardown.
• summary.json records per-protocol comparison metadata: engine and version (Trino's from the coordinator's own /v1/info), connector version, image reference/digest, requested/ready worker counts, catalog/schema identity, and the UTC session time zone. query_results.csv gets a row per protocol per query.
• No latency thresholds and no CI gating.
• The existing PGWire-only frozen perf scenario is unchanged, with a test asserting it contains no Trino steps.

Testing

Written test-first. New coverage: API auth/validation/status/error mapping, fail-closed missing reader identity, rendered coordinator/worker topology, image pin and explicit resources, read-only role and Secret wiring, absence of writer credentials, readiness requiring all workers, idempotent provision, partial-provision cleanup, ownership-safe cleanup, lifecycle-client polling and terminal failure, mixed PGWire/Trino catalog applicability, scenario DAG and teardown ordering, and no secret values in responses/logs/artifacts. Fake Kubernetes clients and fake resolvers throughout — no live cluster needed for unit tests.

Passing: go test ./tests/perf/... ./tests/mw-dev/..., go test ./controlplane/... ./configresolve/..., the -tags kubernetes lane, just lint (0 issues), git diff --check.

Blocked on the companion charts PR

The feature is disabled and fail-closed until the charts reader resources are deployed. Charts must provide:

• a per-Duckling read-only S3 role (s3:ListBucket + s3:GetObject only), assumable by the control-plane role;
• a dedicated metadata-Postgres reader role and password Secret;
• RBAC letting the control plane read that Secret by exact name;
• and publish all three on the Duckling CR as status.benchmarkReader.{metadataUser, credentialSecretRef{name,namespace,key}, s3ReadOnlyRoleArn} — the exact contract this resolver reads.

The dev benchmark has not been run against dev; it cannot be until those exist. That plan item is left unchecked.

Docs

Scenario runbook (config table, API table, lifecycle, artifacts, failure recovery), a load-bearing-contract section in CLAUDE.md, and updated plan checkboxes.

🤖 Generated with Claude Code

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	15	14	0
E2E/journey files	0	0	0
Workflow files	0	1	0

Signals

• Test cases: +87 / -0
• Assertions: +447 / -5
• Skips or known failures added: 3
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: likely reduced

Warnings

• New skips or known failures (likely reduced)
  • 3 skip/allowlist line(s) added

[bill-ph]

Fixed in 00a232e — and the finding turned out to have a sibling.

Confirmed. cmd/duckgres-controlplane/main.go built its own ControlPlaneConfig literal and omitted TrinoBenchmark, so every DUCKGRES_TRINO_BENCHMARK_* variable was resolved into memory and then discarded: lifecycle nil, routes 503. Exactly as described, and invisible to the scenario run because that builds the root Dockerfile all-in-one binary.

A field-by-field diff of the two literals found a second drift, pre-existing and unrelated to this PR: UserSecretKey is also missing from the production one. DUCKGRES_USER_SECRET_KEY never reaches the production control plane, so CREATE PERSISTENT SECRET is rejected with 0A000 in production even when the chart supplies a key. Same blind spot for the same reason.

Fix

Rather than add two lines and leave the bug class in place, the two ~56-field literals are collapsed into one assembly site: configresolve.ControlPlaneConfig. The four genuinely per-binary values (the binary's own TLS/ACME-adjusted server.Config, its socket-dir and config-path flags, its already-running metrics server) arrive via ControlPlaneOverrides. 124 lines of duplicated literal → 12. A knob added to Resolved now reaches both binaries by construction.

Tripwire

configresolve/controlplane_test.go checks mapping coverage in both directions and asserts nothing about runtime values — zero is a legitimate value for an unset TTL, an empty PriorityClass meaning "headroom disabled", or a false feature gate, so a "must be non-zero" test would have degenerated into an assertion about production configuration:

• destination coverage — every field of the produced config must be movable by some input (compare an all-zero build against an all-sentinel build); a field nothing can move has no source;
• input influence — setting exactly one Resolved field at a time must change the output; a field that changes nothing is parsed-then-discarded.

Both exemption maps carry stated structural reasons rather than being a dumping ground: Server arrives through the overrides, SessionInitTimeout reaches the control plane inside the embedded server.Config, and HealthCheckInterval has no flag/env/YAML source at all (defaulted in RunControlPlane). The influence test currently covers all 47 Resolved fields.

CLAUDE.md's Configuration section now states that a resolved knob reaches the control plane only through that function, and calls out that duplicated config assembly deserves a field-by-field review whenever separate Dockerfiles select different entrypoints — which is precisely how both of these survived.

just lint clean, git diff --check clean, and the configresolve / controlplane / root-package suites pass on both the default and -tags kubernetes builds.