Skip to content

Dev multi-worker Trino benchmark lifecycle (fail-closed) #1049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bill-ph wants to merge 2 commits into from
Open

Dev multi-worker Trino benchmark lifecycle (fail-closed) #1049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
162cb62
Add dev multi-worker Trino benchmark lifecycle (fail-closed)
bill-ph Aug 10, 2026
64a9ffe
Assemble the control-plane config in one place
bill-ph Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/scenario-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,15 @@ on:
required: false
default: ""
type: string
trino_benchmark_image:
description: >-
Pinned Trino+Brikk image for the posthog_frozen_trino_perf scenario.
Empty (the default) leaves the Trino benchmark lifecycle disabled.
Never pass credentials here — the metadata reader password and the
read-only S3 role are charts-created cluster resources.
required: false
default: ""
type: string
schedule:
- cron: "17 8 * * *"

Expand Down Expand Up @@ -73,6 +82,9 @@ jobs:
PR_NUMBER: ${{ github.run_id }}
NAMESPACE: duckgres-ci-pr-${{ github.run_id }}
DUCKGRES_SCENARIO_MAX_RUNTIME: 4h
# Opt-in only: with no pinned image the control plane keeps the Trino
# benchmark lifecycle disabled and its API answers 503.
DUCKGRES_TRINO_BENCHMARK_IMAGE: ${{ inputs.trino_benchmark_image || '' }}
DUCKGRES_SCENARIO_GO_TEST_TIMEOUT: 4h15m
# Add process headroom for repeated full-dataset pgwire aggregates.
DUCKGRES_K8S_WORKER_CPU_REQUEST: "2"
Expand Down
80 changes: 77 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ In topologies 2 and 3, the control plane exposes only PostgreSQL wire protocol t

### Key Components

- **main.go / config_resolution.go**: CLI flags; effective config resolution (CLI > env > YAML > defaults), including env-only K8s knobs.
- **main.go / cmd/**: entry points. `main.go` is the all-in-one binary (all four modes); `cmd/duckgres-controlplane` is the production control plane, `cmd/duckgres-worker` the production worker.
- **configresolve/**: CLI flags (`cliflags.go`) and effective config resolution (`resolve.go`; CLI > env > YAML > defaults), including env-only K8s knobs. `controlplane.go` is the SINGLE place a resolved config becomes a `controlplane.ControlPlaneConfig` — both control-plane entry points call it (see Configuration below).
- **server/** — PG wire protocol server and DuckDB execution
- Wire protocol & connections: `server.go`, `conn.go`, `conn_errors.go`, `conn_query_exec.go`, `conn_results.go`, `conn_copy.go`, `conn_extended_query.go`, `conn_pg_stat_activity.go`, `conn_cursor.go`, `protocol.go`, `exports.go`
- Execution: `executor.go`, `flight_executor.go`, `chsql.go`, `transient.go`
Expand Down Expand Up @@ -140,7 +141,7 @@ Key CLI flags for control-plane mode:
- K8s pool: `--k8s-worker-image`, `--k8s-worker-namespace`, `--k8s-control-plane-id`, `--k8s-worker-port`, `--k8s-worker-secret`, `--k8s-worker-configmap`, `--k8s-worker-image-pull-policy`, `--k8s-worker-service-account` (no global worker cap — per-org `Org.MaxWorkers`, 0=unbounded, is the only cap)
- AWS / STS: `--aws-region`
- Compute-usage billing needs no config: metering is always on for the remote backend and billing PULLS usage over the internal-secret-authed HTTP API (`GET /api/v1/billing/usage` + `POST /api/v1/billing/ack`). See `docs/design/billing-pull-api.md` and "Compute-Usage Billing" below.
- Pod scheduling knobs (CPU/memory requests, node selector, tolerations) are env-only — see `config_resolution.go`.
- Pod scheduling knobs (CPU/memory requests, node selector, tolerations) are env-only — see `configresolve/resolve.go`.

Key CLI flags for duckdb-service mode:
- `--duckdb-listen` (e.g., `unix:///...` or `:8816`)
Expand All @@ -150,14 +151,34 @@ Key CLI flags for duckdb-service mode:

## Configuration

Configuration is resolved in `config_resolution.go` with the following precedence (highest to lowest):
Configuration is resolved in `configresolve/resolve.go` with the following precedence (highest to lowest):
1. CLI flags (`--port`, `--config`, etc.)
2. Environment variables (`DUCKGRES_PORT`, etc.)
3. YAML config file
4. Built-in defaults

Note: `--mode` is CLI-only (not loadable from YAML/env). A handful of K8s pod-scheduling knobs are env-only (no CLI flag).

**A resolved knob only reaches the control plane through
`configresolve.ControlPlaneConfig`** (`configresolve/controlplane.go`). That is
the SINGLE assembly site for `controlplane.ControlPlaneConfig`, shared by both
entry points — the all-in-one `duckgres --mode control-plane` (`main.go`) and the
production `cmd/duckgres-controlplane` (`Dockerfile.controlplane`, its own CD
pipeline). It exists because the two binaries previously each carried a
hand-maintained ~56-field literal with nothing forcing them to agree, and two
knobs had silently drifted out of the PRODUCTION one — `DUCKGRES_USER_SECRET_KEY`
and every `DUCKGRES_TRINO_BENCHMARK_*` variable were resolved into memory and
then dropped, so the features they configure were dead in production while the
mw-dev scenario (which builds the all-in-one binary) passed. Add a knob to
`Resolved` and wire it in that one function; never re-introduce a second literal.
`configresolve/controlplane_test.go` is the tripwire, and it checks MAPPING
COVERAGE in both directions rather than runtime values (zero is a legitimate
value for an unset TTL, an empty PriorityClass, or a false feature gate): every
destination field must be movable by some input, and every `Resolved` field must
change the output. Its two exemption maps are structural facts with stated
reasons, not a dumping ground. **When two Dockerfiles select different
entrypoints, treat duplicated config assembly as a field-by-field review item.**

## Keep docs in sync with behavior

When you change a behavior, default, flag, or invariant that is documented
Expand Down Expand Up @@ -1201,6 +1222,59 @@ entrypoint), `controlplane/reshard_pod.go` (spawner) +
cnpg→ext positive path is unit-only (harness lacks the RDS password);
cnpg→cnpg positive path needs a second mw-dev shard (follow-up).

## Trino Benchmark Lifecycle (dev-only, `kubernetes` tag) — LOAD-BEARING CONTRACT

An opt-in, disposable side-by-side benchmark: one Duckgres worker over PGWire
vs a multi-worker Trino cluster reading the SAME per-run DuckLake snapshot.
Scenario: `tests/mw-dev/scenario/scenarios/posthog_frozen_trino_perf.yaml`;
runbook: `docs/runbooks/scenario-runner.md`. Pieces: `controlplane/
trino_benchmark_api.go` (untagged types + admin-authenticated routes),
`trino_benchmark_manager.go` (k8s lifecycle), `trino_benchmark_reader.go` +
`trino_benchmark_reader_k8s.go` (reader identity), `tests/mw-dev/scenario/trino/`
(lifecycle steps + HTTP client), `tests/perf/drivers/trino/` (statement driver).

- **Fail-closed, with NO writer-credential fallback.** Env-only knobs
(`DUCKGRES_TRINO_BENCHMARK_*`, `configresolve/resolve.go`) default the feature
OFF; enabling it without a pinned image leaves the lifecycle unbuilt; and a
built lifecycle still refuses to provision unless the charts-created reader
identity resolves in full. `buildTrinoReaderIdentity` additionally REJECTS a
configuration whose reader S3 role or reader database user equals the
warehouse writer identity. Never add a degrade path here.
- **Credentials never cross the API boundary.** The lifecycle API returns only
cluster ID, state, endpoint, requested/ready worker counts, and the pinned
image. The metadata reader password exists in control-plane memory for exactly
one hop — read by exact `SecretReference` and written into the cluster-owned
short-lived Secret — and is never logged, returned, or stored on a struct.
Reader status is read through `DucklingClient.GetStatusWithoutCredentials`, so
resolving a reader never pulls the tenant WRITER password into memory.
- **Ownership labels are the cleanup boundary.** Every object carries
`app.kubernetes.io/name=duckgres-trino-benchmark` +
`duckgres.posthog.com/trino-benchmark-cluster=<id>` +
`duckgres.posthog.com/org=<org>`; every delete lists by BOTH the app and
cluster labels. Cleanup is idempotent, safe after a partial provision, and
structurally unable to reach a worker pod, another benchmark cluster, or the
charts-created reader Secret.
- **Readiness means the WHOLE requested topology**: coordinator ready AND
`ReadyWorkers >= RequestedWorkers`. A ready state with no endpoint stays a
polling state; `failed` is terminal and stops the poller. Provision is
idempotent for an identical request (200 vs 202) and 409s on a different org,
image, or worker count — never a silent adoption.
- **S3 access is an assumed read-only role, never static keys.** The catalog
properties set `s3.iam-role`/`s3.role-session-name` (renewable credentials);
`s3.aws-access-key`/`s3.aws-secret-key` must never appear.
- **UTC is pinned in both engines** (`-Duser.timezone=UTC` in jvm.config, the
driver's `X-Trino-Time-Zone`), or cross-engine TIMESTAMPTZ predicates diverge.
- **Artifacts record what ran, not what was intended**: `summary.json`
`environments[]` carries engine/version, connector version, image reference,
requested/ready workers, catalog/schema, and UTC per protocol. No thresholds,
no CI gating.
- Touching any of this → update `controlplane/trino_benchmark_api_test.go`,
`trino_benchmark_api_authz_test.go`, `trino_benchmark_manager_test.go`,
`trino_benchmark_reader_test.go`, `trino_benchmark_reader_k8s_test.go`,
`configresolve/resolve_trino_benchmark_test.go`,
`tests/mw-dev/scenario/trino/*_test.go`, `tests/mw-dev/run_sh_test.go`, and
the scenario DAG assertions in `tests/mw-dev/scenario/runner_test.go`.

## TODO Reference

`TODO.md` is a lightweight backlog for ideas that do not yet have a better
Expand Down
67 changes: 6 additions & 61 deletions cmd/duckgres-controlplane/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,66 +224,11 @@ func main() {
_ = metricsSrv.Shutdown(ctx)
}()

cpCfg := controlplane.ControlPlaneConfig{
Config: cfg,
Process: controlplane.ProcessConfig{
MinWorkers: resolved.ProcessMinWorkers,
MaxWorkers: resolved.ProcessMaxWorkers,
},
SocketDir: *socketDir,
ConfigPath: *configFile,
WorkerQueueTimeout: resolved.WorkerQueueTimeout,
WorkerIdleTimeout: resolved.WorkerIdleTimeout,
RetireOnSessionEnd: resolved.ProcessRetireOnSessionEnd,
HandoverDrainTimeout: resolved.HandoverDrainTimeout,
MetricsServer: metricsSrv,
WorkerBackend: resolved.WorkerBackend,
ConfigStoreConn: resolved.ConfigStoreConn,
ConfigPollInterval: resolved.ConfigPollInterval,
InternalSecret: resolved.InternalSecret,
InternalSecretFallbacks: resolved.InternalSecretFallbacks,
ReadOnlySecret: resolved.ReadOnlySecret,
ReadOnlySecretFallbacks: resolved.ReadOnlySecretFallbacks,
SNIRoutingMode: resolved.SNIRoutingMode,
ManagedHostnameSuffixes: resolved.ManagedHostnameSuffixes,
MetadataHostnameSuffixes: resolved.MetadataHostnameSuffixes,
MetadataProxyMaxConns: resolved.MetadataProxyMaxConns,
DucklingBucketSuffix: resolved.DucklingBucketSuffix,
DuckLakeDefaultSpecVersion: resolved.DuckLakeDefaultSpecVersion,

AdmissionReclaimerMaxReservations: resolved.AdmissionReclaimerMaxReservations,
K8s: controlplane.K8sConfig{
WorkerImage: resolved.K8sWorkerImage,
WorkerNamespace: resolved.K8sWorkerNamespace,
ControlPlaneID: resolved.K8sControlPlaneID,
WorkerPort: resolved.K8sWorkerPort,
WorkerSecret: resolved.K8sWorkerSecret,
WorkerConfigMap: resolved.K8sWorkerConfigMap,
ImagePullPolicy: resolved.K8sWorkerImagePullPolicy,
ServiceAccount: resolved.K8sWorkerServiceAccount,
WorkerCPURequest: resolved.K8sWorkerCPURequest,
WorkerMemoryRequest: resolved.K8sWorkerMemoryRequest,
WorkerNodeSelector: resolved.K8sWorkerNodeSelector,
WorkerTolerationKey: resolved.K8sWorkerTolerationKey,
WorkerTolerationValue: resolved.K8sWorkerTolerationValue,
AllowClientWorkerProfile: resolved.K8sAllowClientWorkerProfile,
WorkerPriorityClassName: resolved.K8sWorkerPriorityClassName,
PlaceholderImage: resolved.K8sPlaceholderImage,
PlaceholderPriorityClassName: resolved.K8sPlaceholderPriorityClassName,
WorkerProfileMinCPU: resolved.K8sWorkerProfileMinCPU,
WorkerProfileMaxCPU: resolved.K8sWorkerProfileMaxCPU,
WorkerProfileMinMemory: resolved.K8sWorkerProfileMinMemory,
WorkerProfileMaxMemory: resolved.K8sWorkerProfileMaxMemory,
WorkerMaxTTL: resolved.K8sWorkerMaxTTL,
WorkerDefaultTTL: resolved.K8sWorkerDefaultTTL,
ExploratoryTierEnabled: resolved.K8sExploratoryTierEnabled,
ExploratoryWorkerCPU: resolved.K8sExploratoryWorkerCPU,
ExploratoryWorkerMemory: resolved.K8sExploratoryWorkerMemory,
ExploratoryWorkerTTL: resolved.K8sExploratoryWorkerTTL,
ReshardPodCPU: resolved.K8sReshardPodCPU,
ReshardPodMemory: resolved.K8sReshardPodMemory,
AWSRegion: resolved.AWSRegion,
},
}
cpCfg := configresolve.ControlPlaneConfig(resolved, configresolve.ControlPlaneOverrides{
Server: cfg,
SocketDir: *socketDir,
ConfigPath: *configFile,
MetricsServer: metricsSrv,
})
controlplane.RunControlPlane(cpCfg)
}
111 changes: 111 additions & 0 deletions configresolve/controlplane.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
package configresolve

import (
"net/http"

"github.com/posthog/duckgres/controlplane"
"github.com/posthog/duckgres/server"
)

// Control-plane config assembly.
//
// This is the SINGLE place a resolved config becomes a
// controlplane.ControlPlaneConfig. Both control-plane entry points call it: the
// all-in-one `duckgres --mode control-plane` (main.go) and the production
// `cmd/duckgres-controlplane` (Dockerfile.controlplane, its own CD pipeline).
//
// It is one function on purpose. The two binaries previously each carried a
// hand-maintained 56-field literal with nothing forcing them to agree, and both
// DUCKGRES_USER_SECRET_KEY and every DUCKGRES_TRINO_BENCHMARK_* variable had
// drifted out of the production one — resolved into memory, then dropped on the
// floor, so the features they configure were dead in production while the
// mw-dev scenario (which runs the all-in-one binary) passed. Adding a knob to
// Resolved and wiring it here now reaches both binaries, and
// configresolve/controlplane_test.go fails if either half of that contract
// breaks again.

// ControlPlaneOverrides carries the few values that are NOT part of the
// resolved config because each binary owns them: its own (already
// TLS/ACME-adjusted) server config, the flags it parsed, and the metrics server
// it already started.
type ControlPlaneOverrides struct {
// Server is the binary's server.Config AFTER it has applied its own TLS /
// ACME adjustments — not resolved.Server.
Server server.Config
// SocketDir and ConfigPath come from the binary's own flags.
SocketDir string
ConfigPath string
// MetricsServer is the already-running metrics server, shut down during a
// handover. Nil when the binary runs none.
MetricsServer *http.Server
}

// ControlPlaneConfig assembles the control-plane config both binaries boot
// from. Every field is either derived from resolved or taken from overrides;
// there is no third source.
func ControlPlaneConfig(resolved Resolved, overrides ControlPlaneOverrides) controlplane.ControlPlaneConfig {
return controlplane.ControlPlaneConfig{
Config: overrides.Server,
Process: controlplane.ProcessConfig{
MinWorkers: resolved.ProcessMinWorkers,
MaxWorkers: resolved.ProcessMaxWorkers,
},
SocketDir: overrides.SocketDir,
ConfigPath: overrides.ConfigPath,
MetricsServer: overrides.MetricsServer,
WorkerQueueTimeout: resolved.WorkerQueueTimeout,
WorkerIdleTimeout: resolved.WorkerIdleTimeout,
RetireOnSessionEnd: resolved.ProcessRetireOnSessionEnd,
HandoverDrainTimeout: resolved.HandoverDrainTimeout,
WorkerBackend: resolved.WorkerBackend,
ConfigStoreConn: resolved.ConfigStoreConn,
ConfigPollInterval: resolved.ConfigPollInterval,
InternalSecret: resolved.InternalSecret,
InternalSecretFallbacks: resolved.InternalSecretFallbacks,
ReadOnlySecret: resolved.ReadOnlySecret,
ReadOnlySecretFallbacks: resolved.ReadOnlySecretFallbacks,
UserSecretKey: resolved.UserSecretKey,
SNIRoutingMode: resolved.SNIRoutingMode,
ManagedHostnameSuffixes: resolved.ManagedHostnameSuffixes,
MetadataHostnameSuffixes: resolved.MetadataHostnameSuffixes,
MetadataProxyMaxConns: resolved.MetadataProxyMaxConns,
DucklingBucketSuffix: resolved.DucklingBucketSuffix,
DuckLakeDefaultSpecVersion: resolved.DuckLakeDefaultSpecVersion,

AdmissionReclaimerMaxReservations: resolved.AdmissionReclaimerMaxReservations,

K8s: controlplane.K8sConfig{
WorkerImage: resolved.K8sWorkerImage,
WorkerNamespace: resolved.K8sWorkerNamespace,
ControlPlaneID: resolved.K8sControlPlaneID,
WorkerPort: resolved.K8sWorkerPort,
WorkerSecret: resolved.K8sWorkerSecret,
WorkerConfigMap: resolved.K8sWorkerConfigMap,
ImagePullPolicy: resolved.K8sWorkerImagePullPolicy,
ServiceAccount: resolved.K8sWorkerServiceAccount,
WorkerCPURequest: resolved.K8sWorkerCPURequest,
WorkerMemoryRequest: resolved.K8sWorkerMemoryRequest,
WorkerNodeSelector: resolved.K8sWorkerNodeSelector,
WorkerTolerationKey: resolved.K8sWorkerTolerationKey,
WorkerTolerationValue: resolved.K8sWorkerTolerationValue,
AllowClientWorkerProfile: resolved.K8sAllowClientWorkerProfile,
WorkerPriorityClassName: resolved.K8sWorkerPriorityClassName,
PlaceholderImage: resolved.K8sPlaceholderImage,
PlaceholderPriorityClassName: resolved.K8sPlaceholderPriorityClassName,
WorkerProfileMinCPU: resolved.K8sWorkerProfileMinCPU,
WorkerProfileMaxCPU: resolved.K8sWorkerProfileMaxCPU,
WorkerProfileMinMemory: resolved.K8sWorkerProfileMinMemory,
WorkerProfileMaxMemory: resolved.K8sWorkerProfileMaxMemory,
WorkerMaxTTL: resolved.K8sWorkerMaxTTL,
WorkerDefaultTTL: resolved.K8sWorkerDefaultTTL,
ExploratoryTierEnabled: resolved.K8sExploratoryTierEnabled,
ExploratoryWorkerCPU: resolved.K8sExploratoryWorkerCPU,
ExploratoryWorkerMemory: resolved.K8sExploratoryWorkerMemory,
ExploratoryWorkerTTL: resolved.K8sExploratoryWorkerTTL,
ReshardPodCPU: resolved.K8sReshardPodCPU,
ReshardPodMemory: resolved.K8sReshardPodMemory,
TrinoBenchmark: resolved.TrinoBenchmark,
AWSRegion: resolved.AWSRegion,
},
}
}
Loading
Loading