ducklake: add CockroachDB metadata-store flavor#781
Open
benben wants to merge 3 commits into
Open
Conversation
Lets a tenant declare its external Postgres-protocol metadata store is
CockroachDB. The worker then emits two postgres_scanner GLOBALs before
ATTACH:
SET GLOBAL pg_use_text_protocol = true
SET GLOBAL pg_use_ctid_scan = false
CRDB has not implemented binary COPY (cockroachdb#96590) and does not
expose ctid row addressing, so postgres_scanner's default scan path
fails on the first metadata ATTACH with
COPY (...) TO STDOUT (FORMAT "binary"): ERROR: at or near "binary":
syntax error: unimplemented
Plumbing:
- configstore.ManagedWarehouseMetadataStore gains a typed Flavor field
("" → postgres, "cockroachdb" → CRDB). Zero value preserves the old
behaviour for every existing row.
- provisioning API accepts metadata_store.external.flavor; unknown
values reject the request rather than silently degrading.
- SharedWorkerActivator copies the configstore flavor onto
DuckLakeConfig.MetadataStoreFlavor regardless of which
buildDuckLakeConfigFrom*() path resolved the rest of the
infrastructure — the field doesn't live on the Duckling CR.
- server.buildDuckLakePreAttachStatements emits the two CRDB GLOBALs
when MetadataStoreFlavor == "cockroachdb"; the SET GLOBALs are
process-wide on the worker's DuckDB instance, so the control plane
is expected to pin a tenant to workers of one flavor for its
lifetime — no flavor flips on the same worker.
Not addressed in this PR — out-of-scope upstream gaps from
PostHog/ducklake#24:
- ducklake_expire_snapshots() / ducklake_flush_inlined_data() still
fail on CRDB (ctid DELETE path; needs a routing change in the
DuckLake extension to go through postgres_execute).
- Concurrent writers lose ~24% of commits to "40001 restart
transaction" because DuckLake's RetryOnError() only matches the
Postgres "duplicate key ... unique" message.
So this PR unblocks ATTACH + DDL + DML + queries on a CRDB-backed
external metadata store, but a CRDB-backed catalog is not yet
production-grade for snapshot expiry or concurrent-writer workloads.
Mirrors container-image-controlplane-cd.yml + container-image-worker-cd.yml but only fires on workflow_dispatch from the GitHub UI, only builds arm64 (mw-dev is arm64-only), and only tags with mw-dev-<short-sha> + mw-dev-latest. Doesn't touch :latest so the prod CD pipeline keeps owning that tag. Pick controlplane / worker / both from the dispatch input. Worker pins the same default DuckDB row the CD matrix uses (1.5.3 + cred-refresh httpfs + posthog.4 ducklake); the env block calls out the dependency so if the CD matrix moves, this workflow has to follow. Use to validate a feature branch against real mw-dev infra (cnpg-shards, external RDS, Lakekeeper, Crossplane) before merging.
A workflow file only becomes dispatchable from the GitHub UI once it lives on the default branch, so adding it on this feature branch wouldn't have helped — the dispatch dropdown only lists workflows that already exist on main. Use the existing Container Image CD (workflow_dispatch enabled) on this ref instead — it pushes the image and ships a commit_state_update to PostHog/charts, bumping state.duckgres.image.sha. From charts, dispatch ArgoCD Deploy with app=duckgres to promote image.sha → image.dev; the image-argo helper consumes image.dev for the dev environment and image.prod for prod, so prod stays pinned to its existing sha.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lets a tenant declare its external Postgres-protocol metadata store is CockroachDB instead of stock PostgreSQL. When the flavor is
cockroachdb, the worker emits two postgres_scanner GLOBALs before ATTACH:```sql
SET GLOBAL pg_use_text_protocol = true
SET GLOBAL pg_use_ctid_scan = false
```
CRDB has not implemented binary COPY (cockroachdb#96590) and does not expose ctid row addressing, so postgres_scanner's default scan path fails on the first metadata ATTACH with:
Session-level `SET` doesn't help — DuckLake's internal metadata connection doesn't see session-local state — so the settings have to be GLOBAL, and they're process-wide on the worker's DuckDB instance. The control plane is expected to pin a tenant to workers of one flavor for its lifetime; no flavor flips on the same worker.
Plumbing
Migration: GORM auto-migrate adds the column. Existing rows get NULL/empty → postgres → no behaviour change.
Out of scope (upstream gaps from PostHog/ducklake#24)
This PR unblocks ATTACH + DDL + DML + queries on a CRDB-backed external metadata store, but a CRDB-backed catalog is not yet production-grade:
Migrating an existing tenant
For a tenant that was provisioned against CRDB before this lands (no flavor on the configstore row), one SQL after rollout:
```sql
UPDATE managed_warehouse_metadata_stores
SET flavor = 'cockroachdb'
WHERE org_id = '';
```
Then end the worker session (tenant deactivation) — next activation reads the updated flavor and ATTACH succeeds.
Test plan
🤖 Generated with Claude Code