Skip to content

ducklake: add CockroachDB metadata-store flavor#781

Open
benben wants to merge 3 commits into
mainfrom
ben/ducklake-metadata-flavor-cockroachdb
Open

ducklake: add CockroachDB metadata-store flavor#781
benben wants to merge 3 commits into
mainfrom
ben/ducklake-metadata-flavor-cockroachdb

Conversation

@benben

@benben benben commented Jun 15, 2026

Copy link
Copy Markdown
Member

What

Lets a tenant declare its external Postgres-protocol metadata store is CockroachDB instead of stock PostgreSQL. When the flavor is cockroachdb, the worker emits two postgres_scanner GLOBALs before ATTACH:

```sql
SET GLOBAL pg_use_text_protocol = true
SET GLOBAL pg_use_ctid_scan = false
```

CRDB has not implemented binary COPY (cockroachdb#96590) and does not expose ctid row addressing, so postgres_scanner's default scan path fails on the first metadata ATTACH with:

COPY (SELECT NULL FROM "public"."ducklake_metadata" LIMIT 1) TO STDOUT (FORMAT "binary"): ERROR: at or near "binary": syntax error: unimplemented

Session-level `SET` doesn't help — DuckLake's internal metadata connection doesn't see session-local state — so the settings have to be GLOBAL, and they're process-wide on the worker's DuckDB instance. The control plane is expected to pin a tenant to workers of one flavor for its lifetime; no flavor flips on the same worker.

Plumbing

Layer Change
`controlplane/configstore/models.go` Typed `Flavor MetadataStoreFlavor` field on `ManagedWarehouseMetadataStore` (`""` → postgres, `"cockroachdb"` → CRDB). Zero value preserves old behaviour for every existing row.
`controlplane/provisioning/api.go` Provision API accepts `metadata_store.external.flavor`. Unknown values reject the request with a 400 rather than silently degrading to postgres — a typo would otherwise only surface as "DuckLake migration check failed" at first activation.
`controlplane/shared_worker_activator.go` Flavor lives only on the configstore row (not on the Duckling CR), so it's copied onto `DuckLakeConfig.MetadataStoreFlavor` regardless of which `buildDuckLakeConfigFrom*()` path resolved the rest of the infrastructure.
`server/ducklake/config.go` `MetadataStoreFlavor string` on the worker `Config`.
`server/server.go` `buildDuckLakePreAttachStatements` emits the two CRDB GLOBALs when flavor == "cockroachdb", in stable order after the existing pg_pool GLOBALs so the test diff stays small.

Migration: GORM auto-migrate adds the column. Existing rows get NULL/empty → postgres → no behaviour change.

Out of scope (upstream gaps from PostHog/ducklake#24)

This PR unblocks ATTACH + DDL + DML + queries on a CRDB-backed external metadata store, but a CRDB-backed catalog is not yet production-grade:

  • `ducklake_expire_snapshots()` / `ducklake_flush_inlined_data()` still fail on CRDB. They route DELETEs through DuckDB's DML planner, which addresses rows by ctid — CRDB doesn't expose it. Fix needs a routing change in the DuckLake extension (route through `postgres_execute` like every other metadata write). Catalog grows forever until that lands.
  • Concurrent writers lose ~24% of commits to `SQLSTATE 40001 restart transaction` because DuckLake's `RetryOnError()` only matches the Postgres "duplicate key ... unique" message. One-line upstream patch (match `"restart transaction"`) verified in CockroachDB metadata-store support report + e2e test harness ducklake#24.

Migrating an existing tenant

For a tenant that was provisioned against CRDB before this lands (no flavor on the configstore row), one SQL after rollout:

```sql
UPDATE managed_warehouse_metadata_stores
SET flavor = 'cockroachdb'
WHERE org_id = '';
```

Then end the worker session (tenant deactivation) — next activation reads the updated flavor and ATTACH succeeds.

Test plan

  • `go test ./server -run TestBuildDuckLakePreAttachStatements` — 8 subtests cover defaults, pgbouncer, CRDB flavor, composition order, and unknown-flavor fallback.
  • `go test ./controlplane/provisioning -run TestProvisionExternalFlavor` — 4 subtests cover omitted/explicit-postgres/cockroachdb/unknown.
  • `go test ./controlplane/...` and `./server/...` — all green.
  • Manual: provision a CRDB-backed external duckling on mw-dev, confirm first connect activates without the binary-COPY error.
  • Manual: existing PG-backed external duckling on the same worker still attaches with no extra latency.

🤖 Generated with Claude Code

benben added 2 commits June 15, 2026 10:26
Lets a tenant declare its external Postgres-protocol metadata store is
CockroachDB. The worker then emits two postgres_scanner GLOBALs before
ATTACH:

  SET GLOBAL pg_use_text_protocol = true
  SET GLOBAL pg_use_ctid_scan     = false

CRDB has not implemented binary COPY (cockroachdb#96590) and does not
expose ctid row addressing, so postgres_scanner's default scan path
fails on the first metadata ATTACH with

  COPY (...) TO STDOUT (FORMAT "binary"): ERROR: at or near "binary":
  syntax error: unimplemented

Plumbing:

- configstore.ManagedWarehouseMetadataStore gains a typed Flavor field
  ("" → postgres, "cockroachdb" → CRDB). Zero value preserves the old
  behaviour for every existing row.
- provisioning API accepts metadata_store.external.flavor; unknown
  values reject the request rather than silently degrading.
- SharedWorkerActivator copies the configstore flavor onto
  DuckLakeConfig.MetadataStoreFlavor regardless of which
  buildDuckLakeConfigFrom*() path resolved the rest of the
  infrastructure — the field doesn't live on the Duckling CR.
- server.buildDuckLakePreAttachStatements emits the two CRDB GLOBALs
  when MetadataStoreFlavor == "cockroachdb"; the SET GLOBALs are
  process-wide on the worker's DuckDB instance, so the control plane
  is expected to pin a tenant to workers of one flavor for its
  lifetime — no flavor flips on the same worker.

Not addressed in this PR — out-of-scope upstream gaps from
PostHog/ducklake#24:

- ducklake_expire_snapshots() / ducklake_flush_inlined_data() still
  fail on CRDB (ctid DELETE path; needs a routing change in the
  DuckLake extension to go through postgres_execute).
- Concurrent writers lose ~24% of commits to "40001 restart
  transaction" because DuckLake's RetryOnError() only matches the
  Postgres "duplicate key ... unique" message.

So this PR unblocks ATTACH + DDL + DML + queries on a CRDB-backed
external metadata store, but a CRDB-backed catalog is not yet
production-grade for snapshot expiry or concurrent-writer workloads.
Mirrors container-image-controlplane-cd.yml + container-image-worker-cd.yml
but only fires on workflow_dispatch from the GitHub UI, only builds arm64
(mw-dev is arm64-only), and only tags with mw-dev-<short-sha> +
mw-dev-latest. Doesn't touch :latest so the prod CD pipeline keeps owning
that tag.

Pick controlplane / worker / both from the dispatch input. Worker pins the
same default DuckDB row the CD matrix uses (1.5.3 + cred-refresh httpfs +
posthog.4 ducklake); the env block calls out the dependency so if the CD
matrix moves, this workflow has to follow.

Use to validate a feature branch against real mw-dev infra (cnpg-shards,
external RDS, Lakekeeper, Crossplane) before merging.
Comment thread .github/workflows/manual-deploy-mw-dev.yml Fixed
A workflow file only becomes dispatchable from the GitHub UI once it
lives on the default branch, so adding it on this feature branch
wouldn't have helped — the dispatch dropdown only lists workflows that
already exist on main.

Use the existing Container Image CD (workflow_dispatch enabled) on
this ref instead — it pushes the image and ships a
commit_state_update to PostHog/charts, bumping
state.duckgres.image.sha. From charts, dispatch ArgoCD Deploy with
app=duckgres to promote image.sha → image.dev; the image-argo helper
consumes image.dev for the dev environment and image.prod for prod,
so prod stays pinned to its existing sha.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants