Skip to content

feat(sdk:go): Phase 11.11c — cross-pool sibling shape via path registry (SQLR-22)#134

Merged
joaoh82 merged 2 commits into
mainfrom
worktree-phase-11-11c-go-sibling
May 11, 2026
Merged

feat(sdk:go): Phase 11.11c — cross-pool sibling shape via path registry (SQLR-22)#134
joaoh82 merged 2 commits into
mainfrom
worktree-phase-11-11c-go-sibling

Conversation

@joaoh82
Copy link
Copy Markdown
Owner

@joaoh82 joaoh82 commented May 11, 2026

Summary

  • Closes the last open item from Phase 11's SDK arc. Pre-11.11c the Go SDK took a real engine Connection::open for every sql.Open(…) call — which deadlocked against itself on flock(LOCK_EX) whenever two *sql.DB instances pointed at the same file, or a single pool grew past one connection.
  • New process-level path registry keyed by canonical absolute path. The first opener pays for a real sqlrite_open; subsequent openers mint sibling handles off it via the FFI's sqlrite_connect_sibling (shipped in 11.8), sharing Arc<Mutex<Database>> underneath. Refcounted: the last sibling out fires sqlrite_close on the primary.
  • Three new tests cover cross-*sql.DB state sharing, BEGIN CONCURRENT across separate pools with a real Busy + retry (final value 0 + 1 + 100 = 101), and the refcount-to-zero roundtrip.

The headline cross-pool demo

db1, _ := sql.Open("sqlrite", "accounts.sqlrite")
db2, _ := sql.Open("sqlrite", "accounts.sqlrite") // sibling, shares state with db1

db1.Exec("PRAGMA journal_mode = mvcc")

ctx := context.Background()
a, _ := db1.Conn(ctx); defer a.Close()
b, _ := db2.Conn(ctx); defer b.Close()

a.ExecContext(ctx, "BEGIN CONCURRENT")
b.ExecContext(ctx, "BEGIN CONCURRENT")
a.ExecContext(ctx, "UPDATE accounts SET balance = balance + 50 WHERE id = 1")
b.ExecContext(ctx, "UPDATE accounts SET balance = balance + 100 WHERE id = 1")
a.ExecContext(ctx, "COMMIT")                  // wins
_, err := b.ExecContext(ctx, "COMMIT")        // → wrapped sqlrite.ErrBusy
errors.Is(err, sqlrite.ErrBusy)               // true

Architecture

sql.Open("sqlrite", "accounts.sqlrite")   ─┐
sql.Open("sqlrite", "accounts.sqlrite")   ─┼──> canonicalPath → registryMu
sql.Open("sqlrite", "./accounts.sqlrite") ─┘                       │
                                                                   ▼
                            ┌──────────────────────────────────────┐
                            │ pathRegistry["/abs/accounts.sqlrite"]│
                            │   primary: *SqlriteConnection (hidden│
                            │            ─ owned by registry)      │
                            │   refcount: N (outstanding siblings) │
                            └──────────────────────────────────────┘
                                          │ connect_sibling
                                          ▼
                            ┌──────────────────────────────────────┐
                            │  Each *conn owns its own sibling     │
                            │  handle. Close() drops refcount;     │
                            │  hitting 0 closes the primary +      │
                            │  removes the entry.                  │
                            └──────────────────────────────────────┘

Scope decisions (v0)

Open mode Registry path
File-backed read-write ✅ Routes through registry
:memory: ❌ Bypasses — isolated by design (matches SQLite)
Read-only (sqlrite.OpenReadOnly) ❌ Bypasses — shared flock(LOCK_SH) already coexists with other readers
Symlinks Not resolved. Lexical key only (filepath.Abs + filepath.Clean). Callers needing symlink-equality pass os.EvalSymlinks-canonicalized paths

Where this fits in the Phase 11 SDK arc

SDK Sibling API Status
C FFI sqlrite_connect_sibling ✅ 11.8
Python conn.connect() ✅ 11.8
Node.js db.connect() ✅ 11.8
Go path registry + sqlrite_connect_sibling 11.11c (this PR)
WASM (deferred — single-threaded runtime)

Docs

  • sdk/go/README.md — new "Multi-handle reads + writes" section with the cross-pool runnable example + BEGIN CONCURRENT retry-loop demo + caveats.
  • docs/concurrent-writes.md — SDK propagation table updated; the "still constructs its own backing DB" note replaced with the registry's actual behaviour.
  • docs/_index.md — Phase 11 summary blurb refreshed to reflect end-to-end completion.
  • docs/roadmap.md — Phase 11.11c promoted to ✅ shipped; active-frontier blurb refreshed.
  • docs/design-decisions.md — new §12i covering the registry rationale, lock order, scope decisions, symlink caveat.

Test plan

  • cargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks --all-targets
  • cargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks — 615/615
  • cargo fmt --all -- --check
  • cargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks --all-targets
  • cargo doc --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks --no-deps
  • cd sdk/go && go test -count=1 -timeout 120s ./... — all existing + 3 new tests pass in ~1s
  • CI green

What's left of Phase 11

This wraps the SDK arc. The only remaining items are deferred-by-design or foundation work:

  • Phase 11.10 — Indexes under MVCC (deferred-by-design; Turso punted on the same problem).
  • Checkpoint-drain follow-up — the parked half of 11.9. Enables set_journal_mode(Mvcc → Wal) once MvStore is drainable.

🤖 Generated with Claude Code

…ry (SQLR-22)

Closes the last open item from Phase 11's SDK arc. Pre-11.11c the Go
SDK took a real engine `Connection::open` for every `sql.Open(…)`
call — which deadlocked against itself on `flock(LOCK_EX)` whenever
two `*sql.DB` instances pointed at the same file, or a single pool
grew past one connection. The C / Python / Node SDKs already had
sibling-handle shapes since Phase 11.8; Go was the holdout because
`database/sql`'s pool model expects `driver.Open` to be cheap +
idempotent and SQLRite's exclusive flock collided with that contract.

The fix: a process-level path registry in `sdk/go/sqlrite.go` keyed
by canonical absolute path (`filepath.Abs` + `filepath.Clean`). For
file-backed read-write opens:

1. First opener pays for a real `sqlrite_open` → handle stored as a
   hidden "primary" in the registry, refcount = 0.
2. Subsequent openers mint a sibling via the FFI's
   `sqlrite_connect_sibling(primary)` (shipped in 11.8). Each
   `*conn` owns its own sibling; refcount++.
3. Close: refcount--. When 0, the registry closes the primary and
   removes the entry.

Lock order: `c.mu` → `registryMu`, never the reverse. `newConn`
holds only `registryMu` (the `*conn` doesn't exist yet); `conn.Close`
takes `c.mu` first, then `registryMu`.

Scope (v0):
- `:memory:` opens bypass the registry — each is its own DB by
  design (matches SQLite).
- Read-only opens (`sqlrite.OpenReadOnly`) bypass too — they take
  a shared `flock(LOCK_SH)` that already coexists with other
  readers.
- Symlinks are NOT resolved; the key is lexical. Callers needing
  symlink-equality canonicalize via `os.EvalSymlinks`.

Tests (`sdk/go/sqlrite_test.go`):
- `TestTwoSqlOpenOnSameFileShareState` — two `*sql.DB`s on the
  same path see each other's writes immediately, bidirectional.
- `TestBeginConcurrentAcrossSqlOpenInstances` — pinned
  `*sql.Conn`s from two different pools each hold their own
  `BEGIN CONCURRENT`; A's commit wins, B's hits `ErrBusy` and
  retries; final value matches `0 + 1 + 100 = 101`.
- `TestRegistryRefcountDropsToZeroOnLastClose` — after the last
  sibling closes, a fresh `sql.Open` on the same path succeeds
  (proves the flock was released and the entry removed).

Docs:
- `sdk/go/README.md` — new "Multi-handle reads + writes" section
  with the cross-pool runnable example + BEGIN CONCURRENT retry
  loop demo. Caveats called out (`:memory:` isolation, read-only
  bypass, symlinks).
- `docs/concurrent-writes.md` — SDK propagation table refreshed
  to reflect Go cross-pool support; the "still constructs its own
  backing DB" note replaced with the registry's actual behaviour.
- `docs/_index.md` — Phase 11 summary blurb updated to reflect
  the end-to-end story (only 11.10 indexes-under-MVCC and the
  checkpoint-drain follow-up remain).
- `docs/roadmap.md` — Phase 11.11c promoted to ✅ shipped; active-
  frontier blurb refreshed.
- `docs/design-decisions.md` — new §12i covering the registry
  rationale, lock order, scope decisions, and symlink caveat.

Workspace: 615/615 Rust tests pass; `go test ./...` in `sdk/go`
covers all existing tests plus the 3 new ones in ~1s. fmt +
clippy + doc all clean.

Phase 11's SDK arc is now done end-to-end: C / Python / Node / Go
all mint sibling handles that share `Arc<Mutex<Database>>`; the
11.7 retryable-error machinery is exerciseable cross-pool from
every shipped SDK. Only WASM remains untouched (single-threaded
runtime, sibling story doesn't apply).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel Bot commented May 11, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
rust-sqlite Ready Ready Preview, Comment May 11, 2026 1:40pm

Request Review

… method

`t.Context()` was added in Go 1.24, but `sdk/go/go.mod` declares
Go 1.21 and CI runs an older toolchain. The three new tests added
in 11.11c (`TestBeginConcurrentAcrossSqlOpenInstances` +
`TestRegistryRefcountDropsToZeroOnLastClose`) used `t.Context()`
which compiled locally on Go 1.24+ but broke CI on Go 1.21.

Replace all three sites with `context.Background()` (available
since Go 1.7), add the `context` import.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joaoh82 joaoh82 merged commit 786f379 into main May 11, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant