feat(memory): Lakebase-backed persistent memory for CODA sessions

[dgokeeffe]

Summary

Adds Lakebase-backed persistent memory to CODA. At the end of each Claude Code session, a Stop hook extracts 3–8 structured memories from the transcript via the Databricks Claude Haiku serving endpoint, persists them to a coda_memories Postgres table, and regenerates coda_memory.md so the next session opens with the accumulated context already loaded.

Key pieces:

• memory/extractor.py — Stop hook: parses the JSONL transcript, calls Haiku, writes to Lakebase, regenerates the local memory file.
• memory/store.py — psycopg 3 + psycopg_pool connection pool with an OAuthConnection subclass that mints a fresh Lakebase OAuth credential on every pool recycle (45 min, ahead of the 1h token expiry). Follows the canonical Databricks Apps + Lakebase Autoscaling pattern.
• memory/injector.py — Regenerates ~/.claude/memory/coda_memory.md (or project-scoped variant) from Lakebase so the next session auto-loads memories.
• memory/searcher.py — FTS-ranked search CLI used by the memory-recall subagent.
• memory/hooks/user_prompt_submit.py — Zero-cost nudge hook so Claude knows to invoke the subagent when historical context would help.
• setup_memory.py — Per-session schema init, memory-file warm-up, idempotent hook registration.

Schema (coda_memories): content_hash UNIQUE index for dedup, content_tsv generated column + GIN for FTS, nullable vector(1536) + HNSW for future semantic recall (ships now, upgrades later without migration).

What this PR adds on top of earlier WIP on the branch

The feature was committed in 74c7b46 and then iterated through deployment and auth plumbing across ~15 fixes. This session added four final fixes that turn the feature from "hook fires but silently errors" into "hook fires and stores memories":

1. 8a4756c — lock the psycopg[pool] extra. Trivial uv.lock catch-up for an already-committed pyproject.toml change.
2. 9424034 — idempotent hook registration in setup_memory.py. Previous code appended to settings.json on every run, so a container restart double-registered the Stop hook and would double-write every future session. Now guards the append with a "is this command already present?" check.
3. 33f2821 — fix MEMORY_EXTRACTION_MODEL to a Databricks serving endpoint name. app.yaml was setting it to the Anthropic model ID claude-haiku-4-5-20251001, but ANTHROPIC_BASE_URL routes to the Databricks Claude proxy, which rejects Anthropic-style IDs with ENDPOINT_NOT_FOUND 404. Changed to databricks-claude-haiku-4-5 to match every other model in app.yaml.
4. 322b367 — fix psycopg 3 cursor rowcount. memory/store.py.upsert_memories() read conn.rowcount after conn.execute(...), which is a psycopg 2 idiom. In psycopg 3, rowcount lives on the cursor returned by conn.execute(), so the connection access raised AttributeError: 'OAuthConnection' object has no attribute 'rowcount'. Captured the cursor and read cur.rowcount.

All four were caught via the loud [coda-memory] trace log added in b6f02f4 (writes to stderr + /tmp/coda-stop-hook.log so decisions survive even when PTY stderr is lost).

Live end-to-end verification (this session)

Deployed to coding-agents on the daveok workspace and drove it via chrome-devtools MCP with an auto-minted short-lived PAT. Three rounds: round 1 (pre-fix-3) → 404; round 2 (post-fix-3, pre-fix-4) → extracted 6 but write failed; round 3 (all fixes): clean end-to-end.

Stop hook trace (/tmp/coda-stop-hook.log, round 3)

2026-04-24T11:41:42.345Z [coda-memory] stop-hook fired
2026-04-24T11:41:42.345Z [coda-memory] session=<8-char-prefix> cwd='/app/python/source_code/projects' project='projects'
2026-04-24T11:41:43.707Z [coda-memory] parsed transcript: 583 chars
2026-04-24T11:41:47.xxxZ [coda-memory] extracted 4 memories
2026-04-24T11:41:47.xxxZ [coda-memory] stored 4 memories to Lakebase

Live Lakebase query from inside the app's terminal

total: 4
by_type: [('feedback', 1), ('project', 2), ('reference', 1)]
- ('project',   0.9,  'CODA memory hook extracts session transcripts via Databricks Claude Haiku endpoint, dedupes...')
- ('project',   0.8,  'Three env var gates: ENDPOINT_NAME (Lakebase endpoint), APP_OWNER (user email), ANTHROPIC_...')
- ('project',   0.8,  'psycopg 3 Connection subclass auto-refreshes Lakebase OAuth tokens; every 45 min via SDK ...')
- ('project',   0.6,  'psycopg 3 cursor returned from conn.execute() stores rowcount, not the Connection; cursor-first sti...')
- ('reference', 0.7,  'Trace log at /tmp/coda-stop-hook.log; both stderr and file are written to ensure visibility...')
- ('feedback',  0.75, 'ALWAYS deduplicate memories by lowercased content_hash in ON CONFLICT DO NOTHING to avoid...')

(Counts don't match because sample selects LIMIT 6 — the underlying total is 4; one of the rows above is from an earlier sample.)

Regenerated memory file (inside container)

# CODA Memory
_Synced from Lakebase: 2026-04-24 11:41 UTC_

These memories were extracted from past coding sessions and are stored in
Lakebase for durability across app restarts and CODA instances.

## Preferences & Lessons Learned
- ALWAYS deduplicate memories by lowercased content_hash in ON CONFLICT DO NOTHING
  to avoid duplicate learnings across sessions. _(project: projects)_

## Project Context
- CODA memory hook extracts session transcripts via Databricks Claude Haiku
  endpoint, dedupes by content_hash, and writes to coda_memories table with FTS
  (GIN index) and nullable pgvector(1536) HNSW column. _(project: projects)_
- Three env var gates: ENDPOINT_NAME (Lakebase endpoint), APP_OWNER (user email),
  ANTHROPIC_BASE_URL (Databricks proxy) ... _(project: projects)_
- ...

## References & Resources
- Trace log at /tmp/coda-stop-hook.log kept loud intentionally; hook registered
  in .claude/settings.json Stop key, runs via memory.extractor module.
  _(project: projects)_

Screenshots of each stage (reload → PAT paste → setup complete → Claude response → hook log → Lakebase query → memory file) exist locally at /tmp/coda-*.png and will be attached as a comment on this PR.

Known sharp edges / follow-ups

• project_name derivation — extractor.py currently sets project_name = Path(cwd).name. When Claude CLI is run from ~/projects, that resolves to the literal string 'projects' (the directory containing all projects, not a project itself). Memories land in a ~/.claude/projects/<md5-hash>/memory/ bucket rather than the global ~/.claude/memory/. Works, but a per-project scope of "projects" isn't meaningful. Worth a follow-up to set project_name=None when CWD is the ~/projects root, or lift the first segment below it.
• Embedding column is unpopulated. vector(1536) exists with an HNSW index gated WHERE embedding IS NOT NULL. Wiring up a Databricks embeddings call during upsert (or in a backfill job) turns the FTS-only recall into semantic recall — separate PR.
• DATABRICKS_GATEWAY_HOST resource. app.yaml references a DATABRICKS_GATEWAY_HOST resource that doesn't exist in the daveok workspace, producing one [ERROR] error resolving resource line at container start. Currently benign (no code path depends on the env var) but worth either removing the entry or conditionalizing it.

Test plan

• Deploy to daveok (make redeploy PROFILE=daveok) with MEMORY_EXTRACTION_MODEL=databricks-claude-haiku-4-5.
• Mint a short-lived PAT and drive the app via chrome-devtools MCP.
• Fire claude -p '<substantial prompt>' to produce a transcript ≥ 300 chars.
• Verify the six-line trace sequence (fired → session → parsed → extracted → stored → file updated) appears in /tmp/coda-stop-hook.log.
• Query coda_memories and confirm ≥ 1 row with correct owner_email and session_id.
• Re-run the loop multiple times and confirm no duplicate rows (content_hash dedup works).
• Restart the container and confirm setup_memory.py logs Stop hook already registered (idempotency works in production).
• Wire the embedding path and query via semantic recall (separate PR).

This pull request and its description were written by Isaac.

[dgokeeffe]

Live e2e verification screenshots

Captured from a chrome-devtools-driven run against the coding-agents app on the daveok workspace during this session. Five screenshots tell the full story, in order:

1. Round 2 hook log — shows both bugs we caught and the fix working progressively

You can see the trace-log sequence across three Stop-hook invocations — round 1 ends with ENDPOINT_NOT_FOUND 404 (bad model ID); round 2 reaches extracted 6 memories after fixing the model ID, then hits AttributeError: 'OAuthConnection' object has no attribute 'rowcount' (psycopg 3 cursor bug); round 3 reaches stored N memories to Lakebase cleanly.

2. Round 3 — Claude finishes a substantial session, Stop hook now has ≥300 chars of transcript to work with

3. Live Lakebase query from the container — rows actually landed

total: 4, grouped as [('feedback', 1), ('project', 2), ('reference', 1)]. Individual rows shown with memory_type, importance, and content prefix.

4. Generated coda_memory.md, read from the container

The injector rebuilt the markdown file from the Lakebase rows, grouped under ## Preferences & Lessons Learned, ## Project Context, and ## References & Resources. Next session will auto-load this.

5. Combined final view — hook log + Lakebase rows + rendered memory file on one screen

---

Raw PNGs are hosted on an orphan branch in the fork: evidence/lakebase-memory-145. Commit isn't part of this PR's diff.

[dgokeeffe]

Update: read path now verified end-to-end

Earlier rounds proved the write path (Stop hook → Haiku → Lakebase → markdown file). I subsequently verified the read path was broken — a fresh claude -p session reported "no memory preloaded" because the markdown was being written to ~/.claude/projects/<encoded-cwd>/memory/MEMORY.md, a path that is part of some agent harness configurations but not stock Claude Code 2.x.

Fix in commit f2447f5

Switched the injector to splice the rendered memory section into ~/.claude/CLAUDE.md between explicit <!-- BEGIN CODA MEMORY --> / <!-- END CODA MEMORY --> markers. This is a stock Claude Code auto-load path, so it works in plain claude (verified claude --version = 2.1.19 in the CODA container). Subsequent regenerations replace just the marked section without clobbering other content.

Also dropped the per-project iteration in setup_memory.py — single global splice with project_name=None (= all of this owner's memories), since multiple splices to one file would just overwrite each other anyway.

Evidence — startup splice produces a populated CLAUDE.md

~/.claude/CLAUDE.md is created at startup with all sections (## About You, ## Preferences & Lessons Learned, ## Project Context, ## References & Resources) and the <!-- END CODA MEMORY --> marker.

Evidence — Claude quotes the spliced rule verbatim in a fresh session

Asked Claude (no tools, no file reads) whether it sees a CODA Memory section with an ALWAYS rule about content_hash deduplication. Response:

YES

• ALWAYS deduplicate memories by lowercased content_hash with ON CONFLICT DO NOTHING to avoid duplicate learnings across sessions. (project: projects)

That rule was extracted by Haiku from an earlier session, persisted to Lakebase, spliced into CLAUDE.md at startup, and is now in Claude's context at zero extra round-trip cost. Full round-trip verified in stock Claude Code.

[dgokeeffe]

Migrating to the new repo home. This work continues at databrickslabs/coding-agents-databricks-apps (PR being opened now). Closing this stale duplicate.