Add test corpus, oracle tooling, and the implementation loop - #2
Merged
Conversation
Build the conformance corpus and workflow ahead of parser implementation: - cmd/regenerate-parse: extracts literal SQL from SQLite's TCL test suite (do_execsql_test, do_catchsql_test, and single-call do_test bodies) and records ground truth from a real SQLite build — a small embedded C oracle compiled against the pinned 3.53.4 amalgamation that splits statements shell-style and prepares each one against :memory: without executing. Downloads are pinned by SHA-256 and cached in .sqlite/ (gitignored). - parser/testdata: 4,685 cases across 69 corpus files with raw per-statement oracle results (ok, or exact error message and offset), plus metadata sidecars marking every case as todo. - internal/testfile: corpus file format, metadata sidecars, and the accept/reject classification policy (syntax-family errors must be reproduced verbatim; semantic prepare errors mean the parse succeeded). - internal/tclextract: conservative TCL extraction; anything it cannot take verbatim is counted and skipped, never guessed at. - cmd/next-test: picks the next todo case to implement. - parser, ast: API skeletons (Parse, parser.Error, Node/Stmt/Expr) so the harness compiles; parser/parser_test.go runs the corpus, skipping todo cases by default and clearing them from metadata under -check-parse. - CLAUDE.md, README.md, testdata provenance README, MIT LICENSE, CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjRSyzyQ5AGtfH8DZYcfc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the conformance corpus and development workflow ahead of parser implementation (PLAN.md milestones 1–2 groundwork).
The corpus
4,685 cases across 69 files (~1.2 MB in
parser/testdata/), extracted from SQLite's own test suite:parser1,tokenize,keyword1,select1–selectH,expr/expr2,in–in5,between,with*,window1–windowE/windowerr,alter*,altertab*,alterdropcol*,trigger1,upsert1–upsert5,returning1, and thee_select/e_expr/e_createtable/e_insert/e_update/e_deleteevidence files.Every case stores the raw per-statement verdict from a real SQLite build:
ok, or result code + byte offset + exact error message. The harness derives the requirement from it: meyer must reject with the identical message when SQLite's parser rejected (near "X": syntax error,unrecognized token: "X",incomplete input), and must accept everything else — including statements that failed semantically (no such table), since those parsed fine. All 4,685 cases start astodoin metadata sidecars.Tooling
cmd/regenerate-parse— downloads the pinned SQLite 3.53.4 release (SHA-256-verified, cached in gitignored.sqlite/), compiles an embedded C oracle against the amalgamation, extracts literal SQL from the TCL scripts (do_execsql_test,do_catchsql_test, and single-calldo_test { catchsql {…} }bodies), and writes corpus + merged metadata. The oracle splits statements shell-style viasqlite3_completeand prepares each independently against:memory:withsqlite3_prepare_v2— nothing executes, so cases are order-independent. Extraction is conservative and loud: of 5,643 blocks found, 958 skipped (TCL substitution / non-literal bodies), every skip counted in the output.cmd/next-test— picks the next todo case (file with fewest remaining todos first).parser/parser_test.go— corpus harness: skips todo cases by default; with-check-parse, re-runs them and removes newly passing ones from the metadata (logged asPARSE PASSES NOW), so progress commits alongside code.internal/testfile(corpus format + classification policy) andinternal/tclextract(TCL extraction), both with unit tests.Scaffolding
parserandastAPI skeletons: the finalParse(ctx, io.Reader) ([]ast.Stmt, error)signature andparser.Error{Message, Offset}; every input currently errors, which is what keeps all corpus cases todo.CLAUDE.md(workflow + rules: never hand-edit corpus files, zero dependencies, no parser generators),parser/testdata/README.md(provenance + pinned hashes, public-domain note), MITLICENSE, updatedREADME.md, GitHub Actions CI (go build/go vet/go test -race).Notes
syntax error after column name "b") are currently classified as semantic, so meyer is permitted to accept them for now. The pattern list lives ininternal/testfileand can tighten later without regenerating, because the corpus stores raw oracle output.go build ./...,go vet ./..., andgo test ./...are green (corpus subtests all skip as todo).The implementation loop is ready:
go run ./cmd/next-test→ implement →go test ./parser -check-parse.🤖 Generated with Claude Code
https://claude.ai/code/session_014vjRSyzyQ5AGtfH8DZYcfc
Generated by Claude Code