sqlite: replace the ANTLR parser with meyer (#4535)

* sqlite: replace the ANTLR parser with meyer

meyer is a hand-written recursive descent parser for SQLite's dialect,
ported from src/parse.y and src/tokenize.c and checked against SQLite
itself. Moving to it deletes 1.2 MB of generated ANTLR code and the
antlr4-go dependency, and replaces a grammar that only approximated
SQLite with one that accepts what SQLite accepts.

parse.go calls meyer and slices statement extents out of the node spans;
convert.go maps meyer's tree onto sqlc's AST; reserved.go now defers to
meyer's keyword table, which the hand-maintained list was missing
MATERIALIZED and RETURNING from.

SQLite has no schema-qualified function call, so sqlc.arg() and its
siblings are not SQL meyer will parse. The parser folds "sqlc.name(" to
"sqlc_name(" over the token stream -- one byte for another, so every
offset still points into the original source, and string literals and
comments are untouched -- and the converter puts the schema back.

Behavior the old grammar got wrong and this corrects:

- "col TEXT NULL" was read as a column of type "TEXTNULL", so the column
  generated as interface{} rather than a nullable string. This changes
  one golden file.
- GROUP BY was dropped whenever the query also had a WHERE clause, from
  an off-by-one over the ANTLR expression list. Now that GROUP BY reaches
  the compiler, its column references are validated, which SQLite's
  implicit rowid would have failed; the compiler now recognizes rowid,
  _rowid_ and oid for SQLite.
- Comparison operators other than "=" were all recorded as "=", and IS,
  ISNULL, NOTNULL and LIKE reached the AST as "=" too. They now carry
  their own spelling, and lang.IsComparisonOperator learned the
  keyword-spelled comparisons so they still type as bool.
- ORDER BY, DISTINCT, WITH RECURSIVE, ON CONFLICT, OVER and FILTER were
  parsed and then discarded, so ast.Format dropped them from the SQL it
  rebuilds. All are carried through now.
- Quoted identifiers other than "..." kept their quotes; float literals
  became the integer 0.

One capability is lost: the pinned SQLite build meyer targets defines
neither SQLITE_ENABLE_UPDATE_DELETE_LIMIT nor SQLITE_UDL_CAPABLE_PARSER,
so UPDATE and DELETE no longer accept ORDER BY or LIMIT. No test covered
it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

* sqlite: drop parse_test.go

The end-to-end tests are the acceptance gate for the parser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

* sqlite: drop catalog_test.go

The end-to-end tests cover the catalog the schema builds. Removing the
test leaves newTestCatalog with no callers, so it goes too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

* sqlite: drop the analyzer test

The end-to-end tests exercise the analyzer against a real database.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

* sqlite: drop the sqlc.arg fold, now that preprocessing handles it

#4537 rewrites sqlc's syntax to native placeholders above the engines, and
SQLite is one of the preprocessed dialects, so sqlc.arg() and @name never
reach the parser. The fold this branch carried is dead.

One gap follows from the pass's rule that a statement whose sqlc syntax
did not validate is copied through for the engine to parse: that assumes
the engine can parse it. SQLite cannot, so an invalid sqlc.arg() surfaced
as `near "(": syntax error` and the preprocessor's own diagnostics were
never reported -- from parseCatalog, since a schema file and a query file
are often the same file. On a parse failure the compiler now reports the
sqlc syntax errors recorded for that file, if any, in place of the failure
they produced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

* sqlite: update to meyer v0.1.1, restoring UPDATE/DELETE LIMIT

v0.1.1 adds parser.Options, whose UpdateDeleteLimit field compiles in the
grammar SQLITE_ENABLE_UPDATE_DELETE_LIMIT does. The option selects between
two forms of SQLite's own rules, so meyer cannot tell which one a caller
wants from the SQL alone; sqlc has always accepted ORDER BY and LIMIT on
UPDATE and DELETE, so it asks for them.

UpdateStmt and DeleteStmt grow OrderBy and Limit fields to match. sqlc's
AST has somewhere to put the LIMIT and nowhere to put the ORDER BY, which
is what the ANTLR converter did with them too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: kyleconroy

go.mod

@@ -5,7 +5,6 @@ go 1.26.0
 toolchain go1.26.5
 
 require (
-	github.com/antlr4-go/antlr/v4 v4.13.1
 	github.com/cubicdaiya/gonp v1.0.4
 	github.com/davecgh/go-spew v1.1.1
 	github.com/fatih/structtag v1.2.0
@@ -22,6 +21,7 @@ require (
 	github.com/spf13/pflag v1.0.10
 	github.com/sqlc-dev/doubleclick v1.0.0
 	github.com/sqlc-dev/marino v0.1.0
+	github.com/sqlc-dev/meyer v0.1.1
 	github.com/sqlc-dev/zetajones v0.1.0
 	github.com/tetratelabs/wazero v1.12.0
 	github.com/wasilibs/go-pgquery v0.0.0-20250409022910-10ac41983c07
@@ -36,6 +36,7 @@ require (
 require (
 	cel.dev/expr v0.25.2 // indirect
 	filippo.io/edwards25519 v1.2.0 // indirect
+	github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect

go.sum

@@ -83,6 +83,8 @@ github.com/sqlc-dev/doubleclick v1.0.0 h1:2/OApfQ2eLgcfa/Fqs8WSMA6atH0G8j9hHbQIg
 github.com/sqlc-dev/doubleclick v1.0.0/go.mod h1:ODHRroSrk/rr5neRHlWMSRijqOak8YmNaO3VAZCNl5Y=
 github.com/sqlc-dev/marino v0.1.0 h1:8Fn13vFhx7OUcmDFfRZdf3zARAbNl04Lcy74211ZpIw=
 github.com/sqlc-dev/marino v0.1.0/go.mod h1:mQxC2dgDE0DWHMb2B5jZNk7KToJuS6wnxnffBfYnq08=
+github.com/sqlc-dev/meyer v0.1.1 h1:BAeZcfgLyTnk9f90DyGEKXPrHxtgvVD/DTM6awq2kUY=
+github.com/sqlc-dev/meyer v0.1.1/go.mod h1:pS4USCRf/SLjWtaMcnTo4YrEEFKBj8CyyqlxcVUJQH8=
 github.com/sqlc-dev/zetajones v0.1.0 h1:VeG0atx6lNABr9V2bSI5vL9DvOKTHX0XjMqWUE/rv40=
 github.com/sqlc-dev/zetajones v0.1.0/go.mod h1:dU1DxwqC6Cahbpnw16KpH1J2waWRDMdwyDSvovMZR4I=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=

internal/compiler/compile.go

@@ -48,7 +48,12 @@ func (c *Compiler) parseCatalog(schemas []string) error {
 		// but don't update the catalog - the database will be the source of truth
 		stmts, err := c.parser.Parse(strings.NewReader(contents))
 		if err != nil {
-			merr.Add(filename, contents, 0, err)
+			// A schema file and a query file are often the same file, so a
+			// query's sqlc syntax can fail here. Look for an explanation
+			// before reporting the syntax error it caused.
+			if reported := addSyntaxErrors(merr, filename, contents, preprocess.File(c.conf.Engine, contents)); !reported {
+				merr.Add(filename, contents, 0, err)
+			}
 			continue
 		}
 
@@ -111,7 +116,9 @@ func (c *Compiler) parseQueries(o opts.Parser) (*Result, error) {
 
 		stmts, err := c.parser.Parse(strings.NewReader(pp.Text))
 		if err != nil {
-			merr.Add(filename, src, 0, err)
+			if reported := addSyntaxErrors(merr, filename, src, pp); !reported {
+				merr.Add(filename, src, 0, err)
+			}
 			continue
 		}
 		for _, stmt := range stmts {
@@ -162,3 +169,32 @@ func (c *Compiler) parseQueries(o opts.Parser) (*Result, error) {
 		Queries: q,
 	}, nil
 }
+
+// addSyntaxErrors reports every sqlc syntax error the preprocessor recorded
+// for a file, in source order, and says whether there were any. Locations come
+// back in the rewritten text's coordinates, so they are mapped through Origin
+// to point at what the user wrote.
+//
+// A statement whose sqlc syntax did not validate is copied through for the
+// engine to parse, which assumes the engine can parse it. SQLite cannot: it
+// has no schema-qualified function call, so a bad sqlc.arg() is a syntax error
+// there rather than a call the preprocessor's message can be attached to.
+// These messages name the cause, so they are reported in place of the failure
+// they produced; anything else wrong with the file surfaces on the next run.
+func addSyntaxErrors(merr *multierr.Error, filename, src string, pp *preprocess.Result) bool {
+	var found bool
+	for _, stmt := range pp.Statements() {
+		if stmt.Err == nil {
+			continue
+		}
+		found = true
+		loc := pp.Origin(stmt.Start)
+		var e *sqlerr.Error
+		if errors.As(stmt.Err, &e) && e.Location != 0 {
+			loc = pp.Origin(e.Location)
+			e.Location = loc
+		}
+		merr.Add(filename, src, loc, stmt.Err)
+	}
+	return found
+}

internal/compiler/output_columns.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"fmt"
 
+	"github.com/sqlc-dev/sqlc/internal/config"
 	"github.com/sqlc-dev/sqlc/internal/sql/ast"
 	"github.com/sqlc-dev/sqlc/internal/sql/astutils"
 	"github.com/sqlc-dev/sqlc/internal/sql/catalog"
@@ -69,7 +70,7 @@ func (c *Compiler) outputColumns(qc *QueryCatalog, node ast.Node) ([]*Column, er
 
 		if n.GroupClause != nil {
 			for _, item := range n.GroupClause.Items {
-				if err := findColumnForNode(item, tables, targets); err != nil {
+				if err := c.findColumnForNode(item, tables, targets); err != nil {
 					return nil, err
 				}
 			}
@@ -85,7 +86,7 @@ func (c *Compiler) outputColumns(qc *QueryCatalog, node ast.Node) ([]*Column, er
 					if !ok {
 						continue
 					}
-					if err := findColumnForNode(sb.Node, tables, targets); err != nil {
+					if err := c.findColumnForNode(sb.Node, tables, targets); err != nil {
 						return nil, fmt.Errorf("%v: if you want to skip this validation, set 'strict_order_by' to false", err)
 					}
 				}
@@ -101,7 +102,7 @@ func (c *Compiler) outputColumns(qc *QueryCatalog, node ast.Node) ([]*Column, er
 						if !ok {
 							continue
 						}
-						if err := findColumnForNode(caseExpr.Xpr, tables, targets); err != nil {
+						if err := c.findColumnForNode(caseExpr.Xpr, tables, targets); err != nil {
 							return nil, fmt.Errorf("%v: if you want to skip this validation, set 'strict_order_by' to false", err)
 						}
 					}
@@ -713,14 +714,36 @@ func outputColumnRefs(res *ast.ResTarget, tables []*Table, node *ast.ColumnRef)
 	return cols, nil
 }
 
-func findColumnForNode(item ast.Node, tables []*Table, targetList *ast.List) error {
+func (c *Compiler) findColumnForNode(item ast.Node, tables []*Table, targetList *ast.List) error {
 	ref, ok := item.(*ast.ColumnRef)
 	if !ok {
 		return nil
 	}
+	if c.isImplicitColumnRef(ref) {
+		return nil
+	}
 	return findColumnForRef(ref, tables, targetList)
 }
 
+// isImplicitColumnRef reports whether ref names a column that every table in
+// the dialect has without declaring it, and which therefore never appears in
+// a catalog built from the schema. SQLite gives each ordinary table a rowid,
+// reachable under three spellings.
+func (c *Compiler) isImplicitColumnRef(ref *ast.ColumnRef) bool {
+	if c.conf.Engine != config.EngineSQLite {
+		return false
+	}
+	parts := stringSlice(ref.Fields)
+	if len(parts) == 0 {
+		return false
+	}
+	switch parts[len(parts)-1] {
+	case "rowid", "_rowid_", "oid":
+		return true
+	}
+	return false
+}
+
 func findColumnForRef(ref *ast.ColumnRef, tables []*Table, targetList *ast.List) error {
 	parts := stringSlice(ref.Fields)
 	var alias, name string

internal/endtoend/testdata/case_named_params/sqlite/go/models.go

@@ -12,7 +12,3 @@ func NewCatalog() *catalog.Catalog {
 		Extensions: map[string]struct{}{},
 	}
 }
-
-func newTestCatalog() *catalog.Catalog {
-	return catalog.New("main")
-}