core: rewrite sqlc.arg() to native bind parameters in a preprocessing pass above the engines

[kyleconroy]

Fixes #4536.

sqlc.arg(), sqlc.narg(), sqlc.slice(), sqlc.embed() and @name are sqlc syntax, not SQL, but they used to survive all the way through every engine's parser. The pipeline only worked because each converter happened to round-trip them as an ast.FuncCall with Func.Schema == "sqlc", and that knowledge was spread across named.IsParamFunc, three validators, two AST rewriters and the PostgreSQL engine.

The preprocessor

New package internal/sql/preprocess rewrites the query text to native SQL before any engine parser runs. A dialect-parameterized lexer skips comments, string literals, quoted identifiers and dollar-quoted strings, then replaces each construct with the engine's placeholder and records what it replaced.

syntax	becomes
sqlc.arg(name) / sqlc.narg(name)	the dialect's native placeholder
sqlc.slice(name)	the placeholder wrapped in /*SLICE:name*/
sqlc.embed(table)	table.*
@name	the native placeholder, where @ is sqlc syntax

engine	placeholder	@name
postgresql	$1	sqlc syntax
mysql	?	user variable, left alone
sqlite	?1	sqlc syntax

GoogleSQL and ClickHouse are deliberately not preprocessed. They are getting their own parameter system, so they should not inherit sqlc syntax just because the preprocessor is dialect-driven. An engine with no entry in the dialect table is passed through untouched — File returns the source unchanged with an empty side table — so sqlc.arg() reaches their parsers as the function call it looks like and fails there. Absence from the table is the switch, and dialect.go says so.

The side table the preprocessor emits alongside the rewritten text holds the parameter set, embed spans, slice spans, placeholder numbers, the placeholder-style validation, and an offset map back to the original source.

Compiler

The compiler consumes that side table instead of walking the AST. Three parts needed care:

• Numbering is authoritative from the preprocessor. Engines number bind parameters in AST-conversion order, which isn't source order — MySQL's convertSelectStmt converts ORDER BY before WHERE, so order_by_binds came out with its params swapped until the compiler started applying the preprocessor's numbers by placeholder location.
• Embeds are matched by location rather than node identity, which distinguishes a rewritten t.* from one the user wrote.
• Placeholder-style validation moved up too. Keeping validate.ParamRef post-rewrite broke two cases, because it counted sqlc's own placeholders as user-written. The mixed-style and numbering-gap checks now run in the preprocessor over what the user actually typed.

Deleted

internal/sql/rewrite, named.IsParamFunc/IsParamSign, validate.SqlcFunctions, validate.In, validate.ParamRef, and the sqlc.arg awareness in internal/engine/postgresql/utils.go. A missed astutils container node can no longer silently drop a parameter or an embed.

Engine changes

• GoogleSQL converts @name to a ParamRef. That is native GoogleSQL syntax rather than sqlc syntax, and the AST rewriter that used to turn it into one is gone, so the converter has to. Parameter names now come from the column the parameter is compared against, which is what every GoogleSQL golden already expected.
• ClickHouse tags binary and unary operators as A_Expr_Kind_OP and seeds comparison and arithmetic operators. Those are what let the analysis core type a WHERE clause at all — without them even WHERE id = ? fails. analyze_params/clickhouse covers them with native ? placeholders. This is independent of the preprocessor; happy to split it out if you'd rather this PR not touch the ClickHouse engine.

Tests

internal/sql/preprocess/testdata/<engine>/<case>/ holds input.sql, output.sql, side_table.json and — when the input is invalid — stderr.txt. TestRewrite runs one subtest per directory: 69 cases (42 postgresql, 17 mysql, 10 sqlite); -update regenerates the goldens.

side_table.json is rendered by reading the result back through the same API the compiler uses, so every case covers parameter names and nullability, embed and slice spans, numbering and the offset map:

{"number": 1, "location": 47, "origin": 34, "name": "ids", "named": true, "slice": true}

location is the ? after the /*SLICE:ids*/ marker, where the engine reports the node; origin is the sqlc.slice( it replaced.

The cases were chosen by auditing every .sql file in internal/endtoend and examples/, grouping each sqlc.* call by engine, function and argument shape, and diffing that inventory against the goldens. Every shape the corpus uses now has a test for its engine. The ones that exercise a distinct path:

• MySQL reads "x" as a string, so a double-quoted argument keeps its case, unlike a bare reference which is folded (coalesce_params/mysql, overrides_go_types/mysql)
• $1 is not a placeholder in MySQL and is left alone (create_view/mysql)
• -- @sqlc-vet-disable is a vet directive in a PostgreSQL query file, where @name is sqlc syntax; it survives only because the lexer never looks inside a comment (vet_disable)
• PostgreSQL's @> operator next to a string holding a $-path, with a cast on the argument (table_function/postgresql)

go test --tags=examples ./... passes with no failures.

Behaviour changes worth reviewing

• sqlc_arg_invalid goldens changed. Errors now name the offending text (got "$1") instead of an AST type (got *ast.ParamRef), and point at the call rather than the statement start.
• A bare identifier in sqlc.arg(FooBar) is lowercased for PostgreSQL and MySQL, matching what those parsers did, and preserved for SQLite. It's a FoldIdentifier flag in the dialect table if that split is wrong.

🤖 Generated with Claude Code