Skip to content

[BUG] tokenizer.Reset() does not clear dialect state → pooled tokenizers leak dialect/keywords across callers #524

Description

@mem

Bug Description

Tokenizer.Reset() (called by PutTokenizer before returning an instance to the sync.Pool) resets input, position, line tracking, and the logger, but not the dialect field or the keywords table. Because a Tokenizer obtained from the pool can have had its dialect changed via SetDialect, returning it to the pool leaves a non-default dialect in a shared, reusable instance. The next GetTokenizer() caller can receive a tokenizer configured for the wrong dialect (e.g. MySQL when it expected the PostgreSQL default), causing silently incorrect tokenization/parsing.

To Reproduce

  1. Get a tokenizer from the pool
  2. Set Dialect
  3. Put it back
  4. Get a tokenizer from the pool
  5. Validate that the dialect is set to the default value

Steps 4. and 5. might need to be repeated on a loop.

Code Example

func TestPoolDialectLeak(t *testing.T) {
    a := GetTokenizer()
    a.SetDialect(keywords.DialectMySQL)
    PutTokenizer(a)

    b := GetTokenizer()
    defer PutTokenizer(b)
    if b.Dialect() != keywords.DialectPostgreSQL {
        t.Fatalf("LEAK: got dialect %q, want default %q", b.Dialect(), keywords.DialectPostgreSQL)
    }
}

SQL Query

N/A

Expected Behavior

The GetTokenizer behavior is consistent, or there's a explit note in the docs about having to set the dialect each time a tokenizer is retrieved from the pool.

Actual Behavior

You might get a tokenizer with a different dialect than the default.

Environment

  • OS: Linux
  • Go Version: 1.26
  • GoSQLX Version: 1.14.0

Additional Context

N/A

Possible Solution

  1. Set the tokenizer back to the default when Reset() is called.
  2. Require setting the tokenizer explicitly after GetTokenizer()

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions