Bug Description
Tokenizer.Reset() (called by PutTokenizer before returning an instance to the sync.Pool) resets input, position, line tracking, and the logger, but not the dialect field or the keywords table. Because a Tokenizer obtained from the pool can have had its dialect changed via SetDialect, returning it to the pool leaves a non-default dialect in a shared, reusable instance. The next GetTokenizer() caller can receive a tokenizer configured for the wrong dialect (e.g. MySQL when it expected the PostgreSQL default), causing silently incorrect tokenization/parsing.
To Reproduce
- Get a tokenizer from the pool
- Set Dialect
- Put it back
- Get a tokenizer from the pool
- Validate that the dialect is set to the default value
Steps 4. and 5. might need to be repeated on a loop.
Code Example
func TestPoolDialectLeak(t *testing.T) {
a := GetTokenizer()
a.SetDialect(keywords.DialectMySQL)
PutTokenizer(a)
b := GetTokenizer()
defer PutTokenizer(b)
if b.Dialect() != keywords.DialectPostgreSQL {
t.Fatalf("LEAK: got dialect %q, want default %q", b.Dialect(), keywords.DialectPostgreSQL)
}
}
SQL Query
N/A
Expected Behavior
The GetTokenizer behavior is consistent, or there's a explit note in the docs about having to set the dialect each time a tokenizer is retrieved from the pool.
Actual Behavior
You might get a tokenizer with a different dialect than the default.
Environment
- OS: Linux
- Go Version: 1.26
- GoSQLX Version: 1.14.0
Additional Context
N/A
Possible Solution
- Set the tokenizer back to the default when
Reset() is called.
- Require setting the tokenizer explicitly after
GetTokenizer()
Bug Description
Tokenizer.Reset() (called by PutTokenizer before returning an instance to the sync.Pool) resets input, position, line tracking, and the logger, but not the dialect field or the keywords table. Because a Tokenizer obtained from the pool can have had its dialect changed via SetDialect, returning it to the pool leaves a non-default dialect in a shared, reusable instance. The next GetTokenizer() caller can receive a tokenizer configured for the wrong dialect (e.g. MySQL when it expected the PostgreSQL default), causing silently incorrect tokenization/parsing.
To Reproduce
Steps 4. and 5. might need to be repeated on a loop.
Code Example
SQL Query
N/A
Expected Behavior
The
GetTokenizerbehavior is consistent, or there's a explit note in the docs about having to set the dialect each time a tokenizer is retrieved from the pool.Actual Behavior
You might get a tokenizer with a different dialect than the default.
Environment
Additional Context
N/A
Possible Solution
Reset()is called.GetTokenizer()