Skip to content

feat: add script to preprocess and tokenize #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mazzma12 wants to merge 15 commits into
base: main
Choose a base branch
from
Open

feat: add script to preprocess and tokenize #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
5cc7b31
wip
mazzma12 Mar 9, 2026
04239f6
handle jsonl and modify build make
mazzma12 Mar 9, 2026
d527d1f
add -mode flag
mazzma12 Mar 9, 2026
3d908b6
remove single and double quotes
mazzma12 Mar 10, 2026
59decef
lexer: handle multiline comment after keyword
blotus Mar 10, 2026
24c5dbb
wip
mazzma12 Mar 9, 2026
743ac5e
handle jsonl and modify build make
mazzma12 Mar 9, 2026
2689efd
add -mode flag
mazzma12 Mar 9, 2026
7d6f3ef
remove single and double quotes
mazzma12 Mar 10, 2026
03e6e43
Feat: remove SPACE from model
mazzma12 Mar 10, 2026
d54a327
Revert "Feat: remove SPACE from model"
mazzma12 Mar 11, 2026
24612ba
feat: add data sample for benchmark
mazzma12 Mar 13, 2026
883e538
Merge remote-tracking branch 'refs/remotes/origin/feat/script' into f…
blotus Mar 13, 2026
0aeb6b6
handle multiline comment after identifier
blotus Mar 13, 2026
b9719cf
add missing file
blotus Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,13 @@ build:
@echo "Building $(BINARY_NAME)..."
@mkdir -p $(BUILD_DIR)
go build -o $(BUILD_DIR)/$(BINARY_NAME) ./cmd/sqllexer
go build -o $(BUILD_DIR)/$(BINARY_NAME)v2 ./cmd/sqlprocessor

# Install the binary to GOPATH/bin
install: build
@echo "Installing $(BINARY_NAME)..."
cp $(BUILD_DIR)/$(BINARY_NAME) $(shell go env GOPATH)/bin/
cp $(BUILD_DIR)/$(BINARY_NAME)v2 $(shell go env GOPATH)/bin/

# Run tests
test:
Expand All @@ -43,4 +45,4 @@ help:
@echo " test - Run tests"
@echo " bench - Run benchmarks"
@echo " clean - Clean build artifacts"
@echo " help - Show this help message"
@echo " help - Show this help message"
Loading