Support Unicode property classes (\p{…} / \P{…})

[DaliVana]

Summary

Regex.compile rejects all Unicode property escapes (\p{…} and \P{…}). This includes general categories (\p{L}, \p{N}, \p{Lu}, \p{Nd}, …), scripts (\p{Greek}, \p{Han}, …), blocks, and binary properties (\p{White_Space}, \p{Alphabetic}, …). These are standardized in [UTS #18 (Unicode Regular Expressions)][tr18] and are required to express tokenizer pre-tokenization patterns faithfully.

Version

• zig-regex: 0.1.1
• Zig: 0.16.0

Reproduction

const Regex = @import("regex").Regex;

test "unicode property classes" {
    inline for (.{ "\\p{L}+", "\\p{N}+", "\\p{Lu}", "\\P{L}", "[^\\p{L}\\p{N}]" }) |pat| {
        var re = try Regex.compile(std.testing.allocator, pat);
        re.deinit();
    }
}

Expected
All Unicode property forms compile and match per the Unicode Character Database (UCD), mirroring the UTS #18 / PCRE / fancy-regex semantics:

General categories: one- and two-letter (\p{L}, \p{Lu}, \p{N}, \p{Nd}, \p{P}, \p{Z}, …) — see UAX #44 §5.7.1 General_Category
Negation: \P{…} and \p{^…}
Scripts and script extensions: \p{Greek}, \p{Script=Han}, \p{scx=Latin}, … — see UAX #24 (Script property)
Binary properties: \p{White_Space}, \p{Alphabetic}, \p{Emoji}, …
Usable both standalone and inside character classes, e.g. [^\p{L}\p{N}], with property/value names matched loosely per UAX #44-LM3 (loose matching) (case/_/-/space-insensitive)

Actual
Compilation fails. The lexer has no case for p/P after a backslash, so it falls through to else => RegexError.InvalidEscapeSequence (src/parser.zig:107). Inside [...] the class parser likewise rejects it (parseCharClass → InvalidCharacterClass, src/parser.zig:722-723).

Why this matters
The canonical GPT-4 BPE split pattern is:

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]|\s[\r\n]|\s+(?!\S)|\s+ Without Unicode property classes this can only be approximated with ASCII [A-Za-z]/[0-9], which silently mis-tokenizes any non-ASCII text (accented Latin, CJK, Cyrillic, emoji, etc.). zig-regex already supports the harder parts of this pattern (lookahead), so property classes are the main remaining blocker for a faithful port — and they're broadly useful well beyond tokenization.

Proposed scope
Full \p{…}/\P{…} support backed by generated UCD tables, targeting UTS #18 "Basic Unicode Support" (RL1.2):

General categories — both abbreviated (L, Lu, Nd, P, Z, …) and the super-categories (L, M, N, P, S, Z, C).
Negation — \P{…} and the \p{^…} form.
Scripts / script extensions — \p{Script=…} / \p{sc=…} and \p{Script_Extensions=…} / \p{scx=…}, plus the bare \p{Greek} shorthand.
Binary properties — at minimum White_Space, Alphabetic, Uppercase, Lowercase, Noncharacter_Code_Point; ideally the full set.
Loose name matching per UAX #44-LM3.
Works identically standalone and inside [...], and composes with negated classes ([^\p{L}\p{N}]) and existing quantifiers.
Implementation suggestion: vendor a small code-generator that emits codepoint range tables from the UCD data files
(DerivedGeneralCategory.txt, Scripts.txt, ScriptExtensions.txt, PropList.txt, PropertyAliases.txt, PropertyValueAliases.txt) at build time, pin the Unicode version, and resolve \p{…} to range sets the existing matcher already understands. Phasing is fine — general categories + negation first (covers the tokenizer use case), scripts/binary properties next.

I'm happy to test a branch against a real BPE tokenizer port (ASCII and non-ASCII corpora) if that helps validate correctness.

References
UTS #18 — Unicode Regular Expressions (property syntax & conformance levels)
UAX #44 — Unicode Character Database (General_Category, loose matching UAX44-LM3)
UAX #24 — Unicode Script Property
UCD data files (latest)
Prior art: Rust regex crate Unicode support, PCRE2 \p docs

[chrisbbreuer]

Thanks for reporting this!

I have already worked quite a bit on a proper foundation here in the last couple of days, and should be able to get this finalized rather sooner than later now.

Keeping you updated here. Thanks again!

[chrisbbreuer]

Update: I tightened up the Unicode property escape implementation for this issue.

What is now covered locally:

• \p{...} / \P{...} compile as standalone atoms with the default Regex.compile path.
• Property escapes also work inside bracket classes, including examples like [^\p{L}\p{N}].
• General categories, binary properties, Script= / sc=, Script_Extensions= / scx=, and bare script shorthands like \p{Greek} resolve through the Unicode data tables.
• \p{^...} negation is supported, including double-negation behavior with \P{^...}.
• Property and property-value names now use loose matching for case, _, -, and spaces.

I added regression coverage for the reproduction from this issue plus matching checks for letters, numbers, negated classes, loose spellings, binary White_Space, and script shorthand.

If there are any other pain points for you, let me know and I will get them knocked out here 😊