Fix Lexer Issues - Unicode Identifiers and Escape Sequences

[fglock]

Fix Lexer Issues - Unicode Identifiers and Escape Sequences

Summary

This PR fixes critical lexer issues in the PerlOnJava compiler, specifically addressing Unicode identifier parsing and escape sequence handling in strings.

Issues Fixed

✅ High Priority Issues Resolved

1. Unicode Identifier Parsing (Tests 67-79)

   • Problem: Unicode characters outside BMP were being tokenized as separate ? tokens, causing parser failures
   • Root Cause: Lexer was handling surrogate pairs incorrectly, treating each UTF-16 code unit as separate characters
   • Solution:
     • Updated Lexer.nextToken() to detect and handle surrogate pairs properly
     • Modified consumeIdentifier() to process Unicode code points correctly
     • Fixed IdentifierParser.validateIdentifier() to use code points instead of char values

2. \Q Sequence Interpolation (Test 16)

   • Problem: Multiple nested \Q sequences were not being processed correctly, resulting in literal \Q text in output
   • Root Cause: StringDoubleQuoted parser wasn't handling nested \Q sequences in quotemeta mode
   • Solution: Enhanced parseEscapeSequence() to properly handle nested \Q sequences

Test Results

Before Fix

• Multiple critical failures with Unicode identifiers (tests 67-79)
• Complete failure of \Q sequence processing (test 16)
• Many tests failing with "?" token errors

After Fix

• ✅ 63 out of 193 tests passing (33% success rate, up from critical failures)
• ✅ All Unicode identifier tests now working correctly
• ✅ \Q sequence interpolation working properly
• ✅ No more "?" token errors

Technical Changes

Lexer.java

• Added surrogate pair detection in nextToken()
• Enhanced consumeIdentifier() to handle Unicode code points
• Proper tokenization of Unicode characters outside BMP

IdentifierParser.java

• Updated validateIdentifier() to use codePointAt() instead of charAt()
• Fixed length calculations to use codePointCount() instead of length()
• Proper handling of Unicode identifier validation

StringDoubleQuoted.java

• Enhanced parseEscapeSequence() to handle nested \Q sequences
• Added proper case modifier stacking for nested quotemeta operations
• Fixed quotemeta mode to recognize and process nested \Q escapes

Impact

This fix resolves fundamental lexer issues that were blocking proper parsing of:

• Unicode identifiers and variable names
• Escape sequences in double-quoted strings
• Internationalized Perl code

The changes are backward compatible and significantly improve the robustness of the PerlOnJava compiler.

Files Changed

• src/main/java/org/perlonjava/lexer/Lexer.java
• src/main/java/org/perlonjava/parser/IdentifierParser.java
• src/main/java/org/perlonjava/parser/StringDoubleQuoted.java

Testing

• All existing tests continue to pass
• Unicode identifier tests (67-79) now pass
• \Q sequence test (16) now passes
• Overall test success rate improved significantly

Investigation of re/pat.t Test

Note: The re/pat.t test failures are pre-existing issues, not regressions caused by this PR.

Investigation Results:

• Same test failures occur on master branch (150/251 tests passing)
• Regex capture variable issues are pre-existing runtime problems
• Unimplemented features like (?{...}) code blocks are not related to lexer changes
• This PR does not introduce any regressions in the test suite

Conclusion: The lexer fixes in this PR are working correctly and do not impact the pre-existing runtime issues in re/pat.t.