Fix byte/character offset mixup in computed constants output - #148
Draft
rootkiller6788 wants to merge 1 commit into
Draft
Fix byte/character offset mixup in computed constants output#148rootkiller6788 wants to merge 1 commit into
rootkiller6788 wants to merge 1 commit into
Conversation
The dynamic constant propagation analysis records source locations as UTF-16 code-unit offsets (Babel/ESTree positions), but DumpJsirAnalysisResult sliced the original UTF-8 source with those offsets directly. For sources containing non-ASCII characters this misreported the source segment shown next to each computed constant. Convert the source to UTF-16 before slicing and convert the selected segment back to UTF-8 for output, reusing the existing Utf8ToUtf16 / Utf16ToUtf8 helpers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
jsir_genruns dynamic constant propagation on a script containing non-ASCII characters (e.g. Cyrillic text in a comment), theComputed constantssection reports the wrong source segment.Fixes #22.
Root cause
DumpJsirAnalysisResultinmaldoca/js/ir/jsir_gen_lib.ccformats each computed constant withFrom [start, end): ...by callingoriginal_source.substr(start_offset, end_offset - start_offset). However:original_sourceis the raw UTF-8 byte string.start_offset/end_offsetare UTF-16 code-unit offsets. They come fromJsirTriviaAttrlocations, which are populated from Babel/ESTree positions (node->start()/node->end()inast_to_jsir.handwritten.cc). ESTree source positions are UTF-16 code units.Slicing a byte string with code-unit offsets diverges as soon as any non-ASCII character appears before the constant, producing the misaligned output described in the issue.
Fix
Convert the source to UTF-16 once with the existing
Utf8ToUtf16helper, slice it using the UTF-16 offsets, and convert the selected segment back to UTF-8 withUtf16ToUtf8for output. Adds the:utf16dependency to thejsir_gen_libBUILD target.Testing
Manual verification of the logic: offsets recorded by the analysis are UTF-16 code-unit offsets (confirmed against
pass.ccand the Babel position provenance), and the fix applies the existing, already-usedUtf8ToUtf16/Utf16ToUtf8round-trip so byte offsets and code-unit offsets no longer get conflated.