Broken syntax highlighting

[eregon]

Description

Ruby LSP Information

VS Code Version

1.107.1

Ruby LSP Extension Version

0.9.33

Ruby LSP Server Version

0.26.5

Ruby LSP Add-ons

Ruby Version

3.4.7

Ruby Version Manager

chruby

Installed Extensions

Click to expand

• cmake (0.0.17)
• cmake-tools (1.21.36)
• code-spell-checker (4.4.0)
• cpptools (1.29.3)
• cpptools-extension-pack (1.3.1)
• cpptools-themes (2.0.0)
• debugpy (2025.18.0)
• python (2026.0.0)
• ruby-lsp (0.9.33)
• rust-analyzer (0.3.2761)
• test-adapter-converter (0.2.1)
• vscode-github-actions (0.30.0)
• vscode-jsonnet (0.7.2)
• vscode-pylance (2025.10.4)
• vscode-test-explorer (2.22.1)

Ruby LSP Settings

Click to expand

Workspace

{}

User

{
  "enabledFeatures": {
    "codeActions": true,
    "diagnostics": false,
    "documentHighlights": true,
    "documentLink": true,
    "documentSymbols": true,
    "foldingRanges": true,
    "formatting": true,
    "hover": true,
    "inlayHint": true,
    "onTypeFormatting": true,
    "selectionRanges": true,
    "semanticHighlighting": true,
    "completion": true,
    "codeLens": false,
    "definition": true,
    "workspaceSymbol": true,
    "signatureHelp": true,
    "typeHierarchy": true
  },
  "featuresConfiguration": {},
  "addonSettings": {},
  "rubyVersionManager": {
    "identifier": "auto"
  },
  "customRubyCommand": "",
  "formatter": "auto",
  "linters": null,
  "bundleGemfile": "",
  "testTimeout": 30,
  "branch": "",
  "pullDiagnosticsOn": "both",
  "useBundlerCompose": false,
  "bypassTypechecker": false,
  "rubyExecutablePath": "",
  "indexing": {},
  "erbSupport": true,
  "featureFlags": {},
  "sigOpacityLevel": "1"
}

Expectation

I would expect syntax highlighting to always be 100% correct for ruby-lsp, at least when the code is valid Ruby code, because I'd assume it uses Prism.
In practice I frequently see syntax highlighting break, making me think it might use some regexps instead, which are well known to work poorly for a grammar as complex as Ruby's.

Reproduction steps

• Download https://gist.github.com/eregon/7c46aee1400d5228292978512ef49882
• Open it in VSCode
• Notice syntax highlighting is broken from line 176 onward, as none of the end are correctly highlighted as keywords.

(FWIW gist syntax highlighting doesn't break: https://gist.github.com/eregon/7c46aee1400d5228292978512ef49882#file-generate-cext-trampoline-rb-L176)

[reese]

TL;DR for Shopify folks -- is it possible to somehow have strings/heredocs added to the use cases for semantic highlighting? They're probably some of the most complex parts of the grammar for highlighting and cause ~most of the breakages I run into.

---

I recently poked around the syntax highlighting stuff for #3909 (though am quite certain it's not related to any of my specific changes, that PR was a very minor change), but the context for how this works is that it is in fact a big pile of regex. There is also semantic highlighting which does use Prism, but that only applies to a very limited subset of items (which does not include strings):

The strategy taken by Ruby LSP is to only return tokens for syntax that is ambiguous in Ruby (as opposed to all existing tokens) to optimize for performance.

That makes it sound like the folks at Shopify have seen enough performance issues that they intentionally didn't go down the route of implementing more there, but would love to hear more context on that from folks who initially decided on that. I'd be curious if there's any wiggle room for more syntactic features to be added to that list, because heredocs and strings are the source of constant pain for Ruby parsing, and almost all the syntax highlighting breakages I see fall into those two buckets.

In this particular case, it's extra complex, because heredocs with specific delimiter names (like <<C, <<JS, <<RUBY, <<XML, etc.) will highlight the string contents in that language. The problem with that is that we can embed Ruby inside those heredocs as in the case above, but the syntax highlighters sometimes get confused by which part is a Ruby string interpolation and which part is actually valid syntax for that particular language, and I think Prism is particularly suited to knowing that distinction, and it may be valuable for it to use semantic highlighting to override the categorization for the contents of InterpolatedStringNodes etc.

I have a PR (#3920) that I think will at least mitigate this by not "leaking" where the lang-specific syntax highlighters get confused, but it is at best a stop-gap solution and doesn't really highlight the contents of the string interpolation.

[eregon]

is it possible to somehow have strings/heredocs added to the use cases for semantic highlighting?

I think this would be good, at least for heredocs.
Indeed over the years my impression is most broken syntax highlighting in various editors/etc on Ruby code happened due to heredocs (which are absolutely horrid to handle and basically need a full Ruby lexer to handle correctly.
I think heredocs are also not super frequent (only a small fraction of the files have heredocs) so it seems unlikely to affect performance too badly.

[vinistock]

Thanks for the report!

I would expect syntax highlighting to always be 100% correct for ruby-lsp, at least when the code is valid Ruby code, because I'd assume it uses Prism.

This assumption is not exactly correct. VS Code lags considerably if the language server returns too many semantic tokens. Because of this, we even added a limit to the document size in which we can provide semantic highlighting (here), which is the same thing that the built-in TypeScript extension does.

Note that this is the editor lagging, not the language server. For large documents, we were seeing a <20ms response from the server and >2seconds round trip from the editor.

In practice, highlighting is a combination of the regular TextMate highlighting and semantic highlighting. The rule of thumb is: if something can be highlighted correctly without ambiguity, it should happen in the TextMate grammar. Otherwise, it's semantic highlighting.

As an example:

foo = 1 # non-ambiguous, foo is a local variable
foo     # ambiguous because the TextMate grammar cannot figure out if this is a method call or local var

The TextMate highlighting is much faster than semantic highlighting and doesn't overwhelm the editor, so the more we offload to the grammar, the better.

Concerning heredocs

Heredocs are indeed challenging. If we can move it to semantic highlighting, that would be nice, but I suspect we will still need a combination of grammar + semantic highlighting.

AFAIK, I don't think semantic highlighting can change the language ID inside a special heredoc that identifies another language (e.g.: <<~GRAPHQL). I suspect you will need to use the grammar to switch the language ID and in semantic highlighting, remember that you're inside a heredoc to push semantic tokens inside.

If someone is interested in taking a stab, we're definitely open to improve this.

[RedCMD]

should use provideDocumentRangeSemanticTokens() to not overload VSCode with too many tokens
https://code.visualstudio.com/api/references/vscode-api#DocumentRangeSemanticTokensProvider.provideDocumentRangeSemanticTokens

[vinistock]

We already support semantic tokens range and semantic tokens delta, but we were still seeing considerable lag.

[eregon]

I wonder what other LSP, like the one for Rust, do. I tried the Rust LSP (rust-analyzer) in VSCode recently and found it to be really good and accurate (on ZJIT code at least).
It underlines many identifiers and clicking them brings to the definition of it, so that sounds like it has many "semantic tokens" (but I know very little about that, so maybe not the case).
For example it works flawlessly on https://github.com/ruby/ruby/blob/master/zjit/src/hir.rs and that has ~8k lines.

In comparison I feel ruby-lsp has quite a way to go to get to such a good LSP experience (maybe I'm doing something wrong, but at least I seem to be using the latest version).
I mean this positively, I'm sure there are extra challenges and clearly it's not the same languages and it's far harder to e.g. highlight Ruby code correctly etc.
I just wish ruby-lsp would feel closer to that developer experience that rust-analyzer provides :)

[eregon]

Could we somehow detect if the regexp-based highlighting breaks? For example checking that the last end keyword is correctly highlighted, and if not switch to semantic highlighting?

Or maybe a config/env var/etc to force semantic highlighting?

[RedCMD]

you can't access VSCode's TextMate tokens
so you'll have to parse the file yourself with your own copy of TextMate