Thoughts on language tooling? i.e. Language Server Protocol/tree-sitter

[acxz]

Hello SysML Team!

Please feel free to transfer this issue over to another repo as you see fit.

I was wondering if there are plans to create a Language Server Protocol (LSP) implementation for SysML v2 or creating a tree-sitter parser. Both of these tools will allow sysmlv2 integration to any editor that has support for these tools (which is pretty much any popular editor such as vscode, neo/vim, among others including our pilot implementations of jupyterlab and Eclipse IDE.

If you want to learn a bit more history about the LSP: https://www.eclipse.org/community/eclipse_newsletter/2017/may/article1.php

Also a tldr in case it helps: Tree-sitter provides syntax detail at the source file level in a quick manner (i.e. as you are entering keystrokes), LSP provides semantic detail at the project level at a slightly longer timescale.

[acxz]

It seems like Sensmetry may have a language server implementation with their SysIDE vscode plugin. However, their implementation is embedded within their plugin and would have to be separated out for everyone else (i.e. other editors/plugins) to use.

It makes sense to me that the sysml team be in charge of the language server protocol that will be used by various tools and editors.

[hpdekoning]

Akash, I was just going to reply that Sensmetry’s SysIDE has implemented LSP support. It works very well in their VSCode deployment and provides an excellent start for more development. See also https://github.com/sensmetry/sysml-2ls/tree/main/packages/syside-languageserver . As far as I understand the LSP approach is also considered in the SYSON project (https://github.com/eclipse-syson/syson). Regards, Hans Peter From: Akash Patel ***@***.***> Sent: Thursday, June 13, 2024 5:10 PM To: Systems-Modeling/SysML-v2-Pilot-Implementation ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Systems-Modeling/SysML-v2-Pilot-Implementation] Thoughts on language tooling? i.e. Language Server Protocol/tree-sitter (Issue #571) It seems like Sensmetry may have a language server implementation with their SysIDE vscode plugin<https://marketplace.visualstudio.com/items?itemName=sensmetry.sysml-2ls>. However, their implementation is embedded within their plugin<https://gitlab.com/sensmetry/public/sysml-2ls/-/issues/13> and would have to be separated out for everyone else (i.e. other editors/plugins) to use. It makes sense to me that the sysml team be in charge of the language server protocol that will be used by various tools and editors. — Reply to this email directly, view it on GitHub<#571 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHY4GEDO3CI46MWS2YHSCFTZHGY3ZAVCNFSM6AAAAABJISITMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVHE2TONJUGY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[acxz]

Oh, I didn't realize that SYSON also used LSP! That makes me happy!

I guess my follow up would be a request for a repo where only the LSP implementation exists.

A big selling point of LSP is the fact that you don't have to implement it for every editor/plugin, but unfortunately that seems to be the case for the current sysmlv2 landscape (of course the tools are limited and sysmlv2 is in its infancy).

Would there be any motivation for an "official" LSP implementation that would be used by say the pilot Eclipse, pilot jupyterlab, the syside vscode plugin, syson, etc?

[himi]

Interesting topic. I want to use abstract syntax as well as concrete syntax for further model-aware processing. So I would like to use the LSP feature of xtext ( https://eclipse.dev/Xtext/documentation/340_lsp_support.html) and LSP4E ( https://projects.eclipse.org/projects/technology.lsp4e) However there seems no activity taking this approach so far. Maybe the SysON project takes a similar approach but unfortunately their metamodel is not the same with that of SysMLv2-Pilot-Implementation. So I'm wondering if I would need to seek some way. If it needs much development effort, I will simply use Lezer on CodeMirror for that purpose. LSP is fairly good but not so responsive and not so useful for advanced language processing. With regards,

…

-- Hisashi Miyashita Mgnite Inc.

On Thu, Jun 13, 2024 at 11:17 AM Hans Peter de Koning < ***@***.***> wrote: Akash, I was just going to reply that Sensmetry’s SysIDE has implemented LSP support. It works very well in their VSCode deployment and provides an excellent start for more development. See also https://github.com/sensmetry/sysml-2ls/tree/main/packages/syside-languageserver . As far as I understand the LSP approach is also considered in the SYSON project (https://github.com/eclipse-syson/syson). Regards, Hans Peter From: Akash Patel ***@***.***> Sent: Thursday, June 13, 2024 5:10 PM To: Systems-Modeling/SysML-v2-Pilot-Implementation ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Systems-Modeling/SysML-v2-Pilot-Implementation] Thoughts on language tooling? i.e. Language Server Protocol/tree-sitter (Issue #571) It seems like Sensmetry may have a language server implementation with their SysIDE vscode plugin< https://marketplace.visualstudio.com/items?itemName=sensmetry.sysml-2ls>. However, their implementation is embedded within their plugin< https://gitlab.com/sensmetry/public/sysml-2ls/-/issues/13> and would have to be separated out for everyone else (i.e. other editors/plugins) to use. It makes sense to me that the sysml team be in charge of the language server protocol that will be used by various tools and editors. — Reply to this email directly, view it on GitHub< #571 (comment)>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AHY4GEDO3CI46MWS2YHSCFTZHGY3ZAVCNFSM6AAAAABJISITMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVHE2TONJUGY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>> — Reply to this email directly, view it on GitHub <#571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACSKO47VOBUTUYQZHTHC3ZHGZZBAVCNFSM6AAAAABJISITMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVHE4DANZXHA> . You are receiving this because you are subscribed to this thread.Message ID: <Systems-Modeling/SysML-v2-Pilot-Implementation/issues/571/2165980778@ github.com>

[acxz]

o.O I wasn't aware of xtext, that is definitely a good start to an LSP implementation, but taking a look at SysIDE's implementation might a good first step (they may already support all the LSP features and it might just be a matter of splitting out the implementation so that other editors/plugins can use it. See the syside issue I created)

Maybe the SysON project takes a similar approach but unfortunately their
metamodel is not the same with that of SysMLv2-Pilot-Implementation.

I might be misunderstanding you here, but sysmlv2 is sysmlv2 no matter what tool you are using right? If SYSON is parsing .sysml files, that is all that matters. I don't think an internalized metamodels matters for an LSP implementation, just the language.

LSP is fairly good but not so responsive

I agree that LSP can be slow, but that is why languages have both tree-sitter and LSP, tree-sitter for the super fast incremental updates that happen in the source file and LSP for the semantic understanding at the project level.

and not so useful for advanced language processing.

Could you clarify this? What is your definition of advanced here with respect to an actual feature? Is there something in mind that isn't part of the LSP spec: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/? Is there an alternative you have in mind?

If it needs much development effort, I will simply use Lezer on CodeMirror for that purpose.

Lezer is a parser and falls in the same category as tree-sitter, however, tree-sitter is much more popular and supported in various editors, whereas Lezer was made primarily for CodeMirror and web based editors. I personally think of Lezer as a light-weight javascript implementation of tree-sitter (which is in C). You can find more info on the comparison on Lezer's website and forums. I tried search but couldn't find a Lezer parser for sysmlv2 as of yet, and would recommend that a tree-sitter parser is created/popularized for sysmlv2 over a Lezer parser.

[himi]

Oh, I thought this discussion thread is on GitHub issues. Please use the SysML v2 Release <https://groups.google.com/g/sysml-v2-release> Google Group if you want to continue. It seems you understand most of my points. I can simply say that currently I'm not so interested in supporting various IDEs. Rather, I want to make an advanced intelligent SysMLv2 editor on top of SysMLv2 Pilot Implementation.

…

On Thu, Jun 13, 2024 at 12:13 PM Akash Patel ***@***.***> wrote: o.O I wasn't aware of xtext, that is definitely a good start to an LSP implementation, but taking a look at SysIDE's implementation might a good first step (they may already support all the LSP features and it might just be a matter of splitting out the implementation so that other editors/plugins can use it. See the syside issue I created <https://gitlab.com/sensmetry/public/sysml-2ls/-/issues/13>) Maybe the SysON project takes a similar approach but unfortunately their metamodel is not the same with that of SysMLv2-Pilot-Implementation. I might be misunderstanding you here, but sysmlv2 is sysmlv2 no matter what tool you are using right? If SYSON is parsing .sysml files, that is all that matters. I don't think an internalized metamodels matters for an LSP implementation, just the language. LSP is fairly good but not so responsive I agree that LSP can be slow, but that is why languages have both tree-sitter and LSP, tree-sitter for the super fast incremental updates that happen in the source file and LSP for the semantic understanding at the project level. and not so useful for advanced language processing. Could you clarify this? What is your definition of advanced here with respect to an actual feature? Is there something in mind that isn't part of the LSP spec: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/? Is there an alternative you have in mind? If it needs much development effort, I will simply use Lezer on CodeMirror for that purpose. Lezer is a parser and falls in the same category as tree-sitter, however, tree-sitter is much more popular and supported in various editors, whereas Lezer was made primarily for CodeMirror and web based editors. I personally think of Lezer as a light-weight javascript implementation of tree-sitter (which is in C). You can find more info on the comparison on Lezer's website and forums. I tried search but couldn't find a Lezer parser for sysmlv2 as of yet, and would recommend that a tree-sitter parser is created/popularized for sysmlv2 over a Lezer parser. — Reply to this email directly, view it on GitHub <#571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACSKIY63A4EPZJRLFHN53ZHHAJJAVCNFSM6AAAAABJISITMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGEZDONZYGE> . You are receiving this because you commented.Message ID: <Systems-Modeling/SysML-v2-Pilot-Implementation/issues/571/2166127781@ github.com>

[acxz]

Rather, I want to make an advanced intelligent SysMLv2 editor on top
of SysMLv2 Pilot Implementation.

The Pilot implementations will become advanced intelligent editors if we implement LSP for the sysmlv2 language, fulfilling that request as both the Eclipse and jupyterlab have LSP support.

I can simply say that currently I'm not so interested in supporting various IDEs.

This will be a (fortunate) by-product of using LSP.

I think we are all on the same page here with having better language tooling and code intelligence features for sysmlv2.
Moving forward the question I have is does the sysml team want to be in charge of an LSP implementation for sysml that will be used everywhere else? Or leave this to the community to figure out? And if a community solution does emerge, will this community support be part of the Pilot implementations? Will the Pilot implementations have LSP/tree-sitter support?

Thank you all for listening to my thoughts and I appreciate the discourse!

[dadada]

I've been working on and off for the last few months on a tree-sitter parser for KerML/SysML with the goal of eventually open-sourcing it. So far it can parse and highlight some syntax such as comments, features, classifiers, namespaces, and so on. There is a small test corpus for that and some queries. Limited support for code navigation is also available via tree-sitter locals. What is left is to support all of the syntax and increase the size of the test corpus. It would be great if we could make this a community effort.

[acxz]

That's absolutely amazing @dadada! Do you have an idea on when you would open-source it or open it up to community contributions? Is there a roadmap for your project? Maybe a table with the syntax of the language as specified in the docs here, compared to existing parsing capability in your tree-sitter implementation?

I'm not familiar with tree-sitter locals, are you referring to anything mentioned on tree-sitter/code-navigation-systems?

[dadada]

Is there a roadmap for your project?

No, not so far. Would you be interested in contributing to the project, or know of others that might be interested? I'd like to avoid releasing a half-baked version.

I'm not familiar with tree-sitter locals, are you referring to anything mentioned on tree-sitter/code-navigation-systems?

Yes, I am trying to add support for both local.* queries and for tag-based queries.

[acxz]

I'd like to avoid releasing a half-baked version.

I understand and fully support that.

I won't be able to dedicate any substantial resources, but if you have bite size issues, I'd love to help tackle them. Unfortunately, I'm not aware of anyone else that would be interested.

Realistically, the lsp and tree-sitter impl for sysml should be incorporated into the pilot implementations. I would feel much better, if say the main contributors of the pilot implementations (@seidewitz, @himi, @hpdekoning, etc.) would give confirmation that they would consider a PR that adds lsp/tree-sitter support to the pilot implementations. It seems that @himi would be on board, from replies earlier in this thread.

[EthanJamesLew]

I've put together a neovim plugin sysmlv2.nvim for my own use that leverages syside-2ls LSP and a basic vim syntax highlighter. It would be nice to pull in a tree-sitter based highlighter and navigator, or use some else's vim plugin entirely.

[anduril-cy]

@dadada any progress updates on the tree-sitter grammar? Super interested in this

[Samonjourus]

Not to hijack a post or anything..., but I'm currently working on a tree-sitter parser and am running into a few issues. For example:

I have no idea what an InvocationTypeMember is and it isn't in the pdf. If someone can point me in the right direction, I'd appreciate it... :)

[leon-thomm]

As far as I am aware, SysON doesn't have an LSP yet, and the sensmitry/syside implementation is now proprietary and closed-source. I think a FOSS LSP implementation could help tooling development itself. For example, an LSP can probably be wrapped into an MCP server for AI-assisted model development quite easily.