Content mappers

[Andrew Branch (andrewbranch)]

Implements #2824 (comment)

Overview

Content mappers are external integrations that allow TypeScript to include otherwise unsupported file types in a program. They transform a foreign file’s original text into valid TypeScript syntax and provide mappings between the original and transformed content.

Users specify a set of file extensions to be handled by a content mapper package in a tsconfig.json file:

{
  "compilerOptions": {
    // ...
  },
  "contentMappers": [
    {
      "package": "vue-content-mapper",
      "extensions": [".vue"],
      "options": {
        "strictTemplates": true
      }
    }
  ],
  "include": ["src"] // implicitly includes .vue as well as .ts
}

When contentMappers are specified, tsc must be run with --loadExternalPlugins. VS Code passes --loadExternalPlugins to tsc --lsp only in trusted workspaces; otherwise, contentMappers are ignored in the LSP server.

The package field will be resolved as a Node.js module name. The optional options field must be an object and is passed through to the mapper.

The package.json of the content mapper package must specify a tsContentMapper top-level field describing how to spawn the content mapper process and what compiler options its transform requires. A mapper that reads additional project-specific configuration from external sources beyond those compiler options can additionally declare dynamicConfig: true:

{
  "name": "vue-content-mapper",
  "version": "1.0.0",
  "tsContentMapper": {
    "exec": ["node", "dist/server.js"],
    "compilerOptions": ["module", "jsx", "jsxImportSource"],
    "dynamicConfig": true
  }
}

Note that the content mapper process need not be run with Node.js or implemented in JavaScript; package resolution serves as a convenient way to associate a content mapper with a versioned identity that can be managed alongside other dependencies in a project, but the exec field can specify any command.

Protocol

When constructing a program for a config that specifies contentMappers, module resolution recognizes file lookups for the specified extensions and requests transformed content from mapper processes over STDIO. Content mappers communicate with TypeScript over JSON-RPC. TypeScript sends all requests; mappers do not send requests or notifications. All mappers handle initialize and transform. Mappers declaring dynamicConfig: true additionally handle openProject and closeProject.

type PositionEncoding = "utf-8" | "utf-16";

interface InitializeParams {
    protocolVersion: 1;
    /** The position encodings supported by TypeScript. The mapper must choose one of these encodings. */
    positionEncodings: PositionEncoding[];
    /** BCP 47 locale requested for diagnostics. */
    locale?: string;
}

interface InitializeResult {
    /** Must match the protocolVersion sent in InitializeParams. */
    protocolVersion: 1;
    /** The position encoding the mapper will use for all span mapping positions and diagnostic positions. */
    positionEncoding: PositionEncoding;
    /**
     * The source identifier displayed for mapper-produced diagnostics.
     * Must not be "ts", "tsc", "typescript", or any file extension TypeScript understands.
    */
    diagnosticSource: string;
}

/** This request is sent only to mappers that declare `dynamicConfig: true`. */
interface OpenProjectParams {
    /** Absolute tsconfig path, or an empty string for a project without a config file. */
    configFileName: string;
    /** Opaque process-local handle assigned by TypeScript. */
    projectHandle: string;
    /** Object from the contentMappers entry, when specified. */
    options?: Record<string, unknown>;
    /** The project's effective compiler options. */
    compilerOptions: CompilerOptions;
}

/** This response is required only from mappers that declare `dynamicConfig: true`. */
interface OpenProjectResult {
    /**
     * Stable fingerprint of all dynamically discovered configuration that can affect transforms.
     */
    configIdentity: string;
    /**
     * Absolute file names whose changes may alter configIdentity or transform output.
     * May only be returned when the package declares `dynamicConfig: true`. Do not include
     * the files being transformed; those are watched separately.
     */
    watchedFiles?: string[];
}

interface TransformParams {
    fileName: string;
    /** Original content of the file to be transformed. */
    content: string;
    /** Object from the contentMappers entry, when specified. */
    options?: Record<string, unknown>;
    /** Project handle supplied in openProject. Absent for mappers without `dynamicConfig: true`. */
    projectHandle?: string;
    /** The subset of compiler options that the mapper requested in its package.json. */
    compilerOptions: CompilerOptions;
}

interface MappedOutput {
    /** Valid JS, JSX, TS, TSX, or JSON text that TypeScript can parse, according to the specified `scriptKind`. */
    text: string;
    /** The kind of syntax returned in `text`. Defaults to `ScriptKind.TS` if not specified. */
    scriptKind?: ScriptKind;
    /** Mappings between the original and transformed content. */
    mappings?: SpanMapping[];
}

interface TransformResult extends MappedOutput {
    /** Parse errors in the original content. */
    diagnostics?: MapperDiagnostic[];
    /** Additional generated files associated with this input. */
    supplemental?: MappedOutput[];
}

/** This request is sent only to mappers that declare `dynamicConfig: true`. */
interface CloseProjectParams {
    /** Project handle supplied in openProject. */
    projectHandle: string;
}

/** Positions and lengths are in the specified `positionEncoding`. */
type SpanMapping = [
    generatedStart: number,
    generatedLength: number,
    originalStart: number,
    originalLength: number,
    kind: SpanMapKind,
    features?: SpanMapFeature,
];

enum ScriptKind {
    JS = 1,
    JSX = 2,
    TS = 3,
    TSX = 4,
    JSON = 6,
}

enum SpanMapKind {
    /** Verbatim spans in generated output have the same length and content as their counterparts in original text. */
    Verbatim = 0,
    /** Atom spans in generated output may have different length and content than their counterparts in the original text. */
    Atom = 1,
    /** Alias spans in generated output may have different length and content than their counterparts in the original text, but diagnostics display their original text. */
    Alias = 2,
}

/** Controls which TypeScript language service features may use a span. */
enum SpanMapFeature {
    None = 0,
    Hover = 1 << 0,
    SignatureHelp = 1 << 1,
    Completion = 1 << 2,
    Definition = 1 << 3,
    TypeDefinition = 1 << 4,
    Implementation = 1 << 5,
    SourceDefinition = 1 << 6,
    References = 1 << 7,
    DocumentHighlights = 1 << 8,
    Rename = 1 << 9,
    CallHierarchy = 1 << 10,
    CodeActions = 1 << 11,
    Formatting = 1 << 12,
    InlayHints = 1 << 13,
    SemanticTokens = 1 << 14,
    FoldingRanges = 1 << 15,
    SelectionRanges = 1 << 16,
    LinkedEditing = 1 << 17,
    AutoInsert = 1 << 18,
    DocumentSymbols = 1 << 19,
    CodeLens = 1 << 20,
    /** Enables every language service feature. This is the default when `features` is omitted. */
    All = (CodeLens << 1) - 1,
}

/** Start and length are in the specified `positionEncoding`. */
interface MapperDiagnostic {
    messageText: string;
    start: number;
    length: number;
    code?: number;
}

A mapper may return supplemental outputs when a file contributes more than one TypeScript or JavaScript file, such as an Astro component containing multiple script blocks. TypeScript automatically includes these outputs in the same program as the canonical output, so they participate in binding and type checking without needing to be imported. Supplemental outputs receive compiler-assigned virtual file names based on their order and scriptKind, but those names are not module resolution targets and cannot be imported directly. Imports written inside supplemental outputs resolve relative to the directory containing the original file.

Span maps

For a content mapper to be useful, it needs to provide a mapping between the transformed output and the original content. In the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. Take a simple example:

// original content:
(+ 1 2 "oops")

// transformed content:
add(1, 2, "oops");

// span mapping:
add(1, 2, "oops");
^^^                 [0, 3)   → [1, 2) + atom
    ^               [4, 5)   → [3, 4) 1 verbatim
       ^            [7, 8)   → [5, 6) 2 verbatim
          ^^^^^^    [10, 16) → [7, 13) "oops" verbatim

TypeScript sees and checks the transformed content, in this case producing a diagnostic for the string literal "oops" because it is not a number. The span mapping allows TypeScript to report the diagnostic in the original content, at the correct location of the string literal ([7, 13), instead of [10, 16)).

In this example, add mapped to + with SpanMapKind.Atom, indicating a correspondence between the two spans, but with different lengths and content. If the name add failed to resolve, the displayed diagnostic range would cover +, but the message would still reference the identifier add:

add.lisp:1:2 - error TS2304: Cannot find name 'add'.

1 (+ 1 2 "oops")
   ~

The mapper can use SpanMapKind.Alias instead of SpanMapKind.Atom to indicate that the generated and original text name the same entity. When the diagnostic is rendered, the original text of the alias span (+) will be substituted for the generated text (add) in the diagnostic message:

add.lisp:1:2 - error TS2304: Cannot find name '+'.
1 (+ 1 2 "oops")
   ~

Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. Unlike in Volar, diagnostics in unmappable regions are not discarded. In the CLI, they cause a short snippet of the transformed content to be shown with the diagnostic. A common case may be a content mapper that synthesizes an import statement at the top of the file used in scaffolding. If that import fails to resolve, the user will see:

app.vue:1:26 - error TS2307: Cannot find module '@vue/content-mapper-utils' or its corresponding type declarations.
  This location is in code generated by the content mapper '@vue/content-mapper@1.0.0' and has no corresponding location in the original file.

1 import { scaffolding } from "@vue/content-mapper-utils";
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spans in the generated output must not overlap, but multiple may map to the same span in the original content. In other words, one range in the original content can map to multiple ranges in the transformed content. This can be useful in the language server when combined with SpanMapFeature and SpanMapKind. Broadly speaking, when a language server request is received for a position in a content-mapped file, the handler maps it to every projection whose feature mask includes the requested operation, performs analysis on the transformed content, and maps visible results back through spans that participate in the same feature. This lets a mapper independently select, for example, one projection for hover and another for definitions or references.

The language server currently supports the following features for content-mapped files:

• Diagnostics - always mapped to original content where possible; diagnostics in synthesized regions are collected and reported at the top of the file. Diagnostics are intentionally not represented by a feature flag, so generated code cannot opt out of diagnostic reporting.
• Position-based features - hover, signature help, completions, definitions, type definitions, implementations, source definitions, references, document highlights, rename, call hierarchy, code actions, formatting, linked editing, and auto-insert map incoming positions or ranges through spans participating in their corresponding SpanMapFeature flag.
• Document-wide features - inlay hints, semantic tokens, folding ranges, selection ranges, document symbols, and CodeLens map visible results back only through spans participating in their corresponding flag.
• Text edits - feature participation does not make a mapping edit-safe. Rename, code action, completion, and formatting edits may be written back only through exact, length-preserving SpanMapKind.Verbatim mappings.

Language service requests and visible results can be disabled independently for any span by clearing the corresponding bits, or disabled for all features with SpanMapFeature.None. If features is omitted from the span mapping tuple, it defaults to SpanMapFeature.All, enabling every supported language service feature for that span.

Note

Unlike with Volar, feature participation must be statically determined by the content mapper during transformation. This level of LSP feature mapping is not intended to replace fully custom language servers. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. We expect that ecosystems implementing complex transforms may still want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features. Content mappers provide a baseline editing experience, but they also provide the API foundation for more specialized language servers to build on. Vue tooling, for example, may choose to enable TypeScript features only for selected projections while a separate language server handles the rest, accessing the AST, type, and symbol information of transformed content through an API connection to TypeScript’s language server.

Failure handling

Mappers return diagnostics for unparseable content, and errors in the transformed text itself are handled by TypeScript like any other file. If the mapper fails in an unexpected way (e.g., crashes or doesn’t conform to the protocol), TypeScript reports a localized diagnostic and treats the file as an empty TypeScript file. After five failures in a single project, TypeScript stops attempting to transform files with that content mapper and issues a final diagnostic reporting the failure.

LSP activation

TypeScript’s language server can only know to care about the file extensions registered in contentMappers once the server is running and has discovered a tsconfig.json that specifies them. In the case where a user opens a directory in VS Code and opens a single .vue file, the TypeScript VS Code extension hasn’t even activated, much less spawned a server that knows about a contentMappers registration. To address this, third-party VS Code extensions need to explicitly activate the TypeScript extension and request that it search for content mappers handling open files:

const extension = vscode.extensions.getExtension("TypeScriptTeam.native-preview");
await extension?.activate();

await vscode.commands.executeCommand(
    "typescript.native-preview.discoverContentMappers",
    {
        uris: vscode.workspace.textDocuments
            .filter(document => document.languageId === "vue")
            .map(document => document.uri),
        extensions: [".vue"],
    },
);

Extension-provided content mappers are not yet supported, but a follow-up exploration is planned.

Emit

Content-mapped files are not emitted to JavaScript. When --declaration is enabled, however, declaration files are emitted from the transformed content. The declaration file name for App.svelte is App.d.svelte.ts. Declaration files for supplemental outputs of a file named App.svelte are emitted as App.svelte.0.d.ts, App.svelte.1.d.ts, etc., and are automatically referenced by App.d.svelte.ts. Declaration maps are currently not supported.

Incremental, build, watch, and process consolidation

Content mappers are supported in --incremental, --build, and --watch modes. Each project records sorted mapper transform identities in .tsbuildinfo and compares them during up-to-date checks. Changing an identity forces files handled by that mapper to be transformed again.

For a mapper without dynamicConfig: true, the transform identity is computed without starting its process. It includes the resolved package name and version, the tsconfig entry’s options, and the values of compiler options named by tsContentMapper.compilerOptions. Consequently, incremental and solution-build status checks do not spawn processes for mappers with static configuration.

For a mapper declaring dynamicConfig: true, TypeScript sends openProject to obtain configIdentity before an up-to-date decision. The mapper is responsible for changing configIdentity whenever dynamically discovered configuration that can affect transforms changes.

TypeScript watches the absolute paths returned in watchedFiles. A change invalidates only projects that reported that path, closes their current mapper project configuration, obtains a fresh identity and watch set, and performs a normal project rebuild. Other projects using the same mapper package continue using their existing project handles and the shared process.

Note

Modifying a static-config content mapper implementation during local development will not change its identity, so you’ll need to bump the local package.json version, or use --force or --clean to clear cached outputs if testing with --incremental or --build.

In --build mode with project references, and in some instances in the language server, it’s possible to have a project graph with many projects all defining the same content mapper. To avoid excessive spawning of child processes, TypeScript deduplicates content mapper processes by resolved package name and version. For a mapper with dynamicConfig: true, one process may have many open project handles. Dynamic-config mappers must isolate project-specific state by projectHandle, accept requests for different projects in any order, and release that state on closeProject. Static-config mappers receive transforms without a project handle. Processes remain alive while any project using that package is retained.

API integration

Content-mapped SourceFiles can be inspected by the JavaScript API. For a content-mapped SourceFile, file.text is the transformed text, file.originalText is the original text, and file.spanMap exposes an API for mapping between the two. Regardless of the positionEncoding used by the content mapper, accessing the span map through the JavaScript API always yields UTF-16 positions.

const mapped = file.spanMap.generatedToOriginalPosition(10);
// { position, fidelity }
// See _packages/native-preview/src/ast/spanMap.ts for details.

To-do

• Break out content mapper time in --extendedDiagnostics and LSP logs
• Turn on additional language service features
• Decide whether mapper parse diagnostics prevent text from being used
• Investigate timeout/cancellation in LSP
• Investigate VS Code extension-provided content mappers, similar to the way TS Server plugins could be automatically provided by VS Code extensions
• Investigate if declaration maps can work by double-mapping back to original text
• In VS Code, provide a read-only view of the transformed content for debuggability
• Provide a JavaScript library for implementing the content mapper protocol

[Lyu, Wei-Da (jasonlyu123)]

This is way more powerful than what I expected. It'll definitely save a lot of work for all the non-TS file tooling. Some of the features will just work without needing enhancement.

There are a few questions I want to ask:

• Would it make sense for the content mapper to be able to specify which editor feature to disable in the targeted files?

  Svelte currently has a lot of custom handling in completion and code actions. Mostly to enhance the result from the TypeScript(6) language service and move unmappable entries, and also to reroute the request position to a different position. If there is an IPC API for the target feature, it might be better to take over the feature entirely in Svelte files rather than going through client-side middleware.

• If I understand it correctly, the IPC API won’t automatically sourcemap the request and result position. Is it correct? Or is it just not done yet? Also, does that mean validations like this won’t be done in the IPC API?

• Kind of out of the scope of this PR, but will there be a pure source file parsing api in the future?

  Svelte’s current transformation uses TypeScript 6’s SourceFile api to parse script tags. I tried it before, but it seems like getting a source file in the current TypeScript 7 api is slower than TypeScript 6’s SourceFile. Probably it also does semantic work and the serialisation overhead.

For mapping, Svelte currently uses sourcemap generated by magic-string. We're using the hires option to generate character-to-character mapping. Because it's mostly string manipulation, converting it to span mapping is probably the main problem we'll need to solve on the Svelte side. In the long term, it probably makes sense to rebuild our transformation with this “span-based” mapping in mind. We always have issues with source maps being 1 or 2 characters shorter because the end of the range is generated code. So having a custom mapping solution might not be a bad thing. I am currently experimenting with it in this branch if you’re interested.

[Andrew Branch (andrewbranch)]

Michael Arnaldi (@mikearnaldi) good catches, thank you! I will push some of these to this PR. I think patch 0002 isn’t quite right; I started on a different fix today but didn’t have time to finish it. It got surprisingly complicated. When you map a position at the very end of a span, what to return is ambiguous, but it doesn’t depend on whether that position is part of the last segment in the file. The ambiguity is there at the end of any span, and it’s particularly problematic at the boundary between two original text spans that map to non-contiguous virtual text spans:

// original
pos:  0 1 2 3 4 5
text: f o o b a r
span: A - - B - -

// transformed
pos:  0 1 2 3 4 5 6 7 8
text: f o o b a z b a r
span: A - - X - - B - -

At position 3 in the original text, where do you map to? It looks like span B in the diagram, but if that were a completions request at position 3, that would put the insertion caret right between foo and bar, meaning you’re completing foo. Really, every mapping operation needs to know some context about what it’s doing and specify a left/right affinity. I think query inputs almost always prioritize left, because they often arise from the insertion caret, which renders to the left of the Nth character when it’s at position N. LSP results are almost always ranges, not individual positions, so there’s no ambiguity with those.

[Andrew Branch (andrewbranch)]

Wanted to give a quick update here before I’m out on vacation for a week and ask for feedback on a couple specific things from any implementers, especially anyone coming from Volar.

First, I’m reconsidering the SpanMapPurpose classification I mentioned in the PR description:

/**
 * SpanMapPurpose controls which TypeScript language service features will activate using a span
 * in the transformed content given a request position in the original content.
 */
enum SpanMapPurpose {
  /** Disables all language service features for the span. */
  None = 0,
  /** Used by features that inspect semantic information, such as hover, signature help, and completions. */
  Semantic = 1 << 0,
  /** Used by features that locate symbols, such as definitions, references, rename, and call hierarchy. */
  Navigation = 1 << 1,
  /** Enables both semantic and navigation features. This is the default when `purpose` is omitted. */
  All = Semantic | Navigation,
}

...

This feature mapping is not as granular as what Volar allows, and it must be statically determined by the content mapper during transformation. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. The current proposal goes beyond that, but is intentionally limited in scope. We expect that ecosystems implementing complex transforms will invariably want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features.

I stand by the motivation here, but I think SpanMapPurpose is kind of a leaky abstraction compared to a bit flag per language service feature—smaller is not necessarily simpler.

I also want to get feedback on the constraints I’ve put on overlapping segments. The current rules are

• Mappings must be ordered by start position in the transformer output text, and spans in the transformer output text must not overlap.
• Spans in the original text may be perfect duplicates (i.e., multiple spans in the transformer text may map to the same span in the original text) but otherwise must not overlap.

I’m not sure if Volar has either of these restrictions; I think it at least doesn’t have the latter. I couldn’t think of a realistic example that would rely on overlapping but non-duplicate spans in the original text, and had trouble reasoning about the semantic implications of that kind of mapping.

Another question that came up in the latest design meeting about this was whether content mapper processes would need to read their own configuration files, beyond the compilerOptions that we pass to each transform call. If so, that undermines our current logic for caching transform results in LSP/watch mode, invalidating declaration outputs in --incremental, and consolidating process spawns between different projects. Currently, we only spawn one process per unique name@version, and we cache transforms of individual files by that name@version identifier along with a hash of the compilerOptions that we passed to transform (the subset the mapper declared it relies on in its package.json). If mappers vary their outputs across projects with their own config, this breaks two important features of our current strategy:

1. Currently, the cache key of a transform can be computed without spawning the mapper. This is most important for --build, where we need to check if a referenced project needs to be rebuilt or not. On the initial build, we write the hash of the mapper identity and the compiler options it depends on into the .tsbuildinfo, and then on a later build we can resolve the mapper’s package.json and see if the hash has changed. If the mapper actually depends on other configuration we’re unaware of, this no longer works.
2. In the LSP or in --watch mode, if the mapper reads external config files, we won’t know to invalidate mapped file caches or reload the mapper process when the external config changes.

These are solvable problems, but at a performance and complexity penalty. How do existing transforms work? Are they consuming arbitrary config from the workspace?

---

Lyu, Wei-Da (@jasonlyu123) I just saw your comment as I was about to paste this—I’m on a plane with terrible WiFi so I’ll have to look closer at your links in a week. But your input about specific language feature overrides supports the first change I mentioned I was considering, so that’s helpful feedback! I’m not sure I fully understand the questions about the IPC API. The protocol in this PR is totally separate from the IPC API you can get by importing "typescript". These content mapper processes won’t automatically be able to use the TS API against the tsc process that spawns them. (At the time the transforms run, there is no program, no checker, and ASTs are in the process of being created.) You could theoretically implement a content mapper that imports "typescript" to use that API for whatever reason, but that would spawn a child process to do that, and be a separate communication channel from the compiler owning the program compilation. In the LSP plugin scenario, you’d have tsc --lsp both spawning content mappers and serving API requests to your API client (which could either be installed as an LSP middleware in our client or running its own LSP server for .svelte files). Those API requests would have access to the ASTs produced by content mappers, and can freely map positions between the original and virtual text. But the virtual text is basically treated as the canonical text in the API (it’s what we have an AST for; it’s what gets bound and type checked.) I’m not sure if this was helpful or more confusing, but I really appreciate you trying this out already! Happy to talk in depth when I return.

Also, yes, there will be a way to get a parsed SourceFile without setting up a whole program, so that would help with the script tag parsing. Coming soon.

[Michael Arnaldi (mikearnaldi)]

Michael Arnaldi (@mikearnaldi) good catches, thank you! I will push some of these to this PR. I think patch 0002 isn’t quite right; I started on a different fix today but didn’t have time to finish it. It got surprisingly complicated. When you map a position at the very end of a span, what to return is ambiguous, but it doesn’t depend on whether that position is part of the last segment in the file. The ambiguity is there at the end of any span, and it’s particularly problematic at the boundary between two original text spans that map to non-contiguous virtual text spans:

// original
pos:  0 1 2 3 4 5
text: f o o b a r
span: A - - B - -

// transformed
pos:  0 1 2 3 4 5 6 7 8
text: f o o b a z b a r
span: A - - X - - B - -

At position 3 in the original text, where do you map to? It looks like span B in the diagram, but if that were a completions request at position 3, that would put the insertion caret right between foo and bar, meaning you’re completing foo. Really, every mapping operation needs to know some context about what it’s doing and specify a left/right affinity. I think query inputs almost always prioritize left, because they often arise from the insertion caret, which renders to the left of the Nth character when it’s at position N. LSP results are almost always ranges, not individual positions, so there’s no ambiguity with those.

I am sure my patches are not perfect, I am mostly iterating with agents so please take everything in my repo as a "maybe", the reason for that patch is: when the file is empty or when the cursor is pointed at the end of a file I would not get LSP completions, with my patch I get LSP completions but I am not sure it's correct :)

[Lyu, Wei-Da (jasonlyu123)]

I’m not sure I fully understand the questions about the IPC API. The protocol in this PR is totally separate from the IPC API you can get by importing "typescript".

Yeah. I should have worded it more clearly. The IPC API question is related to the first question. And in both cases, what I want to say is LSP-connected IPC API.

Those API requests would have access to the ASTs produced by content mappers, and can freely map positions between the original and virtual text. But the virtual text is basically treated as the canonical text in the API

I think this should answer my question. The scenario of my question is: when the Svelte language server handles a completions request, whether it's triggered in TS client-side middleware or a standard LSP request, the Svelte language server uses the LSP-connected IPC API to ask about the completion result. In that case, should the parameter sent to the IPC API be a generated position or the source position? Another thing is whether we can ask for positions that are purely generated? From the current implementation, it looks like the LSP-connected IPC API should all use generated positions and won't have mapping validations. So I am mostly asking for confirmation.

[Remco Haszing (remcohaszing)]

First of all, this is a great start! Thanks for working on this. ❤️

I only read the PR description and comments. I have some thoughts and feedback.

---

Content-mapped files are not emitted to JavaScript. When --declaration is enabled, however, declaration files are emitted from the transformed content. The declaration file name for App.svelte is App.d.svelte.ts. Declaration maps are currently not supported.

This is probably correct for many languages, but not all. I believe Svelte, Vue, and Astro files are published to npm as-is.

MDX is different though. MDX is a language that is syntactic sugar for JS(X). It gets compiled to plain JS. It should be treated the same as JSX. For example, the following MDX:

# Hello {props.user.name}

<Avatar user={props.user} />

gets compiled to roughly the same JavaScript as the following JSX:

export default function MDXContent(props) {
  return (
    <>
      <h1>
        Hello {props.user.name}
      </h1>
      <Avatar user={props.user} />
    </>
  )
}

I don’t see a reason to publish to MDX to npm, but I believe the situation is similar for Ember (cc Alex Matchneer (@machty)). I imagine a content mapper should be able to specify how its declarations are emitted.

I’m also not sure how emit should work with mapped files. I can imagine it it works with noEmit or emitDeclarationsOnly. Alternatively the protocol can support a new request type to get the content and sourcemap to emit.

---

type SpanMapping = [
    generatedStart: number,
    generatedLength: number,
    originalStart: number,
    originalLength: number,
    kind: SpanMapKind,
    purpose?: SpanMapPurpose,
];

I really like that generatedLength and originalLength can differ. This is currently not possible in Volar. I do wonder if this causes compatibility issues with Volar. It would be nice to reuse the TypeScript content mapper with other Volar services instead of having to write separate mappers for TypeScript and Volar. This is more of a concern for Volar than for TypeScript. cc Johnson Chu (@johnsoncodehk)

---

enum SpanMapPurpose {
  /** Disables all language service features for the span. */
  None = 0,
  /** Used by features that inspect semantic information, such as hover, signature help, and completions. */
  Semantic = 1 << 0,
  /** Used by features that locate symbols, such as definitions, references, rename, and call hierarchy. */
  Navigation = 1 << 1,
  /** Enables both semantic and navigation features. This is the default when `purpose` is omitted. */
  All = Semantic | Navigation,
}

I think we need for fine-grained control. Notably, actions that read are generally safe, but actions that write may be unsafe, as the edit would be based on mapped content, but be applied to to the actual content. Edits produce valid TypeScript, but might produce content that’s not valid in the original content. For example, say we have an MDX file with YAML frontmatter. The pipe represents the cursor position.

---
|
---

Now autocomplete might turn this into something like:

---
{
  created: new Date(),
  title: ''
}|
---

This completion made sense in TypeScript, but it doesn’t make sense in YAML. There are no Date constructors in YAML. And while the rest happens to be valid YAML, most would consider it non-idiomatic. In this case, I would probably like to enable hover, signature, definitions, references, and hierarchy. But renaming and completions are more dangerous actions to perform.

---

I see that content mappers can specify which compiler options they consume. This is a great start, but they may also need other options. How are users supposed to specify those? Should content mappers support custom compiler options?

{
  "compilerOptions": {
    "mdxRemarkPlugins": [],
    // …
  },
  "contentMappers": [
    {
      "package": "@mdx-js/content-mapper",
      "extensions": [".mdx"]
    }
  ]
}

Or should they get their own config option? In this case, how should extended tsconfigs be handled?

{
  "compilerOptions": {
    // …
  },
  "contentMappers": [
    {
      "package": "@mdx-js/content-mapper",
      "extensions": [".mdx"],
      "options": {
        "remarkPlugins": []
      }
    }
  ]
}

For editor JSON schema support it might also be nicer to use a mapping instead of an array.

{
  "compilerOptions": {
    // …
  },
  "contentMappers": {
    "@mdx-js/content-mapper": {
      "extensions": [".mdx"],
      "options": {
        "remarkPlugins": []
      }
    }
  }
}

---

Can multiple mapped content ranges overlap with the same position in a source file? For example, you can either import or inject JSX components in MDX. So say we have the following MDX, where the pipe is the cursor:

import { Imported } from 'module'

<Imported />
<Injected| />

This is roughly equivalent to the following JSX:

import { Imported } from 'module'

export default function MDXContent(props) {
  return (
    <>
      <Imported />
      <props.components.Injected />
    </>
  )
}

This means that while typing JSX, it’s nice to get completions from imported members as well as members on the props.components type. To achive this in Volar, we roughly map this to the following virtual content:

import { Imported } from 'module'

export default function MDXContent(props) {
  return (
    <>
      <Imported />
      <props.components.Injected| />
    </>
  )
}

Injected|

Notice that there are two cursors in the virtual content.

---

A related issue for content mappers is microsoft/TypeScript#31894. This doesn’t need to be resolved in the first iteration, but it’s a big pain point in the current Volar based approach that I want to highlight.

A scenario:

Say you want to add a content mapper to JSX support to TypeScript. You could make the JSX behaviour configurable using some types on a namespace named JSX. It’s up to the user, such as @types/react to define these types. They should specify for example JSX.Element, and JDX.IntrinsicElements. Now several years later you decide to add a new type, JSX.ElementType. This breaks @types/react, because it didn’t specify JSX.ElementType. Not specifying this type just became a type error.

JSX support is builtin to TypeScript. So they broke their own rule. JSX.ElementType is optional. In fact, any type can be omitted from the JSX namespace. This will lead to weird behaviour, but it doesn’t make the type checker fail. The JSX namespace can even be defined in different places.

It’s common for other content mappers to depend on types that may or may not be defined. Currently this is done by depending on undefined behaviour of /*unresolved*/ any, and the behaviour has been broken a couple of times already.

---

I feel like it would be useful to add support for injecting a custom TypeScript file into the program. I believe some Volar integrations add boilerplate that could be reused to avoid the need to re-parse them all.

But another situation I have in mind is Next.js. They support typed routes by emitting a TypeScript file and continuously updating it when you run the server. I feel like they could abuse the content mapper system by making the user commit an empty file named whatever.next-types, then implement a content mapper for .next-types files to inject their code.

IMO this might as well be supported without burdening the end user with the hassle of requiring an empty file for the content mapper.

[Lyu, Wei-Da (jasonlyu123)]

The reason for that patch is: when the file is empty or when the cursor is pointed at the end of a file I would not get LSP completions

About completion, I found another problem. Completion position is currently mapped to the right of the cursor. If the span right after the symbol is a SpanMapKind.Atom, completion will be skipped. For example, Svelte transform {a} in the markup to a;. The mapping from } to ; needs to be a SpanMapKind.Atom because the text has changed. Wondering if it would be better to treat the completion position as the end of a range. Something like this

	ranges := l.converters.FromLSPRange(file, lsproto.Range{
		Start: lsproto.Position{Line: LSPPosition.Line, Character: LSPPosition.Character - 1},
		End:   LSPPosition,
	}, spanmap.FeatureCompletion)

	position := int(ranges[0].Span.End())

I also want to get feedback on the constraints I’ve put on overlapping segments. The current rules are

• Mappings must be ordered by start position in the transformer output text, and spans in the transformer output text must not overlap.
• Spans in the original text may be perfect duplicates (i.e., multiple spans in the transformer text may map to the same span in the original text) but otherwise must not overlap.

One problem I found is with adding a quote to an identifier. Svelte transforms element attributes to an object literal.

<button foo="" >

to

{ svelteHTML.createElement("element", { "foo":"",});}

Because JavaScript's identifier rules differ from HTML attribute rules, attributes need to be surrounded by quotes. This caused a conflict with diagnostics and references-related features. The diagnostics range includes the quote, but references don't. If I made the quote mapped to the first character, document highlight will highlight foo as f"oo". But when the quote doesn't have a mapping, the diagnostics range might be mapped to a nearby position. A workaround I found is to map the first quote to the first character of the identifier, while leaving the original length as 0 so it doesn't overlap. Not sure if it counts as not overlapping. Or it's just that the current validation doesn't complain about it.

Another question about mapping: should the span be broken down by identifier or tokens? It seems so in your example. Or can it just be one span? For example, should an unedited expression like document.getElement('') be broken down into document, ., getElement, (, '' and )? From my testing, it seems like just one span also works,

[Andrew Branch (andrewbranch)]

I found another problem. Completion position is currently mapped to the right of the cursor.

It’s actually the same problem Michael Arnaldi (@mikearnaldi) found. I have a fix stashed but it’s more complicated than I’d like so I’m working on some other things while I think about it.

Thanks for the quoted attribute example; I think we can relax the span overlap rules a little bit.

Another question about mapping: should the span be broken down by identifier or tokens? It seems so in your example. Or can it just be one span? For example, should an unedited expression like document.getElement('') be broken down into document, ., getElement, (, '' and )?

This is totally up to the content mapper implementation, but generally speaking, there’s no reason to have contiguous spans with the same kind/features. You should aim for as few spans as it takes to get the behavior you want.

[Andrew Branch (andrewbranch)]

Made a few significant updates and edited the PR description:

• replaced SpanMapPurpose with per-LSP-feature bit flags
• added two different ways of getting extra configuration options into content mappers

[Andrew Branch (andrewbranch)]

Another significant change: a content mapper may now emit additional supplemental files as part of any Transform response. PR description updated again.

[John Reilly (johnnyreilly)]

Quick question. I've been beavering away on adding 7.1 TS support to ts-loader:

TypeStrong/ts-loader#1704

One of the incomplete pieces is custom transformers: https://github.com/TypeStrong/ts-loader#getcustomtransformers

It's possible I've asked this elsewhere and forgotten the answer, if so apologies! But I'm wondering if this functionality is likely to cover what transformers did in the TS version of the API? See: https://github.com/microsoft/TypeScript/blob/b465fdbfe175304d9b977da137b2c178ae1091d3/src/compiler/program.ts#L2693

[Andrew Branch (andrewbranch)]

No, but custom transformers are still planned, mentioned in #4830. I’ll add ts-loader to the list of projects that needs it!