TS conformance Phase 1: errors-baseline runner (#80)#118
Merged
Conversation
…odule (#81) Stands up the foundation for the TypeScript conformance runner (epic #80): new xUnit project mirroring SharpTS.Test262/, and microsoft/TypeScript vendored at external/typescript/ pinned to v5.5.4. The runner itself lands in #84. - external/typescript submodule, shallow=true, pinned to v5.5.4 (c8a7d589e) - TypeScriptConformancePaths: TryFindRoot/ConformanceDir/BaselinesDir/LibDir/ TryFindProjectDir helpers (mirror Test262Paths) - SmokeTest verifies submodule layout resolves; soft-skips when uninitialized - README documents submodule init, Windows core.longpaths requirement, future SHARPTS_TSCONFORMANCE_UPDATE_BASELINE=1 switch - Project NOT in SharpTS.sln; SharpTS.csproj excludes the new dir from its default Compile glob (same pattern as existing Test262/Tests exclusions)
Parses the `// @<key>: <value>` header that drives every TS conformance test, and splits the source at `// @filename:` boundaries into virtual files. - TypeScriptConformanceMetadata record: typed Phase-1 fields (Target, Module, Jsx, Strict, NoImplicitAny, StrictNullChecks, NoEmit, Lib) + RawDirectives bag for the lower-priority ones (declaration, sourceMap, experimentalDecorators, useDefineForClassFields, isolatedModules, ...) so skip-logic in #84 can key off any directive. - Parser scans every line for `// @key: value` (matches TS harness behavior: directives can appear anywhere, including after a @filename:). Non-filename directive lines remain in the body as comments to keep source line numbers aligned with what TS baselines reference. - 13 unit tests + 1 corpus smoke pass that parses every .ts in the conformance corpus to verify no file throws (parallelized; full suite runs in ~270ms).
…hing (#95) Infrastructure half of #95. Adds a nullable `string? TsCode` field that flows from throw site → SharpTSException → Diagnostic, ready for the conformance runner (#84) to diff against tsc's *.errors.txt baselines. - Diagnostic record: new `TsCode` parameter (6th positional, default null). - SharpTSException: new `tsCode:` parameter on the (code, message, ...) ctors. - TypeCheckException: new `tsCode:` parameter on all 3 location-bearing ctors. - DiagnosticTsCodeTests: 7 tests pinning the contract — defaults, propagation through TypeCheckException, and that TsCode does NOT leak into ToHumanFormat or ToMsBuildFormat (existing IDE/snapshot output is unchanged). All 332 throw sites in TypeSystem/ continue to compile unchanged — tagging them with TSnnnn codes is the second half of #95, in a follow-up commit.
Part 2 of #95. Threads tsCode through every TypeCheckException (and its subclass exceptions) site under TypeSystem/, mapping each to the closest canonical TypeScript diagnostic code. Consumed by the conformance runner (#84) to diff against tsc's *.errors.txt baselines. Coverage: - 334 sites tagged with tsCode: "TSnnnn" - ~15 SharpTS-only sites left untagged with explanatory comments (parser internals, conditional-type recursion-depth limit, @namespace decorator validation, internal-invariant switch arms — no canonical TS analogue) Top TS codes by frequency: TS2322 (assignment): 39 TS2345 (argument not assignable): 36 TS2554 (wrong arity): 35 TS2314 (generic type args): 32 TS2339 (property doesn't exist): 22 TS2365 (operator type mismatch): 11 long tail of single-use codes for specific TS diagnostics Also extends Part 1's infrastructure to the 4 TypeCheckException subclasses (TypeMismatchException, UndefinedMemberException, InvalidCallException, TypeOperationException) — they now accept optional tsCode: parameter and forward to base, unblocking the 8 throw sites that use them. Lower-confidence mappings worth a follow-up review (the closest TS code was used; conformance failures may surface refinements): - TS18013 for "Cannot access private field outside class body" - TS2851 for `using` on non-disposable - TS1108/1116/1163/1308/1061 for parser-shaped throws - TS4112/4113 for override auto-accessor errors - TS1257 for "Required tuple element after optional" - TS2474 for const enum member referenced before definition Verification: - dotnet build SharpTS.csproj: clean - dotnet test --filter ~DiagnosticTsCodeTests|~TypeChecker: 539/539 passed
Runs SharpTS's lexer/parser/type-checker over a TS conformance test, collects diagnostics, and diffs them against the test's *.errors.txt baseline. Mirrors SharpTS.Test262 in shape; the pipeline is much simpler since there's no execution stage. New files: - TypeScriptConformanceOutcome.cs — 6-bucket enum (Pass/Fail/ParseError/ TypeCheckError/Skipped/HarnessError) + Result record - ErrorsBaselineParser.cs — parses *.errors.txt header lines into (line, code) tuples; ignores indented continuation lines - TypeScriptConformanceRunner.cs — RunOne(testPath) does the full pipeline - TypeScriptConformanceBaseline.cs — committed-baseline read/write/diff, mirror of Test262Baseline shape Match strategy (per #84): (line, tsCode) tuples. Column intentionally dropped (TS rewords messages between versions; column drift is endemic). File is implicit per-test for single-file cases. Lib-drift skip filter (per #83): tests where our checker produced no errors but tsc expected only TS2339/TS2304/TS2551/TS7053 ("missing surface") errors bucket as Skipped:lib-drift rather than Fail. Conservative — only fires on the strongest signal of lib-version divergence vs a real checker bug. Multi-file tests bucketed as Skipped:multi-file-deferred for now. Cross-file resolution (imports, declaration merging across virtual files) is follow-up work; #84 is single-file scope. The hand-picked test from acceptance criteria (types/conditional/conditionalTypes1.ts) currently lands in ParseError because line 48 has `declare const ...` syntax our parser doesn't fully support — exactly the kind of actionable divergence point this runner exists to surface. 16 new unit tests: - 5 ErrorsBaselineParser (single header, continuation lines, multi-file paths, empty, non-header) - 4 TypeScriptConformanceRunner (hand-picked acceptance, harness error on missing file, directive skip short-circuit, multi-file deferred bucket) - 7 TypeScriptConformanceBaseline (encode bucket, diff: empty/regression/ new-pass/skipped-reason-change/removed-entry) Subset enumeration + InterpretedBaseline() loop are #85's scope.
Wires up the InterpretedBaseline test fact that enumerates a configured subset, runs each test through TypeScriptConformanceRunner, and diffs the results against a committed baseline. Mirrors SharpTS.Test262 structure. Subset choice: types/typeRelationships/assignmentCompatibility/ (70 tests) — smaller than expressions/binaryOperators (lib-heavy) and types/conditional (parser-blocked); dense type-system mechanics aligned with what SharpTS already type-checks well. Initial bucket distribution: ParseError = 57 (81%) Fail = 9 (13%) Pass = 4 ( 6%) Skipped = 0 The high ParseError rate is a real finding, not a runner bug: ~all blocked tests fail on `declare function` syntax our parser doesn't yet support (also blocks conditionalTypes1.ts from #84 acceptance). Supporting ambient declarations would likely unblock a major chunk of the conformance corpus and is worth tracking as a follow-up — but out of scope here. The point of #85 is to establish the ratchet, not pre-fix what the ratchet surfaces. Pass=4/70 looks small but the baseline's job is to lock in current behavior so regressions can't slip past, not to look impressive on day one. Test262's initial baseline had the same shape. New files: - TypeScriptConformanceConfig.cs — JSON config record + skip-list loaders - TypeScriptConformanceTests.cs — InterpretedBaseline() fact + summary + bucket-diff logging - config/subset.json — folder list + 5s timeout + skip refs - config/skip-directives.txt — 6 directives wholesale-skipped (experimentalDecorators, jsx, etc.) - config/skip-tests.txt — empty by-path escape hatch (structure) - baselines/interpreted.txt — 70-line committed baseline Modified: - TypeScriptConformanceRunner.cs — accepts skipTests set; checks before reading the file (escape hatch for tests that crash the runner) Diff harness verified end-to-end: - Re-run with no source change: clean no-op (229ms). - Tampered baseline (4 Pass→Fail entries): hard-failed with actionable "4 new passes" message, listing exact paths that drifted. Per #84/#85 design notes still in effect: - (line, tsCode) match strategy; column ignored. - Lib-drift filter buckets ~zero tests in this subset (no missing-global patterns hit) — will exercise more on lib-heavy folders later. - Multi-file tests bucketed Skipped:multi-file-deferred (none in this folder).
Closes Phase-1 work for #80 epic. Updates: - README.md — new Testing section after Project Status; documents both conformance projects, their submodule init, run commands, and baseline update env vars. Calls out the Windows long-paths requirement up front. - STATUS.md — new "17. CONFORMANCE TEST SUITES" section between .NET Interop and Breaking Changes; lists current pass rate for each suite (Test262: 10132/10132 on subset; TS conformance: 4/70 = 5.7% on assignmentCompatibility); explains why the high ParseError rate is the parser's `declare function` gap, not a runner bug. - CLAUDE.md — brief Conformance Suites section so future sessions know the runners exist, the bucket model, and the env-var switches. - SharpTS.TypeScriptConformance/README.md — refresh from #81's scaffolding- era version to reflect the now-landed runner: full bucket model, match strategy, lib-drift filter design, configuration files, and pointers to #80/#83/#95/#99. - SharpTS.Test262/README.md — new file, mirrors the TS conformance README's structure (per #89 acceptance: "create one for both"). Minimal parallel docs so neither project lacks a project-level README. A new contributor can now clone the repo, init either submodule, run the corresponding conformance suite, and update its baseline without asking questions — the #89 acceptance bar.
feat: expand TS conformance subset to types/conditional (#85) Add tests/cases/conformance/types/conditional (9 tests) to the conformance subset and record the baseline. All 9 currently bucket as ParseError — the folder exercises parser features SharpTS does not yet support (declare function, mapped types `[K in keyof T]`, indexed access on a type literal, leading-& intersections). The baseline captures the current state so parser improvements surface as bucket changes. @
added 6 commits
June 5, 2026 23:35
Gap 2 of #119. The tuple ([...]) and inline-object ({...}) cases in ParsePrimaryType early-returned, bypassing the array-suffix / indexed-access loop at the end of the method. As a result `{ [K in keyof T]: T[K] }[keyof T]`, `{ a: number }["a"]`, `[string, number][0]`, and `[string, number][]` all failed with "Expect ';' after type alias". Fold both cases into the if/else chain (assign typeName instead of returning, matching the existing parenthesized-type case) so the postfix [...] suffix applies to braced/tuple type literals, not just named types. Conformance baseline: assignmentCompatBetweenTupleAndArray.ts moves ParseError -> Fail. Other conditional-folder files remain ParseError, now blocked only by Gap 1 (declare function) and Gap 3 (leading-operator unions).
Gap 1 of #119. `declare` previously hard-required `class` to follow (Parser.Declarations.cs `Consume(TokenType.CLASS, ...)`), so `declare function`/`const`/`let`/`var`/`enum`/`interface`/`type`/`namespace` all failed with "Expect 'class' after 'declare'". Replace the hard consume with an AmbientDeclaration() dispatch that routes to the existing declaration parsers. The bodyless/initializer-less ambient forms are already supported by those parsers (FunctionDeclaration via its overload-signature path, AmbientVarDeclaration with no initializer), so this is purely a dispatch change. Decorators remain rejected on non-class declare forms. Conformance baseline: conditionalTypes2.ts ParseError -> Fail, intersectionIncludingPropFromGlobalAugmentation.ts ParseError -> Pass. conditionalTypes1.ts now blocked only by Gap 3 (leading-operator unions).
Gap 3 of #119. A leading `|` or `&` before the first member of a union or intersection type was rejected ("Expect type") — only the infix forms `A | B` and `A & B` parsed. This shape is common when each member is written on its own line: type D<T> = & { x: number } & { y: number }; Tolerate an optional leading operator at the start of ParseUnionType / ParseIntersectionType before parsing the first member. Safe because these are only reached in type context, where a leading `|`/`&` is unambiguously a type operator. No committed-subset baseline change: the conditional files using this shape are also blocked by other unsupported syntax (e.g. `keyof this`), so they remain ParseError for now. Verified via direct repros.
`this` was not accepted in type position — ParsePrimaryType listed the keyword-types that can start a named type (void/null/never/...) but not THIS, so `): this`, `keyof this`, and `this[K]` all failed with "Expect type". Add TokenType.THIS to the accepted primary-type tokens (lexeme "this"). The existing keyof prefix and array-suffix/indexed-access loop then cover `keyof this` and `this[K]` for free. Parsing only — no checker-side this-typing. No committed-subset baseline change: conditionalTypes1.ts (the motivating file) is now additionally blocked by generic function-type signatures `(<U extends boolean>(a: U) => never)`, and no other subset file was blocked solely on `this`. Verified via direct repros.
Two recursion bugs in the type-string resolver, both reachable once more complex type syntax began parsing (the #119/#120 parser work): 1. TryParseIndexedAccessType recursed on `$"{result}{remaining}"` for chained indexed access (T[K][J]). Since IndexedAccess.ToString() round-trips to "T[K]", the reconstructed string equalled the input and the call never made progress -- infinite recursion on ANY chained access. Replaced with an iterative loop that consumes each `[...]` / `[]` suffix against the already-built result. 2. A mutual cycle (ToTypeInfo -> ParseGenericTypeReference -> ResolveGenericType -> ParseInlineObjectTypeInfo -> ParseMappedTypeInfo -> TryParseIndexedAccessType -> ToTypeInfo) could recurse unboundedly on self-referential mapped/indexed type strings. Added a depth backstop on ToTypeInfo (MaxTypeResolutionDepth) that throws a catchable TypeCheckException (TS2456) instead of letting an uncatchable StackOverflowException tear down the process. A StackOverflowException cannot be caught, so without these the conformance runner aborts mid-run ("Test Run Aborted") rather than bucketing the file.
A function type with leading type parameters -- `<T>(x: T) => T`, `<U extends boolean>(a: U) => never` -- was not parseable: ParsePrimaryType had no branch for a leading `<` in type position, so it failed with "Expect type". Add a branch that parses `<...>` via ParseTypeParameters, then the function body via the existing ParseFunctionTypeBody, emitting `<T>(params) => ret`. Extract the type-parameter-to-string rendering from ParseMethodSignature into a shared FormatTypeParams helper. This was the last parser blocker on conditionalTypes1.ts (`(<U extends boolean> (a: U) => never) extends ...`). Conformance baseline: 9 tests off ParseError (5 -> Pass, 4 -> Fail), mostly the assignmentCompatibility call-signature family. Buckets now 54 ParseError / 15 Fail / 10 Pass (was 63/11/5). Depends on the companion typecheck recursion fix -- conditionalTypes1.ts would otherwise stack-overflow the checker once it parses.
This was referenced Jun 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TypeScript conformance test runner — Phase 1 (errors baseline)
Implements the Phase-1 deliverables of epic #80: a parallel conformance runner (mirroring
SharpTS.Test262/) that diffs our type checker's diagnostics againstmicrosoft/TypeScript's*.errors.txtbaselines, matching on(file, line, TSnnnn)tuples.Closes #81, #82, #84, #85, #89. Implements #95 and the runner-side lib-drift filter of #83.
What's included
SharpTS.TypeScriptConformance/project (standalone, not inSharpTS.sln) +external/typescriptsubmodule pinned to v5.5.4 (c8a7d58).// @target,@strict,@lib,@filenamemulti-file split, …).TsCodefield onDiagnostic; 333 type-check throw sites tagged withTSnnnncodes acrossTypeSystem/.Pass/Fail/ParseError/TypeCheckError/Skipped/HarnessError),SHARPTS_TSCONFORMANCE_UPDATE_BASELINE=1switch, per-test timeout, lib-drift skip filter (TS Conformance: resolve lib.d.ts from vendored TypeScript checkout #83).types/typeRelationships/assignmentCompatibility/(70 files).Provenance / why now
This work was authored on a worktree branch a month ago and never opened as a PR. It was 459 commits behind
main. This PR is that work rebased onto currentorigin/main, with conflicts resolved:SharpTS.csproj— kept both the Test262.Worker and TypeScriptConformance compile-exclusion blocks.TypeChecker.Properties.New.cs—mainintentionally relaxednewon a non-class callee to defer to runtime (no throw). Keptmain's behavior; the obsoleteTS2351throw site that TS Conformance: assign TSnnnn diagnostic codes at type-check throw sites #95 had tagged no longer exists.Verification
dotnet build SharpTS.csproj— clean (1 pre-existing unrelated warning inCompilation/).dotnet build+dotnet teston the conformance project — 31/31 pass.UPDATE_BASELINE=1: zero content drift vs. the committed baseline (byte-identical modulo EOL), confirming the rebased checker produces the same diagnostics on this subset.Windows caveat
Initializing the submodule requires
git config core.longpaths true—microsoft/TypeScripthas baseline filenames exceedingMAX_PATH. A handful of files undertests/baselines/reference/still fail to check out on Windows even with long paths enabled; these are not undertests/cases/conformance/and do not affect the runner. (Worth a note in the README / a follow-up for the CI doc.)Not in this PR (tracked follow-ups)
tsclib.d.ts into the checker.