Add a conformance suite for HermitCrab, with its adequacy argument - #480
Open
johnml1135 wants to merge 16 commits into
Open
Add a conformance suite for HermitCrab, with its adequacy argument#480johnml1135 wants to merge 16 commits into
johnml1135 wants to merge 16 commits into
Conversation
johnml1135
force-pushed
the
integrate-conformance-framework
branch
from
August 19, 2026 19:46
d826869 to
8f9dcea
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #480 +/- ##
==========================================
- Coverage 73.53% 73.41% -0.13%
==========================================
Files 449 546 +97
Lines 37633 55283 +17650
Branches 5174 7527 +2353
==========================================
+ Hits 27673 40585 +12912
- Misses 8824 12898 +4074
- Partials 1136 1800 +664 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
johnml1135
force-pushed
the
integrate-conformance-framework
branch
from
August 20, 2026 06:27
47b6149 to
d0b5448
Compare
Fixtures with per-fixture hashes, a published manifest, and a documented contract for running an external engine against them. This is what a consumer receives: the fixtures, the manifest, and PROTOCOL.md, and nothing else in this branch. Also a census of the compilations the pinned compiler produced, a denominator for rule interactions, and per-phase parse observation. The interaction ledger is stated as a corpus statistic rather than a bound. It counts what these grammars happen to contain, which is not a claim about what a grammar could contain.
A denominator read off the corpus measures the corpus. So the interface inventory comes from the DTD, and RequiredToLoad splits into RequiredByDtd and RequiredByLoader, because "the DTD demands it" and "the loader demands it" are different facts about a grammar. The surfaces no consumer reads are named rather than quietly counted, and the host configuration the fixtures assume becomes part of the contract. Real grammars supply the SHAPE of a feature and never its scale.
A construct can be present, referenced correctly, load cleanly, and change no parse. Only severance that changes a parse counts as evidence. Adds the interaction-chain layer, per-grammar witness traceability that holds presence and witness apart, exercises: corroboration, and fold-in candidates. The witness ledger records every fixture that witnesses a surface, not just the first. The earlier strictly-better tie-break silently preferred a fabricated edge case over a language grammar on every tie, which hid how much coverage the real grammars already carried.
The doer skill, the review skill, the measurement, and the revision loop, with the suite documentation shipped alongside the fixtures. Measurement and revision stay separate deliberately. Doing both in one pass makes a change impossible to attribute to either.
Morpher.Synthesize sizes its Parallel.ForEach off ProcessorCount, so a mutant could return a different verdict per run. Pinning DOTNET_PROCESSOR_COUNT=1 in the child makes two independent sweeps byte-identical, which is the property the drift gates already assumed and did not have. Test child processes are bounded to the test's own lifetime. Adds the conformance workflow, the fixture index, and the RuleName/FailureAllomorph threading that lets an infinite-loop crash name the rule responsible. The expensive sweep runs on a schedule, not on every pull request. The dead-rule step is advisory: --coverage-report reports dead rules and still exits 0, and making that a hard failure while 14 exist would fail CI on its first run.
An obligation is worth covering only where the engine can act on it, the DTD can declare it, and a real FieldWorks project can produce it. Failing any one of the three is an exclusion, not a gap. Layer one is FailureReason, HermitCrab's own 23-member enumeration of the decisions it makes when declining to apply something. Witness status comes from a real traced run, and six of those gates are reachable through no XML attribute at all, so an attribute-keyed denominator cannot see them. Layer three is HCLoader, the component that turns a LibLCM model into an HC grammar and therefore integrates both the model and what is reachable from it. Of 83 subjects, 61 are producible and 22 are not. Keying MC/DC to the gates also fixes a miscount: the attribute-keyed ledger emits four arms per chain but can certify at most one, because it reads the arm off the reader attribute's spelling. The older ledger is kept as a cross-check on writer-reader chains, which the gate ledger does not measure, and the docs say which number is the claim. Ledger assertions read the checked-in file; only a freshness check recomputes, behind [Explicit]. An earlier version reswept four times in the default suite.
Fifty-one unmet obligations, each assigned a bucket from the blocker its own ledger records, with the construct stated in plain language so a reader can judge whether covering it is worth anything. The split that matters: thirteen are a HARNESS gap, not a coverage gap, and no amount of authoring touches them. Seven control arms already have their word in the corpus and fail only because GrammarRuleIndex cannot resolve Allomorph, MorphologicalInput, AffixTemplate or PhonologicalSubrule to a fired-rule id. Six more are reachable only through element content, for which no severance primitive exists. Also writes down the engine and tooling facts each author had been rediscovering at real cost: severance is fixture-wide and can unblock as easily as block; a DTD #REQUIRED attribute can never be severed, which makes both co-occurrence gates unwitnessable by any word; outputPartOfSpeech overrides rather than sets; stem names compare by object identity; and a Timeout is a statement about the machine, not the grammar. check-obligation-feasibility.ps1 answers the three mechanical questions before a budget is committed, and claims only that no known mechanism forbids a witness.
No grammar in the corpus declared an MPR gate on a phonological subrule, so exception features and minor rules -- one of the most ordinary things a real grammar does -- had no coverage at all. HCLoader emits both at lines 2057 and 2058, from FLEx's own Required and Excluded rule features. Satisfied cells go from four to nine, and the recurring shape is worth knowing: a feature the ROOT presets cannot witness a required-gate chain, because severing it can only turn a passing word into a failing one. What works is a feature an affix confers or destroys before the reader checks it. Two items are foreclosed by the engine rather than unattempted. Stem names compare by bare object identity, and a root-preset feature meets a plain set-membership test, so in both cases removing the payload can only break a match and never repair one. Raises the mutant budget from 45s to 180s and drops the confirmation retry. A Timeout roved to a new fixture that proved Unobservable in isolation while needing about 128s uncontended, and genuine non-termination has never once been observed here.
Every figure was read from a checked-in ledger at the time of writing. Interface edges: 60 declared, 44 present, 19 witnessed. Obligation cells: 346 enumerated, 18 worth covering, 8 defensible. Gate arms: 46, of which 42 are worth covering and 14 are evidenced, giving 5 of 23 gates both MC/DC arms. Those are findings, not a backlog, and the docs now say so. The apparatus is frozen here on purpose: it established what the fixtures witness and produced the map of what remains, and its marginal return then fell to roughly nothing. obligation-triage.tsv classifies every unmet obligation by the blocker its own ledger records, and most of the remainder is impossible or strained rather than merely undone. One row changed what it MEASURES, and the docs say so rather than swapping a number quietly. The interaction-chain row reported chains whose writer and reader are each evidenced somewhere, which two separate words satisfy; it now reports chains with a same-word paired witness. Resolving a control arm to its nearest rule ancestor makes three more attributable. Four others never will be: Allomorph is always a child of LexicalEntry and never of a rule, and the DTD gives AffixTemplate no id at all, so neither has any identity to attribute a control to. An adversarial review then broke one claim. The affix-conferred blocking cell cited HCLoader.cs:1717 as its read, but that line populates ExcludedMprFeatures only from slot.ReferringObjects.OfType<ILexEntryInflType>(), FieldWorks' own irregular-form blocking. The witness is behaviourally genuine and the fixture stays as an engine test, but the claim is withdrawn. The general fact matters more than the cell. Producibility is keyed per attribute and does not compose along a chain, and no gate in this repository can catch that, because it requires reading another one. The branch's own working documents come out with it. Twenty plans, censuses and rationale files are deleted and their reasoning moves to the pull request, which is the one place a reader always looks and which survives being read a year later. Two were genuinely durable and move into conformance/docs instead: the HermitCrab XML semantic catalogue, and the pipeline design. Nothing shipped now points at a document a consumer does not receive. That eviction was not cosmetic. docs/conformance-migration-ledger.md was still load-bearing in code: it sat in ConformanceManifestGenerator's AdditionalSourceFiles, which is hashed into every manifest's SourceHash, so deleting the file would have permanently recorded an absent contributor to that hash, and a parametrized test would have thrown on a missing file.
Two independent reasons every job failed. global.json pinned SDK 10.0.303 with rollForward disabled. Hosted runners carry 10.0.110, 10.0.204, 10.0.302 and 10.0.400 -- never 303 -- so the pin was unsatisfiable by construction and all three jobs died on their first dotnet invocation. That version was one developer's local SDK. The pin now states a floor with latestFeature instead of an exact patch. The part that matters is unchanged: the census still analyses this repository with the same Roslyn that builds it, taken from $(MSBuildToolsPath)/Roslyn/bincore rather than a NuGet package. An exact patch bought nothing beyond that, because SdkVersion feeds a graph hash that is recomputed every run and compared against nothing checked in, so any .NET 10 SDK is self-consistent. Pinning one that is not installable everywhere only made the repository unbuildable. Second, ci.yml gated on the semantic-coverage CLI, which exits non-zero whenever the catalog is incomplete. The catalog is INTENTIONALLY an incomplete proposal backlog, and unclassified-mapping is its expected state: the CLI reported zero new gaps, zero stale lines and zero unbacked quotients while still failing. The test named TheCheckedInCatalogMapsEveryRealSurfaceExactlyOnce already runs that identical audit under dotnet test, and asserts both that the catalog is incomplete and that unclassified-mapping is the only diagnostic class allowed, which is a stronger gate than an exit code and can express the intent. The redundant step is gone and the comment names its real gate. The CLI's exit code conflating "not complete" with "regressed" is left as it is, and recorded in the pull request rather than changed under time pressure.
CI runs dotnet csharpier check, and 200 files in this branch had never been run through the formatter. Formatting them exposed a genuine disagreement between two pre-existing tools: .editorconfig sets csharp_prefer_braces to when_multiline, so a brace-less if is legal with a one-line body and an IDE0011 error once the body wraps. CSharpier wrapped 83 such bodies, all of them long argument lists, which turned legal code into build errors under TreatWarningsAsErrors. Master never hits this because its brace-less bodies are short enough not to wrap. Adding braces satisfies both tools, so neither .editorconfig nor the CSharpier configuration is touched -- the repository's style is the repository's own call. Convergence took two passes. The second was not redundant: while the Conformance project failed to build, the Tests project could not compile at all, so six of the 83 sites stayed invisible until the first pass fixed its project reference. Verified that no logic moved, rather than assuming it: comparing token streams with whitespace and braces stripped, every difference across all 108 changed files is either a trailing comma CSharpier adds to a multi-line initialiser or a using-directive reorder. Both are inert in C#.
The compilation-graph hash is relocation-invariant so a graph captured on one machine can be verified on another, which means the path vocabulary has to understand a Windows-shaped path and a Unix-shaped one regardless of the host. LogicalPathTokens carries its own IsAbsolute and NormalizeAbsolute for exactly that reason. Three places reached for the platform's own path handling instead, and all three failed only on Linux. A csc switch is syntactically an absolute path on Unix, so switch shape must be decided before any path handling: /noconfig carries no value, /doc:/tmp/x.xml carries one, and /home/me/src/X.cs is a real path. The discriminator is what follows the switch name -- nothing, a colon, or a further separator. The ancestor .editorconfig probe used Path.GetFileName, whose notion of a separator is the host's, so a Windows-shaped path there returned the whole string as its filename on Linux and the probe silently declined a file it should have admitted. It now segments with this class's own rules. Finally, a mismatch reason gave only a count. A count cannot be diagnosed from a CI log on a machine you cannot reach, so it now names the words and their expected and actual analyses, truncated and capped at five.
Master replaced the SINGLE_THREADED compile symbol with a runtime MaxDegreeOfParallelism, so the engine no longer contains any code that exists in only some configurations. Two SemanticCoverage tests were reading engine code as their live specimen and lost it. Both point the census at real repository source rather than at a string literal, which is the whole reason they exist -- the synthetic-source tests beside each already cover the same logic on fixtures. So they still read real source, but source whose shape this branch is answerable for. The local-function keying test reads FailureRuleAttributor's Walk in place of Morpher's deleted GenerateSynthesis. The configuration-only census test reads Morpher's surviving #if OUTPUT_ANALYSES region, scoped to the type rather than to one method signature, so that an upstream rename cannot present itself here as a census defect. The second test's claim is narrower than it was, and the name now says so: it witnessed a method that existed only under a symbol, and no such method remains anywhere in the engine. What it still witnesses is a configuration-gated region censused under exactly the configurations that contain it. Its expected configuration string was confirmed by running the test, not derived.
johnml1135
force-pushed
the
integrate-conformance-framework
branch
from
August 20, 2026 09:10
d0b5448 to
9dfb2f6
Compare
Threading rule.Name down to the 256-node throw site cost a required parameter on two public constructors, SynthesisRewriteRuleSpec and EpenthesisSynthesisRewriteSubruleSpec. Both ship in the HermitCrab NuGet package, so that is a source and binary break for anyone constructing them. Nothing read what it bought. InfiniteLoopException.RuleName has no reader anywhere in the repository: the conformance runner mentions the exception only in a comment, the fixture that provokes it asserts expect_crash and not a message, and the engine test asserts only Throws.TypeOf<InfiniteLoopException>(). A capability with no consumer does not justify breaking a published constructor, so the four files go back to what master has. The 256-node cap itself is untouched, and edge-cases/simultaneous- epenthesis-cascade still pins it.
GrammarHealthChecker and GrammarHealthFinding are a diagnostic feature, not part of the conformance suite: nothing in conformance/ or in the Conformance project calls them, and their only caller here was their own test file. They are wanted in FieldWorks on a schedule of their own, so they ship in #475 instead, which now carries the newer copies this branch had developed. Two tests go with them that cannot follow them there. Both load real conformance fixtures, and one uses the Conformance project's Fixture.DiscoverAll, so neither compiles on a branch that has no fixture tree. They belong here rather than in #475, and should come back once that lands; until then they are recoverable from 9dfb2f6. What remains of this branch's engine footprint is Trace.FailureAllomorph, its assignment in TraceManager from an argument that method already received and discarded, and the inert SemanticBranch capture point.
SemanticBranch was introduced by this branch's first commit as a way for engine code to declare a semantic path that the census would then count. It was never wired up. No engine or tool code has called SemanticBranch.Hit in any commit here, no ledger, catalog or coverage file has ever carried a branch: id or a branch-marker surface, and no document names it as a mechanism anyone intends to use. Every coverage number this branch reports comes from severance sweeps and trace evidence instead. So it was a public type in the shipped netstandard2.0 package with no consumer, and the census carried a fail-closed resolution path that could never fire. It could not move to the Conformance project either: that project references the engine and not the reverse, so a marker channel living there could never be called from the engine that is its only intended caller. Removed with it: CollectMarkers and its two resolution helpers in CSharpInventoryReader, the branch-marker catalog family, and three tests that existed only to exercise markers. Three further tests used a marker as an incidental probe and keep their subject. Two lose nothing -- the if/else-if arms keep an inert body, and the reachable-versus-dead claim was already carried by the xml-read assertions beside it. The third is narrower than it was and its name now says so: it asserted that an unresolved marker in dead code does not trip the fail-closed check, and with no such check left it now asserts only that an unresolved call reached from dead code raises no diagnostic. What remains of this branch's engine footprint is Trace.FailureAllomorph and its assignment in TraceManager: two files, thirteen lines, no new public type, and both read by FailureRuleAttributor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a conformance suite for HermitCrab: 33 fixtures, 446 cases, and a published protocol
(
conformance/PROTOCOL.md) that an alternative implementation can be checked against. It has alreadypinned two C# HermitCrab defects — one of them fixed upstream as #471 — and found one bug in a
reimplementation.
Why it is 85,000 lines. About a fifth is the deliverable a consumer reads: the fixtures, the
manifest, and the protocol. The rest measures how much of the engine those fixtures actually witness,
and publishes what it cannot claim. Commit 1 is the deliverable and stands alone — if you review
one commit, review that one. The other fifteen are the adequacy argument and can be judged
separately, or not at all.
Where to look
conformance/PROTOCOL.md— the published contract. Is it implementable without reading anything else?conformance/docs/README.md— itsSTATUSsection draws the freeze line. Is it in the right place?--coverage-reportreports 15 dead rules and still exits 0, so that CI step is advisory, not blocking.Allomorphhas no rule ancestor,AffixTemplatehas noid.Deliberately not here
conformance/obligation-triage.tsvclassifies every unmet obligation by the blocker its own ledgerrecords; most of the remainder is impossible or strained rather than merely undone.
fieldworks-producibility.tsv— it snapshots a repository this one does not depend on.Verification. Full suite 570 passed, 1 skipped, 0 failed. Conformance self-check 33/33 including
pathological. Coarse construct parity 29/29 in-scope. Zero-warning build with
TreatWarningsAsErrorsand
GenerateDocumentationFileon.gitlintclean on all sixteen commits. Measured through theprotocol, a reimplementation passes 30 of 33 fixtures; the three misses are one documented divergence
where that engine is correct, and two from a single ordering bug in it. The
[Explicit]regenerationsweeps run on a schedule, not per pull request.
Rebased onto master's analysis-cascade memo. Master replaced the
SINGLE_THREADEDcompile symbolwith a runtime
MaxDegreeOfParallelism, so this branch's fix to theSINGLE_THREADEDbuild ofMorpher.GenerateWordsno longer has anything to fix: the branch now leavesMorpher.csexactly asmaster has it. Two consequences a reviewer should weigh rather than take on trust. The census still
declares
single-threadedandcombinedbuild profiles that now compile identically to the othertwo — left declared rather than removed, because the symbol is still legal and the apparatus is
frozen, but they are two censuses that can no longer distinguish anything. And two tests had used
engine code as their live specimen — the
GenerateSynthesislocal function, and theSINGLE_THREADED-onlySynthesizeoverload — both of which master deleted; they now read specimensthis branch owns, or a region that survives, so an upstream refactor cannot present itself here as a
census defect.
Three things came out of this branch rather than into it. The rule-name threading that let the
epenthesis infinite-loop throw name its rule is gone: nothing ever read
InfiniteLoopException.RuleName,and paying for it meant adding a required parameter to two public constructors
(
SynthesisRewriteRuleSpec,EpenthesisSynthesisRewriteSubruleSpec) in the shipped package — a realbreak for an unused capability. And
GrammarHealthChecker/GrammarHealthFindingnow ship separately in#475, where they are wanted on their own schedule for FieldWorks; nothing in the conformance suite calls
them. Two tests that ran the checker over the real fixtures went with them out of this branch and belong
back here once #475 lands; they are preserved in this branch's history at
9dfb2f67.Third,
SemanticBranchis gone. This branch's first commit added it so engine code could declare asemantic path for the census to count, and it was never wired up: no engine or tool code ever called
SemanticBranch.Hit, no ledger or catalog ever carried abranch:id, and no document names it as amechanism anyone intends to use. Every coverage number above comes from severance sweeps and trace
evidence instead. Relocating it was not an option either — the Conformance project references the engine
and not the reverse, so a marker channel living there could never be called by its only intended caller.
Its census support and three marker-only tests went with it; three further tests kept their subject and
swapped the probe, one ending up narrower than it was, which its new name records.
What is left in the engine is two files and twelve lines.
TracegainsFailureAllomorph;TraceManagerassigns it from anallomorphargument that method already received and threw away. Nonew public type, nothing removed, no signature changed — a consumer of the HermitCrab package sees one
new property on a trace node and nothing else.
Reading this a year from now — start here
This branch deleted twenty of its own working documents — plans, censuses, per-fixture rationale,
migration ledgers — and moved their reasoning here. That was deliberate, not loss. Those documents
described how the suite was built; the accordions below describe what it decided and why, which is the
part that stays useful.
Two documents were genuinely durable and moved into the tree rather than here:
conformance/docs/hc-xml-semantic-catalog.md(HermitCrab XML semantics) andconformance/docs/pipeline-design.md. Everything shipped now lives underconformance/, so a consumerwho receives only that directory has no pointer they cannot follow.
If you are picking this up to extend it, read
conformance/docs/decisions-and-lessons.mdfirst, andconformance/docs/severance-mechanics.mdbefore writing any fixture. The second one exists becausethree separate efforts each rediscovered the same engine facts at considerable cost.
The layer cake — what measures what
Five denominators, each stricter than the one above it, each with its own ledger under
conformance/:semantic-coverage-counterfactuals.tsvinterface-inventory.tsv,interface-witness.tsvinteraction-chains.tsvdataflow-obligations.tsvgate-obligations.tsvThe pipeline behind them is inventory → evidence → proof → completeness, described in
conformance/docs/pipeline-design.md.The single most important distinction, and the one this suite got wrong five separate times before
getting it right: a construct being present in a grammar is not the same as its being witnessed.
Presence means it is declared and loads. Witnessed means severing it changes a parse. Only the second
transfers to another implementation, because only the second is observable from outside.
Decisions, and why
Denominators come from the DTD and the engine, never from the corpus. A denominator read off the
corpus measures the corpus. The cost of this choice is that it weights an obscure template-ordering gate
equally with plural suffixation — deliberately decoupled from the distribution real projects occupy.
MC/DC is keyed to
FailureReason, HermitCrab's own enumeration of its decisions, not to DTD attributepairs. The attribute-keyed version failed in both directions: it emitted four MC/DC arms per chain but
could certify at most one, because it inferred the arm from the reader attribute's spelling; and it had
no row at all for six gates that no XML attribute reaches. Keying to the engine's own 23 gates fixes
both. The older ledger is kept as a cross-check on writer/reader chains, which the gate ledger does not
measure.
An obligation is worth covering only if three authorities permit it: the engine can act on it
(
FailureReason), the DTD can declare it (HermitCrabInput.dtd), and a real FieldWorks project canproduce it (
HCLoader.cs). Failing any one is an exclusion, not a gap. This is what took the honesttarget from 346 cells to 18.
A Timeout is a statement about the machine, not the grammar. Verdicts are pinned at zero timeouts and
the mutant budget is 180s, because every timeout ever investigated here resolved to a real verdict on a
quieter machine — two of them to
Evidenced, having suppressed real coverage while they stood.The apparatus is frozen rather than finished. Its marginal return fell to roughly nothing once it had
established what the fixtures witness and produced the map of what remains.
conformance/docs/README.mdnames what is live, what is frozen, and what nobody should extend.
Paths not taken
Plan-shaped fixture recipes. Erased by minimisation; the real axis turned out to be which compiler
emits the grammar, not how the plan was shaped.
A hand-rolled memory watchdog and a machine-wide reservation ledger for bounding sweep cost. Written,
then deleted in favour of an OS primitive: the kernel enforces at allocation time with no sampling
interval to lose a spike in.
A counted semaphore for build concurrency. Replaced by N named mutexes after it deadlocked every
worktree on the machine — a counted semaphore never restores its count when a holder dies, and in agent
workflows holders die constantly.
NIST SP 800-142 as the justification for stopping at all-uses rather than all-du-paths. Cited, then
withdrawn on inspection: it argues the opposite. Replaced with the classical path-explosion argument.
Corpus-derived coverage denominators. Rejected for the reason in Decisions above, with the cost
acknowledged rather than hidden.
Fabricating narrow grammars per uncovered area. Rejected in favour of adding realistic features to
existing typologically-coherent grammars. One real Bantu grammar closed gaps that 33 synthetic fixtures
had missed, which is the strongest single empirical result on this branch.
Reversals, invisible in the squashed history
A coverage claim was withdrawn after an adversarial review broke it. The affix-conferred blocking cell
cited
HCLoader.cs:1717as its read. That line populatesExcludedMprFeaturesonly fromslot.ReferringObjects.OfType<ILexEntryInflType>()— FieldWorks' own irregular-form blocking — never froman affix output. Both citations were real references to real assignments, and the edit passed every
automated gate including the 33/33 self-check. The witness is behaviourally genuine so the fixture stays
as an engine test, but the claim is gone and the defensible figure is 8 of 18 rather than 9.
That miscitation had been promoted into the authoring skill as its canonical worked legitimate
example, so it would have taught the next author to repeat it. Replaced with a worked example that reads
the engine first, plus a section on citations that look load-bearing and are not.
A crash expectation was retired.
metathesis-comparison-crashexpected the engine to throw, and itsown header said the expectation should come out once the engine was fixed. #471 fixed it; nobody removed
the expectation. A reimplementation had been recorded as diverging on that fixture when in fact our
expectation was stale — a divergence entry that would have laundered an out-of-date fixture into an
accepted engine difference.
The authoring harness was revised once, and the revision did not raise its yield (1/6, then 1/5). It
converted false negatives into verified negatives, which is worth more than the number suggests: five
cells had been abandoned on an argument that was simply wrong.
Surprising findings
MC/DC on
FailureReasoncovers only the engine's rejection half. It says nothing about thetransformation half — rewriting, metathesis, epenthesis, reduplication, overwrite semantics, stratum
ordering. A reimplementation diverges by producing a different signature, not by misjudging a gate, and
all three defects this suite has found were of that kind. The suite's own founding defect, MPR-overwrite
order dependence, is not a
FailureReasoneither.Raising the mutant budget produced more coverage than most authoring did. 45s to 180s lifted evidenced
witness rows from 76 to 85, because timed-out mutants were being recorded as no-evidence.
Severance is fixture-wide, and that inverts the intuition. It strips an attribute from every element at
once, so removing a payload can unblock a word as easily as block one. Reasoning about it per-element
makes a same-word fail-to-pass flip look impossible when it is routine — the most common wrong conclusion
on this branch.
Deleting the working documents nearly corrupted a manifest hash.
docs/conformance-migration-ledger.mdsat inConformanceManifestGenerator.AdditionalSourceFiles, whichis hashed into every manifest's
SourceHash. Deleting the file would have permanently recorded an absentcontributor to that hash, and a parametrized test would have thrown on a missing file. The tidying step was
the dangerous one.
What this does NOT authorize
It does not establish that the fixture set is complete. It establishes what the fixture set witnesses,
publishes the gap, and names which parts of the gap are impossible to close.
It does not make the coverage ledgers a consumer obligation. A consumer implements
PROTOCOL.mdandproduces the same parses. Nothing else here is measured against them, and the ledgers ship only because a
claim you cannot inspect is a claim you must take on trust.
It does not license extending the apparatus. The severance primitives, the authoring-yield loop, and the
producibility snapshot are named in
conformance/docs/README.mdas things not to extend.It is not precedent for coverage denominators derived from an engine's internals in general. That choice
was right here because a second implementation must match this engine.
Deferred, and what would unblock it
Making the dead-rule gate blocking.
--coverage-reportreports 15 dead rules and exits 0. Turning ithard requires addressing those 15 first, or CI fails on its first run. Both halves belong in one change.
A severance primitive for element content. Three gates (
Environments,Pattern,DisjunctiveAllomorph) are reachable only through child elements, and the tool can only remove attributes.Named as do not extend because the campaign it would serve has ended.
Instrumenting one dead raise site.
AnalysisCompoundingRule.csdiscards a POS-mismatched non-head witha bare
continueand no trace call, so the corresponding synthesis-side gate is unobservable. One linewould fix it, and it is a genuine engine improvement independent of coverage.
A cheap flag for
construct-claim-corroboration.tsv. It is a fixture-read artifact bundled behind anexpensive severance sweep, so changing one label costs a full regeneration.
Differential testing between two engines. Both already speak
PROTOCOL.md. Running them against eachother — on generated grammars, or better, on the grammars real projects produce — measures the
transformation half this apparatus does not. This is where the next real gain is, and it is not here.
Recommended next steps — in priority order
Ordered by expected value, from what was actually measured here rather than from what would be tidy.
1. Differential testing between two engines — the highest-value channel, and it is not in this
branch. Both engines already speak
PROTOCOL.md, the signature algorithm is shared, and both areconformant today, so the expensive parts are already paid for. Run them against each other and minimise
each disagreement into a fixture. This attacks the transformation half of the engine that no layer here
measures, and every hit arrives as a discriminating fixture, which is the deliverable.
Two flavours, and the second is stronger. On generated grammars you must answer "which engine is
right?" for every disagreement, and some will land where behaviour is simply unspecified. On the grammars
real projects produce, this engine is definitionally the oracle — a reimplementation exists to
reproduce it — so a disagreement is a defect report rather than a question, and the triage cost that
usually kills fuzzing efforts collapses.
Before scaling either: run a pilot and measure the disagreement rate and the fraction that prove to be
real bugs. If roughly one in five disagreements is a real bug, this is excellent. If it is one in fifty,
triage will eat the budget and the real-project variant is the better bet. That number is a day of work
and it decides the strategy.
2. Make the dead-rule gate actually gate.
--coverage-reportreports 15 dead rules and exits 0. Theexit code and the 15 rules have to move together — flipping the exit code alone fails CI on its first run,
and a gate that fails on arrival gets switched off and then protects nothing. A fixture declaring rules no
word exercises is a coverage finding in its own right.
3. Instrument the one dead raise site.
AnalysisCompoundingRule.csdiscards a part-of-speech-mismatchednon-head with a bare
continueand no trace call, so the corresponding synthesis-side gate can never beobserved. One line. It is a genuine diagnostic improvement whether or not anyone cares about coverage, and
it happens to make two obligations reachable.
4. Give
construct-claim-corroboration.tsvits own regeneration flag. It is derived from readingfixtures, with no severance involved, but it is bundled behind a sweep that takes ten minutes. Changing one
label currently costs a full regeneration; this was paid four times in a single merge.
What not to do, and why it is listed here. Do not extend the coverage apparatus. Measured authoring
yield was one in five, and a single configuration change — raising a mutant budget — produced more coverage
than most authoring runs did.
conformance/obligation-triage.tsvclassifies every unmet obligation by itsblocker, and most of the remainder is impossible or strained rather than merely undone. Read that file
before assuming a number should be higher; it was built to be a permission to stop.
If you do pick up the funnel anyway, start with
conformance/docs/severance-mechanics.mdand runconformance/tools/check-obligation-feasibility.ps1before committing any budget. Three separate effortseach rediscovered the same engine facts at considerable cost, and that file exists so a fourth does not.
Evidence behind the pitch
Two C# HermitCrab defects.
metathesis-comparison-crashpinned a comparator invariant violation("Only nodes from the same list can be compared"), fixed upstream as #471.
simultaneous-epenthesis-cascadepins a 256-node infinite-loop cap viaexpect_crash.One reimplementation bug. Two fixtures fail there for a single cause:
multipleApplicationOrder="rightToLeftIterative"produces the left-to-right result when a rule has multiple firing sites in one pass. Twelve of the 33
fixtures had never been run against that engine; eleven passed and the twelfth found this.
The 15 dead rules. Verified directly:
--coverage-reportprints*** 15 DEAD RULE(S) (exercised by zero words) ***and exits 0, and the checked-inrules.csvcarries the same 15 rows with an empty
wordscolumn.Producibility does not compose. Both
MorphologicalOutput.MPRFeaturesandMorphologicalInput.excludedMPRFeaturesare individually producible byHCLoader; the chain between themis not, because the two are populated from different FLEx fields that can never hold the same value. The
same reader is reachable from a lexical entry (
HCLoader.cs:746), which is why one cell using it isgenuine and another is not.
Four unattributable arms. Per the DTD,
Allomorphis always a child ofLexicalEntryand never of arule, and
AffixTemplateis never given anid. Neither has an identity a Control arm could name.Reproducibility. Two independent sweeps produce byte-identical ledgers after pinning
DOTNET_PROCESSOR_COUNT=1in the mutant child process;Morpher.Synthesizesizes itsParallel.ForEachoff
ProcessorCount, so a mutant could otherwise return a different verdict per run.Preflight review details
Code Review Summary
Branch: integrate-conformance-framework
Base: origin/master
Date: 2026-08-19 (re-verified 2026-08-20 after rebasing onto master's analysis-cascade memo)
Review model: Claude Opus 5
Files changed: 622 (88,452 insertions, 5 deletions)
Overview
A conformance suite for HermitCrab: 33 fixtures and 446 cases with a published adapter protocol, so
an alternative implementation can be checked against the C# engine. It has already pinned two C# HC
defects (one fixed upstream as #471) and found one bug in a reimplementation.
Shipped alongside it is a coverage apparatus that measures how much of the engine those fixtures
actually witness. That apparatus is deliberately frozen rather than finished, and the branch says so
in
conformance/docs/README.md: its marginal return fell to roughly nothing once it had establishedwhat the fixtures witness and produced the map of what remains. Roughly 18% of the branch is the
deliverable a consumer reads; the rest is the adequacy argument and its instrumentation.
The review found no open Critical findings. The most serious issue was found by an adversarial pass
and fixed before this summary: a coverage claim resting on a citation that did not support it.
Contract/API Changes
conformance/PROTOCOL.md(a 3-argument batch invocation and a 5-column TSVreply). A consumer needs this and the fixtures; nothing else in the branch.
hc-conformance:--adapter,--capabilities, and a family of--write-*regeneration flags with matching check flags.
GrammarRuleIndexgainsResolveAncestorRuleId.CounterfactualGate.DefaultTimeoutbecomesinternalso one constant serves both gates.FailureAllomorphon failed trace nodes (Trace), populated inTraceManagerfrom an argument that method already received and discarded. Two files, twelve addedlines, no new public type. Both are read by
FailureRuleAttributor, which lives in the Conformanceproject rather than the engine.
gains a required parameter. (An earlier revision of this branch did add a required parameter to
SynthesisRewriteRuleSpecandEpenthesisSynthesisRewriteSubruleSpec; that change is withdrawn.)Findings
Critical - Must address before merge
adversarial pass traced
HCLoader.cs:1717and found it populatesExcludedMprFeaturesonly fromslot.ReferringObjects.OfType<ILexEntryInflType>(), FieldWorks' irregular-form blocking, never froman affix output. The witness is behaviourally genuine so the fixture stays as an engine test, but the
claim is withdrawn and the defensible figure is 8 of 18 rather than 9.)
example (fixed during review: replaced with the compound-category edit, which read the engine
first and kept a failed attempt in its commit message, plus a new section on citations that look
load-bearing and are not.)
Important - Should address before merge
--coverage-reportreports dead rules and exits 0, so the CI dead-rule step is advisoryrather than blocking. Verified directly: it prints
*** 15 DEAD RULE(S) (exercised by zero words) ***and exits 0, and the checked-in
rules.csvcarries the same 15 rows with an emptywordscolumn.Deliberate for now - making it blocking while 15 exist would fail CI on its first run - and documented
in the workflow comment. The 15 dead rules are themselves a coverage finding. (This review first
recorded 14; the ledger has said 15 throughout, so the count was misread, not changed.)
repository can catch that, because it requires reading a different repository. This is the hole the
Critical finding above came through. Recorded in
conformance/docs/severance-mechanics.md.conformance/fieldworks-producibility.tsvis a hand-researched snapshot of an externalrepository with no drift check. Deliberate: the repo does not depend on FieldWorks, so a drift test
would fail for anyone without it checked out. Every verdict carries file and line citations so it can
be re-derived.
headline table still read 9 arms and 1 both-arm gate against the ledger's 14 and 5. An earlier commit
meant to publish correct numbers had fixed a second table and missed the one a reader sees first.)
Minor - Consider
Allomorphis always a child ofLexicalEntryandnever of a rule, and the DTD gives
AffixTemplatenoid, so neither has an identity a Control armcould name. Recorded rather than left as an open gap.
construct-claim-corroboration.tsvis a fixture-read artifact bundled behind an expensiveseverance sweep, so changing one label costs a full regeneration. It wants its own flag.
without making them fewer. Recorded in
conformance/harness-runs/first-pass-yield.tsv.plan-doc pointers, over-length lines, absence narration, and non-ASCII punctuation; history collapsed
to 9 commits whose messages pass gitlint.)
///summary was attached to the wrong method (fixed during review: a summary describingRunCoverageEvidencesat aboveRunConformanceManifest, so two doc blocks merged into one wrongdoc.)
Required Validation / Evidence
Run and green:
dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests- 570 passed, 1 skipped,0 failed (571 total). Re-run after the rebase and after all three scope reductions; the difference
from the 561 first recorded here is master's own analysis-cascade memo tests arriving in the base,
less the seven that left with the grammar health checker and the three that tested branch markers.
python conformance/parity-check.py- 29/29 in-scope constructs.TreatWarningsAsErrorsandGenerateDocumentationFileon.gitlint --ignore body-is-missing --commits origin/master..HEAD- exit 0 on all 16 commits.(zero non-comment changed lines).
PROTOCOL.md: 30 of 33 fixtures pass;the 3 misses are one documented divergence where the other engine is correct, and two from a single
ordering bug in that engine.
Re-verified after rebasing onto master's analysis-cascade memo, which removed the
SINGLE_THREADEDcompile symbol this branch had been fixing the build of. The branch no longer modifies
Morpher.cs.Two tests that used engine code as their live specimen (
GenerateSynthesis, and theSINGLE_THREADED-onlySynthesizeoverload) were retargeted onto specimens this branch owns or aregion that survives; both were the census apparatus reading real source, and neither assertion
changed in substance.
Not run, deliberately:
[Explicit]freshness sweeps that regenerate ledgers from scratch (7-16 minutes each). They runon a schedule in
counterfactual-coverage.yml, not per pull request.Positive Observations
does not care about coverage theory.
defended. The general fact behind it - producibility not composing along a chain - is now documented.
and recomputed rather than predicted evidence.
conformance/obligation-triage.tsvis a machine-derived permission to stop, which is a more usefulartifact than a higher coverage number would have been.
stay cheap. An earlier version reswept four times in the default suite and was fixed.
Interview Notes
Author's stated purpose, in their own words: a conformance suite so an alternative implementation can
be checked against the C# engine, which has already caught two C# HC bugs and one bug in the
reimplementation, plus a coverage apparatus measuring what the fixtures witness and publishing what it
cannot claim.
On whether to finish the apparatus: the author's judgement, after a meta review, is that the system is
half-built and has value as it stands, that building it further may not pay, and that discarding it
would be foolish. The decision recorded in the branch is to freeze rather than finish, and to say so in
the artifact. The
STATUSsection inconformance/docs/README.mdwas added for exactly this reason,and it names what is live, what is frozen, what nobody should extend, and where the next real gain is.
On what the apparatus does not measure: MC/DC keyed to
FailureReasoncovers the engine's rejectionhalf. It says nothing about the transformation half, which is where a reimplementation actually
diverges - all three defects found so far were of that kind. This is stated in the
STATUSsectionrather than left for a reader to discover.
Unresolved and accepted: the advisory-only dead-rule gate, and the absence of any mechanism in this
repository that can catch a non-composing producibility chain.
In-Review Quality Check
Comment hygiene, commit messages, and the withdrawn claim were all fixed during review. After each, the
conformance self-check and the ledger tests were re-run; the full suite was run after the last change
that could affect behaviour.
Suggested Review Focus
one - the other fifteen are the adequacy argument and can be judged separately.
conformance/PROTOCOL.mdis the published contract. Is it complete enough to implement againstwithout reading any other file?
STATUSsection inconformance/docs/README.md- does it draw the freeze line where youwould draw it?
This change is