Skip to content

Add a conformance suite for HermitCrab, with its adequacy argument - #480

Open
johnml1135 wants to merge 16 commits into
masterfrom
integrate-conformance-framework
Open

Add a conformance suite for HermitCrab, with its adequacy argument#480
johnml1135 wants to merge 16 commits into
masterfrom
integrate-conformance-framework

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Adds a conformance suite for HermitCrab: 33 fixtures, 446 cases, and a published protocol
(conformance/PROTOCOL.md) that an alternative implementation can be checked against. It has already
pinned two C# HermitCrab defects — one of them fixed upstream as #471 — and found one bug in a
reimplementation.

Why it is 85,000 lines. About a fifth is the deliverable a consumer reads: the fixtures, the
manifest, and the protocol. The rest measures how much of the engine those fixtures actually witness,
and publishes what it cannot claim. Commit 1 is the deliverable and stands alone — if you review
one commit, review that one. The other fifteen are the adequacy argument and can be judged
separately, or not at all.

Where to look

  • conformance/PROTOCOL.md — the published contract. Is it implementable without reading anything else?
  • conformance/docs/README.md — its STATUS section draws the freeze line. Is it in the right place?
  • --coverage-report reports 15 dead rules and still exits 0, so that CI step is advisory, not blocking.
  • Producibility is keyed per attribute and does not compose along a chain; no gate here can catch that.
  • Four gate arms can never be attributed: Allomorph has no rule ancestor, AffixTemplate has no id.

Deliberately not here

  • The apparatus is frozen, not finished: 14 of 46 gate arms evidenced, 8 of 18 defensible cells.
    conformance/obligation-triage.tsv classifies every unmet obligation by the blocker its own ledger
    records; most of the remainder is impossible or strained rather than merely undone.
  • No severance primitive for element content, so three gates are unwitnessable by construction.
  • No drift check on fieldworks-producibility.tsv — it snapshots a repository this one does not depend on.

Verification. Full suite 570 passed, 1 skipped, 0 failed. Conformance self-check 33/33 including
pathological. Coarse construct parity 29/29 in-scope. Zero-warning build with TreatWarningsAsErrors
and GenerateDocumentationFile on. gitlint clean on all sixteen commits. Measured through the
protocol, a reimplementation passes 30 of 33 fixtures; the three misses are one documented divergence
where that engine is correct, and two from a single ordering bug in it. The [Explicit] regeneration
sweeps run on a schedule, not per pull request.

Rebased onto master's analysis-cascade memo. Master replaced the SINGLE_THREADED compile symbol
with a runtime MaxDegreeOfParallelism, so this branch's fix to the SINGLE_THREADED build of
Morpher.GenerateWords no longer has anything to fix: the branch now leaves Morpher.cs exactly as
master has it. Two consequences a reviewer should weigh rather than take on trust. The census still
declares single-threaded and combined build profiles that now compile identically to the other
two — left declared rather than removed, because the symbol is still legal and the apparatus is
frozen, but they are two censuses that can no longer distinguish anything. And two tests had used
engine code as their live specimen — the GenerateSynthesis local function, and the
SINGLE_THREADED-only Synthesize overload — both of which master deleted; they now read specimens
this branch owns, or a region that survives, so an upstream refactor cannot present itself here as a
census defect.

Three things came out of this branch rather than into it. The rule-name threading that let the
epenthesis infinite-loop throw name its rule is gone: nothing ever read InfiniteLoopException.RuleName,
and paying for it meant adding a required parameter to two public constructors
(SynthesisRewriteRuleSpec, EpenthesisSynthesisRewriteSubruleSpec) in the shipped package — a real
break for an unused capability. And GrammarHealthChecker/GrammarHealthFinding now ship separately in
#475, where they are wanted on their own schedule for FieldWorks; nothing in the conformance suite calls
them. Two tests that ran the checker over the real fixtures went with them out of this branch and belong
back here once #475 lands; they are preserved in this branch's history at 9dfb2f67.

Third, SemanticBranch is gone. This branch's first commit added it so engine code could declare a
semantic path for the census to count, and it was never wired up: no engine or tool code ever called
SemanticBranch.Hit, no ledger or catalog ever carried a branch: id, and no document names it as a
mechanism anyone intends to use. Every coverage number above comes from severance sweeps and trace
evidence instead. Relocating it was not an option either — the Conformance project references the engine
and not the reverse, so a marker channel living there could never be called by its only intended caller.
Its census support and three marker-only tests went with it; three further tests kept their subject and
swapped the probe, one ending up narrower than it was, which its new name records.

What is left in the engine is two files and twelve lines. Trace gains FailureAllomorph;
TraceManager assigns it from an allomorph argument that method already received and threw away. No
new public type, nothing removed, no signature changed — a consumer of the HermitCrab package sees one
new property on a trace node and nothing else.


Reading this a year from now — start here

This branch deleted twenty of its own working documents — plans, censuses, per-fixture rationale,
migration ledgers — and moved their reasoning here. That was deliberate, not loss. Those documents
described how the suite was built; the accordions below describe what it decided and why, which is the
part that stays useful.

Two documents were genuinely durable and moved into the tree rather than here:
conformance/docs/hc-xml-semantic-catalog.md (HermitCrab XML semantics) and
conformance/docs/pipeline-design.md. Everything shipped now lives under conformance/, so a consumer
who receives only that directory has no pointer they cannot follow.

If you are picking this up to extend it, read conformance/docs/decisions-and-lessons.md first, and
conformance/docs/severance-mechanics.md before writing any fixture. The second one exists because
three separate efforts each rediscovered the same engine facts at considerable cost.

The layer cake — what measures what

Five denominators, each stricter than the one above it, each with its own ledger under conformance/:

layer denominator demonstrated source
Unit surfaces 264 grammar-observable 110 semantic-coverage-counterfactuals.tsv
Interface edges 60 declared, 44 present 19 interface-inventory.tsv, interface-witness.tsv
Interaction chains 40 declared, 26 exercised 9 paired interaction-chains.tsv
Obligation cells 346 enumerated, 18 worth covering 8 defensible dataflow-obligations.tsv
Gate arms (MC/DC) 46, 42 worth covering 14 gate-obligations.tsv

The pipeline behind them is inventory → evidence → proof → completeness, described in
conformance/docs/pipeline-design.md.

The single most important distinction, and the one this suite got wrong five separate times before
getting it right: a construct being present in a grammar is not the same as its being witnessed.
Presence means it is declared and loads. Witnessed means severing it changes a parse. Only the second
transfers to another implementation, because only the second is observable from outside.

Decisions, and why

Denominators come from the DTD and the engine, never from the corpus. A denominator read off the
corpus measures the corpus. The cost of this choice is that it weights an obscure template-ordering gate
equally with plural suffixation — deliberately decoupled from the distribution real projects occupy.

MC/DC is keyed to FailureReason, HermitCrab's own enumeration of its decisions, not to DTD attribute
pairs.
The attribute-keyed version failed in both directions: it emitted four MC/DC arms per chain but
could certify at most one, because it inferred the arm from the reader attribute's spelling; and it had
no row at all for six gates that no XML attribute reaches. Keying to the engine's own 23 gates fixes
both. The older ledger is kept as a cross-check on writer/reader chains, which the gate ledger does not
measure.

An obligation is worth covering only if three authorities permit it: the engine can act on it
(FailureReason), the DTD can declare it (HermitCrabInput.dtd), and a real FieldWorks project can
produce it (HCLoader.cs). Failing any one is an exclusion, not a gap. This is what took the honest
target from 346 cells to 18.

A Timeout is a statement about the machine, not the grammar. Verdicts are pinned at zero timeouts and
the mutant budget is 180s, because every timeout ever investigated here resolved to a real verdict on a
quieter machine — two of them to Evidenced, having suppressed real coverage while they stood.

The apparatus is frozen rather than finished. Its marginal return fell to roughly nothing once it had
established what the fixtures witness and produced the map of what remains. conformance/docs/README.md
names what is live, what is frozen, and what nobody should extend.

Paths not taken

Plan-shaped fixture recipes. Erased by minimisation; the real axis turned out to be which compiler
emits the grammar, not how the plan was shaped.

A hand-rolled memory watchdog and a machine-wide reservation ledger for bounding sweep cost. Written,
then deleted in favour of an OS primitive: the kernel enforces at allocation time with no sampling
interval to lose a spike in.

A counted semaphore for build concurrency. Replaced by N named mutexes after it deadlocked every
worktree on the machine — a counted semaphore never restores its count when a holder dies, and in agent
workflows holders die constantly.

NIST SP 800-142 as the justification for stopping at all-uses rather than all-du-paths. Cited, then
withdrawn on inspection: it argues the opposite. Replaced with the classical path-explosion argument.

Corpus-derived coverage denominators. Rejected for the reason in Decisions above, with the cost
acknowledged rather than hidden.

Fabricating narrow grammars per uncovered area. Rejected in favour of adding realistic features to
existing typologically-coherent grammars. One real Bantu grammar closed gaps that 33 synthetic fixtures
had missed, which is the strongest single empirical result on this branch.

Reversals, invisible in the squashed history

A coverage claim was withdrawn after an adversarial review broke it. The affix-conferred blocking cell
cited HCLoader.cs:1717 as its read. That line populates ExcludedMprFeatures only from
slot.ReferringObjects.OfType<ILexEntryInflType>() — FieldWorks' own irregular-form blocking — never from
an affix output. Both citations were real references to real assignments, and the edit passed every
automated gate including the 33/33 self-check. The witness is behaviourally genuine so the fixture stays
as an engine test, but the claim is gone and the defensible figure is 8 of 18 rather than 9.

That miscitation had been promoted into the authoring skill as its canonical worked legitimate
example
, so it would have taught the next author to repeat it. Replaced with a worked example that reads
the engine first, plus a section on citations that look load-bearing and are not.

A crash expectation was retired. metathesis-comparison-crash expected the engine to throw, and its
own header said the expectation should come out once the engine was fixed. #471 fixed it; nobody removed
the expectation. A reimplementation had been recorded as diverging on that fixture when in fact our
expectation was stale — a divergence entry that would have laundered an out-of-date fixture into an
accepted engine difference.

The authoring harness was revised once, and the revision did not raise its yield (1/6, then 1/5). It
converted false negatives into verified negatives, which is worth more than the number suggests: five
cells had been abandoned on an argument that was simply wrong.

Surprising findings

MC/DC on FailureReason covers only the engine's rejection half. It says nothing about the
transformation half — rewriting, metathesis, epenthesis, reduplication, overwrite semantics, stratum
ordering. A reimplementation diverges by producing a different signature, not by misjudging a gate, and
all three defects this suite has found were of that kind. The suite's own founding defect, MPR-overwrite
order dependence, is not a FailureReason either.

Raising the mutant budget produced more coverage than most authoring did. 45s to 180s lifted evidenced
witness rows from 76 to 85, because timed-out mutants were being recorded as no-evidence.

Severance is fixture-wide, and that inverts the intuition. It strips an attribute from every element at
once, so removing a payload can unblock a word as easily as block one. Reasoning about it per-element
makes a same-word fail-to-pass flip look impossible when it is routine — the most common wrong conclusion
on this branch.

Deleting the working documents nearly corrupted a manifest hash.
docs/conformance-migration-ledger.md sat in ConformanceManifestGenerator.AdditionalSourceFiles, which
is hashed into every manifest's SourceHash. Deleting the file would have permanently recorded an absent
contributor to that hash, and a parametrized test would have thrown on a missing file. The tidying step was
the dangerous one.

What this does NOT authorize

It does not establish that the fixture set is complete. It establishes what the fixture set witnesses,
publishes the gap, and names which parts of the gap are impossible to close.

It does not make the coverage ledgers a consumer obligation. A consumer implements PROTOCOL.md and
produces the same parses. Nothing else here is measured against them, and the ledgers ship only because a
claim you cannot inspect is a claim you must take on trust.

It does not license extending the apparatus. The severance primitives, the authoring-yield loop, and the
producibility snapshot are named in conformance/docs/README.md as things not to extend.

It is not precedent for coverage denominators derived from an engine's internals in general. That choice
was right here because a second implementation must match this engine.

Deferred, and what would unblock it

Making the dead-rule gate blocking. --coverage-report reports 15 dead rules and exits 0. Turning it
hard requires addressing those 15 first, or CI fails on its first run. Both halves belong in one change.

A severance primitive for element content. Three gates (Environments, Pattern,
DisjunctiveAllomorph) are reachable only through child elements, and the tool can only remove attributes.
Named as do not extend because the campaign it would serve has ended.

Instrumenting one dead raise site. AnalysisCompoundingRule.cs discards a POS-mismatched non-head with
a bare continue and no trace call, so the corresponding synthesis-side gate is unobservable. One line
would fix it, and it is a genuine engine improvement independent of coverage.

A cheap flag for construct-claim-corroboration.tsv. It is a fixture-read artifact bundled behind an
expensive severance sweep, so changing one label costs a full regeneration.

Differential testing between two engines. Both already speak PROTOCOL.md. Running them against each
other — on generated grammars, or better, on the grammars real projects produce — measures the
transformation half this apparatus does not. This is where the next real gain is, and it is not here.

Recommended next steps — in priority order

Ordered by expected value, from what was actually measured here rather than from what would be tidy.

1. Differential testing between two engines — the highest-value channel, and it is not in this
branch. Both engines already speak PROTOCOL.md, the signature algorithm is shared, and both are
conformant today, so the expensive parts are already paid for. Run them against each other and minimise
each disagreement into a fixture. This attacks the transformation half of the engine that no layer here
measures, and every hit arrives as a discriminating fixture, which is the deliverable.

Two flavours, and the second is stronger. On generated grammars you must answer "which engine is
right?" for every disagreement, and some will land where behaviour is simply unspecified. On the grammars
real projects produce, this engine is definitionally the oracle — a reimplementation exists to
reproduce it — so a disagreement is a defect report rather than a question, and the triage cost that
usually kills fuzzing efforts collapses.

Before scaling either: run a pilot and measure the disagreement rate and the fraction that prove to be
real bugs
. If roughly one in five disagreements is a real bug, this is excellent. If it is one in fifty,
triage will eat the budget and the real-project variant is the better bet. That number is a day of work
and it decides the strategy.

2. Make the dead-rule gate actually gate. --coverage-report reports 15 dead rules and exits 0. The
exit code and the 15 rules have to move together — flipping the exit code alone fails CI on its first run,
and a gate that fails on arrival gets switched off and then protects nothing. A fixture declaring rules no
word exercises is a coverage finding in its own right.

3. Instrument the one dead raise site. AnalysisCompoundingRule.cs discards a part-of-speech-mismatched
non-head with a bare continue and no trace call, so the corresponding synthesis-side gate can never be
observed. One line. It is a genuine diagnostic improvement whether or not anyone cares about coverage, and
it happens to make two obligations reachable.

4. Give construct-claim-corroboration.tsv its own regeneration flag. It is derived from reading
fixtures, with no severance involved, but it is bundled behind a sweep that takes ten minutes. Changing one
label currently costs a full regeneration; this was paid four times in a single merge.

What not to do, and why it is listed here. Do not extend the coverage apparatus. Measured authoring
yield was one in five, and a single configuration change — raising a mutant budget — produced more coverage
than most authoring runs did. conformance/obligation-triage.tsv classifies every unmet obligation by its
blocker, and most of the remainder is impossible or strained rather than merely undone. Read that file
before assuming a number should be higher; it was built to be a permission to stop.

If you do pick up the funnel anyway, start with conformance/docs/severance-mechanics.md and run
conformance/tools/check-obligation-feasibility.ps1 before committing any budget. Three separate efforts
each rediscovered the same engine facts at considerable cost, and that file exists so a fourth does not.

Evidence behind the pitch

Two C# HermitCrab defects. metathesis-comparison-crash pinned a comparator invariant violation
("Only nodes from the same list can be compared"), fixed upstream as #471.
simultaneous-epenthesis-cascade pins a 256-node infinite-loop cap via expect_crash.

One reimplementation bug. Two fixtures fail there for a single cause: multipleApplicationOrder="rightToLeftIterative"
produces the left-to-right result when a rule has multiple firing sites in one pass. Twelve of the 33
fixtures had never been run against that engine; eleven passed and the twelfth found this.

The 15 dead rules. Verified directly: --coverage-report prints
*** 15 DEAD RULE(S) (exercised by zero words) *** and exits 0, and the checked-in rules.csv
carries the same 15 rows with an empty words column.

Producibility does not compose. Both MorphologicalOutput.MPRFeatures and
MorphologicalInput.excludedMPRFeatures are individually producible by HCLoader; the chain between them
is not, because the two are populated from different FLEx fields that can never hold the same value. The
same reader is reachable from a lexical entry (HCLoader.cs:746), which is why one cell using it is
genuine and another is not.

Four unattributable arms. Per the DTD, Allomorph is always a child of LexicalEntry and never of a
rule, and AffixTemplate is never given an id. Neither has an identity a Control arm could name.

Reproducibility. Two independent sweeps produce byte-identical ledgers after pinning
DOTNET_PROCESSOR_COUNT=1 in the mutant child process; Morpher.Synthesize sizes its Parallel.ForEach
off ProcessorCount, so a mutant could otherwise return a different verdict per run.

Preflight review details

Code Review Summary

Branch: integrate-conformance-framework

Base: origin/master

Date: 2026-08-19 (re-verified 2026-08-20 after rebasing onto master's analysis-cascade memo)

Review model: Claude Opus 5

Files changed: 622 (88,452 insertions, 5 deletions)

Overview

A conformance suite for HermitCrab: 33 fixtures and 446 cases with a published adapter protocol, so
an alternative implementation can be checked against the C# engine. It has already pinned two C# HC
defects (one fixed upstream as #471) and found one bug in a reimplementation.

Shipped alongside it is a coverage apparatus that measures how much of the engine those fixtures
actually witness. That apparatus is deliberately frozen rather than finished, and the branch says so
in conformance/docs/README.md: its marginal return fell to roughly nothing once it had established
what the fixtures witness and produced the map of what remains. Roughly 18% of the branch is the
deliverable a consumer reads; the rest is the adequacy argument and its instrumentation.

The review found no open Critical findings. The most serious issue was found by an adversarial pass
and fixed before this summary: a coverage claim resting on a citation that did not support it.

Contract/API Changes

  • New published contract: conformance/PROTOCOL.md (a 3-argument batch invocation and a 5-column TSV
    reply). A consumer needs this and the fixtures; nothing else in the branch.
  • New CLI surface on hc-conformance: --adapter, --capabilities, and a family of --write-*
    regeneration flags with matching check flags.
  • GrammarRuleIndex gains ResolveAncestorRuleId.
  • CounterfactualGate.DefaultTimeout becomes internal so one constant serves both gates.
  • Engine footprint, in full: a FailureAllomorph on failed trace nodes (Trace), populated in
    TraceManager from an argument that method already received and discarded. Two files, twelve added
    lines, no new public type. Both are read by FailureRuleAttributor, which lives in the Conformance
    project rather than the engine.
  • No public HermitCrab API removed or altered in a breaking way, and no existing public constructor
    gains a required parameter. (An earlier revision of this branch did add a required parameter to
    SynthesisRewriteRuleSpec and EpenthesisSynthesisRewriteSubruleSpec; that change is withdrawn.)

Findings

Critical - Must address before merge

  • A coverage claim rested on a citation that does not support it (fixed during review: an
    adversarial pass traced HCLoader.cs:1717 and found it populates ExcludedMprFeatures only from
    slot.ReferringObjects.OfType<ILexEntryInflType>(), FieldWorks' irregular-form blocking, never from
    an affix output. The witness is behaviourally genuine so the fixture stays as an engine test, but the
    claim is withdrawn and the defensible figure is 8 of 18 rather than 9.)
  • The miscitation had been promoted into the authoring skill as its canonical worked LEGITIMATE
    example
    (fixed during review: replaced with the compound-category edit, which read the engine
    first and kept a failed attempt in its commit message, plus a new section on citations that look
    load-bearing and are not.)

Important - Should address before merge

  • --coverage-report reports dead rules and exits 0, so the CI dead-rule step is advisory
    rather than blocking. Verified directly: it prints *** 15 DEAD RULE(S) (exercised by zero words) ***
    and exits 0, and the checked-in rules.csv carries the same 15 rows with an empty words column.
    Deliberate for now - making it blocking while 15 exist would fail CI on its first run - and documented
    in the workflow comment. The 15 dead rules are themselves a coverage finding. (This review first
    recorded 14; the ledger has said 15 throughout, so the count was misread, not changed.)
  • Producibility is keyed per attribute and does not compose along a chain, and no gate in this
    repository can catch that, because it requires reading a different repository. This is the hole the
    Critical finding above came through. Recorded in conformance/docs/severance-mechanics.md.
  • conformance/fieldworks-producibility.tsv is a hand-researched snapshot of an external
    repository
    with no drift check. Deliberate: the repo does not depend on FieldWorks, so a drift test
    would fail for anyone without it checked out. Every verdict carries file and line citations so it can
    be re-derived.
  • The primary funnel table understated its own claim by roughly 3x (fixed during review: the
    headline table still read 9 arms and 1 both-arm gate against the ledger's 14 and 5. An earlier commit
    meant to publish correct numbers had fixed a second table and missed the one a reader sees first.)

Minor - Consider

  • Four gate arms can never be attributed. Allomorph is always a child of LexicalEntry and
    never of a rule, and the DTD gives AffixTemplate no id, so neither has an identity a Control arm
    could name. Recorded rather than left as an open gap.
  • construct-claim-corroboration.tsv is a fixture-read artifact bundled behind an expensive
    severance sweep
    , so changing one label costs a full regeneration. It wants its own flag.
  • Measured authoring yield is 1 of 5, unmoved by a revision that made the failures honest
    without making them fewer. Recorded in conformance/harness-runs/first-pass-yield.tsv.
  • 88 comment-hygiene violations and 25 gitlint violations (fixed during review: internal
    plan-doc pointers, over-length lines, absence narration, and non-ASCII punctuation; history collapsed
    to 9 commits whose messages pass gitlint.)
  • A /// summary was attached to the wrong method (fixed during review: a summary describing
    RunCoverageEvidence sat above RunConformanceManifest, so two doc blocks merged into one wrong
    doc.)

Required Validation / Evidence

Run and green:

  • Full suite: dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests - 570 passed, 1 skipped,
    0 failed
    (571 total). Re-run after the rebase and after all three scope reductions; the difference
    from the 561 first recorded here is master's own analysis-cascade memo tests arriving in the base,
    less the seven that left with the grammar health checker and the three that tested branch markers.
  • Coarse construct parity: python conformance/parity-check.py - 29/29 in-scope constructs.
  • Conformance self-check including pathological: 33 passed, 0 failed, 0 skipped.
  • Zero-warning build with TreatWarningsAsErrors and GenerateDocumentationFile on.
  • gitlint --ignore body-is-missing --commits origin/master..HEAD - exit 0 on all 16 commits.
  • Comment-hygiene audit re-run after cleanup - 0 violations, and the cleanup verified comment-only
    (zero non-comment changed lines).
  • Differential measurement against a reimplementation through PROTOCOL.md: 30 of 33 fixtures pass;
    the 3 misses are one documented divergence where the other engine is correct, and two from a single
    ordering bug in that engine.

Re-verified after rebasing onto master's analysis-cascade memo, which removed the SINGLE_THREADED
compile symbol this branch had been fixing the build of. The branch no longer modifies Morpher.cs.
Two tests that used engine code as their live specimen (GenerateSynthesis, and the
SINGLE_THREADED-only Synthesize overload) were retargeted onto specimens this branch owns or a
region that survives; both were the census apparatus reading real source, and neither assertion
changed in substance.

Not run, deliberately:

  • The [Explicit] freshness sweeps that regenerate ledgers from scratch (7-16 minutes each). They run
    on a schedule in counterfactual-coverage.yml, not per pull request.
  • Manual validation in any host application. None was performed and none is claimed.

Positive Observations

  • The suite has found three real defects, which is the only argument for it that survives a reader who
    does not care about coverage theory.
  • An adversarial review broke one of the branch's own claims, and the claim was withdrawn rather than
    defended. The general fact behind it - producibility not composing along a chain - is now documented.
  • The presence-versus-witness discipline transferred into fixture quality: control words, minimal pairs,
    and recomputed rather than predicted evidence.
  • conformance/obligation-triage.tsv is a machine-derived permission to stop, which is a more useful
    artifact than a higher coverage number would have been.
  • Ledger assertions read the checked-in file and only a freshness check recomputes, so the cheap gates
    stay cheap. An earlier version reswept four times in the default suite and was fixed.

Interview Notes

Author's stated purpose, in their own words: a conformance suite so an alternative implementation can
be checked against the C# engine, which has already caught two C# HC bugs and one bug in the
reimplementation, plus a coverage apparatus measuring what the fixtures witness and publishing what it
cannot claim.

On whether to finish the apparatus: the author's judgement, after a meta review, is that the system is
half-built and has value as it stands, that building it further may not pay, and that discarding it
would be foolish. The decision recorded in the branch is to freeze rather than finish, and to say so in
the artifact. The STATUS section in conformance/docs/README.md was added for exactly this reason,
and it names what is live, what is frozen, what nobody should extend, and where the next real gain is.

On what the apparatus does not measure: MC/DC keyed to FailureReason covers the engine's rejection
half. It says nothing about the transformation half, which is where a reimplementation actually
diverges - all three defects found so far were of that kind. This is stated in the STATUS section
rather than left for a reader to discover.

Unresolved and accepted: the advisory-only dead-rule gate, and the absence of any mechanism in this
repository that can catch a non-composing producibility chain.

In-Review Quality Check

Comment hygiene, commit messages, and the withdrawn claim were all fixed during review. After each, the
conformance self-check and the ledger tests were re-run; the full suite was run after the last change
that could affect behaviour.

Suggested Review Focus

  • Commit 1 alone is the deliverable a consumer reads. If you only review one commit, review that
    one - the other fifteen are the adequacy argument and can be judged separately.
  • conformance/PROTOCOL.md is the published contract. Is it complete enough to implement against
    without reading any other file?
  • The advisory-only dead-rule gate: acceptable as documented, or should it block before merge?
  • The STATUS section in conformance/docs/README.md - does it draw the freeze line where you
    would draw it?

This change is Reviewable

@johnml1135
johnml1135 requested a review from ddaspit August 19, 2026 19:27
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from d826869 to 8f9dcea Compare August 19, 2026 19:46
@codecov-commenter

codecov-commenter commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.41%. Comparing base (5d26fac) to head (c0ac5c9).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #480      +/-   ##
==========================================
- Coverage   73.53%   73.41%   -0.13%     
==========================================
  Files         449      546      +97     
  Lines       37633    55283   +17650     
  Branches     5174     7527    +2353     
==========================================
+ Hits        27673    40585   +12912     
- Misses       8824    12898    +4074     
- Partials     1136     1800     +664     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from 47b6149 to d0b5448 Compare August 20, 2026 06:27
Fixtures with per-fixture hashes, a published manifest, and a documented
contract for running an external engine against them. This is what a
consumer receives: the fixtures, the manifest, and PROTOCOL.md, and
nothing else in this branch.

Also a census of the compilations the pinned compiler produced, a
denominator for rule interactions, and per-phase parse observation.

The interaction ledger is stated as a corpus statistic rather than a
bound. It counts what these grammars happen to contain, which is not a
claim about what a grammar could contain.
A denominator read off the corpus measures the corpus. So the interface
inventory comes from the DTD, and RequiredToLoad splits into
RequiredByDtd and RequiredByLoader, because "the DTD demands it" and
"the loader demands it" are different facts about a grammar.

The surfaces no consumer reads are named rather than quietly counted,
and the host configuration the fixtures assume becomes part of the
contract.

Real grammars supply the SHAPE of a feature and never its scale.
A construct can be present, referenced correctly, load cleanly, and
change no parse. Only severance that changes a parse counts as evidence.

Adds the interaction-chain layer, per-grammar witness traceability that
holds presence and witness apart, exercises: corroboration, and fold-in
candidates.

The witness ledger records every fixture that witnesses a surface, not
just the first. The earlier strictly-better tie-break silently preferred
a fabricated edge case over a language grammar on every tie, which hid
how much coverage the real grammars already carried.
The doer skill, the review skill, the measurement, and the revision
loop, with the suite documentation shipped alongside the fixtures.

Measurement and revision stay separate deliberately. Doing both in one
pass makes a change impossible to attribute to either.
Morpher.Synthesize sizes its Parallel.ForEach off ProcessorCount, so a
mutant could return a different verdict per run. Pinning
DOTNET_PROCESSOR_COUNT=1 in the child makes two independent sweeps
byte-identical, which is the property the drift gates already assumed
and did not have.

Test child processes are bounded to the test's own lifetime.

Adds the conformance workflow, the fixture index, and the
RuleName/FailureAllomorph threading that lets an infinite-loop crash
name the rule responsible. The expensive sweep runs on a schedule, not
on every pull request.

The dead-rule step is advisory: --coverage-report reports dead rules and
still exits 0, and making that a hard failure while 14 exist would fail
CI on its first run.
An obligation is worth covering only where the engine can act on it, the
DTD can declare it, and a real FieldWorks project can produce it.
Failing any one of the three is an exclusion, not a gap.

Layer one is FailureReason, HermitCrab's own 23-member enumeration of the
decisions it makes when declining to apply something. Witness status
comes from a real traced run, and six of those gates are reachable
through no XML attribute at all, so an attribute-keyed denominator cannot
see them.

Layer three is HCLoader, the component that turns a LibLCM model into an
HC grammar and therefore integrates both the model and what is reachable
from it. Of 83 subjects, 61 are producible and 22 are not.

Keying MC/DC to the gates also fixes a miscount: the attribute-keyed
ledger emits four arms per chain but can certify at most one, because it
reads the arm off the reader attribute's spelling. The older ledger is
kept as a cross-check on writer-reader chains, which the gate ledger does
not measure, and the docs say which number is the claim.

Ledger assertions read the checked-in file; only a freshness check
recomputes, behind [Explicit]. An earlier version reswept four times in
the default suite.
Fifty-one unmet obligations, each assigned a bucket from the blocker its
own ledger records, with the construct stated in plain language so a
reader can judge whether covering it is worth anything.

The split that matters: thirteen are a HARNESS gap, not a coverage gap,
and no amount of authoring touches them. Seven control arms already have
their word in the corpus and fail only because GrammarRuleIndex cannot
resolve Allomorph, MorphologicalInput, AffixTemplate or
PhonologicalSubrule to a fired-rule id. Six more are reachable only
through element content, for which no severance primitive exists.

Also writes down the engine and tooling facts each author had been
rediscovering at real cost: severance is fixture-wide and can unblock as
easily as block; a DTD #REQUIRED attribute can never be severed, which
makes both co-occurrence gates unwitnessable by any word;
outputPartOfSpeech overrides rather than sets; stem names compare by
object identity; and a Timeout is a statement about the machine, not the
grammar.

check-obligation-feasibility.ps1 answers the three mechanical questions
before a budget is committed, and claims only that no known mechanism
forbids a witness.
No grammar in the corpus declared an MPR gate on a phonological subrule,
so exception features and minor rules -- one of the most ordinary things
a real grammar does -- had no coverage at all. HCLoader emits both at
lines 2057 and 2058, from FLEx's own Required and Excluded rule features.

Satisfied cells go from four to nine, and the recurring shape is worth
knowing: a feature the ROOT presets cannot witness a required-gate
chain, because severing it can only turn a passing word into a failing
one. What works is a feature an affix confers or destroys before the
reader checks it.

Two items are foreclosed by the engine rather than unattempted. Stem
names compare by bare object identity, and a root-preset feature meets a
plain set-membership test, so in both cases removing the payload can
only break a match and never repair one.

Raises the mutant budget from 45s to 180s and drops the confirmation
retry. A Timeout roved to a new fixture that proved Unobservable in
isolation while needing about 128s uncontended, and genuine
non-termination has never once been observed here.
Every figure was read from a checked-in ledger at the time of writing.
Interface edges: 60 declared, 44 present, 19 witnessed. Obligation cells:
346 enumerated, 18 worth covering, 8 defensible. Gate arms: 46, of which
42 are worth covering and 14 are evidenced, giving 5 of 23 gates both
MC/DC arms.

Those are findings, not a backlog, and the docs now say so. The
apparatus is frozen here on purpose: it established what the fixtures
witness and produced the map of what remains, and its marginal return
then fell to roughly nothing. obligation-triage.tsv classifies every
unmet obligation by the blocker its own ledger records, and most of the
remainder is impossible or strained rather than merely undone.

One row changed what it MEASURES, and the docs say so rather than
swapping a number quietly. The interaction-chain row reported chains
whose writer and reader are each evidenced somewhere, which two separate
words satisfy; it now reports chains with a same-word paired witness.

Resolving a control arm to its nearest rule ancestor makes three more
attributable. Four others never will be: Allomorph is always a child of
LexicalEntry and never of a rule, and the DTD gives AffixTemplate no id
at all, so neither has any identity to attribute a control to.

An adversarial review then broke one claim. The affix-conferred blocking
cell cited HCLoader.cs:1717 as its read, but that line populates
ExcludedMprFeatures only from
slot.ReferringObjects.OfType<ILexEntryInflType>(), FieldWorks' own
irregular-form blocking. The witness is behaviourally genuine and the
fixture stays as an engine test, but the claim is withdrawn.

The general fact matters more than the cell. Producibility is keyed per
attribute and does not compose along a chain, and no gate in this
repository can catch that, because it requires reading another one.

The branch's own working documents come out with it. Twenty plans, censuses
and rationale files are deleted and their reasoning moves to the pull
request, which is the one place a reader always looks and which survives
being read a year later. Two were genuinely durable and move into
conformance/docs instead: the HermitCrab XML semantic catalogue, and the
pipeline design. Nothing shipped now points at a document a consumer does
not receive.

That eviction was not cosmetic. docs/conformance-migration-ledger.md was
still load-bearing in code: it sat in ConformanceManifestGenerator's
AdditionalSourceFiles, which is hashed into every manifest's SourceHash, so
deleting the file would have permanently recorded an absent contributor to
that hash, and a parametrized test would have thrown on a missing file.
Two independent reasons every job failed.

global.json pinned SDK 10.0.303 with rollForward disabled. Hosted runners
carry 10.0.110, 10.0.204, 10.0.302 and 10.0.400 -- never 303 -- so the pin
was unsatisfiable by construction and all three jobs died on their first
dotnet invocation. That version was one developer's local SDK.

The pin now states a floor with latestFeature instead of an exact patch.
The part that matters is unchanged: the census still analyses this
repository with the same Roslyn that builds it, taken from
$(MSBuildToolsPath)/Roslyn/bincore rather than a NuGet package. An exact
patch bought nothing beyond that, because SdkVersion feeds a graph hash
that is recomputed every run and compared against nothing checked in, so
any .NET 10 SDK is self-consistent. Pinning one that is not installable
everywhere only made the repository unbuildable.

Second, ci.yml gated on the semantic-coverage CLI, which exits non-zero
whenever the catalog is incomplete. The catalog is INTENTIONALLY an
incomplete proposal backlog, and unclassified-mapping is its expected
state: the CLI reported zero new gaps, zero stale lines and zero unbacked
quotients while still failing. The test named
TheCheckedInCatalogMapsEveryRealSurfaceExactlyOnce already runs that identical
audit under dotnet test, and asserts both that
the catalog is incomplete and that unclassified-mapping is the only
diagnostic class allowed, which is a stronger gate than an exit code and
can express the intent. The redundant step is gone and the comment names
its real gate.

The CLI's exit code conflating "not complete" with "regressed" is left as
it is, and recorded in the pull request rather than changed under time
pressure.
CI runs dotnet csharpier check, and 200 files in this branch had never been
run through the formatter. Formatting them exposed a genuine disagreement
between two pre-existing tools: .editorconfig sets csharp_prefer_braces to
when_multiline, so a brace-less if is legal with a one-line body and an
IDE0011 error once the body wraps. CSharpier wrapped 83 such bodies, all of
them long argument lists, which turned legal code into build errors under
TreatWarningsAsErrors.

Master never hits this because its brace-less bodies are short enough not
to wrap. Adding braces satisfies both tools, so neither .editorconfig nor
the CSharpier configuration is touched -- the repository's style is the
repository's own call.

Convergence took two passes. The second was not redundant: while the
Conformance project failed to build, the Tests project could not compile at
all, so six of the 83 sites stayed invisible until the first pass fixed its
project reference.

Verified that no logic moved, rather than assuming it: comparing token
streams with whitespace and braces stripped, every difference across all
108 changed files is either a trailing comma CSharpier adds to a multi-line
initialiser or a using-directive reorder. Both are inert in C#.
The compilation-graph hash is relocation-invariant so a graph captured on
one machine can be verified on another, which means the path vocabulary
has to understand a Windows-shaped path and a Unix-shaped one regardless
of the host. LogicalPathTokens carries its own IsAbsolute and
NormalizeAbsolute for exactly that reason. Three places reached for the
platform's own path handling instead, and all three failed only on Linux.

A csc switch is syntactically an absolute path on Unix, so switch shape
must be decided before any path handling: /noconfig carries no value,
/doc:/tmp/x.xml carries one, and /home/me/src/X.cs is a real path. The
discriminator is what follows the switch name -- nothing, a colon, or a
further separator.

The ancestor .editorconfig probe used Path.GetFileName, whose notion of a
separator is the host's, so a Windows-shaped path there returned the whole
string as its filename on Linux and the probe silently declined a file it
should have admitted. It now segments with this class's own rules.

Finally, a mismatch reason gave only a count. A count cannot be diagnosed
from a CI log on a machine you cannot reach, so it now names the words and
their expected and actual analyses, truncated and capped at five.
Master replaced the SINGLE_THREADED compile symbol with a runtime
MaxDegreeOfParallelism, so the engine no longer contains any code that
exists in only some configurations. Two SemanticCoverage tests were
reading engine code as their live specimen and lost it.

Both point the census at real repository source rather than at a string
literal, which is the whole reason they exist -- the synthetic-source
tests beside each already cover the same logic on fixtures. So they still
read real source, but source whose shape this branch is answerable for.
The local-function keying test reads FailureRuleAttributor's Walk in
place of Morpher's deleted GenerateSynthesis. The configuration-only
census test reads Morpher's surviving #if OUTPUT_ANALYSES region, scoped
to the type rather than to one method signature, so that an upstream
rename cannot present itself here as a census defect.

The second test's claim is narrower than it was, and the name now says
so: it witnessed a method that existed only under a symbol, and no such
method remains anywhere in the engine. What it still witnesses is a
configuration-gated region censused under exactly the configurations
that contain it. Its expected configuration string was confirmed by
running the test, not derived.
Threading rule.Name down to the 256-node throw site cost a required
parameter on two public constructors, SynthesisRewriteRuleSpec and
EpenthesisSynthesisRewriteSubruleSpec. Both ship in the HermitCrab
NuGet package, so that is a source and binary break for anyone
constructing them.

Nothing read what it bought. InfiniteLoopException.RuleName has no
reader anywhere in the repository: the conformance runner mentions the
exception only in a comment, the fixture that provokes it asserts
expect_crash and not a message, and the engine test asserts only
Throws.TypeOf<InfiniteLoopException>(). A capability with no consumer
does not justify breaking a published constructor, so the four files
go back to what master has.

The 256-node cap itself is untouched, and edge-cases/simultaneous-
epenthesis-cascade still pins it.
GrammarHealthChecker and GrammarHealthFinding are a diagnostic feature,
not part of the conformance suite: nothing in conformance/ or in the
Conformance project calls them, and their only caller here was their own
test file. They are wanted in FieldWorks on a schedule of their own, so
they ship in #475 instead, which now carries the newer copies this
branch had developed.

Two tests go with them that cannot follow them there. Both load real
conformance fixtures, and one uses the Conformance project's
Fixture.DiscoverAll, so neither compiles on a branch that has no fixture
tree. They belong here rather than in #475, and should come back once
that lands; until then they are recoverable from 9dfb2f6.

What remains of this branch's engine footprint is Trace.FailureAllomorph,
its assignment in TraceManager from an argument that method already
received and discarded, and the inert SemanticBranch capture point.
SemanticBranch was introduced by this branch's first commit as a way for
engine code to declare a semantic path that the census would then count.
It was never wired up. No engine or tool code has called SemanticBranch.Hit
in any commit here, no ledger, catalog or coverage file has ever carried a
branch: id or a branch-marker surface, and no document names it as a
mechanism anyone intends to use. Every coverage number this branch reports
comes from severance sweeps and trace evidence instead.

So it was a public type in the shipped netstandard2.0 package with no
consumer, and the census carried a fail-closed resolution path that could
never fire. It could not move to the Conformance project either: that
project references the engine and not the reverse, so a marker channel
living there could never be called from the engine that is its only
intended caller.

Removed with it: CollectMarkers and its two resolution helpers in
CSharpInventoryReader, the branch-marker catalog family, and three tests
that existed only to exercise markers.

Three further tests used a marker as an incidental probe and keep their
subject. Two lose nothing -- the if/else-if arms keep an inert body, and
the reachable-versus-dead claim was already carried by the xml-read
assertions beside it. The third is narrower than it was and its name now
says so: it asserted that an unresolved marker in dead code does not trip
the fail-closed check, and with no such check left it now asserts only
that an unresolved call reached from dead code raises no diagnostic.

What remains of this branch's engine footprint is Trace.FailureAllomorph
and its assignment in TraceManager: two files, thirteen lines, no new
public type, and both read by FailureRuleAttributor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants