From b5604c37148da89e54e871a789fbd4e12b360b29 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 18 Aug 2026 07:01:16 -0400 Subject: [PATCH 1/3] Memoize HermitCrab's sequential analysis cascade Adds AnalysisStateKey/AnalysisScope/Word.ReplayOnto and wires a memo table into both the mrule cascade and the affix-template battery, so repeated analysis-cascade states reached via different rule-unapplication orders are computed once and replayed rather than re-expanded. Off by default (the existing parallel cascade is unchanged); opt in via Morpher(maxDegreeOfParallelism: 1), which selects the new sequential+memo path. Ported from an archived prototype (parse-optimization-archive) with stronger verification: the acceptance gate is analysis-set equality (canonical morpheme-signature sets), not byte-identical objects, since a memo-replayed Word is not guaranteed field-for-field identical to a freshly-computed one. Verified via unit tests (key order-invariance, replay graft correctness, in-flight re-entry guard) plus corpus runs against three real grammars (Sena, Indonesian, Amharic) with zero analysis-set divergences. On a known-pathological word, sequential+memo measured 6.2x faster than the parallel default (isolated to ~6.3x attributable to the memo itself, not threading); aggregate corpus evidence and the typical-word tradeoff are in memoization.md, along with honestly-reported open gaps. --- memoization.md | 420 ++++++++++++++++++ .../AnalysisScope.cs | 72 +++ .../AnalysisStateKey.cs | 110 +++++ .../AnalysisStratumRule.cs | 68 ++- .../MemoizedCombinationRuleCascade.cs | 122 +++++ .../Morpher.cs | 19 +- src/SIL.Machine.Morphology.HermitCrab/Word.cs | 71 +++ .../AnalysisStateKeyTests.cs | 148 ++++++ .../MemoCorpusVerification.cs | 234 ++++++++++ .../MemoizedCombinationRuleCascadeTests.cs | 189 ++++++++ .../MorpherTests.cs | 298 +++++++++++++ 11 files changed, 1736 insertions(+), 15 deletions(-) create mode 100644 memoization.md create mode 100644 src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs create mode 100644 src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs create mode 100644 src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs create mode 100644 tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs create mode 100644 tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs create mode 100644 tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs diff --git a/memoization.md b/memoization.md new file mode 100644 index 000000000..2ce3359b8 --- /dev/null +++ b/memoization.md @@ -0,0 +1,420 @@ +# Memoization for HermitCrab analysis — minimal, verified port to master + +**Branch:** `feature/memoization` (at master `d9deb167`). +**Goal:** capture the prototype's demonstrated speedup (a pathological Bantu-template +word ~5×; a 312-word Sena heavy set 1,051s → 388s) with a small, human-reviewable diff, +no RUSTIFY rearchitecture, and stronger verification than the prototype had. + +Real grammar/corpus content (Sena, Amharic, Indonesian) is never committed to this repo, +including individual word forms in prose — this doc refers to specific test words only by +generic labels (heavy word H1, H2, ...) rather than by their actual spelling. + +## 1. Provenance and evidence base + +The design is not new. It was built, measured, and corpus-verified on the +`parse-optimization` branch (2026-07-02/03), which was stacked on the unmerged RUSTIFY +commit and later deleted; the full 23-commit chain has been recovered and preserved as the +local branch **`parse-optimization-archive`**. The Rust port (PanGloss `hc-memo`, +`hc-rules/stratum.rs`) faithfully re-implements the same design and adds test patterns and +two hygiene lessons, but no new architecture. Primary references: + +- `parse-optimization-archive` — `AnalysisScope.cs`, `AnalysisStateKey.cs`, + `MemoizedCombinationRuleCascade.cs`, `Word.ReplayOnto` (commits `3522fc41` Phases 0-3, + `6ea0536a` Phase 3b), plus `parse-optimization.md` and + `docs/hermitcrab-parse-algorithm-analysis.md` in that tree. +- PanGloss — `hc-memo/src/lib.rs`, `hc-rules/src/stratum.rs:751-955`, + `hc-rules/tests/memo_gate.rs`, `hc-parse/tests/simultaneous_conformance.rs`. +- fst-advisor branch docs — verification-gate methodology (golden signature dumps, + per-grammar gating order, soundness batteries). + +**Where the speedup actually is (fair sequential baselines, measured on the archive):** + +| Mechanism | Effect | +|---|---| +| mrule-cascade positive memo + nogood cache (Phases 2/3) | ~10% (30.5s → 27.4s on heavy word H1). 2,555 real expansions vs 118,162 hits — hits are cheap because guard clauses already reject fast. | +| **template-battery memo (Phase 3b)** | **The real win: 93% of instrumented wall time.** 38,840 battery invocations vs ~2,581 unique keys. H1 27.4s → 6.1s; H2 102.7s → 26.9s; Sena 312-set 1,051s → 388s. | + +Both mechanisms share one key type and one replay primitive, so the minimal port includes +both; the template memo is the piece that must not be dropped if scope is ever cut. + +## 2. What the design is (one page) + +All memoization is scoped to a single `Morpher.ParseWord` call and applies only to the +**sequential** Unordered-order (template/Bantu) analysis cascade. + +- **`AnalysisScope`** — carrier object threaded through `Word` clones like `CurrentTrace` + (reference-shared; excluded from `FreezeImpl`/`ValueEquals`). Holds two tables + + in-flight guards + a 100,000-entry growth cap (OOM guard; correctness-neutral): + - `Memo` — mrule-cascade results (empty list = nogood, non-empty = positive), + - `TemplateMemo` — affix-template-battery results (separate table: same key space, + different computation), + - `InProgress` (+ a separate `TemplateInProgress`, a Rust-side clarity improvement worth + adopting) — re-entry guards; a hit falls through to unmemoized expansion for that call. +- **`AnalysisStateKey`** (readonly struct) — `(Shape, Stratum, SyntacticFeatureStruct, + RealizationalFeatureStruct, NonHeadCount, multiset of per-rule unapplication counts)`. + Order-invariant by construction: the multiset collapses the k! orderings that cause the + 98.4% expansion redundancy. Hash uses cached frozen hashes + commutative XOR over the + multiset; equality compares full values (never trust a bare hash — entries store the full + key). The constructor **freezes shape/FSs defensively on read** (known landmine: + `AnalysisAffixTemplateRule.Apply` reassigns an unfrozen `SyntacticFeatureStruct` after + the Word is frozen; that setter has no `CheckFrozen()` guard). +- **`MemoEntry`** — `{ Results: IReadOnlyList, MruleTrailPrefixLength, + NonHeadPrefixLength }`. +- **`Word.ReplayOnto(query, mruleTrailPrefixLen, nonHeadPrefixLen)`** — the replay graft: + clone the stored result, replace the trail/non-head *prefix* with the query node's own, + keep the suffix, re-freeze. Sound because everything strictly inside a memoized subtree + is a deterministic function of the key fields alone; only the two ordered structures the + key collapses to counts need grafting. +- **`MemoizedCombinationRuleCascade`** (mrule cascade) and an `ApplyTemplateBattery` + hook in `AnalysisStratumRule` (template battery): both do key → `TryGetValue` → replay + on hit / compute + `TryAdd` on miss. +- **Tracing bypasses the memo entirely** (`ParseWord` only installs `AnalysisScope` when + `!IsTracing`): traces must stay byte-identical to the unmemoized engine. + +**Design rules locked in from the research (each maps to a specific finding):** + +1. **No pruning gate may be coupled to memo presence.** The prototype bundled the Phase-5 + `HasReachableRoot` lexical gate into the memo-on path; Rust deliberately removed the + coupling to preserve `memo-on == memo-off` as a true invariant. We port the memo only — + no Phase 5, no gating — so the toggle is a pure optimization knob. +2. **Key-completeness is the primary correctness risk** (packrat/Ford; Bazel cache + doctrine). The port includes a written audit: every field read by any + `MorphologicalRules/Analysis*.cs` rule is either in the key or provably irrelevant + (`IsPartial` and `_isLastAppliedRuleFinal` are grep-verified unread). This audit lives + as a code comment on `AnalysisStateKey` and must be re-run when analysis rules change. +3. **Trail push and multiset increment are one atomic unit.** `record`-style bookkeeping + pairing (`_mruleApps.Add` + unapplied-count increment) must never drift; replay's + prefix/suffix split depends on multiset-equality ⇒ prefix-length-equality. +4. **Budget interaction (future):** if complexity-cap ever lands, a budget-interrupted + subtree must never be written to the memo ("exhausted subtree is not memoized" — the + PanGloss ground rule). Not needed now (master has no budgets) but recorded as a standing + invariant next to the cap constant. +5. **Per-parse lifetime, pre-sized where cheap** — no cross-word or process-global caching. + (A word→analysis-set cache above the engine is sound and orthogonal — explicitly out of + scope here; see fst-advisor `HYBRID_FST_FEASIBILITY.md` §6.) + +## 3. What master is missing (the port deltas) + +Master's HC tree is unchanged since the RUSTIFY fork point (`78350670`), so the substrate +the memo needs — `Word._mruleApps`/`_nonHeadApps`/`Clone`/`Freeze`, +`Shape/FeatureStruct.GetFrozenHashCode/ValueEquals`, `CombinationRuleCascade` — exists +**byte-identical** on master. Deltas: + +| Delta | Adaptation | +|---|---| +| `TOffset` is `ShapeNode` on master, `int` on RUSTIFY | Mechanical: port against `IRule` etc. | +| **Master has no runtime sequential/parallel toggle** — cascade choice is `#if SINGLE_THREADED`, which no csproj defines, so the shipped library always runs the parallel cascade and the memo path would be unreachable | **Prerequisite work** (the only genuinely new work): add a runtime knob to `Morpher` (RUSTIFY's `maxDegreeOfParallelism` ctor param is the reference design). Default preserves current parallel behavior. Scoped to the analysis side only — `SynthesisStratumRule` is untouched; synthesis has no equivalent memo. | +| No COW `Shape`/`FeatureStruct` (RUSTIFY-only) | Omit Phase 10a-1/7b polish. `ReplayOnto` does a real deep clone on master — slower than the prototype, never wrong. Expect somewhat less than the prototype's exact numbers; measure. | +| `InstrumentedRule`/rule-stats infra, `TrailDirectedRuleCascade` (Phase 0/1), `hc batch` command | Not load-bearing for the memo. Skip from the shipped diff; a slim corpus-signature harness is built as test-only tooling instead. | +| Master's `.Distinct(FreezableEqualityComparer)` in `AnalysisStratumRule` | Keep (harmless with the memo's deduped output; removal is a separate evaluation). | + +## 4. Implementation structure + +Built and gated in five logical stages (not five separate PR commits — the shipped diff +is squashed to one): + +**Stage 1 — prerequisite: runtime sequential-cascade toggle.** +`Morpher` gains `MaxDegreeOfParallelism` (ctor param; default = current parallel +behavior). `AnalysisStratumRule` selects sequential vs parallel cascade at runtime +instead of `#if SINGLE_THREADED`. No memoization yet. Gate: full unit suite green; +behavior byte-identical at default; a test pins sequential == parallel output on an +existing grammar. + +**Stage 2 — key + scope + replay primitives, with their unit tests.** +New `AnalysisStateKey`, `AnalysisScope`; additive `Word` members (`ReplayOnto`, +`UnappliedRuleCounts`, `MorphologicalRuleTrailLength`, `AnalysisScope` property). +Nothing consumes them yet. Tests ported from the archive + PanGloss `hc-memo` unit +battery: key order-invariance over permuted unapplication sequences, hash consistency, +sensitivity to non-head count and differing multisets, replay prefix/suffix graft, +in-flight guard semantics. The key-completeness audit comment lands here. + +**Stage 3 — mrule-cascade memo (positive + nogood).** +`MemoizedCombinationRuleCascade` wired into `AnalysisStratumRule`'s Unordered sequential +path; `ParseWord` installs `AnalysisScope` when sequential and `!IsTracing`. Tests: +hit-counter-guarded equivalence test (compounding grammar, from archive `MorpherTests`), +plus the **new adversarial fixtures the prototype never had** (§5, items a-b). + +**Stage 4 — template-battery memo (the 5×).** +`TemplateMemo` + `ApplyTemplateBattery` in `AnalysisStratumRule`. Tests: the +affix-template equivalence test (two commuting prefix rules, hit-counter-guarded), and +the memo-on/off gate exercising both tables in one parse (PanGloss `memo_gate.rs` shape). + +**Stage 5 — corpus verification harness + benchmark evidence.** +Test-only `[Explicit]` corpus runner in the HC test project (modeled on +`FstSenaBenchmark.cs`: env-var driven, per-word watchdog), comparing canonical +analysis-set signatures against uncommitted local grammars. Results recorded in §5/§6. + +Ship the memo **on by default whenever the cascade runs sequentially** (it is a pure +optimization once the invariant tests pass); the *default cascade mode* stays parallel in +this PR — flipping the library default to sequential+memo (which the corpus numbers say is +faster anyway) is proposed as a follow-up PR with its own benchmark evidence, so the +behavior-default change and the mechanism land as separately revertable units. + +## 5. Verification plan (stronger than the prototype's) + +Per-commit unit gates as above, plus: + +- **Memo-on/off analysis-set equality as the standing acceptance gate** (the metamorphic + relation: equal keys ⇒ equal result sets) — **not byte-for-byte object equality.** The + gate compares canonical signature sets (sorted `join("+", morphemeIds)+":"+rootIndex`), + because `ReplayOnto`'s grafted `Word` is not guaranteed field-for-field identical to a + freshly-computed one (e.g. `ShapeNode`/annotation object identity differs) even when it + represents the same analysis. Every equivalence test asserts non-vacuously (memo hit + counters > 0, both tables non-empty where applicable). +- **(a) SelfOpaquing ≥2-iteration fixture — the confirmed-bug shape (resolved: not reproduced, + wired in as a standing regression).** PanGloss's rust conformance suite documents a + confirmed C#-oracle bug in this exact code path (a `RewriteApplicationMode.Simultaneous` + epenthesis rule against root 19's `"b+ubu"`; `AnalysisRewriteRule` compiles this shape as + `ReapplyType.SelfOpaquing`, a repeat-until-fixpoint loop). Before trusting stage 3, this + was reconstructed directly against `parse-optimization-archive`'s own `AnalysisScope` + using the real `RewriteRuleTests.EpenthesisRules` rule definition: parsing `"buibui"` + gave the same result (1 parse) under every tracing/call-order combination tried + (non-traced, traced, repeated, order-swapped) — no divergence. The real minimal fixture + (`conformance/rewrite/simultaneous-epenthesis/`) is not present in this checkout to test + against directly (its own Rust test is `#[ignore]`'d for the same reason). Per the + advisor consult: this does not block stage 3 — the general memo-on/off analysis-set + equality gate is strictly stronger than one fixture, and PanGloss's own Rust analysis + reached the identical conclusion for their side (memo-sound on this shape; the ≥2-iteration + loop×memo interaction specifically "remains untested" because no available fixture drives + the loop past one iteration). **Shipped:** `MorpherTests.ParseWord_MemoOnMatchesMemoOff_ + ForSelfOpaquingSimultaneousEpenthesis` wires this exact fixture in as a standing + regression case (memo-on/off equality plus a pinned-value assertion), and the + ≥2-iteration gap is recorded here verbatim as an open, honest limitation — not silently + dropped. +- **(b) Trail-order-observable grammar — closed at the right layer, not through `ParseWord`.** + `Morpher.ParseWord` does not return raw analysis-cascade words — it feeds them through a + full **synthesis** re-derivation (`Synthesize`/`LexicalLookup`/`PermuteRules`), which + re-explores rule orderings on its own, so a `ParseWord`-level probe is the wrong + instrument for this: an attempt via a disabled-merge grammar variant returned 0 results + (most likely `MergeEquivalentAnalyses=false` breaking that synthesis path some other way, + not evidence about trail-order observability one way or the other — not chased further, + since it isn't the right layer regardless). **What the actual safety net is:** the + standing memo-on/off analysis-*set*-equality gate catches the class of bug that matters + — a dropped or spuriously-added analysis, exactly the PanGloss 0-vs-1 shape — because a + dropped analysis never reaches synthesis to be re-normalized away. Internal analysis + trail order, by contrast, is plausibly a don't-care for the returned output set, which + is also why it doesn't need a `ParseWord`-level test. The graft primitive itself IS + pinned by a real red/green test, at the layer where it's actually observable: + `AnalysisStateKeyTests.ReplayOnto_Grafts*` (stage 2, direct construction) and + `MemoizedCombinationRuleCascadeTests.Apply_PositiveReplayMatchesUnmemoizedResultSet_ + IncludingTrailOrder` (stage 3, a synthetic 3-rule cascade run directly against + `MemoizedCombinationRuleCascade` — bypassing `Morpher` so a real commuting-order replay + can be forced — comparing the memoized run's result set against an unmemoized run + *including* `MorphemesInApplicationOrder`, i.e. trail order). Verified this actually + discriminates: temporarily breaking `ReplayOnto` to skip the prefix graft made both this + test and the stage-2 unit tests fail, confirming they are not vacuous. +- **Corpus gates, in the fst-advisor order:** Indonesian 121/121 first (cheap iteration); + guarded Sena slice (first 60 words, 5s/word watchdog); then the Sena 312-word heavy set. + Compare sequential-memo (`maxDegreeOfParallelism: 1`, the only sequential configuration + that exists post-stage-3 — the memo is unconditional whenever the cascade is sequential, + so "sequential-unmemoized" is no longer a reachable end-to-end configuration; that + comparison is instead covered at the unit level, see §5(b)) against parallel-unmemoized + master default — the actual user-visible claim. Must be **analysis-set identical** — same + canonical signature set, not byte-identical objects — including negative/no-parse words + (nogoods cache no-parses too — both sides must produce the empty set). Amharic + (machine-local, gitignored) as the alphabet-stress smoke test; never commit real-grammar + fixtures (privacy constraint). +- **Reporting** (standing requirement): every measured claim ships with p50/p95 per-word + latency, aggregate wall, and hit/miss counts (`DiagMemoHits`/`DiagNogoodHits`, split — + not just totals) — and stage 5 must assert `DiagMemoHits > 0` somewhere in the corpus + run, so the positive-replay path can't ship having never actually fired end-to-end. + +**Stage 5 result (local uncommitted grammar — Sena, both slices):** ran +`MemoCorpusVerification` against the local `samples/data/sena-hc.xml` / +`sena-words.txt` (never committed — grammar-privacy constraint). + +| Slice | Timeout | Compared | Timed out | No-parse (both) | Divergences | Mrule memo (pos/nogood) | Template memo (pos/nogood) | +|---|---|---|---|---|---|---|---| +| First 60 | 5s | 54 | 6 | 11 | **0** | 8,652 / 59,156 | 10,481 / 3 | +| Full 312 | 8s | 252 | 60 | 87 | **0** | 201,051 / 1,692,696 | 286,581 / 29 | + +Zero divergences across both runs, hundreds of thousands of memo hits on both +tables (not vacuous), on real analysis-rule content the toy-grammar unit tests +(stages 2-4) structurally cannot reach — this is the empirical confirmation +of `AnalysisStateKey`'s key-completeness audit. The full-312 run's timeouts +(60 words, 8s/side budget) are the same known pathological Bantu template +words the archive prototype measured; p50 678.7ms / p95 +7,997.2ms per word (memo-on + memo-off combined) on the successfully-compared +words. Full per-word timing table is in the run's `TestContext` output, not +committed (never commit real-word signature data). + +**Indonesian (121/121, PanGloss's local `indonesian-hc.xml`/`-words.txt`):** +**0 divergences**, 0 timeouts, p50 37.2ms / p95 346.0ms. Mrule memo: 64 +positive / 157 nogood. Template memo: **0 positive / 410 nogood** — Indonesian +has no Bantu-style affix-template redundancy, so the template table only ever +proves subtrees empty here, never replays; matches §6's own prediction that +typical (non-template) grammars see little upside. Per-heavy-word timing on +the 10 slowest words shows memo-on (sequential) is actually *slower* than +memo-off (parallel default) by ~10-15% on this grammar — expected and +consistent with §6: the memo only pays for itself where order-variant +redundancy exists, and sequential-single-core loses to parallel-multi-core +when it doesn't. This is the honest "no regression in the wrong direction on +non-Bantu grammars" finding, not a null result — the comparison being measured +here is user-visible (sequential-memo vs parallel-unmemoized default), not the +memo mechanism isolated from single- vs multi-threading (see §5's note on why +"sequential-unmemoized" isn't independently constructible once the mrule-cascade +memo is wired in). + +**Amharic (machine-local, gitignored, alphabet-stress smoke test):** 30 words, +8s/side timeout. **0 divergences** on the 6 words that finished both sides +within budget (10 timed out — Amharic's Ge'ez-script grammar is known to be +build/parse-heavy; 14 had no parse on either side). Mrule memo: 14 positive / +113 nogood. Template memo: 41 positive / 2 nogood — both fired. Confirms the +key/replay machinery is script-agnostic (Ge'ez abugida segments, not Latin +transliteration) with no divergence. No Amharic word forms appear in this doc +or any committed file, per the standing grammar-privacy constraint. + +**Summary across all three real grammars tested: zero divergences, in every +case with both memo tables exercised non-vacuously.** This is the strongest +available evidence for `AnalysisStateKey`'s key-completeness audit short of +running the entire, much larger production corpora. + +**Critical follow-up (advisor-flagged): the aggregate corpus runs above prove +nothing about the heavy words, because they timed out and were excluded from +the equality gate.** 60 of the full-312 Sena run's words never got compared — +those are exactly the template-heavy words the memo exists for, and the ones +where a key-completeness miss would actually surface. A short timeout also +means the only completed timings showed memo-on *slower* (Indonesian, +Amharic) — sequential-memo vs parallel-unmemoized, not evidence of the actual +speedup goal. Closed with a targeted, longer-timeout run on heavy word **H1** +(known-pathological, present in the Sena heavy-word set) at 240s: + +- **Soundness on a heavy word, actually executed:** memo-on vs memo-off — + analysis-set identical (0 divergences). Mrule memo: 29,736 positive / + 88,426 nogood hits. Template memo: 37,512 positive / 0 nogood hits — this + is the key-completeness confirmation the excluded corpus runs above could + not provide. +- **User-visible timing:** memo-on (sequential+memo) **22.1s** vs memo-off + (today's shipped parallel default) **136.1s** — **6.2×**. +- **Isolated memo contribution** (mutation-tested: temporarily disabled the + `AnalysisScope` install in `Morpher.ParseWord`, reverted after — same + technique as the ReplayOnto mutation test in §5(b)): sequential-no-memo + **138.0s** vs parallel-no-memo (unchanged) **141.5s** — parallelism alone + contributes essentially nothing on this word (~2%). Compared against + sequential-**memo**'s 22.1s: the memo mechanism itself is responsible for + essentially the entire **~6.3× speedup**, not a threading artifact. + +This exceeds the plan's own target (≥3× on heavy-word-class words) despite +master lacking the prototype's copy-on-write `Shape`/`FeatureStruct` — the +`ReplayOnto` deep-clone tax feared in §6 below did not dominate here. + +**Second heavy word, H2 — soundness inconclusive, not a clean win.** Two +independent runs at a 400s per-side timeout both had the memo-on +(sequential+memo) side fail to finish within 400s — i.e. this word is *not* +analysis-set-verified at this depth; the "Passed" result in the first run +reflects the harness's by-design exclusion of timed-out words from the +divergence check, not a completed comparison. This is a materially +different outcome from H1 and is reported honestly rather than folded into +the "0 divergences" summary above. Notably, the archive prototype's own +measurement (§1's table above) had H2 at 102.7s → 26.9s with the template +memo — master's ported memo-on run did not complete this word within 400s, +~15× slower than the archive's memoized figure. Plausible explanations +(unconfirmed): master's deep-clone `ReplayOnto` (vs the archive's +copy-on-write `Shape`/`FeatureStruct`) may cost materially more on this +word's particular redundancy shape than it did on H1; or H2 exercises a +rule-graph region where the memo's hit rate is lower and raw expansion +dominates. Not chased further with a longer timeout, per the same +bounded-effort judgment applied to the other unmeasured heavy words below — +documented as unmeasured/slower-than-expected, a concrete follow-up, not a +blocker for this PR (soundness is unaffected either way: an incomplete run +makes no claim, positive or negative, about key-completeness for this +word). + +Remaining honest gap: H1 is the one heavy word with a completed, positive +measurement; H2 is a heavy word with an *incomplete* one (see above); the +other 58 timed-out full-312-run words (and Indonesian/Amharic's timed-out +words) are unmeasured at this depth — a natural follow-up, not a blocker, +given the confirmed pattern (soundness holds, memo dominates the win) on +the one word measured to completion. + +**Aggregate corpus evidence: count-based and wall-clock, reported +separately (see the harness's own two-line report format).** A Sena +sub-corpus (words 1-70, 200s per-side timeout — long enough for the one +tractable heavy word in this range to complete on both sides, per §5's H1 +measurement above) gives, of the 69 words that completed within budget: +**53/69 (77%) faster under memo, 16/69 (23%) slower** — count-based, this +is the "yes, some words are slower" half of the picture. **Wall-clock: +memo-on total 114.6s vs memo-off total 277.8s across those same 69 +words — 2.42× faster in aggregate** — this is the "but a lot are faster, +and it nets out strongly positive" half. The one word that timed out at +200s (a second tractable-but-slow heavy word, distinct from both H1 and +H2) is excluded from both numbers. The harness's timeout wraps both the +memo-on and memo-off calls in one try block per word, so it cannot record +which side actually timed out for this word — if it was memo-on (the more +likely case here, since it's attempted first and this class of word is +exactly where the memo's own cost is least understood, see the H2 +discussion below), the word's true memo-off time is genuinely unmeasured, +not "presumably at least as slow." **This 2.42× figure should be read as +this slice's measured result, not asserted as a guaranteed lower bound** +in either direction — a longer timeout is the only way to actually find +out. Both memo tables fired heavily and non-vacuously on this slice +(mrule: tens of thousands of positive/nogood hits; template: tens of +thousands positive, near-zero nogood). + +**Is the typical-word slowdown "just" losing a thread? Mostly, not +purely.** Isolated on Indonesian (no template redundancy — the cleanest +instrument for this, since the template memo never replays there) via +three repeated runs per configuration to average out run-to-run JIT/GC +noise (single-run absolute times varied by as much as 60% between +otherwise-identical runs of the unchanged parallel baseline — only +within-run memo-on-vs-memo-off ratios are trustworthy here, not +across-run absolute milliseconds): + +| Configuration vs parallel-default | Aggregate ratio (3 reps) | Implied slowdown (1/ratio − 1) | +|---|---|---| +| sequential + memo (shipped opt-in) | 0.80×, 0.80×, 0.78× | ~25-28% | +| sequential, memo forced off (mutation-tested) | 0.69×, 0.80×, 0.68× | ~25-47% | + +The sequential-*no-memo* slowdown is consistently at or above the +sequential-*memo* slowdown, never below it. So most of the typical-word +regression genuinely is the lost thread (both configurations are slower +than parallel-default, by a similar order of magnitude) — but it is not +*purely* that: even on a grammar where the template memo never once +replays, the mrule-cascade's own cache still fires (64 positive / 157 +nogood hits on this corpus) and measurably claws back some of the +thread-loss penalty rather than adding to it. Three reps each is enough to +see the direction of the effect, not enough to state a precise recovered +percentage with confidence — a larger repeated-trial run would sharpen +this if the exact number ever matters for the default-flip decision. + +**Is the H2 shortfall actually the deep-clone tax?** Plausible, not +confirmed — resist stating it as established. The one piece of direct +evidence is that H2's memo fired heavily before timing out (tens of +thousands of positive hits on both tables, §5 above) — the memo is +clearly not idle on this word, which favors "replay/clone cost per hit is +what's expensive" over "the memo doesn't engage." But that is a +directional tell, not a measurement: distinguishing "expensive replay" from +"just a lot more raw expansion than H1" requires profiling where memo-on +wall time actually goes (`Clone`/`Freeze`/`ReplayOnto` vs raw rule +expansion) on H2 specifically. Scoped as a follow-up, not undertaken here. + +## 6. Expected outcome and honest caveats + +- Pathological template-heavy words: prototype showed ~5× (with COW clones). Master's + deep-clone `ReplayOnto` was expected to give back some of that; **measured directly on + heavy word H1 (§5 addendum): ~6.3× isolated memo contribution, ~6.2× on the + user-visible sequential-memo-vs-parallel-default comparison — met and exceeded the + ≥3× target**, meaning the feared `ReplayOnto` deep-clone tax did not dominate on this + word. The ≥50% Sena-heavy-set aggregate target remains unmeasured (that run's heavy + words timed out before completing the equality gate, §5 addendum) — a natural + follow-up. A second heavy word, H2, did NOT complete within a 400s memo-on budget (§5 + addendum) — the first concrete (if inconclusive) counter-data-point to the + deep-clone-tax question, since the archive's COW-based prototype handled this same + word in 26.9s. If a future heavy-set measurement shows replay cost dominating after + all, this is the evidence that would motivate porting `Shape` COW as its own + follow-up — do not fold it into this PR pre-emptively. +- Typical words (Indonesian-class) get measurably *slower* under sequential+memo + (~25-28% in aggregate, §5 addendum) — mostly the cost of losing a thread, not the memo + itself (isolated via a sequential-no-memo comparison, same addendum). The corpus gate + proves "no divergence" there, not "no regression" — this is exactly why the memo ships + off by default rather than flipping the library's default cascade mode. +- The memo is inert for `Ordered`-strata-only grammars and for `ParallelCombinationRuleCascade` + (no natural subtree-completion moment in its breadth-first walk — same reason the + prototype left it unmemoized). + +## 7. Out of scope (deliberately) + +Phase 0/1 instrumentation and synthesis-side `TrailDirectedRuleCascade`; Phase 4/5 gates +(measured inert or unsound-coupled); RUSTIFY COW polish (7b/10a); packed forest (9c); +word-level result caching above the engine; any interned-ID key redesign (`rust-conversion.md` +§6.3 is an unshipped plan — needs its own measurement if ever wanted). diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs new file mode 100644 index 000000000..73d6cc99c --- /dev/null +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs @@ -0,0 +1,72 @@ +using System.Collections.Concurrent; +using System.Collections.Generic; + +namespace SIL.Machine.Morphology.HermitCrab +{ + /// + /// Per-parse cache carrier threaded through clones exactly like + /// -- reference-shared, excluded from Word.FreezeImpl/ + /// Word.ValueEquals so existing dedup semantics are unchanged. Holds the analysis-cascade memo + /// (memoization.md) -- see . + /// One instance per call: entries are state facts + /// about a specific parse (a state key does not encode the target surface word), so sharing this + /// across concurrent parses of different words would be unsound without also scoping the key to the + /// word -- that cross-word extension is explicitly out of scope. + /// Thread-safe because a single word's analysis can itself run in parallel + /// (ParallelCombinationRuleCascade, used when is not + /// 1) -- though today only the sequential cascade actually reads/writes this. + /// + internal sealed class AnalysisScope + { + // OOM guard: a positive memo holds actual Word lists (not just a boolean like the nogood case), so + // it is the one that can grow unboundedly on a pathological word. Past the cap, new subtrees are + // simply not memoized -- correctness is unaffected, only the hit rate degrades. No corpus word + // seen so far has come close to this cap; deliberately untested against an actual overflow. + private const int MaxMemoEntries = 100_000; + + public ConcurrentDictionary Memo { get; } = + new ConcurrentDictionary(); + + // Second table, same key space, different computation: the affix-template battery result for a + // state (AnalysisStratumRule.ApplyTemplateBattery), as opposed to the mrule-cascade subtree result + // above. Kept separate because a state can be memoized in one table but not (yet) the other. + public ConcurrentDictionary TemplateMemo { get; } = + new ConcurrentDictionary(); + + // Keys currently under expansion on some call stack -- guards the in-flight re-entry case (a + // multiApp cascade can reach the same state again before its own first expansion has completed, + // e.g. via a self-loop). A hit here must fall through to plain, unmemoized expansion rather than + // read a nonexistent/partial entry or deadlock; see MemoizedCombinationRuleCascade.ApplyRules. The + // template battery needs no equivalent guard: ApplyTemplateBattery's call is eager and + // self-contained (no template<->mrule mutual recursion happens inside it). + public ConcurrentDictionary InProgress { get; } = + new ConcurrentDictionary(); + + public bool HasMemoCapacity => Memo.Count < MaxMemoEntries; + + public bool HasTemplateMemoCapacity => TemplateMemo.Count < MaxMemoEntries; + } + + /// + /// A memoized analysis-cascade subtree or template-battery result (memoization.md). + /// empty = the "nogood" case (subtree/battery proved to yield nothing); non-empty = the positive case, + /// replayable onto a differently-ordered arrival at the same via + /// , using / + /// to split each stored result's trail/non-heads into the (discarded, replaced) prefix and the (kept) + /// subtree-local suffix. There is no "budget exhausted / incomplete" flag: this engine has no step/time + /// budget infrastructure, so every recorded subtree was explored to full completion. + /// + internal sealed class MemoEntry + { + public MemoEntry(IReadOnlyList results, int mruleTrailPrefixLength, int nonHeadPrefixLength) + { + Results = results; + MruleTrailPrefixLength = mruleTrailPrefixLength; + NonHeadPrefixLength = nonHeadPrefixLength; + } + + public IReadOnlyList Results { get; } + public int MruleTrailPrefixLength { get; } + public int NonHeadPrefixLength { get; } + } +} diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs new file mode 100644 index 000000000..ca19fef0b --- /dev/null +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs @@ -0,0 +1,110 @@ +using System; +using System.Collections.Generic; +using SIL.Machine.Annotations; +using SIL.Machine.FeatureModel; + +namespace SIL.Machine.Morphology.HermitCrab +{ + /// + /// Order-independent identity of an analysis-cascade node, used by the memo cache in + /// 's Unordered-mode morphological rule cascade (memoization.md). Two + /// Words with an equal key are guaranteed to make identical decisions in every analysis-side + /// morphological rule that cascade can invoke -- verified by inspecting what each one reads from its + /// input (key-completeness audit, re-run whenever an Analysis*.cs rule changes): + /// + /// : Shape (FST pattern match) and + /// (unifiability gate) plus a per-rule unapplication count. + /// : adds + /// (MaxStemCount gate) -- it never reads the non-heads' own content, only the count. + /// : adds + /// . + /// + /// but never the ORDER those rules were unapplied in -- exactly the redundancy this key collapses. + /// Deliberately excludes fields Word.ValueEquals includes for a different purpose (result + /// dedup): the unapplication trail as an ordered SEQUENCE (replaced here by an order-independent + /// multiset) and _isLastAppliedRuleFinal/IsPartial, which are not read by any + /// analysis-side rule (grep-verified against every file matching MorphologicalRules/Analysis*.cs + /// and PhonologicalRules/Analysis*.cs). + /// + internal readonly struct AnalysisStateKey : IEquatable + { + private readonly Shape _shape; + private readonly Stratum _stratum; + private readonly FeatureStruct _syntacticFS; + private readonly FeatureStruct _realizationalFS; + private readonly int _nonHeadCount; + private readonly IReadOnlyDictionary _ruleCounts; + private readonly int _hashCode; + + public AnalysisStateKey(Word word) + { + _shape = word.Shape; + _stratum = word.Stratum; + _syntacticFS = word.SyntacticFeatureStruct; + _realizationalFS = word.RealizationalFeatureStruct; + _nonHeadCount = word.NonHeadCount; + _ruleCounts = word.UnappliedRuleCounts; + + // Defensive: AnalysisAffixTemplateRule.Apply reassigns SyntacticFeatureStruct to a fresh, + // unfrozen clone AFTER the owning Word is already frozen (no CheckFrozen() guard on that + // setter). Freeze() is idempotent and safe to call again here. + _shape.Freeze(); + _syntacticFS.Freeze(); + _realizationalFS.Freeze(); + + int hash = 17; + hash = hash * 31 + _shape.GetFrozenHashCode(); + hash = hash * 31 + (_stratum?.GetHashCode() ?? 0); + hash = hash * 31 + _syntacticFS.GetFrozenHashCode(); + hash = hash * 31 + _realizationalFS.GetFrozenHashCode(); + hash = hash * 31 + _nonHeadCount; + if (_ruleCounts != null) + { + // XOR, not the usual *31 rolling combine: the multiset is unordered, so the combination + // must be commutative -- two dictionaries with the same entries built up in different + // unapplication orders must hash identically. + int multisetHash = 0; + foreach (KeyValuePair kvp in _ruleCounts) + multisetHash ^= (kvp.Key.GetHashCode() * 397) ^ kvp.Value; + hash = hash * 31 + multisetHash; + } + _hashCode = hash; + } + + public override int GetHashCode() => _hashCode; + + public override bool Equals(object obj) => obj is AnalysisStateKey other && Equals(other); + + public bool Equals(AnalysisStateKey other) + { + if (_hashCode != other._hashCode) + return false; + if (_nonHeadCount != other._nonHeadCount || !ReferenceEquals(_stratum, other._stratum)) + return false; + if (!_shape.ValueEquals(other._shape)) + return false; + if (!_syntacticFS.ValueEquals(other._syntacticFS) || !_realizationalFS.ValueEquals(other._realizationalFS)) + return false; + return RuleCountsEqual(_ruleCounts, other._ruleCounts); + } + + private static bool RuleCountsEqual( + IReadOnlyDictionary a, + IReadOnlyDictionary b + ) + { + int aCount = a?.Count ?? 0; + int bCount = b?.Count ?? 0; + if (aCount != bCount) + return false; + if (aCount == 0) + return true; + foreach (KeyValuePair kvp in a) + { + if (!b.TryGetValue(kvp.Key, out int otherCount) || otherCount != kvp.Value) + return false; + } + return true; + } + } +} diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs index 752e08153..eb3d7246b 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs @@ -47,19 +47,17 @@ public AnalysisStratumRule(Morpher morpher, Stratum stratum) ); break; case MorphologicalRuleOrder.Unordered: -#if SINGLE_THREADED - _mrulesRule = new CombinationRuleCascade( - mrules, - true, - FreezableEqualityComparer.Default - ); -#else - _mrulesRule = new ParallelCombinationRuleCascade( - mrules, - true, - FreezableEqualityComparer.Default - ); -#endif + // Sequential (and memoized, see MemoizedCombinationRuleCascade) when the caller caps + // within-word parallelism; parallel cascade otherwise. + _mrulesRule = + morpher.MaxDegreeOfParallelism == 1 + ? (IRule) + new MemoizedCombinationRuleCascade(mrules, FreezableEqualityComparer.Default) + : new ParallelCombinationRuleCascade( + mrules, + true, + FreezableEqualityComparer.Default + ); break; } } @@ -188,9 +186,51 @@ private IEnumerable ApplyMorphologicalRules(Word input) } } + // Test/reporting hooks (memoization.md's standing hit/miss-count requirement), mirroring + // MemoizedCombinationRuleCascade's DiagMemoHits/DiagNogoodHits split. + internal static long DiagTemplateMemoHits; + internal static long DiagTemplateNogoodHits; + + // Runs the affix-template battery for `input`, memoized by AnalysisStateKey (memoization.md), + // same replay mechanism as MemoizedCombinationRuleCascade -- see AnalysisScope.InProgress for + // why no re-entry guard is needed here. Measured motivation (an archive prototype benchmark on + // a Bantu-template grammar): this battery accounted for the large majority of parse wall time + // once the mrule cascade's own memo had already shrunk its own share to near-nothing. + private IEnumerable ApplyTemplateBattery(Word input) + { + AnalysisScope scope = input.AnalysisScope; + if (scope == null || _morpher.MaxDegreeOfParallelism != 1) + return _templatesRule.Apply(input); + + var key = new AnalysisStateKey(input); + if (scope.TemplateMemo.TryGetValue(key, out MemoEntry entry)) + { + if (entry.Results.Count == 0) + { + DiagTemplateNogoodHits++; + return Enumerable.Empty(); + } + var replayed = new List(entry.Results.Count); + foreach (Word stored in entry.Results) + replayed.Add(stored.ReplayOnto(input, entry.MruleTrailPrefixLength, entry.NonHeadPrefixLength)); + DiagTemplateMemoHits++; + return replayed; + } + + var results = new List(_templatesRule.Apply(input)); + if (scope.HasTemplateMemoCapacity) + { + scope.TemplateMemo.TryAdd( + key, + new MemoEntry(results, input.MorphologicalRuleTrailLength, input.NonHeadCount) + ); + } + return results; + } + private IEnumerable ApplyTemplates(Word input) { - foreach (Word tempOutWord in _templatesRule.Apply(input).Distinct(FreezableEqualityComparer.Default)) + foreach (Word tempOutWord in ApplyTemplateBattery(input).Distinct(FreezableEqualityComparer.Default)) { switch (_stratum.MorphologicalRuleOrder) { diff --git a/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs new file mode 100644 index 000000000..bad620f3a --- /dev/null +++ b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs @@ -0,0 +1,122 @@ +using System.Collections.Generic; +using SIL.Machine.Annotations; +using SIL.Machine.Rules; + +namespace SIL.Machine.Morphology.HermitCrab +{ + /// + /// Drop-in replacement for the sequential on + /// Unordered-order analysis strata (memoization.md). Before expanding a node, checks whether an + /// earlier expansion elsewhere in the same word's analysis -- reached via a different unapplication + /// order, but with an equal -- already searched this exact state: + /// + /// proved empty (the "nogood" case) -> skip straight to "no results"; + /// produced results (the positive case) -> replay them () instead + /// of re-searching: clone each stored result and graft the CURRENT arrival's own trail/non-head + /// prefix onto the stored subtree-local suffix (see and + /// for why only the prefix -- never the suffix -- needs replacing). + /// + /// Scoped to the sequential cascade only. The parallel cascade + /// (ParallelCombinationRuleCascade, used when is + /// not 1) is a level-by-level breadth-first walk with no natural "this subtree is fully expanded" + /// moment to hang a memo write on, so it is left unmemoized -- callers that want this optimization + /// should construct their with maxDegreeOfParallelism: 1. + /// + internal class MemoizedCombinationRuleCascade : RuleCascade + { + // Test/reporting hooks (memoization.md's standing hit/miss-count requirement): DiagMemoHits counts + // positive replays (a stored non-empty subtree grafted onto a new arrival); DiagNogoodHits counts + // nogood hits (a stored EMPTY subtree short-circuited without any replay work). The equivalence + // tests that cover the replay path assert DiagMemoHits is nonzero so they can never go vacuous -- + // a memo that silently stopped firing would otherwise look exactly like a passing test. + internal static long DiagMemoHits; + internal static long DiagNogoodHits; + + public MemoizedCombinationRuleCascade( + IEnumerable> rules, + IEqualityComparer comparer + ) + : base(rules, true, comparer) { } + + public override IEnumerable Apply(Word input) + { + var output = new HashSet(Comparer); + ApplyRules(input, output); + return output; + } + + // Returns every result produced strictly within the subtree rooted at `input` (i.e. by applying + // one or more rules starting from `input`, at any depth) -- NOT including `input` itself. This is + // both the return value callers use and the value memoized against `input`'s AnalysisStateKey + // once the subtree finishes, so a later differently-ordered arrival at the same state can replay + // it via Word.ReplayOnto instead of re-searching. + private List ApplyRules(Word input, HashSet output) + { + AnalysisScope scope = input.AnalysisScope; + // See Word.AnalysisScope's doc for when this is null. + if (scope == null) + return ApplyRulesRaw(input, output); + + var key = new AnalysisStateKey(input); + + if (scope.Memo.TryGetValue(key, out MemoEntry entry)) + { + if (entry.Results.Count == 0) + { + DiagNogoodHits++; + return new List(); + } + var replayed = new List(entry.Results.Count); + foreach (Word storedResult in entry.Results) + { + Word replay = storedResult.ReplayOnto( + input, + entry.MruleTrailPrefixLength, + entry.NonHeadPrefixLength + ); + output.Add(replay); + replayed.Add(replay); + } + DiagMemoHits++; + return replayed; + } + + // In-flight re-entry guard, see AnalysisScope.InProgress. + if (!scope.InProgress.TryAdd(key, 0)) + return ApplyRulesRaw(input, output); + + List results; + try + { + results = ApplyRulesRaw(input, output); + } + finally + { + scope.InProgress.TryRemove(key, out _); + } + + // Past the cap, keep searching correctly, just stop growing the table (AnalysisScope.HasMemoCapacity). + if (scope.HasMemoCapacity) + scope.Memo.TryAdd(key, new MemoEntry(results, input.MorphologicalRuleTrailLength, input.NonHeadCount)); + + return results; + } + + private List ApplyRulesRaw(Word input, HashSet output) + { + var local = new List(); + for (int i = 0; i < Rules.Count; i++) + { + foreach (Word result in ApplyRule(Rules[i], i, input)) + { + local.Add(result); + output.Add(result); + // avoid infinite loop -- same guard CombinationRuleCascade uses + if (!Comparer.Equals(input, result)) + local.AddRange(ApplyRules(result, output)); + } + } + return local; + } + } +} diff --git a/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs b/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs index 0ecd5633e..9aed10d34 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs @@ -29,10 +29,13 @@ public class Morpher : IMorphologicalAnalyzer, IMorphologicalGenerator private readonly ReadOnlyObservableCollection _morphemes; private readonly IList _lexicalPatterns = new List(); - public Morpher(ITraceManager traceManager, Language lang) + public Morpher(ITraceManager traceManager, Language lang, int maxDegreeOfParallelism = 0) { _lang = lang; _traceManager = traceManager; + // Must be set before CompileAnalysisRule: AnalysisStratumRule picks a sequential vs. parallel + // cascade for Unordered-order analysis strata at construction time based on this value. + MaxDegreeOfParallelism = maxDegreeOfParallelism; _allomorphTries = new Dictionary(); var morphemes = new ObservableList(); foreach (Stratum stratum in _lang.Strata) @@ -84,6 +87,15 @@ public ITraceManager TraceManager /// public bool MergeEquivalentAnalyses { get; set; } + /// + /// Caps parallelism used within a single parse's Unordered-order analysis-rule cascade. A value of + /// 1 selects the sequential cascade, which is also the only one eligible for the analysis-cascade + /// memo (see ); any other value (default 0) keeps the + /// existing parallel cascade. Set via the constructor: it influences how the analysis rules are + /// compiled. + /// + public int MaxDegreeOfParallelism { get; } + public Func LexEntrySelector { get; set; } public Func RuleSelector { get; set; } @@ -115,6 +127,11 @@ public IEnumerable ParseWord(string word, out object trace, bool guessRoot Shape shape = _lang.SurfaceStratum.CharacterDefinitionTable.Segment(word); var input = new Word(_lang.SurfaceStratum, shape); + // Only the sequential cascade reads AnalysisScope (see MemoizedCombinationRuleCascade); + // skip the allocation on the parallel path, and while tracing (tracing must stay + // byte-identical to the unmemoized engine). + if (!_traceManager.IsTracing && MaxDegreeOfParallelism == 1) + input.AnalysisScope = new AnalysisScope(); input.Freeze(); if (_traceManager.IsTracing) _traceManager.AnalyzeWord(_lang, input); diff --git a/src/SIL.Machine.Morphology.HermitCrab/Word.cs b/src/SIL.Machine.Morphology.HermitCrab/Word.cs index 9b29429e9..c98946fba 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/Word.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/Word.cs @@ -90,6 +90,7 @@ protected Word(Word word) _isLastAppliedRuleFinal = word._isLastAppliedRuleFinal; _isPartial = word._isPartial; CurrentTrace = word.CurrentTrace; + AnalysisScope = word.AnalysisScope; _disjunctiveAllomorphIndices = word._disjunctiveAllomorphIndices.ToDictionary( kvp => kvp.Key, kvp => new HashSet(kvp.Value) @@ -212,6 +213,16 @@ public IEnumerable MorphemesInApplicationOrder public object CurrentTrace { get; set; } + /// + /// Carrier for the analysis-cascade memo (memoization.md). Reference-shared like + /// , deliberately excluded from FreezeImpl and + /// ValueEquals so existing dedup semantics are unchanged. Null for words never routed + /// through (e.g. words built directly by + /// rule-level unit tests) or while tracing, in which case the cascade that reads this must fall + /// back to unmemoized behavior rather than throw. + /// + internal AnalysisScope AnalysisScope { get; set; } + public bool IsPartial { get { return _isPartial; } @@ -338,6 +349,12 @@ internal int GetUnapplicationCount(IMorphologicalRule mrule) return numUnapplies; } + /// + /// The full per-rule unapplication-count multiset backing , for + /// (order-independent analysis-cascade memoization, memoization.md). + /// + internal IReadOnlyDictionary UnappliedRuleCounts => _mrulesUnapplied; + /// /// Notifies this word synthesis that the specified morphological rule has applied. /// @@ -392,6 +409,14 @@ internal int NonHeadCount get { return _nonHeadApps.Count; } } + /// + /// Length of the morphological-rule trail so far -- _mruleApps.Count. Recorded alongside + /// at the point an 's subtree is + /// memoized (memoization.md), so a later differently-ordered arrival at the same key knows where + /// its own trail ends and the memoized subtree's suffix begins -- see . + /// + internal int MorphologicalRuleTrailLength => _mruleApps.Count; + internal void NonHeadUnapplied(Word nonHead) { CheckFrozen(); @@ -450,6 +475,52 @@ internal IList ExpandAlternatives() return alternatives; } + /// + /// Re-parents a Word computed while exploring the subtree below some analysis-cascade node N onto + /// -- a different Word that reached the same + /// as N via a different morphological-rule unapplication order + /// (memoization.md's positive memo; see ). Everything + /// computed strictly WITHIN the subtree -- deeper shape/feature edits, and any rules or non-heads + /// unapplied below N -- is a deterministic function of N's content alone (Shape, both + /// FeatureStructs, the rule-unapplication multiset, and non-head count all match between N and + /// by definition of an equal key), so it is kept as-is from `this`. + /// Only the two ORDERED structures the key deliberately summarizes as counts/multisets -- the + /// morphological-rule trail and the non-head list -- have their PREFIX (whatever was accumulated + /// before reaching N) replaced with 's own actual prefix, since arrival + /// order can only ever affect that part. + /// + /// The word that hit the memo -- its trail/non-heads become the new prefix. + /// + /// _mruleApps.Count of N at the moment its subtree was memoized -- everything in `this`'s + /// trail from this index on is the subtree-local suffix to keep. + /// + /// Same, for _nonHeadApps. + internal Word ReplayOnto(Word queryNode, int mruleTrailPrefixLength, int nonHeadPrefixLength) + { + Word clone = Clone(); + + List mruleSuffix = clone._mruleApps.GetRange( + mruleTrailPrefixLength, + clone._mruleApps.Count - mruleTrailPrefixLength + ); + clone._mruleApps.Clear(); + clone._mruleApps.AddRange(queryNode._mruleApps); + clone._mruleApps.AddRange(mruleSuffix); + clone._mruleAppIndex = clone._mruleApps.Count - 1; + + List nonHeadSuffix = clone._nonHeadApps.GetRange( + nonHeadPrefixLength, + clone._nonHeadApps.Count - nonHeadPrefixLength + ); + clone._nonHeadApps.Clear(); + clone._nonHeadApps.AddRange(queryNode._nonHeadApps.CloneItems()); + clone._nonHeadApps.AddRange(nonHeadSuffix); + clone._nonHeadAppIndex = clone._nonHeadApps.Count - 1; + + clone.Freeze(); + return clone; + } + public Allomorph GetAllomorph(Annotation morph) { var alloID = (string)morph.FeatureStruct.GetValue(HCFeatureSystem.Allomorph); diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs new file mode 100644 index 000000000..29469988f --- /dev/null +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs @@ -0,0 +1,148 @@ +using NUnit.Framework; +using SIL.Machine.FeatureModel; +using SIL.Machine.Morphology.HermitCrab.MorphologicalRules; + +namespace SIL.Machine.Morphology.HermitCrab; + +// Unit battery for the analysis-cascade memo's primitives (memoization.md Stage 2): AnalysisStateKey's +// order-invariance over the rule-unapplication multiset, its sensitivity to every other field the key +// captures, and Word.ReplayOnto's prefix/suffix graft. Nothing in production code consumes these yet +// (that is Stage 3) -- these tests pin the primitives' own contract in isolation, independent of any +// particular cascade wiring. +[TestFixture] +public class AnalysisStateKeyTests : HermitCrabTestBase +{ + [Test] + public void Equals_IsInvariantOverUnapplicationOrder_ForEqualMultisets() + { + var ruleA = new AffixProcessRule { Name = "ruleA" }; + var ruleB = new AffixProcessRule { Name = "ruleB" }; + + // Same multiset {ruleA: 2, ruleB: 1}, built up in two genuinely different orders -- + // wordY touches ruleB first, unlike wordX, so this also exercises different backing + // dictionary insertion order, not just a different position for a repeated rule. + Word wordX = NewTestWord(); + wordX.MorphologicalRuleUnapplied(ruleA); + wordX.MorphologicalRuleUnapplied(ruleB); + wordX.MorphologicalRuleUnapplied(ruleA); + wordX.Freeze(); + + Word wordY = NewTestWord(); + wordY.MorphologicalRuleUnapplied(ruleB); + wordY.MorphologicalRuleUnapplied(ruleA); + wordY.MorphologicalRuleUnapplied(ruleA); + wordY.Freeze(); + + var keyX = new AnalysisStateKey(wordX); + var keyY = new AnalysisStateKey(wordY); + + Assert.That(keyX.GetHashCode(), Is.EqualTo(keyY.GetHashCode())); + Assert.That(keyX.Equals(keyY), Is.True); + } + + [Test] + public void Equals_False_WhenUnapplicationMultisetsDiffer() + { + var ruleA = new AffixProcessRule { Name = "ruleA" }; + + Word wordX = NewTestWord(); + wordX.MorphologicalRuleUnapplied(ruleA); + wordX.Freeze(); + + Word wordY = NewTestWord(); + wordY.MorphologicalRuleUnapplied(ruleA); + wordY.MorphologicalRuleUnapplied(ruleA); + wordY.Freeze(); + + Assert.That(new AnalysisStateKey(wordX).Equals(new AnalysisStateKey(wordY)), Is.False); + } + + [Test] + public void Equals_False_WhenNonHeadCountDiffers() + { + Word wordX = NewTestWord(); + wordX.Freeze(); + + Word wordY = NewTestWord(); + Word nonHead = NewTestWord(); + nonHead.Freeze(); + wordY.NonHeadUnapplied(nonHead); + wordY.Freeze(); + + Assert.That(new AnalysisStateKey(wordX).Equals(new AnalysisStateKey(wordY)), Is.False); + } + + [Test] + public void Equals_False_WhenSyntacticFeatureStructDiffers() + { + Word wordX = NewTestWord(); + wordX.SyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value; + wordX.Freeze(); + + Word wordY = NewTestWord(); + wordY.SyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("N").Value; + wordY.Freeze(); + + Assert.That(new AnalysisStateKey(wordX).Equals(new AnalysisStateKey(wordY)), Is.False); + } + + [Test] + public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForMruleTrail() + { + var ruleA = new AffixProcessRule { Name = "ruleA" }; + var ruleB = new AffixProcessRule { Name = "ruleB" }; + var ruleC = new AffixProcessRule { Name = "ruleC" }; + + // The memoized node's own trail was [ruleA, ruleB] at the moment its subtree was recorded, with + // ruleA as the length-1 prefix (whatever led to this state) and ruleB as the subtree-local suffix + // that must survive the graft. + Word memoized = NewTestWord(); + memoized.MorphologicalRuleUnapplied(ruleA); + memoized.MorphologicalRuleUnapplied(ruleB); + memoized.Freeze(); + + // A different arrival at the same AnalysisStateKey, via a different prefix: [ruleC]. + Word query = NewTestWord(); + query.MorphologicalRuleUnapplied(ruleC); + query.Freeze(); + + Word replayed = memoized.ReplayOnto(query, mruleTrailPrefixLength: 1, nonHeadPrefixLength: 0); + + Assert.That(replayed.MorphologicalRules, Is.EqualTo(new IMorphologicalRule[] { ruleC, ruleB })); + } + + [Test] + public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForNonHeads() + { + // The memoized node had already unapplied one non-head (its own prefix, at the memo point) before + // its subtree unapplied a second one -- that second one is the subtree-local suffix to keep. The + // two non-heads are built from distinct lexical entries (32 vs 33) so a wrong-prefix graft (e.g. + // one that kept storedNonHead instead of subtreeNonHead, or reversed the GetRange window) is + // actually distinguishable via RootAllomorph identity, not just NonHeadCount. + Word storedNonHead = NewTestWord("32"); + storedNonHead.Freeze(); + Word subtreeNonHead = NewTestWord("33"); + subtreeNonHead.Freeze(); + Word memoized = NewTestWord("32"); + memoized.NonHeadUnapplied(storedNonHead); + memoized.NonHeadUnapplied(subtreeNonHead); + memoized.Freeze(); + + // Query reached the same key with a different (empty) non-head prefix. + Word query = NewTestWord("32"); + query.Freeze(); + + Word replayed = memoized.ReplayOnto(query, mruleTrailPrefixLength: 0, nonHeadPrefixLength: 1); + + // Query's (empty) prefix + the memoized subtree's suffix (subtreeNonHead) = 1 non-head. + Assert.That(replayed.NonHeadCount, Is.EqualTo(1)); + Assert.That(replayed.CurrentNonHead.RootAllomorph, Is.SameAs(subtreeNonHead.RootAllomorph)); + Assert.That(replayed.CurrentNonHead.RootAllomorph, Is.Not.SameAs(storedNonHead.RootAllomorph)); + } + + private Word NewTestWord(string entryId = "32") + { + var word = new Word(Entries[entryId].PrimaryAllomorph, FeatureStruct.New().Value) { Stratum = Morphophonemic }; + return word; + } +} diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs new file mode 100644 index 000000000..be195a77f --- /dev/null +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs @@ -0,0 +1,234 @@ +using System.Diagnostics; +using NUnit.Framework; + +namespace SIL.Machine.Morphology.HermitCrab; + +/// +/// Stage 5 of memoization.md: the real-data verification the toy-grammar unit tests (stages 2-4) +/// cannot provide. The mrule/template unit tests use 2-3-rule synthetic grammars specifically because +/// they let a specific redundant-order scenario be forced deterministically -- but they cannot exercise +/// whether 's key-completeness audit actually holds against the FULL +/// analysis-side rule set a real grammar invokes (AnalysisAffixProcessRule, +/// AnalysisCompoundingRule, AnalysisRealizationalAffixProcessRule, and every phonological +/// rule feeding into the states those rules read). If the key is missing a field some real rule +/// consults, it manifests as a divergence on some real word, nowhere else -- this harness is that check. +/// +/// [Explicit] and env-var driven, modeled on FstSenaBenchmark.cs's convention (this repo never commits +/// real morphological grammars or word lists -- see the standing grammar-privacy constraint -- so this +/// test ships with zero embedded grammar/word content and writes no output files, only TestContext +/// lines, to avoid leaking derived corpus data such as signature dumps into a committed path): +/// $env:HC_MEMO_GRAMMAR = "...\sena-hc.xml" +/// $env:HC_MEMO_WORDS = "...\sena-words.txt" +/// $env:HC_MEMO_MAX_WORDS = "60" # optional, default 60 +/// $env:HC_MEMO_TIMEOUT_MS = "5000" # optional, default 5000 (per-word watchdog) +/// dotnet test --filter "FullyQualifiedName~MemoCorpusVerification" +/// +/// Deliberately NOT the archive's 900-line parallel-benchmark harness (16-way scheduling, GC heap-limit +/// watchdog): per design, the only new runtime configuration this port introduces is single-threaded +/// (maxDegreeOfParallelism: 1), so there is no new concurrency to stress-test here -- just a +/// per-word timeout so one pathological word can't hang the whole run. +/// +[TestFixture] +[Explicit("Manual corpus verification against a local, uncommitted real grammar; not part of CI.")] +public class MemoCorpusVerification +{ + [Test] + public void MemoOnMatchesMemoOff_AnalysisSetIdentical_OnRealCorpus() + { + (Language language, List words) = Load(); + + var memoOff = new Morpher(new TraceManager(), language); + var memoOn = new Morpher(new TraceManager(), language, maxDegreeOfParallelism: 1); + int timeoutMs = int.TryParse(Environment.GetEnvironmentVariable("HC_MEMO_TIMEOUT_MS"), out int t) ? t : 5000; + + long mruleHitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; + long mruleNogoodsBefore = MemoizedCombinationRuleCascade.DiagNogoodHits; + long templateHitsBefore = AnalysisStratumRule.DiagTemplateMemoHits; + long templateNogoodsBefore = AnalysisStratumRule.DiagTemplateNogoodHits; + + var elapsedMsPerWord = new List(); + var perWordTimes = new List<(string Word, double OnMs, double OffMs)>(); + var divergences = new List(); + var timedOut = new List(); + int noParseBoth = 0; + + foreach (string word in words) + { + List onSignatures; + List offSignatures; + double onMs; + double offMs; + try + { + var swOn = Stopwatch.StartNew(); + onSignatures = RunWithTimeout(() => Signatures(memoOn, word), timeoutMs); + swOn.Stop(); + onMs = swOn.Elapsed.TotalMilliseconds; + + var swOff = Stopwatch.StartNew(); + offSignatures = RunWithTimeout(() => Signatures(memoOff, word), timeoutMs); + swOff.Stop(); + offMs = swOff.Elapsed.TotalMilliseconds; + } + catch (TimeoutException) + { + timedOut.Add(word); + continue; + } + elapsedMsPerWord.Add(onMs + offMs); + perWordTimes.Add((word, onMs, offMs)); + + if (onSignatures.Count == 0 && offSignatures.Count == 0) + noParseBoth++; + + if (!onSignatures.SequenceEqual(offSignatures)) + { + divergences.Add( + $"{word}: memo-on={{{string.Join(",", onSignatures)}}} vs " + + $"memo-off={{{string.Join(",", offSignatures)}}}" + ); + } + } + + elapsedMsPerWord.Sort(); + double p50 = Percentile(elapsedMsPerWord, 0.50); + double p95 = Percentile(elapsedMsPerWord, 0.95); + double totalMs = elapsedMsPerWord.Sum(); + + TestContext.Out.WriteLine($"words attempted: {words.Count}, timed out (>{timeoutMs}ms): {timedOut.Count}"); + TestContext.Out.WriteLine($"words with no parse on both sides: {noParseBoth}"); + TestContext.Out.WriteLine($"aggregate wall: {totalMs:F1} ms, p50: {p50:F1} ms, p95: {p95:F1} ms"); + + // Count-based vs wall-clock aggregates, reported SEPARATELY on purpose: a corpus is typically + // bimodal (many cheap words the memo makes slightly slower by losing a thread; a few pathological + // words the memo makes drastically faster) -- collapsing that into one ratio hides which regime a + // reader is in. "Most words are a bit slower" and "the corpus finishes much faster in total" are + // both true simultaneously and neither contradicts the other. + int fasterCount = perWordTimes.Count(x => x.OnMs < x.OffMs); + int slowerCount = perWordTimes.Count(x => x.OnMs > x.OffMs); + int tiedCount = perWordTimes.Count - fasterCount - slowerCount; + double totalOnMs = perWordTimes.Sum(x => x.OnMs); + double totalOffMs = perWordTimes.Sum(x => x.OffMs); + TestContext.Out.WriteLine( + $"count-based: {fasterCount}/{perWordTimes.Count} words faster under memo, " + + $"{slowerCount}/{perWordTimes.Count} slower, {tiedCount} tied" + ); + TestContext.Out.WriteLine( + $"wall-clock: memo-on total {totalOnMs:F1} ms vs memo-off total {totalOffMs:F1} ms " + + $"({(totalOnMs > 0 ? totalOffMs / totalOnMs : 0):F2}x)" + ); + if (timedOut.Count > 0) + { + // NOT a guaranteed lower bound in either direction: the try block above wraps BOTH the + // memo-on and memo-off calls, so a timeout could come from either side -- this harness + // never records which one actually timed out. If it was memo-on, the word's true + // memo-off time is genuinely unmeasured (could be faster OR slower than the ratio above + // implies); only a longer timeout actually resolves it. Report the fact, don't imply a + // direction the data doesn't support. + TestContext.Out.WriteLine( + $"(the {timedOut.Count} timed-out word(s) above are excluded from both aggregates; " + + "re-run with a higher HC_MEMO_TIMEOUT_MS to actually measure them)" + ); + } + // Per-heavy-word attribution (memoization.md's own methodological rule: an aggregate can be + // dominated by cheap words while hiding what pathological words actually do -- report both). + // memo-on is sequential+memo; memo-off is the untouched parallel default, so this is the + // user-visible claim (sequential-memo vs today's shipped behavior), not an isolated measurement + // of the memo mechanism's own contribution in isolation from single- vs multi-threading. + TestContext.Out.WriteLine("heaviest words (by memo-off time), memo-on vs memo-off:"); + foreach ((string w, double onMs2, double offMs2) in perWordTimes.OrderByDescending(x => x.OffMs).Take(10)) + TestContext.Out.WriteLine($" {w}: memo-on {onMs2:F1} ms, memo-off {offMs2:F1} ms"); + TestContext.Out.WriteLine( + $"mrule memo -- positive hits: {MemoizedCombinationRuleCascade.DiagMemoHits - mruleHitsBefore}, " + + $"nogood hits: {MemoizedCombinationRuleCascade.DiagNogoodHits - mruleNogoodsBefore}" + ); + TestContext.Out.WriteLine( + $"template memo -- positive hits: {AnalysisStratumRule.DiagTemplateMemoHits - templateHitsBefore}, " + + $"nogood hits: {AnalysisStratumRule.DiagTemplateNogoodHits - templateNogoodsBefore}" + ); + if (timedOut.Count > 0) + { + // Named, not just counted: a timed-out word is EXCLUDED from the equality gate above, so + // "0 divergences" says nothing about it. These are exactly the candidates for a follow-up + // run with a longer HC_MEMO_TIMEOUT_MS (see memoization.md §5's addendum on why this + // mattered -- the heavy words are precisely the ones the memo, and the key-completeness + // audit, most need to be checked against). + TestContext.Out.WriteLine( + $"timed-out words (excluded from the equality gate above -- re-run with a higher " + + $"HC_MEMO_TIMEOUT_MS to actually check these): {string.Join(", ", timedOut)}" + ); + } + + Assert.That( + divergences, + Is.Empty, + $"{divergences.Count} word(s) diverged between memo-on and memo-off " + + $"(showing up to 10): {string.Join(" | ", divergences.Take(10))}" + ); + Assert.That( + MemoizedCombinationRuleCascade.DiagMemoHits + AnalysisStratumRule.DiagTemplateMemoHits, + Is.GreaterThan(mruleHitsBefore + templateHitsBefore), + "the positive replay path must actually have fired somewhere in this corpus -- otherwise " + + "this run cannot distinguish a working memo from a no-op one" + ); + } + + private static List Signatures(Morpher morpher, string word) + { + try + { + return morpher + .ParseWord(word) + .Select(MorpherTests.WordAnalysisSignature) + .OrderBy(s => s, StringComparer.Ordinal) + .ToList(); + } + catch (InvalidShapeException) + { + // Matches Morpher.AnalyzeWord's own handling: a word list drawn from real text can contain + // strings the grammar's character table doesn't cover (e.g. punctuation) -- not a memo + // concern, both sides would throw identically. + return new List(); + } + } + + // Does NOT cancel `action` on timeout -- Morpher.ParseWord has no cooperative-cancellation + // hook, so a timed-out word's Task keeps running in the background. This can inflate the + // hit/nogood counters and per-word timings reported below with work from an abandoned prior + // word, and piled-up orphaned tasks from several timed-out words in a row can starve the + // thread pool. Acceptable for this [Explicit], never-in-CI diagnostic harness (the equality + // gate itself is unaffected -- a timed-out word is excluded from it either way, see the + // caller), but do not read the reported hit counts/timings as precise when timeouts occurred. + private static T RunWithTimeout(Func action, int timeoutMs) + { + Task task = Task.Run(action); + if (!task.Wait(timeoutMs)) + throw new TimeoutException(); + return task.Result; + } + + private static double Percentile(List sortedValues, double fraction) + { + if (sortedValues.Count == 0) + return 0; + int index = (int)Math.Ceiling(fraction * sortedValues.Count) - 1; + return sortedValues[Math.Clamp(index, 0, sortedValues.Count - 1)]; + } + + private static (Language, List) Load() + { + string? grammarPath = Environment.GetEnvironmentVariable("HC_MEMO_GRAMMAR"); + string? wordsPath = Environment.GetEnvironmentVariable("HC_MEMO_WORDS"); + if (string.IsNullOrEmpty(grammarPath) || string.IsNullOrEmpty(wordsPath)) + Assert.Ignore("set HC_MEMO_GRAMMAR and HC_MEMO_WORDS"); + + int maxWords = int.TryParse(Environment.GetEnvironmentVariable("HC_MEMO_MAX_WORDS"), out int mw) ? mw : 60; + Language language = XmlLanguageLoader.Load(grammarPath!); + List words = File.ReadAllLines(wordsPath!) + .Select(w => w.Trim()) + .Where(w => w.Length > 0) + .Take(maxWords) + .ToList(); + return (language, words); + } +} diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs new file mode 100644 index 000000000..2caf0b61f --- /dev/null +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs @@ -0,0 +1,189 @@ +using NUnit.Framework; +using SIL.Machine.Annotations; +using SIL.Machine.FeatureModel; +using SIL.Machine.Morphology.HermitCrab.MorphologicalRules; +using SIL.Machine.Rules; +using SIL.ObjectModel; + +namespace SIL.Machine.Morphology.HermitCrab; + +// Stage 3 of memoization.md: MemoizedCombinationRuleCascade exercised directly, bypassing +// Morpher/AnalysisStratumRule entirely, so the test can force the exact commuting-order redundancy the +// memo targets without depending on whether a particular morphology grammar's search happens to revisit +// a PRODUCTIVE (not just nogood) state -- MorpherTests' end-to-end compounding grammar, checked via the +// real Morpher pipeline, only ever hits the nogood table on this small a grammar (see its own hit-count +// diagnostic output), so this is the test that pins the POSITIVE replay path non-vacuously. +[TestFixture] +public class MemoizedCombinationRuleCascadeTests : HermitCrabTestBase +{ + [Test] + public void Apply_ReplaysPositiveHit_WhenTwoOrdersReachTheSameKey() + { + var ruleA = new AffixProcessRule { Name = "ruleA" }; + var ruleB = new AffixProcessRule { Name = "ruleB" }; + var ruleC = new AffixProcessRule { Name = "ruleC" }; + + // Each fake rule unapplies its own IMorphologicalRule at most once, so from the initial word, + // trying ruleA-then-ruleB-then-ruleC and ruleB-then-ruleA-then-ruleC reach the SAME + // AnalysisStateKey after the first two steps (multiset {ruleA:1, ruleB:1}) via different orders + // -- exactly the redundancy AnalysisStateKey collapses. Only from that shared state can ruleC + // still apply, so the shared node's own subtree is POSITIVE (one result), not a nogood. + var cascade = new MemoizedCombinationRuleCascade( + new IRule[] + { + new SingleUseUnapplyRule(ruleA), + new SingleUseUnapplyRule(ruleB), + new SingleUseUnapplyRule(ruleC), + }, + FreezableEqualityComparer.Default + ); + + Word initial = NewTestWord(); + initial.AnalysisScope = new AnalysisScope(); + initial.Freeze(); + + long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; + List results = new List(cascade.Apply(initial)); + + // Every leaf where all 3 rules have been unapplied, in whichever of the two orders explored the + // shared {ruleA,ruleB} state first vs via replay, must appear -- and the replay path must have + // actually fired (memoization.md's non-vacuousness requirement). + Assert.That( + results, + Has.Some.Matches(w => + w.GetUnapplicationCount(ruleA) == 1 + && w.GetUnapplicationCount(ruleB) == 1 + && w.GetUnapplicationCount(ruleC) == 1 + ) + ); + Assert.That( + MemoizedCombinationRuleCascade.DiagMemoHits, + Is.GreaterThan(hitsBefore), + "this test's whole point is to force a positive replay -- it must not go vacuous" + ); + } + + [Test] + public void Apply_PositiveReplayMatchesUnmemoizedResultSet_IncludingTrailOrder() + { + // The previous test proves a replay FIRES; this one proves it produces the RIGHT result. A count + // assertion (3 rules applied) would pass even if ReplayOnto grafted the wrong prefix, since counts + // are order-invariant -- comparing MorphemesInApplicationOrder (which walks the trail ReplayOnto + // rewrites, see WordAnalysisSignature's own doc comment in MorpherTests.cs) catches a wrong-prefix + // graft that silently collapses [ruleB,ruleA,ruleC] into a duplicate of [ruleA,ruleB,ruleC]. + var ruleA = new AffixProcessRule { Id = "RULE_A", Name = "ruleA" }; + var ruleB = new AffixProcessRule { Id = "RULE_B", Name = "ruleB" }; + var ruleC = new AffixProcessRule { Id = "RULE_C", Name = "ruleC" }; + IRule[] rules = + { + new SingleUseUnapplyRule(ruleA), + new SingleUseUnapplyRule(ruleB), + new SingleUseUnapplyRule(ruleC), + }; + + Word memoized = NewTestWord(); + memoized.AnalysisScope = new AnalysisScope(); + memoized.Freeze(); + + Word unmemoized = NewTestWord(); + // AnalysisScope left null -- exercises MemoizedCombinationRuleCascade's own unmemoized fallback + // (ApplyRulesRaw), the same path a tracing parse takes. + unmemoized.Freeze(); + + var memoizedCascade = new MemoizedCombinationRuleCascade(rules, FreezableEqualityComparer.Default); + var unmemoizedCascade = new MemoizedCombinationRuleCascade(rules, FreezableEqualityComparer.Default); + + long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; + List memoizedSignatures = memoizedCascade + .Apply(memoized) + .Select(TrailSignature) + .OrderBy(s => s, StringComparer.Ordinal) + .ToList(); + List unmemoizedSignatures = unmemoizedCascade + .Apply(unmemoized) + .Select(TrailSignature) + .OrderBy(s => s, StringComparer.Ordinal) + .ToList(); + + Assert.That( + memoizedSignatures, + Is.EqualTo(unmemoizedSignatures), + "a positive replay must reproduce exactly the unmemoized result set, INCLUDING trail order" + ); + Assert.That( + MemoizedCombinationRuleCascade.DiagMemoHits, + Is.GreaterThan(hitsBefore), + "this test's whole point is to compare a real replay against the unmemoized result -- it " + + "must not go vacuous" + ); + } + + [Test] + public void Apply_FallsBackToUnmemoizedExpansion_WhenKeyIsAlreadyInProgress() + { + // Pins the in-flight re-entry guard (memoization.md's design rule 3 / the InProgress table): + // a multiApp cascade can reach the same AnalysisStateKey again while its own first expansion is + // still on the call stack (e.g. via a self-loop elsewhere in a real grammar's rule graph). + // Rather than construct that reentrancy organically -- the key is monotonic in rule-application + // count for straightforward single-use rules, so a real cyclic re-arrival is hard to force here + // -- this simulates the in-flight state directly: pre-populate InProgress with the exact key + // `initial` will compute, then call Apply and confirm it falls through to a correct, unmemoized + // expansion (ApplyRulesRaw) instead of reading a nonexistent Memo entry or hanging. + var ruleA = new AffixProcessRule { Id = "RULE_A", Name = "ruleA" }; + var cascade = new MemoizedCombinationRuleCascade( + new IRule[] { new SingleUseUnapplyRule(ruleA) }, + FreezableEqualityComparer.Default + ); + + Word initial = NewTestWord(); + var scope = new AnalysisScope(); + initial.AnalysisScope = scope; + initial.Freeze(); + + var key = new AnalysisStateKey(initial); + scope.InProgress.TryAdd(key, 0); + + long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; + List results = new List(cascade.Apply(initial)); + + Assert.That(results, Has.Some.Matches(w => w.GetUnapplicationCount(ruleA) == 1)); + Assert.That( + MemoizedCombinationRuleCascade.DiagMemoHits, + Is.EqualTo(hitsBefore), + "the in-flight fallback must not read/count a memo hit -- it never consults Memo at all" + ); + Assert.That( + scope.Memo.ContainsKey(key), + Is.False, + "the in-flight arrival's OWN key must never be written to Memo (deeper recursive calls for " + + "OTHER keys, reached via ApplyRulesRaw's normal recursion, may still memoize themselves)" + ); + } + + private static string TrailSignature(Word word) => + string.Join("+", word.MorphemesInApplicationOrder.Select(m => m.Id)); + + private Word NewTestWord() + { + return new Word(Entries["32"].PrimaryAllomorph, FeatureStruct.New().Value) { Stratum = Morphophonemic }; + } + + // Minimal stand-in for a compiled analysis morphological rule: unapplies `_rule` against the input + // exactly once (per input), independent of Shape/FeatureStruct pattern matching, so the cascade's + // own commuting-order redundancy can be exercised without compiling a real FST-backed rule. + private sealed class SingleUseUnapplyRule(IMorphologicalRule rule) : IRule + { + private readonly IMorphologicalRule _rule = rule; + + public IEnumerable Apply(Word input) + { + if (input.GetUnapplicationCount(_rule) > 0) + yield break; + + Word result = input.Clone(); + result.MorphologicalRuleUnapplied(_rule); + result.Freeze(); + yield return result; + } + } +} diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs index 3c92898e3..c4e0012cd 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs @@ -535,4 +535,302 @@ IList GetNodes(string pattern) Shape shape = new Segments(Table2, pattern, true).Shape; return shape.GetNodes(shape.Range).ToList(); } + + [Test] + public void ParseWord_SingleThreaded_MatchesParallel_WithCompounding() + { + // Stage 1 of memoization.md: pins the new runtime MaxDegreeOfParallelism toggle to be a pure + // no-op on results before any memoization is wired up. Compounding specifically (not just plain + // affixes) because it's the mechanism the eventual analysis-cascade memo cares about: an affix + // rule that commutes with a compounding rule -- both peers in the same Unordered + // MorphologicalRules cascade -- can revisit an equal AnalysisStateKey via different arrival + // orders, so this grammar is the one later commits' equivalence tests build on. + var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; + var crule = new CompoundingRule { Name = "rule1" }; + Allophonic.MorphologicalRules.Add(crule); + crule.Subrules.Add( + new CompoundingSubrule + { + HeadLhs = { Pattern.New("head").Annotation(any).OneOrMore.Value }, + NonHeadLhs = { Pattern.New("nonHead").Annotation(any).OneOrMore.Value }, + Rhs = { new CopyFromInput("head"), new InsertSegments(Table3, "+"), new CopyFromInput("nonHead") }, + } + ); + + var prefix = new AffixProcessRule + { + Id = "PREFIX", + Name = "prefix", + Gloss = "PAST", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + OutSyntacticFeatureStruct = FeatureStruct + .New(Language.SyntacticFeatureSystem) + .Feature(Head) + .EqualTo(head => head.Feature("tense").EqualTo("past")) + .Value, + }; + Allophonic.MorphologicalRules.Insert(0, prefix); + prefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, + } + ); + + var parallel = new Morpher(TraceManager, Language); + var singleThreaded = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + + foreach (string word in new[] { "pʰutdidat", "pʰutdat" }) + { + List singleResult = singleThreaded.ParseWord(word).ToList(); + List parallelResult = parallel.ParseWord(word).ToList(); + Assert.That( + singleResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(parallelResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"single-threaded parse of '{word}' must match the parallel parse" + ); + } + } + + [Test] + public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithCompounding() + { + // Stage 3 of memoization.md: the mrule-cascade memo is now actually wired in for + // maxDegreeOfParallelism: 1. This is the standing acceptance gate (memoization.md §5) -- + // analysis-set (signature) equality between the memoized single-threaded cascade and the + // unmemoized parallel default -- made non-vacuous by asserting the memo's hit counter actually + // moved, so a memo that silently stopped firing couldn't pass this test by accident. + var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; + var crule = new CompoundingRule { Name = "rule1" }; + Allophonic.MorphologicalRules.Add(crule); + crule.Subrules.Add( + new CompoundingSubrule + { + HeadLhs = { Pattern.New("head").Annotation(any).OneOrMore.Value }, + NonHeadLhs = { Pattern.New("nonHead").Annotation(any).OneOrMore.Value }, + Rhs = { new CopyFromInput("head"), new InsertSegments(Table3, "+"), new CopyFromInput("nonHead") }, + } + ); + + var prefix = new AffixProcessRule + { + Id = "PREFIX", + Name = "prefix", + Gloss = "PAST", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + OutSyntacticFeatureStruct = FeatureStruct + .New(Language.SyntacticFeatureSystem) + .Feature(Head) + .EqualTo(head => head.Feature("tense").EqualTo("past")) + .Value, + }; + Allophonic.MorphologicalRules.Insert(0, prefix); + prefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, + } + ); + + var memoOff = new Morpher(TraceManager, Language); + var memoOn = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + + long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; + long nogoodHitsBefore = MemoizedCombinationRuleCascade.DiagNogoodHits; + foreach (string word in new[] { "pʰutdidat", "pʰutdat" }) + { + List onResult = memoOn.ParseWord(word).ToList(); + List offResult = memoOff.ParseWord(word).ToList(); + Assert.That( + onResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(offResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"memo-on parse of '{word}' must be analysis-set identical to memo-off" + ); + } + TestContext.Out.WriteLine( + $"positive hits: {MemoizedCombinationRuleCascade.DiagMemoHits - hitsBefore}, " + + $"nogood hits: {MemoizedCombinationRuleCascade.DiagNogoodHits - nogoodHitsBefore}" + ); + Assert.That( + MemoizedCombinationRuleCascade.DiagMemoHits + MemoizedCombinationRuleCascade.DiagNogoodHits, + Is.GreaterThan(hitsBefore + nogoodHitsBefore), + "the memo must actually have hit (positive or nogood) at least once on this grammar -- " + + "otherwise this test cannot distinguish a working memo from a no-op one" + ); + } + + [Test] + public void ParseWord_MemoOnMatchesMemoOff_ForSelfOpaquingSimultaneousEpenthesis() + { + // Diagnostic pulled forward per memoization.md §5(a): PanGloss's rust conformance suite documents + // a confirmed C#-oracle nogood-cache bug on this exact fixture shape (a Simultaneous-mode + // epenthesis rule, which AnalysisRewriteRule compiles with ReapplyType.SelfOpaquing -- a + // repeat-until-fixpoint loop -- against root 19's boundary-bearing "b+ubu"). A direct + // reconstruction attempt against this prototype's own AnalysisScope (parse-optimization-archive) + // did not reproduce a divergence for "buibui" under any tracing/order combination tried, and the + // real minimal fixture (rust conformance/rewrite/simultaneous-epenthesis) is not present in this + // checkout to test against directly. Wiring this in as a standing regression case is the + // practical mitigation: the general memo-on/off equality gate (this test) covers it going + // forward, and the ≥2-self-opaquing-iteration case remains an open, documented gap (PanGloss's + // own conclusion too -- no available fixture drives the loop past one iteration). + var highVowel = FeatureStruct + .New(Language.PhonologicalFeatureSystem) + .Symbol(HCFeatureSystem.Segment) + .Symbol("cons-") + .Symbol("voc+") + .Symbol("high+") + .Value; + var highFrontUnrndVowel = FeatureStruct + .New(Language.PhonologicalFeatureSystem) + .Symbol(HCFeatureSystem.Segment) + .Symbol("cons-") + .Symbol("voc+") + .Symbol("high+") + .Symbol("back-") + .Symbol("round-") + .Value; + + var rule4 = new RewriteRule { Name = "rule4", ApplicationMode = RewriteApplicationMode.Simultaneous }; + Allophonic.PhonologicalRules.Add(rule4); + rule4.Subrules.Add( + new RewriteSubrule + { + Rhs = Pattern.New().Annotation(highFrontUnrndVowel).Value, + LeftEnvironment = Pattern.New().Annotation(highVowel).Value, + } + ); + + var memoOff = new Morpher(TraceManager, Language); + var memoOn = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + + foreach (string word in new[] { "buibui", "bubu", "bibu" }) + { + List onResult = memoOn.ParseWord(word).ToList(); + List offResult = memoOff.ParseWord(word).ToList(); + Assert.That( + onResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(offResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"memo-on parse of '{word}' must be analysis-set identical to memo-off" + ); + } + // The traced/correct oracle value for "buibui" (PanGloss's rust conformance fixture): root 19, + // one epenthesized result. Pinned directly, not just compared on-vs-off, so a bug that happened + // to affect both sides identically (e.g. both wrongly returning empty) would still be caught. + Assert.That(memoOn.ParseWord("buibui").Count(), Is.EqualTo(1)); + } + + [Test] + public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithAffixTemplate() + { + // Stage 4 of memoization.md: exercises the template-battery memo + // (AnalysisStratumRule.ApplyTemplateBattery) specifically -- TWO free prefix rules that commute + // with each other and with a template slot suffix. Unapplying di-then-gu vs gu-then-di reaches + // the same AnalysisStateKey (same shape, same rule MULTISET) via a different trail ORDER, so the + // second arrival replays the first arrival's stored template outputs with its own trail prefix + // grafted on (Word.ReplayOnto). One commuting prefix would NOT be enough -- a single rule can + // only unapply once, so no key would ever be re-arrived at and the memo would never fire. + var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; + + var edSuffix = new AffixProcessRule + { + Id = "TPAST", + Name = "template_ed_suffix", + Gloss = "PAST", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + edSuffix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new CopyFromInput("1"), new InsertSegments(Table3, "+d") }, + } + ); + var verbTemplate = new AffixTemplate + { + Name = "verb_template", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + verbTemplate.Slots.Add(new AffixTemplateSlot(edSuffix) { Optional = true }); + Morphophonemic.AffixTemplates.Add(verbTemplate); + + var diPrefix = new AffixProcessRule + { + Id = "TDI", + Name = "template_di_prefix", + Gloss = "DI", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + diPrefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, + } + ); + Morphophonemic.MorphologicalRules.Add(diPrefix); + + var guPrefix = new AffixProcessRule + { + Id = "TGU", + Name = "template_gu_prefix", + Gloss = "GU", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + guPrefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "gu+"), new CopyFromInput("1") }, + } + ); + Morphophonemic.MorphologicalRules.Add(guPrefix); + + var memoOff = new Morpher(TraceManager, Language); + var memoOn = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + + long templateHitsBefore = AnalysisStratumRule.DiagTemplateMemoHits; + long templateNogoodHitsBefore = AnalysisStratumRule.DiagTemplateNogoodHits; + foreach (string word in new[] { "digusagd", "disagd", "gusagd", "sagd", "sag" }) + { + List onResult = memoOn.ParseWord(word).ToList(); + List offResult = memoOff.ParseWord(word).ToList(); + Assert.That( + onResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(offResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"memo-on parse of '{word}' must be analysis-set identical to memo-off" + ); + } + TestContext.Out.WriteLine( + $"template positive hits: {AnalysisStratumRule.DiagTemplateMemoHits - templateHitsBefore}, " + + $"template nogood hits: {AnalysisStratumRule.DiagTemplateNogoodHits - templateNogoodHitsBefore}" + ); + // Guards against this test going vacuous: the template memo's replay path must actually fire + // for this grammar. (As with the mrule-cascade memo, the ReplayOnto graft's effect on final + // signatures is unobservable through ParseWord's synthesis round-trip -- see memoization.md §5(b) + // -- so this counter, not the equivalence assertions above, is what proves the memoized path is + // actually exercised here, not silently bypassed.) + Assert.That( + AnalysisStratumRule.DiagTemplateMemoHits + AnalysisStratumRule.DiagTemplateNogoodHits, + Is.GreaterThan(templateHitsBefore + templateNogoodHitsBefore), + "the template memo must actually have hit (positive or nogood) at least once on this " + + "grammar -- otherwise this test cannot distinguish a working memo from a no-op one" + ); + } + + // Canonical analysis-set signature (memoization.md §5: gates compare this, never byte/object + // equality -- a memo-replayed Word is not guaranteed field-for-field identical to a freshly-computed + // one). AllomorphsInMorphOrder alone would not catch a broken trail/non-head graft (it walks Shape + // annotations, which Word.ReplayOnto never touches) -- MorphemesInApplicationOrder walks + // _mruleApps/_nonHeadApps directly, which is exactly what ReplayOnto rewrites. Root index is included + // so two analyses with the same morpheme sequence but different lexical roots can't collide. + internal static string WordAnalysisSignature(Word word) + { + return string.Join("+", word.AllomorphsInMorphOrder.Select(a => a.Morpheme.Id)) + + "|" + + string.Join("+", word.MorphemesInApplicationOrder.Select(m => m.Id)) + + "|root=" + + word.RootAllomorph.Morpheme.Id; + } } From 5fe9543094a78d058541f62bbbf4651dae85ad45 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 18 Aug 2026 07:01:16 -0400 Subject: [PATCH 2/3] Preserve MaxAlternatives in memoized cascade --- .../AnalysisStratumRule.cs | 2 +- .../MemoizedCombinationRuleCascade.cs | 2 ++ .../MemoizedCombinationRuleCascadeTests.cs | 27 +++++++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs index eb3d7246b..7f40323ce 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs @@ -51,7 +51,7 @@ public AnalysisStratumRule(Morpher morpher, Stratum stratum) // within-word parallelism; parallel cascade otherwise. _mrulesRule = morpher.MaxDegreeOfParallelism == 1 - ? (IRule) + ? (RuleCascade) new MemoizedCombinationRuleCascade(mrules, FreezableEqualityComparer.Default) : new ParallelCombinationRuleCascade( mrules, diff --git a/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs index bad620f3a..a99e3d0ff 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs @@ -75,6 +75,7 @@ private List ApplyRules(Word input, HashSet output) entry.NonHeadPrefixLength ); output.Add(replay); + CheckMaxAlternatives(output.Count); replayed.Add(replay); } DiagMemoHits++; @@ -111,6 +112,7 @@ private List ApplyRulesRaw(Word input, HashSet output) { local.Add(result); output.Add(result); + CheckMaxAlternatives(output.Count); // avoid infinite loop -- same guard CombinationRuleCascade uses if (!Comparer.Equals(input, result)) local.AddRange(ApplyRules(result, output)); diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs index 2caf0b61f..89cd2438b 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs @@ -160,6 +160,33 @@ public void Apply_FallsBackToUnmemoizedExpansion_WhenKeyIsAlreadyInProgress() ); } + [Test] + public void Apply_EnforcesMaxAlternatives_ForRawAndReplayPaths() + { + var ruleA = new AffixProcessRule { Id = "RULE_A", Name = "ruleA" }; + var ruleB = new AffixProcessRule { Id = "RULE_B", Name = "ruleB" }; + IRule[] rules = { new SingleUseUnapplyRule(ruleA), new SingleUseUnapplyRule(ruleB) }; + + var rawCascade = new MemoizedCombinationRuleCascade(rules, FreezableEqualityComparer.Default) + { + MaxAlternatives = 1, + }; + Word rawInitial = NewTestWord(); + rawInitial.AnalysisScope = new AnalysisScope(); + rawInitial.Freeze(); + + Assert.Throws(() => new List(rawCascade.Apply(rawInitial))); + + var replayCascade = new MemoizedCombinationRuleCascade(rules, FreezableEqualityComparer.Default); + Word replayInitial = NewTestWord(); + replayInitial.AnalysisScope = new AnalysisScope(); + replayInitial.Freeze(); + _ = new List(replayCascade.Apply(replayInitial)); + + replayCascade.MaxAlternatives = 1; + Assert.Throws(() => new List(replayCascade.Apply(replayInitial))); + } + private static string TrailSignature(Word word) => string.Join("+", word.MorphemesInApplicationOrder.Select(m => m.Id)); From e37c2888fb4c188b57907723985263297e43c1a7 Mon Sep 17 00:00:00 2001 From: Damien Daspit Date: Tue, 18 Aug 2026 17:23:55 -0400 Subject: [PATCH 3/3] Address review findings in analysis-cascade memo - Clear AnalysisScope entering synthesis, so returned parses no longer pin the per-parse memo tables - Make maxDegreeOfParallelism an enforced cap and retire the dead SINGLE_THREADED toggles it was meant to replace - Keep the template memo off Linear strata, whose key completeness is unaudited, and reject unfrozen words as memo keys - Deduplicate replay/store onto AnalysisScope, drop ReplayOnto's discarded clones, use plain collections on the sequential-only path, and match CombinationRuleCascade's expansion order - Make the diagnostic counters atomic; trim comments to non-obvious constraints and remove memoization.md Co-Authored-By: Claude Opus 5 (1M context) --- memoization.md | 420 ------------------ .../AnalysisAffixTemplateRule.cs | 23 +- .../AnalysisScope.cs | 126 ++++-- .../AnalysisStateKey.cs | 47 +- .../AnalysisStratumRule.cs | 45 +- .../MemoizedCombinationRuleCascade.cs | 82 ++-- .../Morpher.cs | 52 ++- src/SIL.Machine.Morphology.HermitCrab/Word.cs | 99 +++-- .../Rules/ParallelCombinationRuleCascade.cs | 11 + .../AnalysisStateKeyTests.cs | 66 ++- .../AnalysisStratumRuleTests.cs | 94 ++++ .../MemoCorpusVerification.cs | 79 ++-- .../MemoizedCombinationRuleCascadeTests.cs | 48 +- .../MorpherTests.cs | 206 +++++---- 14 files changed, 598 insertions(+), 800 deletions(-) delete mode 100644 memoization.md create mode 100644 tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStratumRuleTests.cs diff --git a/memoization.md b/memoization.md deleted file mode 100644 index 2ce3359b8..000000000 --- a/memoization.md +++ /dev/null @@ -1,420 +0,0 @@ -# Memoization for HermitCrab analysis — minimal, verified port to master - -**Branch:** `feature/memoization` (at master `d9deb167`). -**Goal:** capture the prototype's demonstrated speedup (a pathological Bantu-template -word ~5×; a 312-word Sena heavy set 1,051s → 388s) with a small, human-reviewable diff, -no RUSTIFY rearchitecture, and stronger verification than the prototype had. - -Real grammar/corpus content (Sena, Amharic, Indonesian) is never committed to this repo, -including individual word forms in prose — this doc refers to specific test words only by -generic labels (heavy word H1, H2, ...) rather than by their actual spelling. - -## 1. Provenance and evidence base - -The design is not new. It was built, measured, and corpus-verified on the -`parse-optimization` branch (2026-07-02/03), which was stacked on the unmerged RUSTIFY -commit and later deleted; the full 23-commit chain has been recovered and preserved as the -local branch **`parse-optimization-archive`**. The Rust port (PanGloss `hc-memo`, -`hc-rules/stratum.rs`) faithfully re-implements the same design and adds test patterns and -two hygiene lessons, but no new architecture. Primary references: - -- `parse-optimization-archive` — `AnalysisScope.cs`, `AnalysisStateKey.cs`, - `MemoizedCombinationRuleCascade.cs`, `Word.ReplayOnto` (commits `3522fc41` Phases 0-3, - `6ea0536a` Phase 3b), plus `parse-optimization.md` and - `docs/hermitcrab-parse-algorithm-analysis.md` in that tree. -- PanGloss — `hc-memo/src/lib.rs`, `hc-rules/src/stratum.rs:751-955`, - `hc-rules/tests/memo_gate.rs`, `hc-parse/tests/simultaneous_conformance.rs`. -- fst-advisor branch docs — verification-gate methodology (golden signature dumps, - per-grammar gating order, soundness batteries). - -**Where the speedup actually is (fair sequential baselines, measured on the archive):** - -| Mechanism | Effect | -|---|---| -| mrule-cascade positive memo + nogood cache (Phases 2/3) | ~10% (30.5s → 27.4s on heavy word H1). 2,555 real expansions vs 118,162 hits — hits are cheap because guard clauses already reject fast. | -| **template-battery memo (Phase 3b)** | **The real win: 93% of instrumented wall time.** 38,840 battery invocations vs ~2,581 unique keys. H1 27.4s → 6.1s; H2 102.7s → 26.9s; Sena 312-set 1,051s → 388s. | - -Both mechanisms share one key type and one replay primitive, so the minimal port includes -both; the template memo is the piece that must not be dropped if scope is ever cut. - -## 2. What the design is (one page) - -All memoization is scoped to a single `Morpher.ParseWord` call and applies only to the -**sequential** Unordered-order (template/Bantu) analysis cascade. - -- **`AnalysisScope`** — carrier object threaded through `Word` clones like `CurrentTrace` - (reference-shared; excluded from `FreezeImpl`/`ValueEquals`). Holds two tables + - in-flight guards + a 100,000-entry growth cap (OOM guard; correctness-neutral): - - `Memo` — mrule-cascade results (empty list = nogood, non-empty = positive), - - `TemplateMemo` — affix-template-battery results (separate table: same key space, - different computation), - - `InProgress` (+ a separate `TemplateInProgress`, a Rust-side clarity improvement worth - adopting) — re-entry guards; a hit falls through to unmemoized expansion for that call. -- **`AnalysisStateKey`** (readonly struct) — `(Shape, Stratum, SyntacticFeatureStruct, - RealizationalFeatureStruct, NonHeadCount, multiset of per-rule unapplication counts)`. - Order-invariant by construction: the multiset collapses the k! orderings that cause the - 98.4% expansion redundancy. Hash uses cached frozen hashes + commutative XOR over the - multiset; equality compares full values (never trust a bare hash — entries store the full - key). The constructor **freezes shape/FSs defensively on read** (known landmine: - `AnalysisAffixTemplateRule.Apply` reassigns an unfrozen `SyntacticFeatureStruct` after - the Word is frozen; that setter has no `CheckFrozen()` guard). -- **`MemoEntry`** — `{ Results: IReadOnlyList, MruleTrailPrefixLength, - NonHeadPrefixLength }`. -- **`Word.ReplayOnto(query, mruleTrailPrefixLen, nonHeadPrefixLen)`** — the replay graft: - clone the stored result, replace the trail/non-head *prefix* with the query node's own, - keep the suffix, re-freeze. Sound because everything strictly inside a memoized subtree - is a deterministic function of the key fields alone; only the two ordered structures the - key collapses to counts need grafting. -- **`MemoizedCombinationRuleCascade`** (mrule cascade) and an `ApplyTemplateBattery` - hook in `AnalysisStratumRule` (template battery): both do key → `TryGetValue` → replay - on hit / compute + `TryAdd` on miss. -- **Tracing bypasses the memo entirely** (`ParseWord` only installs `AnalysisScope` when - `!IsTracing`): traces must stay byte-identical to the unmemoized engine. - -**Design rules locked in from the research (each maps to a specific finding):** - -1. **No pruning gate may be coupled to memo presence.** The prototype bundled the Phase-5 - `HasReachableRoot` lexical gate into the memo-on path; Rust deliberately removed the - coupling to preserve `memo-on == memo-off` as a true invariant. We port the memo only — - no Phase 5, no gating — so the toggle is a pure optimization knob. -2. **Key-completeness is the primary correctness risk** (packrat/Ford; Bazel cache - doctrine). The port includes a written audit: every field read by any - `MorphologicalRules/Analysis*.cs` rule is either in the key or provably irrelevant - (`IsPartial` and `_isLastAppliedRuleFinal` are grep-verified unread). This audit lives - as a code comment on `AnalysisStateKey` and must be re-run when analysis rules change. -3. **Trail push and multiset increment are one atomic unit.** `record`-style bookkeeping - pairing (`_mruleApps.Add` + unapplied-count increment) must never drift; replay's - prefix/suffix split depends on multiset-equality ⇒ prefix-length-equality. -4. **Budget interaction (future):** if complexity-cap ever lands, a budget-interrupted - subtree must never be written to the memo ("exhausted subtree is not memoized" — the - PanGloss ground rule). Not needed now (master has no budgets) but recorded as a standing - invariant next to the cap constant. -5. **Per-parse lifetime, pre-sized where cheap** — no cross-word or process-global caching. - (A word→analysis-set cache above the engine is sound and orthogonal — explicitly out of - scope here; see fst-advisor `HYBRID_FST_FEASIBILITY.md` §6.) - -## 3. What master is missing (the port deltas) - -Master's HC tree is unchanged since the RUSTIFY fork point (`78350670`), so the substrate -the memo needs — `Word._mruleApps`/`_nonHeadApps`/`Clone`/`Freeze`, -`Shape/FeatureStruct.GetFrozenHashCode/ValueEquals`, `CombinationRuleCascade` — exists -**byte-identical** on master. Deltas: - -| Delta | Adaptation | -|---|---| -| `TOffset` is `ShapeNode` on master, `int` on RUSTIFY | Mechanical: port against `IRule` etc. | -| **Master has no runtime sequential/parallel toggle** — cascade choice is `#if SINGLE_THREADED`, which no csproj defines, so the shipped library always runs the parallel cascade and the memo path would be unreachable | **Prerequisite work** (the only genuinely new work): add a runtime knob to `Morpher` (RUSTIFY's `maxDegreeOfParallelism` ctor param is the reference design). Default preserves current parallel behavior. Scoped to the analysis side only — `SynthesisStratumRule` is untouched; synthesis has no equivalent memo. | -| No COW `Shape`/`FeatureStruct` (RUSTIFY-only) | Omit Phase 10a-1/7b polish. `ReplayOnto` does a real deep clone on master — slower than the prototype, never wrong. Expect somewhat less than the prototype's exact numbers; measure. | -| `InstrumentedRule`/rule-stats infra, `TrailDirectedRuleCascade` (Phase 0/1), `hc batch` command | Not load-bearing for the memo. Skip from the shipped diff; a slim corpus-signature harness is built as test-only tooling instead. | -| Master's `.Distinct(FreezableEqualityComparer)` in `AnalysisStratumRule` | Keep (harmless with the memo's deduped output; removal is a separate evaluation). | - -## 4. Implementation structure - -Built and gated in five logical stages (not five separate PR commits — the shipped diff -is squashed to one): - -**Stage 1 — prerequisite: runtime sequential-cascade toggle.** -`Morpher` gains `MaxDegreeOfParallelism` (ctor param; default = current parallel -behavior). `AnalysisStratumRule` selects sequential vs parallel cascade at runtime -instead of `#if SINGLE_THREADED`. No memoization yet. Gate: full unit suite green; -behavior byte-identical at default; a test pins sequential == parallel output on an -existing grammar. - -**Stage 2 — key + scope + replay primitives, with their unit tests.** -New `AnalysisStateKey`, `AnalysisScope`; additive `Word` members (`ReplayOnto`, -`UnappliedRuleCounts`, `MorphologicalRuleTrailLength`, `AnalysisScope` property). -Nothing consumes them yet. Tests ported from the archive + PanGloss `hc-memo` unit -battery: key order-invariance over permuted unapplication sequences, hash consistency, -sensitivity to non-head count and differing multisets, replay prefix/suffix graft, -in-flight guard semantics. The key-completeness audit comment lands here. - -**Stage 3 — mrule-cascade memo (positive + nogood).** -`MemoizedCombinationRuleCascade` wired into `AnalysisStratumRule`'s Unordered sequential -path; `ParseWord` installs `AnalysisScope` when sequential and `!IsTracing`. Tests: -hit-counter-guarded equivalence test (compounding grammar, from archive `MorpherTests`), -plus the **new adversarial fixtures the prototype never had** (§5, items a-b). - -**Stage 4 — template-battery memo (the 5×).** -`TemplateMemo` + `ApplyTemplateBattery` in `AnalysisStratumRule`. Tests: the -affix-template equivalence test (two commuting prefix rules, hit-counter-guarded), and -the memo-on/off gate exercising both tables in one parse (PanGloss `memo_gate.rs` shape). - -**Stage 5 — corpus verification harness + benchmark evidence.** -Test-only `[Explicit]` corpus runner in the HC test project (modeled on -`FstSenaBenchmark.cs`: env-var driven, per-word watchdog), comparing canonical -analysis-set signatures against uncommitted local grammars. Results recorded in §5/§6. - -Ship the memo **on by default whenever the cascade runs sequentially** (it is a pure -optimization once the invariant tests pass); the *default cascade mode* stays parallel in -this PR — flipping the library default to sequential+memo (which the corpus numbers say is -faster anyway) is proposed as a follow-up PR with its own benchmark evidence, so the -behavior-default change and the mechanism land as separately revertable units. - -## 5. Verification plan (stronger than the prototype's) - -Per-commit unit gates as above, plus: - -- **Memo-on/off analysis-set equality as the standing acceptance gate** (the metamorphic - relation: equal keys ⇒ equal result sets) — **not byte-for-byte object equality.** The - gate compares canonical signature sets (sorted `join("+", morphemeIds)+":"+rootIndex`), - because `ReplayOnto`'s grafted `Word` is not guaranteed field-for-field identical to a - freshly-computed one (e.g. `ShapeNode`/annotation object identity differs) even when it - represents the same analysis. Every equivalence test asserts non-vacuously (memo hit - counters > 0, both tables non-empty where applicable). -- **(a) SelfOpaquing ≥2-iteration fixture — the confirmed-bug shape (resolved: not reproduced, - wired in as a standing regression).** PanGloss's rust conformance suite documents a - confirmed C#-oracle bug in this exact code path (a `RewriteApplicationMode.Simultaneous` - epenthesis rule against root 19's `"b+ubu"`; `AnalysisRewriteRule` compiles this shape as - `ReapplyType.SelfOpaquing`, a repeat-until-fixpoint loop). Before trusting stage 3, this - was reconstructed directly against `parse-optimization-archive`'s own `AnalysisScope` - using the real `RewriteRuleTests.EpenthesisRules` rule definition: parsing `"buibui"` - gave the same result (1 parse) under every tracing/call-order combination tried - (non-traced, traced, repeated, order-swapped) — no divergence. The real minimal fixture - (`conformance/rewrite/simultaneous-epenthesis/`) is not present in this checkout to test - against directly (its own Rust test is `#[ignore]`'d for the same reason). Per the - advisor consult: this does not block stage 3 — the general memo-on/off analysis-set - equality gate is strictly stronger than one fixture, and PanGloss's own Rust analysis - reached the identical conclusion for their side (memo-sound on this shape; the ≥2-iteration - loop×memo interaction specifically "remains untested" because no available fixture drives - the loop past one iteration). **Shipped:** `MorpherTests.ParseWord_MemoOnMatchesMemoOff_ - ForSelfOpaquingSimultaneousEpenthesis` wires this exact fixture in as a standing - regression case (memo-on/off equality plus a pinned-value assertion), and the - ≥2-iteration gap is recorded here verbatim as an open, honest limitation — not silently - dropped. -- **(b) Trail-order-observable grammar — closed at the right layer, not through `ParseWord`.** - `Morpher.ParseWord` does not return raw analysis-cascade words — it feeds them through a - full **synthesis** re-derivation (`Synthesize`/`LexicalLookup`/`PermuteRules`), which - re-explores rule orderings on its own, so a `ParseWord`-level probe is the wrong - instrument for this: an attempt via a disabled-merge grammar variant returned 0 results - (most likely `MergeEquivalentAnalyses=false` breaking that synthesis path some other way, - not evidence about trail-order observability one way or the other — not chased further, - since it isn't the right layer regardless). **What the actual safety net is:** the - standing memo-on/off analysis-*set*-equality gate catches the class of bug that matters - — a dropped or spuriously-added analysis, exactly the PanGloss 0-vs-1 shape — because a - dropped analysis never reaches synthesis to be re-normalized away. Internal analysis - trail order, by contrast, is plausibly a don't-care for the returned output set, which - is also why it doesn't need a `ParseWord`-level test. The graft primitive itself IS - pinned by a real red/green test, at the layer where it's actually observable: - `AnalysisStateKeyTests.ReplayOnto_Grafts*` (stage 2, direct construction) and - `MemoizedCombinationRuleCascadeTests.Apply_PositiveReplayMatchesUnmemoizedResultSet_ - IncludingTrailOrder` (stage 3, a synthetic 3-rule cascade run directly against - `MemoizedCombinationRuleCascade` — bypassing `Morpher` so a real commuting-order replay - can be forced — comparing the memoized run's result set against an unmemoized run - *including* `MorphemesInApplicationOrder`, i.e. trail order). Verified this actually - discriminates: temporarily breaking `ReplayOnto` to skip the prefix graft made both this - test and the stage-2 unit tests fail, confirming they are not vacuous. -- **Corpus gates, in the fst-advisor order:** Indonesian 121/121 first (cheap iteration); - guarded Sena slice (first 60 words, 5s/word watchdog); then the Sena 312-word heavy set. - Compare sequential-memo (`maxDegreeOfParallelism: 1`, the only sequential configuration - that exists post-stage-3 — the memo is unconditional whenever the cascade is sequential, - so "sequential-unmemoized" is no longer a reachable end-to-end configuration; that - comparison is instead covered at the unit level, see §5(b)) against parallel-unmemoized - master default — the actual user-visible claim. Must be **analysis-set identical** — same - canonical signature set, not byte-identical objects — including negative/no-parse words - (nogoods cache no-parses too — both sides must produce the empty set). Amharic - (machine-local, gitignored) as the alphabet-stress smoke test; never commit real-grammar - fixtures (privacy constraint). -- **Reporting** (standing requirement): every measured claim ships with p50/p95 per-word - latency, aggregate wall, and hit/miss counts (`DiagMemoHits`/`DiagNogoodHits`, split — - not just totals) — and stage 5 must assert `DiagMemoHits > 0` somewhere in the corpus - run, so the positive-replay path can't ship having never actually fired end-to-end. - -**Stage 5 result (local uncommitted grammar — Sena, both slices):** ran -`MemoCorpusVerification` against the local `samples/data/sena-hc.xml` / -`sena-words.txt` (never committed — grammar-privacy constraint). - -| Slice | Timeout | Compared | Timed out | No-parse (both) | Divergences | Mrule memo (pos/nogood) | Template memo (pos/nogood) | -|---|---|---|---|---|---|---|---| -| First 60 | 5s | 54 | 6 | 11 | **0** | 8,652 / 59,156 | 10,481 / 3 | -| Full 312 | 8s | 252 | 60 | 87 | **0** | 201,051 / 1,692,696 | 286,581 / 29 | - -Zero divergences across both runs, hundreds of thousands of memo hits on both -tables (not vacuous), on real analysis-rule content the toy-grammar unit tests -(stages 2-4) structurally cannot reach — this is the empirical confirmation -of `AnalysisStateKey`'s key-completeness audit. The full-312 run's timeouts -(60 words, 8s/side budget) are the same known pathological Bantu template -words the archive prototype measured; p50 678.7ms / p95 -7,997.2ms per word (memo-on + memo-off combined) on the successfully-compared -words. Full per-word timing table is in the run's `TestContext` output, not -committed (never commit real-word signature data). - -**Indonesian (121/121, PanGloss's local `indonesian-hc.xml`/`-words.txt`):** -**0 divergences**, 0 timeouts, p50 37.2ms / p95 346.0ms. Mrule memo: 64 -positive / 157 nogood. Template memo: **0 positive / 410 nogood** — Indonesian -has no Bantu-style affix-template redundancy, so the template table only ever -proves subtrees empty here, never replays; matches §6's own prediction that -typical (non-template) grammars see little upside. Per-heavy-word timing on -the 10 slowest words shows memo-on (sequential) is actually *slower* than -memo-off (parallel default) by ~10-15% on this grammar — expected and -consistent with §6: the memo only pays for itself where order-variant -redundancy exists, and sequential-single-core loses to parallel-multi-core -when it doesn't. This is the honest "no regression in the wrong direction on -non-Bantu grammars" finding, not a null result — the comparison being measured -here is user-visible (sequential-memo vs parallel-unmemoized default), not the -memo mechanism isolated from single- vs multi-threading (see §5's note on why -"sequential-unmemoized" isn't independently constructible once the mrule-cascade -memo is wired in). - -**Amharic (machine-local, gitignored, alphabet-stress smoke test):** 30 words, -8s/side timeout. **0 divergences** on the 6 words that finished both sides -within budget (10 timed out — Amharic's Ge'ez-script grammar is known to be -build/parse-heavy; 14 had no parse on either side). Mrule memo: 14 positive / -113 nogood. Template memo: 41 positive / 2 nogood — both fired. Confirms the -key/replay machinery is script-agnostic (Ge'ez abugida segments, not Latin -transliteration) with no divergence. No Amharic word forms appear in this doc -or any committed file, per the standing grammar-privacy constraint. - -**Summary across all three real grammars tested: zero divergences, in every -case with both memo tables exercised non-vacuously.** This is the strongest -available evidence for `AnalysisStateKey`'s key-completeness audit short of -running the entire, much larger production corpora. - -**Critical follow-up (advisor-flagged): the aggregate corpus runs above prove -nothing about the heavy words, because they timed out and were excluded from -the equality gate.** 60 of the full-312 Sena run's words never got compared — -those are exactly the template-heavy words the memo exists for, and the ones -where a key-completeness miss would actually surface. A short timeout also -means the only completed timings showed memo-on *slower* (Indonesian, -Amharic) — sequential-memo vs parallel-unmemoized, not evidence of the actual -speedup goal. Closed with a targeted, longer-timeout run on heavy word **H1** -(known-pathological, present in the Sena heavy-word set) at 240s: - -- **Soundness on a heavy word, actually executed:** memo-on vs memo-off — - analysis-set identical (0 divergences). Mrule memo: 29,736 positive / - 88,426 nogood hits. Template memo: 37,512 positive / 0 nogood hits — this - is the key-completeness confirmation the excluded corpus runs above could - not provide. -- **User-visible timing:** memo-on (sequential+memo) **22.1s** vs memo-off - (today's shipped parallel default) **136.1s** — **6.2×**. -- **Isolated memo contribution** (mutation-tested: temporarily disabled the - `AnalysisScope` install in `Morpher.ParseWord`, reverted after — same - technique as the ReplayOnto mutation test in §5(b)): sequential-no-memo - **138.0s** vs parallel-no-memo (unchanged) **141.5s** — parallelism alone - contributes essentially nothing on this word (~2%). Compared against - sequential-**memo**'s 22.1s: the memo mechanism itself is responsible for - essentially the entire **~6.3× speedup**, not a threading artifact. - -This exceeds the plan's own target (≥3× on heavy-word-class words) despite -master lacking the prototype's copy-on-write `Shape`/`FeatureStruct` — the -`ReplayOnto` deep-clone tax feared in §6 below did not dominate here. - -**Second heavy word, H2 — soundness inconclusive, not a clean win.** Two -independent runs at a 400s per-side timeout both had the memo-on -(sequential+memo) side fail to finish within 400s — i.e. this word is *not* -analysis-set-verified at this depth; the "Passed" result in the first run -reflects the harness's by-design exclusion of timed-out words from the -divergence check, not a completed comparison. This is a materially -different outcome from H1 and is reported honestly rather than folded into -the "0 divergences" summary above. Notably, the archive prototype's own -measurement (§1's table above) had H2 at 102.7s → 26.9s with the template -memo — master's ported memo-on run did not complete this word within 400s, -~15× slower than the archive's memoized figure. Plausible explanations -(unconfirmed): master's deep-clone `ReplayOnto` (vs the archive's -copy-on-write `Shape`/`FeatureStruct`) may cost materially more on this -word's particular redundancy shape than it did on H1; or H2 exercises a -rule-graph region where the memo's hit rate is lower and raw expansion -dominates. Not chased further with a longer timeout, per the same -bounded-effort judgment applied to the other unmeasured heavy words below — -documented as unmeasured/slower-than-expected, a concrete follow-up, not a -blocker for this PR (soundness is unaffected either way: an incomplete run -makes no claim, positive or negative, about key-completeness for this -word). - -Remaining honest gap: H1 is the one heavy word with a completed, positive -measurement; H2 is a heavy word with an *incomplete* one (see above); the -other 58 timed-out full-312-run words (and Indonesian/Amharic's timed-out -words) are unmeasured at this depth — a natural follow-up, not a blocker, -given the confirmed pattern (soundness holds, memo dominates the win) on -the one word measured to completion. - -**Aggregate corpus evidence: count-based and wall-clock, reported -separately (see the harness's own two-line report format).** A Sena -sub-corpus (words 1-70, 200s per-side timeout — long enough for the one -tractable heavy word in this range to complete on both sides, per §5's H1 -measurement above) gives, of the 69 words that completed within budget: -**53/69 (77%) faster under memo, 16/69 (23%) slower** — count-based, this -is the "yes, some words are slower" half of the picture. **Wall-clock: -memo-on total 114.6s vs memo-off total 277.8s across those same 69 -words — 2.42× faster in aggregate** — this is the "but a lot are faster, -and it nets out strongly positive" half. The one word that timed out at -200s (a second tractable-but-slow heavy word, distinct from both H1 and -H2) is excluded from both numbers. The harness's timeout wraps both the -memo-on and memo-off calls in one try block per word, so it cannot record -which side actually timed out for this word — if it was memo-on (the more -likely case here, since it's attempted first and this class of word is -exactly where the memo's own cost is least understood, see the H2 -discussion below), the word's true memo-off time is genuinely unmeasured, -not "presumably at least as slow." **This 2.42× figure should be read as -this slice's measured result, not asserted as a guaranteed lower bound** -in either direction — a longer timeout is the only way to actually find -out. Both memo tables fired heavily and non-vacuously on this slice -(mrule: tens of thousands of positive/nogood hits; template: tens of -thousands positive, near-zero nogood). - -**Is the typical-word slowdown "just" losing a thread? Mostly, not -purely.** Isolated on Indonesian (no template redundancy — the cleanest -instrument for this, since the template memo never replays there) via -three repeated runs per configuration to average out run-to-run JIT/GC -noise (single-run absolute times varied by as much as 60% between -otherwise-identical runs of the unchanged parallel baseline — only -within-run memo-on-vs-memo-off ratios are trustworthy here, not -across-run absolute milliseconds): - -| Configuration vs parallel-default | Aggregate ratio (3 reps) | Implied slowdown (1/ratio − 1) | -|---|---|---| -| sequential + memo (shipped opt-in) | 0.80×, 0.80×, 0.78× | ~25-28% | -| sequential, memo forced off (mutation-tested) | 0.69×, 0.80×, 0.68× | ~25-47% | - -The sequential-*no-memo* slowdown is consistently at or above the -sequential-*memo* slowdown, never below it. So most of the typical-word -regression genuinely is the lost thread (both configurations are slower -than parallel-default, by a similar order of magnitude) — but it is not -*purely* that: even on a grammar where the template memo never once -replays, the mrule-cascade's own cache still fires (64 positive / 157 -nogood hits on this corpus) and measurably claws back some of the -thread-loss penalty rather than adding to it. Three reps each is enough to -see the direction of the effect, not enough to state a precise recovered -percentage with confidence — a larger repeated-trial run would sharpen -this if the exact number ever matters for the default-flip decision. - -**Is the H2 shortfall actually the deep-clone tax?** Plausible, not -confirmed — resist stating it as established. The one piece of direct -evidence is that H2's memo fired heavily before timing out (tens of -thousands of positive hits on both tables, §5 above) — the memo is -clearly not idle on this word, which favors "replay/clone cost per hit is -what's expensive" over "the memo doesn't engage." But that is a -directional tell, not a measurement: distinguishing "expensive replay" from -"just a lot more raw expansion than H1" requires profiling where memo-on -wall time actually goes (`Clone`/`Freeze`/`ReplayOnto` vs raw rule -expansion) on H2 specifically. Scoped as a follow-up, not undertaken here. - -## 6. Expected outcome and honest caveats - -- Pathological template-heavy words: prototype showed ~5× (with COW clones). Master's - deep-clone `ReplayOnto` was expected to give back some of that; **measured directly on - heavy word H1 (§5 addendum): ~6.3× isolated memo contribution, ~6.2× on the - user-visible sequential-memo-vs-parallel-default comparison — met and exceeded the - ≥3× target**, meaning the feared `ReplayOnto` deep-clone tax did not dominate on this - word. The ≥50% Sena-heavy-set aggregate target remains unmeasured (that run's heavy - words timed out before completing the equality gate, §5 addendum) — a natural - follow-up. A second heavy word, H2, did NOT complete within a 400s memo-on budget (§5 - addendum) — the first concrete (if inconclusive) counter-data-point to the - deep-clone-tax question, since the archive's COW-based prototype handled this same - word in 26.9s. If a future heavy-set measurement shows replay cost dominating after - all, this is the evidence that would motivate porting `Shape` COW as its own - follow-up — do not fold it into this PR pre-emptively. -- Typical words (Indonesian-class) get measurably *slower* under sequential+memo - (~25-28% in aggregate, §5 addendum) — mostly the cost of losing a thread, not the memo - itself (isolated via a sequential-no-memo comparison, same addendum). The corpus gate - proves "no divergence" there, not "no regression" — this is exactly why the memo ships - off by default rather than flipping the library's default cascade mode. -- The memo is inert for `Ordered`-strata-only grammars and for `ParallelCombinationRuleCascade` - (no natural subtree-completion moment in its breadth-first walk — same reason the - prototype left it unmemoized). - -## 7. Out of scope (deliberately) - -Phase 0/1 instrumentation and synthesis-side `TrailDirectedRuleCascade`; Phase 4/5 gates -(measured inert or unsound-coupled); RUSTIFY COW polish (7b/10a); packed forest (9c); -word-level result caching above the engine; any interned-ID key redesign (`rust-conversion.md` -§6.3 is an unshipped plan — needs its own measurement if ever wanted). diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisAffixTemplateRule.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisAffixTemplateRule.cs index 6331e2995..a14114f4a 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisAffixTemplateRule.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisAffixTemplateRule.cs @@ -1,14 +1,12 @@ -using System.Collections.Generic; +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; using System.Linq; +using System.Threading.Tasks; using SIL.Machine.Annotations; using SIL.Machine.FeatureModel; using SIL.Machine.Rules; using SIL.ObjectModel; -#if !SINGLE_THREADED -using System; -using System.Collections.Concurrent; -using System.Threading.Tasks; -#endif namespace SIL.Machine.Morphology.HermitCrab { @@ -47,18 +45,16 @@ public IEnumerable Apply(Word input) inWord.Freeze(); var output = new HashSet(FreezableEqualityComparer.Default); -#if SINGLE_THREADED - ApplySlots(inWord, _rules.Count - 1, output); -#else - ParallelApplySlots(inWord, output); -#endif + if (_morpher.MaxDegreeOfParallelism == 1) + ApplySlots(inWord, _rules.Count - 1, output); + else + ParallelApplySlots(inWord, output); foreach (Word outWord in output) outWord.SyntacticFeatureStruct.Add(fs); return output; } -#if SINGLE_THREADED private void ApplySlots(Word inWord, int index, HashSet output) { for (int i = index; i >= 0; i--) @@ -78,7 +74,7 @@ private void ApplySlots(Word inWord, int index, HashSet output) _morpher.TraceManager.EndUnapplyTemplate(_template, inWord, true); output.Add(inWord); } -#else + private void ParallelApplySlots(Word inWord, HashSet output) { var outStack = new ConcurrentStack(); @@ -126,6 +122,5 @@ private void ParallelApplySlots(Word inWord, HashSet output) output.UnionWith(outStack); } -#endif } } diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs index 73d6cc99c..5ce8c4add 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisScope.cs @@ -1,60 +1,106 @@ -using System.Collections.Concurrent; using System.Collections.Generic; namespace SIL.Machine.Morphology.HermitCrab { /// - /// Per-parse cache carrier threaded through clones exactly like - /// -- reference-shared, excluded from Word.FreezeImpl/ - /// Word.ValueEquals so existing dedup semantics are unchanged. Holds the analysis-cascade memo - /// (memoization.md) -- see . - /// One instance per call: entries are state facts - /// about a specific parse (a state key does not encode the target surface word), so sharing this - /// across concurrent parses of different words would be unsound without also scoping the key to the - /// word -- that cross-word extension is explicitly out of scope. - /// Thread-safe because a single word's analysis can itself run in parallel - /// (ParallelCombinationRuleCascade, used when is not - /// 1) -- though today only the sequential cascade actually reads/writes this. + /// Carrier for the analysis-cascade memo, threaded through clones + /// like and likewise excluded from Word.FreezeImpl/ + /// Word.ValueEquals, so dedup semantics are unchanged. + /// + /// One instance per call. A state key does not + /// encode the target surface word, so sharing a scope across parses of different words would be + /// unsound. + /// + /// + /// Not thread-safe, hence the plain collections: a scope is only installed when + /// is 1. Memoizing the parallel cascade would require + /// concurrent ones. + /// /// internal sealed class AnalysisScope { - // OOM guard: a positive memo holds actual Word lists (not just a boolean like the nogood case), so - // it is the one that can grow unboundedly on a pathological word. Past the cap, new subtrees are - // simply not memoized -- correctness is unaffected, only the hit rate degrades. No corpus word - // seen so far has come close to this cap; deliberately untested against an actual overflow. + // OOM guard, since a positive entry holds Word lists rather than just a flag. Past the cap + // subtrees simply go unmemoized: only the hit rate degrades, never correctness. private const int MaxMemoEntries = 100_000; - public ConcurrentDictionary Memo { get; } = - new ConcurrentDictionary(); + public Dictionary Memo { get; } = new Dictionary(); - // Second table, same key space, different computation: the affix-template battery result for a - // state (AnalysisStratumRule.ApplyTemplateBattery), as opposed to the mrule-cascade subtree result - // above. Kept separate because a state can be memoized in one table but not (yet) the other. - public ConcurrentDictionary TemplateMemo { get; } = - new ConcurrentDictionary(); + // Same key space as Memo, different computation: the affix-template battery's result for a state + // (AnalysisStratumRule.ApplyTemplateBattery). Separate because a state can be memoized in one + // table but not the other. + public Dictionary TemplateMemo { get; } = + new Dictionary(); - // Keys currently under expansion on some call stack -- guards the in-flight re-entry case (a - // multiApp cascade can reach the same state again before its own first expansion has completed, - // e.g. via a self-loop). A hit here must fall through to plain, unmemoized expansion rather than - // read a nonexistent/partial entry or deadlock; see MemoizedCombinationRuleCascade.ApplyRules. The - // template battery needs no equivalent guard: ApplyTemplateBattery's call is eager and - // self-contained (no template<->mrule mutual recursion happens inside it). - public ConcurrentDictionary InProgress { get; } = - new ConcurrentDictionary(); + // Keys still under expansion somewhere on the call stack. A multiApp cascade can reach the same + // state again before its first expansion has finished (e.g. via a self-loop), which must fall + // through to unmemoized expansion rather than read a partial entry or deadlock. The template + // battery needs no equivalent: its call is eager, with no template/mrule mutual recursion inside. + public HashSet InProgress { get; } = new HashSet(); - public bool HasMemoCapacity => Memo.Count < MaxMemoEntries; + /// + /// Replay shared by both memo consumers. False on a miss; on a hit + /// holds the stored results grafted onto , or + /// is empty for a stored-empty ("nogood") entry. The query's non-head prefix is cloned once and + /// shared across this hit's replays, which is safe because each replay freezes immediately and + /// every non-head mutation path is CheckFrozen-guarded. + /// + public bool TryReplay( + Dictionary table, + AnalysisStateKey key, + Word query, + out List replayed + ) + { + if (!table.TryGetValue(key, out MemoEntry entry)) + { + replayed = null; + return false; + } + if (entry.Results.Count == 0) + { + replayed = new List(); + return true; + } + List queryNonHeadPrefix = query.CloneNonHeadsForReplay(); + replayed = new List(entry.Results.Count); + foreach (Word stored in entry.Results) + { + replayed.Add( + stored.ReplayOnto( + query, + entry.MruleTrailPrefixLength, + entry.NonHeadPrefixLength, + queryNonHeadPrefix + ) + ); + } + return true; + } - public bool HasTemplateMemoCapacity => TemplateMemo.Count < MaxMemoEntries; + /// + /// Records a fully-expanded result list against , unless the table is full. + /// + public void Store( + Dictionary table, + AnalysisStateKey key, + Word query, + List results + ) + { + if (table.Count < MaxMemoEntries) + table[key] = new MemoEntry(results, query.MorphologicalRuleTrailLength, query.NonHeadCount); + } } /// - /// A memoized analysis-cascade subtree or template-battery result (memoization.md). - /// empty = the "nogood" case (subtree/battery proved to yield nothing); non-empty = the positive case, - /// replayable onto a differently-ordered arrival at the same via - /// , using / - /// to split each stored result's trail/non-heads into the (discarded, replaced) prefix and the (kept) - /// subtree-local suffix. There is no "budget exhausted / incomplete" flag: this engine has no step/time - /// budget infrastructure, so every recorded subtree was explored to full completion. + /// A memoized subtree or template-battery result. An empty means the state was + /// proved to yield nothing. The two prefix lengths are the trail/non-head counts at the moment of the + /// write, which is where splits a stored result when grafting it onto a + /// new arrival. + /// + /// There is deliberately no "incomplete" flag: only fully-explored subtrees may be recorded. Should a + /// step or time budget ever be added, an interrupted subtree must not be stored. + /// /// internal sealed class MemoEntry { diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs index ca19fef0b..91a91f748 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStateKey.cs @@ -6,25 +6,22 @@ namespace SIL.Machine.Morphology.HermitCrab { /// - /// Order-independent identity of an analysis-cascade node, used by the memo cache in - /// 's Unordered-mode morphological rule cascade (memoization.md). Two - /// Words with an equal key are guaranteed to make identical decisions in every analysis-side - /// morphological rule that cascade can invoke -- verified by inspecting what each one reads from its - /// input (key-completeness audit, re-run whenever an Analysis*.cs rule changes): + /// Order-independent identity of an analysis-cascade node. Two Words with an equal + /// key must make identical decisions in every analysis-side rule the cascade can invoke; that is the + /// memo's correctness contract, so this key-completeness audit of what each rule reads has to be + /// re-run whenever an Analysis*.cs rule changes: /// - /// : Shape (FST pattern match) and - /// (unifiability gate) plus a per-rule unapplication count. + /// : Shape (FST pattern match), + /// (unifiability gate), per-rule unapplication count. /// : adds - /// (MaxStemCount gate) -- it never reads the non-heads' own content, only the count. + /// (MaxStemCount gate) -- never the non-heads' own content, only the count. /// : adds /// . /// - /// but never the ORDER those rules were unapplied in -- exactly the redundancy this key collapses. - /// Deliberately excludes fields Word.ValueEquals includes for a different purpose (result - /// dedup): the unapplication trail as an ordered SEQUENCE (replaced here by an order-independent - /// multiset) and _isLastAppliedRuleFinal/IsPartial, which are not read by any - /// analysis-side rule (grep-verified against every file matching MorphologicalRules/Analysis*.cs - /// and PhonologicalRules/Analysis*.cs). + /// No rule reads the order those rules were unapplied in, which is the redundancy this key collapses, + /// so the trail is reduced to an unordered multiset here. _isLastAppliedRuleFinal and + /// IsPartial are excluded as well: Word.ValueEquals includes them for result dedup, but + /// no analysis-side rule reads them. /// internal readonly struct AnalysisStateKey : IEquatable { @@ -38,6 +35,15 @@ namespace SIL.Machine.Morphology.HermitCrab public AnalysisStateKey(Word word) { + // The cached hash covers live references -- notably Word.UnappliedRuleCounts, the word's own + // mutable dictionary. Keying an unfrozen word would let a later mutation invalidate a stored + // key's hash, silently causing permanent misses or entries that no longer match their bucket. + if (!word.IsFrozen) + throw new ArgumentException( + "The word must be frozen before it can be used as a memo key.", + nameof(word) + ); + _shape = word.Shape; _stratum = word.Stratum; _syntacticFS = word.SyntacticFeatureStruct; @@ -45,9 +51,11 @@ public AnalysisStateKey(Word word) _nonHeadCount = word.NonHeadCount; _ruleCounts = word.UnappliedRuleCounts; - // Defensive: AnalysisAffixTemplateRule.Apply reassigns SyntacticFeatureStruct to a fresh, - // unfrozen clone AFTER the owning Word is already frozen (no CheckFrozen() guard on that - // setter). Freeze() is idempotent and safe to call again here. + // Word.FreezeImpl deliberately leaves SyntacticFeatureStruct unfrozen, and + // AnalysisAffixTemplateRule.Apply mutates it in place on already-frozen Words -- no + // Word-level CheckFrozen guards that path. Freezing here pins the key's view of it, so a + // future rule mutating an already-keyed word throws instead of silently corrupting the table. + // Freeze is idempotent. _shape.Freeze(); _syntacticFS.Freeze(); _realizationalFS.Freeze(); @@ -60,9 +68,8 @@ public AnalysisStateKey(Word word) hash = hash * 31 + _nonHeadCount; if (_ruleCounts != null) { - // XOR, not the usual *31 rolling combine: the multiset is unordered, so the combination - // must be commutative -- two dictionaries with the same entries built up in different - // unapplication orders must hash identically. + // XOR rather than the usual *31 rolling combine: the multiset is unordered, so entries + // accumulated in different unapplication orders must still hash identically. int multisetHash = 0; foreach (KeyValuePair kvp in _ruleCounts) multisetHash ^= (kvp.Key.GetHashCode() * 397) ^ kvp.Value; diff --git a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs index 7f40323ce..8ebfd2d52 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/AnalysisStratumRule.cs @@ -2,6 +2,7 @@ using System.Collections.Generic; using System.Diagnostics; using System.Linq; +using System.Threading; using SIL.Machine.Annotations; using SIL.Machine.Rules; using SIL.ObjectModel; @@ -47,8 +48,6 @@ public AnalysisStratumRule(Morpher morpher, Stratum stratum) ); break; case MorphologicalRuleOrder.Unordered: - // Sequential (and memoized, see MemoizedCombinationRuleCascade) when the caller caps - // within-word parallelism; parallel cascade otherwise. _mrulesRule = morpher.MaxDegreeOfParallelism == 1 ? (RuleCascade) @@ -57,7 +56,10 @@ public AnalysisStratumRule(Morpher morpher, Stratum stratum) mrules, true, FreezableEqualityComparer.Default - ); + ) + { + MaxDegreeOfParallelism = morpher.MaxDegreeOfParallelism, + }; break; } } @@ -186,45 +188,36 @@ private IEnumerable ApplyMorphologicalRules(Word input) } } - // Test/reporting hooks (memoization.md's standing hit/miss-count requirement), mirroring - // MemoizedCombinationRuleCascade's DiagMemoHits/DiagNogoodHits split. + // Counterparts to MemoizedCombinationRuleCascade's counters, for the template table. internal static long DiagTemplateMemoHits; internal static long DiagTemplateNogoodHits; - // Runs the affix-template battery for `input`, memoized by AnalysisStateKey (memoization.md), - // same replay mechanism as MemoizedCombinationRuleCascade -- see AnalysisScope.InProgress for - // why no re-entry guard is needed here. Measured motivation (an archive prototype benchmark on - // a Bantu-template grammar): this battery accounted for the large majority of parse wall time - // once the mrule cascade's own memo had already shrunk its own share to near-nothing. + // The affix-template battery, memoized by AnalysisStateKey against its own table. On + // template-heavy grammars this dominates parse time, which is why it is memoized separately from + // the mrule cascade. See AnalysisScope.InProgress for why no re-entry guard is needed here. private IEnumerable ApplyTemplateBattery(Word input) { + // Scope presence is the single source of truth for whether the memo is active; Morpher decides + // that once, at install time. Linear strata stay unmemoized because the key-completeness audit + // covers only the Unordered cascade. AnalysisScope scope = input.AnalysisScope; - if (scope == null || _morpher.MaxDegreeOfParallelism != 1) + if (scope == null || _stratum.MorphologicalRuleOrder != MorphologicalRuleOrder.Unordered) return _templatesRule.Apply(input); var key = new AnalysisStateKey(input); - if (scope.TemplateMemo.TryGetValue(key, out MemoEntry entry)) + if (scope.TryReplay(scope.TemplateMemo, key, input, out List replayed)) { - if (entry.Results.Count == 0) + if (replayed.Count == 0) { - DiagTemplateNogoodHits++; - return Enumerable.Empty(); + Interlocked.Increment(ref DiagTemplateNogoodHits); + return replayed; } - var replayed = new List(entry.Results.Count); - foreach (Word stored in entry.Results) - replayed.Add(stored.ReplayOnto(input, entry.MruleTrailPrefixLength, entry.NonHeadPrefixLength)); - DiagTemplateMemoHits++; + Interlocked.Increment(ref DiagTemplateMemoHits); return replayed; } var results = new List(_templatesRule.Apply(input)); - if (scope.HasTemplateMemoCapacity) - { - scope.TemplateMemo.TryAdd( - key, - new MemoEntry(results, input.MorphologicalRuleTrailLength, input.NonHeadCount) - ); - } + scope.Store(scope.TemplateMemo, key, input, results); return results; } diff --git a/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs index a99e3d0ff..36f4dd24f 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/MemoizedCombinationRuleCascade.cs @@ -1,34 +1,25 @@ using System.Collections.Generic; +using System.Threading; using SIL.Machine.Annotations; using SIL.Machine.Rules; namespace SIL.Machine.Morphology.HermitCrab { /// - /// Drop-in replacement for the sequential on - /// Unordered-order analysis strata (memoization.md). Before expanding a node, checks whether an - /// earlier expansion elsewhere in the same word's analysis -- reached via a different unapplication - /// order, but with an equal -- already searched this exact state: - /// - /// proved empty (the "nogood" case) -> skip straight to "no results"; - /// produced results (the positive case) -> replay them () instead - /// of re-searching: clone each stored result and graft the CURRENT arrival's own trail/non-head - /// prefix onto the stored subtree-local suffix (see and - /// for why only the prefix -- never the suffix -- needs replacing). - /// - /// Scoped to the sequential cascade only. The parallel cascade - /// (ParallelCombinationRuleCascade, used when is - /// not 1) is a level-by-level breadth-first walk with no natural "this subtree is fully expanded" - /// moment to hang a memo write on, so it is left unmemoized -- callers that want this optimization - /// should construct their with maxDegreeOfParallelism: 1. + /// The sequential plus memoization of each + /// expanded subtree, for Unordered-order analysis strata. A node whose + /// was already searched earlier in this word's analysis, via a + /// different unapplication order, is not searched again: an empty stored result short-circuits, and a + /// non-empty one is replayed onto the current arrival (). + /// + /// The parallel cascade is left unmemoized -- its breadth-first walk never reaches a point where a + /// given subtree is known to be fully expanded, so there is nowhere to hang a memo write. + /// /// - internal class MemoizedCombinationRuleCascade : RuleCascade + internal class MemoizedCombinationRuleCascade : CombinationRuleCascade { - // Test/reporting hooks (memoization.md's standing hit/miss-count requirement): DiagMemoHits counts - // positive replays (a stored non-empty subtree grafted onto a new arrival); DiagNogoodHits counts - // nogood hits (a stored EMPTY subtree short-circuited without any replay work). The equivalence - // tests that cover the replay path assert DiagMemoHits is nonzero so they can never go vacuous -- - // a memo that silently stopped firing would otherwise look exactly like a passing test. + // Read by the equivalence tests to prove the memo actually fired: one that silently stopped + // firing would otherwise look exactly like a passing test. internal static long DiagMemoHits; internal static long DiagNogoodHits; @@ -45,11 +36,8 @@ public override IEnumerable Apply(Word input) return output; } - // Returns every result produced strictly within the subtree rooted at `input` (i.e. by applying - // one or more rules starting from `input`, at any depth) -- NOT including `input` itself. This is - // both the return value callers use and the value memoized against `input`'s AnalysisStateKey - // once the subtree finishes, so a later differently-ordered arrival at the same state can replay - // it via Word.ReplayOnto instead of re-searching. + // Returns the results produced strictly within `input`'s subtree, at any depth, excluding `input` + // itself -- both what callers consume and what gets memoized against `input`'s key. private List ApplyRules(Word input, HashSet output) { AnalysisScope scope = input.AnalysisScope; @@ -59,31 +47,24 @@ private List ApplyRules(Word input, HashSet output) var key = new AnalysisStateKey(input); - if (scope.Memo.TryGetValue(key, out MemoEntry entry)) + if (scope.TryReplay(scope.Memo, key, input, out List replayed)) { - if (entry.Results.Count == 0) + if (replayed.Count == 0) { - DiagNogoodHits++; - return new List(); + Interlocked.Increment(ref DiagNogoodHits); + return replayed; } - var replayed = new List(entry.Results.Count); - foreach (Word storedResult in entry.Results) + foreach (Word replay in replayed) { - Word replay = storedResult.ReplayOnto( - input, - entry.MruleTrailPrefixLength, - entry.NonHeadPrefixLength - ); output.Add(replay); CheckMaxAlternatives(output.Count); - replayed.Add(replay); } - DiagMemoHits++; + Interlocked.Increment(ref DiagMemoHits); return replayed; } // In-flight re-entry guard, see AnalysisScope.InProgress. - if (!scope.InProgress.TryAdd(key, 0)) + if (!scope.InProgress.Add(key)) return ApplyRulesRaw(input, output); List results; @@ -93,16 +74,19 @@ private List ApplyRules(Word input, HashSet output) } finally { - scope.InProgress.TryRemove(key, out _); + scope.InProgress.Remove(key); } - // Past the cap, keep searching correctly, just stop growing the table (AnalysisScope.HasMemoCapacity). - if (scope.HasMemoCapacity) - scope.Memo.TryAdd(key, new MemoEntry(results, input.MorphologicalRuleTrailLength, input.NonHeadCount)); - + scope.Store(scope.Memo, key, input, results); return results; } + // Mirrors the base's multiApp expansion, including its recurse-before-add ordering: the HashSet + // keeps whichever of two comparer-equal results lands first, and Word.ValueEquals ignores + // SyntacticFeatureStruct, so which one survives is observable downstream. Delegating to the base + // is not possible -- it collects into one globally-deduped set, so a subtree result another branch + // already contributed is missing from it, yet must still be recorded here or a later replay of + // this key would return too few results. private List ApplyRulesRaw(Word input, HashSet output) { var local = new List(); @@ -110,12 +94,12 @@ private List ApplyRulesRaw(Word input, HashSet output) { foreach (Word result in ApplyRule(Rules[i], i, input)) { - local.Add(result); - output.Add(result); - CheckMaxAlternatives(output.Count); // avoid infinite loop -- same guard CombinationRuleCascade uses if (!Comparer.Equals(input, result)) local.AddRange(ApplyRules(result, output)); + local.Add(result); + output.Add(result); + CheckMaxAlternatives(output.Count); } } return local; diff --git a/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs b/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs index 9aed10d34..4dd180ca7 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/Morpher.cs @@ -1,6 +1,9 @@ using System; +using System.Collections.Concurrent; using System.Collections.Generic; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SIL.Extensions; using SIL.Machine.Annotations; using SIL.Machine.FeatureModel; @@ -11,12 +14,6 @@ using System.IO; #endif -#if !SINGLE_THREADED -using System.Collections.Concurrent; -using System.Threading.Tasks; -using System.Threading; -#endif - namespace SIL.Machine.Morphology.HermitCrab { public class Morpher : IMorphologicalAnalyzer, IMorphologicalGenerator @@ -88,11 +85,15 @@ public ITraceManager TraceManager public bool MergeEquivalentAnalyses { get; set; } /// - /// Caps parallelism used within a single parse's Unordered-order analysis-rule cascade. A value of - /// 1 selects the sequential cascade, which is also the only one eligible for the analysis-cascade - /// memo (see ); any other value (default 0) keeps the - /// existing parallel cascade. Set via the constructor: it influences how the analysis rules are - /// compiled. + /// Caps the concurrency used within a single parse. A value of 1 runs the parse fully + /// sequentially -- analysis cascade, affix-template unapplication and synthesis alike -- and is the + /// only configuration eligible for the analysis-cascade memo (see + /// ). The default of 0, like any value below 1, leaves + /// concurrency unbounded. Constructor-only, because it determines how the analysis rules compile. + /// + /// This must remain a pure performance knob: nothing that changes which analyses a parse returns + /// may be gated on it, or the memoized and unmemoized configurations stop being comparable. + /// /// public int MaxDegreeOfParallelism { get; } @@ -127,9 +128,8 @@ public IEnumerable ParseWord(string word, out object trace, bool guessRoot Shape shape = _lang.SurfaceStratum.CharacterDefinitionTable.Segment(word); var input = new Word(_lang.SurfaceStratum, shape); - // Only the sequential cascade reads AnalysisScope (see MemoizedCombinationRuleCascade); - // skip the allocation on the parallel path, and while tracing (tracing must stay - // byte-identical to the unmemoized engine). + // Installing a scope is what enables the memo. Never while tracing: traces must stay + // byte-identical to the unmemoized engine. if (!_traceManager.IsTracing && MaxDegreeOfParallelism == 1) input.AnalysisScope = new AnalysisScope(); input.Freeze(); @@ -296,16 +296,26 @@ Stack> permutation in PermuteRules( } } -#if SINGLE_THREADED - private IEnumerable Synthesize(string word, IEnumerable analyses) + private IEnumerable Synthesize(string word, ConcurrentQueue analyses) + { + if (MaxDegreeOfParallelism == 1) + return SynthesizeSequential(word, analyses); + return SynthesizeParallel(word, analyses); + } + + private IEnumerable SynthesizeSequential(string word, IEnumerable analyses) { var matches = new HashSet(FreezableEqualityComparer.Default); + int alternativeCount = 0; foreach (Word analysisWord in analyses) { foreach (Word synthesisWord in LexicalLookup(analysisWord)) { foreach (Word alternative in synthesisWord.ExpandAlternatives()) { + alternativeCount++; + if (MaxAlternatives > 0 && alternativeCount > MaxAlternatives) + throw new MaxAlternativesExceededException("MaxAlternatives exceeded"); foreach (Word validWord in _synthesisRule.Apply(alternative).Where(IsWordValid)) { if (IsMatch(word, validWord)) @@ -316,8 +326,8 @@ private IEnumerable Synthesize(string word, IEnumerable analyses) } return matches; } -#else - private IEnumerable Synthesize(string word, ConcurrentQueue analyses) + + private IEnumerable SynthesizeParallel(string word, ConcurrentQueue analyses) { var matches = new ConcurrentBag(); int alternativeCount = 0; @@ -363,7 +373,6 @@ private IEnumerable Synthesize(string word, ConcurrentQueue analyses throw exception; return matches.Distinct(FreezableEqualityComparer.Default); } -#endif internal IEnumerable SearchRootAllomorphs(Stratum stratum, Shape shape) { @@ -386,6 +395,9 @@ LexEntry entry in SearchRootAllomorphs(input.Stratum, input.Shape) foreach (RootAllomorph allomorph in entry.Allomorphs) { Word newWord = input.Clone(); + // Synthesis never reads the memo, and keeping the reference would pin both tables for + // as long as the caller holds the returned words. + newWord.AnalysisScope = null; newWord.RootAllomorph = allomorph; if (_traceManager.IsTracing) _traceManager.SynthesizeWord(_lang, newWord); @@ -458,6 +470,8 @@ private IEnumerable LexicalGuess(Word input) } // Create a new word that uses the root allomorph. Word newWord = input.Clone(); + // Synthesis never reads the memo; see LexicalLookup. + newWord.AnalysisScope = null; newWord.RootAllomorph = root; if (_traceManager.IsTracing) _traceManager.SynthesizeWord(_lang, newWord); diff --git a/src/SIL.Machine.Morphology.HermitCrab/Word.cs b/src/SIL.Machine.Morphology.HermitCrab/Word.cs index c98946fba..51c3c53cc 100644 --- a/src/SIL.Machine.Morphology.HermitCrab/Word.cs +++ b/src/SIL.Machine.Morphology.HermitCrab/Word.cs @@ -70,6 +70,11 @@ public Word(Stratum stratum, Shape shape) } protected Word(Word word) + : this(word, cloneNonHeadApps: true) { } + + // ReplayOnto passes false: it rebuilds the non-head list wholesale, so cloning it here would be + // discarded work. + private Word(Word word, bool cloneNonHeadApps) { _allomorphs = new Dictionary(word._allomorphs); Stratum = word.Stratum; @@ -84,7 +89,7 @@ protected Word(Word word) _mruleAppIndex = word._mruleAppIndex; _mrulesUnapplied = new Dictionary(word._mrulesUnapplied); _mrulesApplied = new Dictionary(word._mrulesApplied); - _nonHeadApps = new List(word._nonHeadApps.CloneItems()); + _nonHeadApps = cloneNonHeadApps ? new List(word._nonHeadApps.CloneItems()) : new List(); _nonHeadAppIndex = word._nonHeadAppIndex; _obligatorySyntacticFeatures = new IDBearerSet(word._obligatorySyntacticFeatures); _isLastAppliedRuleFinal = word._isLastAppliedRuleFinal; @@ -214,12 +219,12 @@ public IEnumerable MorphemesInApplicationOrder public object CurrentTrace { get; set; } /// - /// Carrier for the analysis-cascade memo (memoization.md). Reference-shared like - /// , deliberately excluded from FreezeImpl and - /// ValueEquals so existing dedup semantics are unchanged. Null for words never routed - /// through (e.g. words built directly by - /// rule-level unit tests) or while tracing, in which case the cascade that reads this must fall - /// back to unmemoized behavior rather than throw. + /// Carrier for the analysis-cascade memo. Reference-shared like + /// and excluded from FreezeImpl/ValueEquals for the same + /// reason. Null while tracing, and for words not routed through + /// at all, so readers must fall back to + /// unmemoized behavior rather than throw. Cleared again on entry to synthesis so returned words do + /// not pin the per-parse tables. /// internal AnalysisScope AnalysisScope { get; set; } @@ -324,6 +329,12 @@ internal void RemoveMorph(Annotation morphAnn) /// indicates that an unknown compounding rule was unapplied. This is used when /// generating a compound word, because the compounding rule is usually not known just /// the non-head allomorph. + /// + /// The trail push and the count increment below must stay in lockstep: + /// splits a stored result on the assumption that equal unapplication multisets imply equal trail + /// lengths. Realizational rules incrementing the count without extending the trail is safe because + /// they do so on both sides of any comparison; any other divergence misaligns the graft. + /// /// internal void MorphologicalRuleUnapplied(IMorphologicalRule mrule) { @@ -350,8 +361,7 @@ internal int GetUnapplicationCount(IMorphologicalRule mrule) } /// - /// The full per-rule unapplication-count multiset backing , for - /// (order-independent analysis-cascade memoization, memoization.md). + /// The full multiset backing , for . /// internal IReadOnlyDictionary UnappliedRuleCounts => _mrulesUnapplied; @@ -409,11 +419,12 @@ internal int NonHeadCount get { return _nonHeadApps.Count; } } + internal IReadOnlyList NonHeads => _nonHeadApps; + /// - /// Length of the morphological-rule trail so far -- _mruleApps.Count. Recorded alongside - /// at the point an 's subtree is - /// memoized (memoization.md), so a later differently-ordered arrival at the same key knows where - /// its own trail ends and the memoized subtree's suffix begins -- see . + /// Length of the morphological-rule trail so far. Recorded with when a + /// subtree is memoized, to mark where a replayed result's kept suffix begins; see + /// . /// internal int MorphologicalRuleTrailLength => _mruleApps.Count; @@ -476,28 +487,35 @@ internal IList ExpandAlternatives() } /// - /// Re-parents a Word computed while exploring the subtree below some analysis-cascade node N onto - /// -- a different Word that reached the same - /// as N via a different morphological-rule unapplication order - /// (memoization.md's positive memo; see ). Everything - /// computed strictly WITHIN the subtree -- deeper shape/feature edits, and any rules or non-heads - /// unapplied below N -- is a deterministic function of N's content alone (Shape, both - /// FeatureStructs, the rule-unapplication multiset, and non-head count all match between N and - /// by definition of an equal key), so it is kept as-is from `this`. - /// Only the two ORDERED structures the key deliberately summarizes as counts/multisets -- the - /// morphological-rule trail and the non-head list -- have their PREFIX (whatever was accumulated - /// before reaching N) replaced with 's own actual prefix, since arrival - /// order can only ever affect that part. + /// Re-parents this Word -- computed while exploring the subtree below some cascade node N -- onto + /// , which reached N's via a different + /// unapplication order. + /// + /// Sound because an equal key means N and agree on Shape, both + /// FeatureStructs, the unapplication multiset and the non-head count, so everything computed + /// inside the subtree is a function of state they share and carries over untouched. Only the two + /// ordered structures the key reduces to counts -- the rule trail and the non-head list -- can + /// differ, and only in the prefix accumulated before reaching N, which is what gets replaced. + /// /// - /// The word that hit the memo -- its trail/non-heads become the new prefix. + /// The word that hit the memo; its trail and non-heads become the prefix. /// - /// _mruleApps.Count of N at the moment its subtree was memoized -- everything in `this`'s - /// trail from this index on is the subtree-local suffix to keep. + /// N's _mruleApps.Count when its subtree was memoized: this word's trail from that index on + /// is the subtree-local suffix to keep. /// /// Same, for _nonHeadApps. - internal Word ReplayOnto(Word queryNode, int mruleTrailPrefixLength, int nonHeadPrefixLength) + /// + /// Pre-cloned non-heads from , so one memo hit clones them once rather + /// than per stored result; see AnalysisScope.TryReplay. Null clones them here instead. + /// + internal Word ReplayOnto( + Word queryNode, + int mruleTrailPrefixLength, + int nonHeadPrefixLength, + IReadOnlyList queryNonHeadPrefix = null + ) { - Word clone = Clone(); + var clone = new Word(this, cloneNonHeadApps: false); List mruleSuffix = clone._mruleApps.GetRange( mruleTrailPrefixLength, @@ -508,19 +526,28 @@ internal Word ReplayOnto(Word queryNode, int mruleTrailPrefixLength, int nonHead clone._mruleApps.AddRange(mruleSuffix); clone._mruleAppIndex = clone._mruleApps.Count - 1; - List nonHeadSuffix = clone._nonHeadApps.GetRange( - nonHeadPrefixLength, - clone._nonHeadApps.Count - nonHeadPrefixLength + // The clone's non-head list starts empty, so it is built as query prefix + this word's + // subtree-local suffix without ever cloning the prefix this word arrived with, which the graft + // discards anyway. + if (queryNonHeadPrefix != null) + clone._nonHeadApps.AddRange(queryNonHeadPrefix); + else + clone._nonHeadApps.AddRange(queryNode._nonHeadApps.CloneItems()); + clone._nonHeadApps.AddRange( + _nonHeadApps.GetRange(nonHeadPrefixLength, _nonHeadApps.Count - nonHeadPrefixLength).CloneItems() ); - clone._nonHeadApps.Clear(); - clone._nonHeadApps.AddRange(queryNode._nonHeadApps.CloneItems()); - clone._nonHeadApps.AddRange(nonHeadSuffix); clone._nonHeadAppIndex = clone._nonHeadApps.Count - 1; clone.Freeze(); return clone; } + // Hoisted out of the per-result loop by AnalysisScope.TryReplay; see ReplayOnto. + internal List CloneNonHeadsForReplay() + { + return new List(_nonHeadApps.CloneItems()); + } + public Allomorph GetAllomorph(Annotation morph) { var alloID = (string)morph.FeatureStruct.GetValue(HCFeatureSystem.Allomorph); diff --git a/src/SIL.Machine/Rules/ParallelCombinationRuleCascade.cs b/src/SIL.Machine/Rules/ParallelCombinationRuleCascade.cs index bed3b25a2..b126b8725 100644 --- a/src/SIL.Machine/Rules/ParallelCombinationRuleCascade.cs +++ b/src/SIL.Machine/Rules/ParallelCombinationRuleCascade.cs @@ -30,8 +30,18 @@ IEqualityComparer comparer ) : base(rules, multiApp, comparer) { } + /// + /// Maximum number of concurrent tasks used by . Values less than 1 (the + /// default is -1) do not limit the degree of parallelism. + /// + public int MaxDegreeOfParallelism { get; set; } = -1; + public override IEnumerable Apply(TData input) { + var parallelOptions = new ParallelOptions + { + MaxDegreeOfParallelism = MaxDegreeOfParallelism >= 1 ? MaxDegreeOfParallelism : -1, + }; var output = new ConcurrentStack(); var from = new ConcurrentStack>>(); from.Push(Tuple.Create(input, !MultipleApplication ? new HashSet() : null)); @@ -43,6 +53,7 @@ public override IEnumerable Apply(TData input) to.Clear(); Parallel.ForEach( from, + parallelOptions, (work, state) => { try diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs index 29469988f..e398bb9e6 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStateKeyTests.cs @@ -4,11 +4,8 @@ namespace SIL.Machine.Morphology.HermitCrab; -// Unit battery for the analysis-cascade memo's primitives (memoization.md Stage 2): AnalysisStateKey's -// order-invariance over the rule-unapplication multiset, its sensitivity to every other field the key -// captures, and Word.ReplayOnto's prefix/suffix graft. Nothing in production code consumes these yet -// (that is Stage 3) -- these tests pin the primitives' own contract in isolation, independent of any -// particular cascade wiring. +// The memo's primitives in isolation, independent of any cascade wiring: AnalysisStateKey's +// order-invariance and field sensitivity, its frozen-word requirement, and ReplayOnto's graft. [TestFixture] public class AnalysisStateKeyTests : HermitCrabTestBase { @@ -18,9 +15,9 @@ public void Equals_IsInvariantOverUnapplicationOrder_ForEqualMultisets() var ruleA = new AffixProcessRule { Name = "ruleA" }; var ruleB = new AffixProcessRule { Name = "ruleB" }; - // Same multiset {ruleA: 2, ruleB: 1}, built up in two genuinely different orders -- - // wordY touches ruleB first, unlike wordX, so this also exercises different backing - // dictionary insertion order, not just a different position for a repeated rule. + // Same multiset {ruleA: 2, ruleB: 1} reached in two different orders. wordY touches ruleB first, + // so the backing dictionaries also differ in insertion order, not just in a repeated rule's + // position. Word wordX = NewTestWord(); wordX.MorphologicalRuleUnapplied(ruleA); wordX.MorphologicalRuleUnapplied(ruleB); @@ -86,6 +83,45 @@ public void Equals_False_WhenSyntacticFeatureStructDiffers() Assert.That(new AnalysisStateKey(wordX).Equals(new AnalysisStateKey(wordY)), Is.False); } + [Test] + public void Constructor_Throws_WhenWordIsNotFrozen() + { + Word unfrozen = NewTestWord(); + + Assert.That(() => new AnalysisStateKey(unfrozen), Throws.ArgumentException); + } + + [Test] + public void ReplayOnto_SharesHoistedQueryPrefix_AcrossOneHitsReplays() + { + Word queryNonHead = NewTestWord("32"); + queryNonHead.Freeze(); + Word query = NewTestWord("32"); + query.NonHeadUnapplied(queryNonHead); + query.Freeze(); + + Word storedNonHead = NewTestWord("32"); + storedNonHead.Freeze(); + Word subtreeNonHead = NewTestWord("33"); + subtreeNonHead.Freeze(); + Word memoized = NewTestWord("32"); + memoized.NonHeadUnapplied(storedNonHead); + memoized.NonHeadUnapplied(subtreeNonHead); + memoized.Freeze(); + + List hoisted = query.CloneNonHeadsForReplay(); + Word first = memoized.ReplayOnto(query, 0, 1, hoisted); + Word second = memoized.ReplayOnto(query, 0, 1, hoisted); + + // Query's 1 non-head prefix plus the stored subtree's 1 non-head suffix. + Assert.That(first.NonHeadCount, Is.EqualTo(2)); + Assert.That(first.CurrentNonHead.RootAllomorph, Is.SameAs(subtreeNonHead.RootAllomorph)); + // Both replays share the one hoisted clone, and it is a clone rather than the query's own instance. + Assert.That(first.NonHeads[0], Is.SameAs(second.NonHeads[0])); + Assert.That(first.NonHeads[0], Is.SameAs(hoisted[0])); + Assert.That(first.NonHeads[0], Is.Not.SameAs(queryNonHead)); + } + [Test] public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForMruleTrail() { @@ -93,15 +129,14 @@ public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForMruleTrail() var ruleB = new AffixProcessRule { Name = "ruleB" }; var ruleC = new AffixProcessRule { Name = "ruleC" }; - // The memoized node's own trail was [ruleA, ruleB] at the moment its subtree was recorded, with - // ruleA as the length-1 prefix (whatever led to this state) and ruleB as the subtree-local suffix - // that must survive the graft. + // Trail [ruleA, ruleB] at the moment of the write: ruleA is the length-1 prefix, ruleB the + // subtree-local suffix that must survive the graft. Word memoized = NewTestWord(); memoized.MorphologicalRuleUnapplied(ruleA); memoized.MorphologicalRuleUnapplied(ruleB); memoized.Freeze(); - // A different arrival at the same AnalysisStateKey, via a different prefix: [ruleC]. + // The same key reached with a different prefix, [ruleC]. Word query = NewTestWord(); query.MorphologicalRuleUnapplied(ruleC); query.Freeze(); @@ -114,11 +149,8 @@ public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForMruleTrail() [Test] public void ReplayOnto_GraftsQueryPrefixOntoStoredSuffix_ForNonHeads() { - // The memoized node had already unapplied one non-head (its own prefix, at the memo point) before - // its subtree unapplied a second one -- that second one is the subtree-local suffix to keep. The - // two non-heads are built from distinct lexical entries (32 vs 33) so a wrong-prefix graft (e.g. - // one that kept storedNonHead instead of subtreeNonHead, or reversed the GetRange window) is - // actually distinguishable via RootAllomorph identity, not just NonHeadCount. + // Distinct lexical entries (32 vs 33) so that a graft keeping the wrong non-head, or reversing the + // GetRange window, is distinguishable by RootAllomorph identity rather than only by count. Word storedNonHead = NewTestWord("32"); storedNonHead.Freeze(); Word subtreeNonHead = NewTestWord("33"); diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStratumRuleTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStratumRuleTests.cs new file mode 100644 index 000000000..b9913b2a9 --- /dev/null +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/AnalysisStratumRuleTests.cs @@ -0,0 +1,94 @@ +using NUnit.Framework; +using SIL.Machine.Annotations; +using SIL.Machine.FeatureModel; +using SIL.Machine.Matching; +using SIL.Machine.Morphology.HermitCrab.MorphologicalRules; + +namespace SIL.Machine.Morphology.HermitCrab; + +// Drives AnalysisStratumRule against a scope the test owns, which is the only way to observe whether the +// template battery memoized anything: through Morpher the scope is created and discarded inside +// ParseWord, and a Linear stratum invokes the battery at most once per distinct state, so it never +// replays there and the hit counters cannot distinguish a memoized run from an excluded one. +[TestFixture] +public class AnalysisStratumRuleTests : HermitCrabTestBase +{ + [Test] + public void Apply_MemoizesTemplateBattery_OnUnorderedStratum() + { + AddVerbTemplate(); + SetRuleOrder(MorphologicalRuleOrder.Unordered); + + AnalysisScope scope = ApplyStratumRule("sagd"); + + Assert.That( + scope.TemplateMemo, + Is.Not.Empty, + "an Unordered stratum must memoize the template battery -- otherwise the negative case below " + + "proves nothing" + ); + } + + [Test] + public void Apply_DoesNotMemoizeTemplateBattery_OnLinearStratum() + { + AddVerbTemplate(); + SetRuleOrder(MorphologicalRuleOrder.Linear); + + AnalysisScope scope = ApplyStratumRule("sagd"); + + Assert.That( + scope.TemplateMemo, + Is.Empty, + "a Linear stratum must not memoize the template battery: AnalysisStateKey's key-completeness " + + "audit covers only the Unordered cascade" + ); + Assert.That( + scope.Memo, + Is.Empty, + "nor may the mrule table be written on a Linear stratum, which runs PermutationRuleCascade" + ); + } + + // Runs one stratum rule over `word` with a scope attached, and hands the scope back for inspection. + private AnalysisScope ApplyStratumRule(string word) + { + var morpher = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + var stratumRule = new AnalysisStratumRule(morpher, Morphophonemic); + + var input = new Word(Morphophonemic, Morphophonemic.CharacterDefinitionTable.Segment(word)); + var scope = new AnalysisScope(); + input.AnalysisScope = scope; + input.Freeze(); + + // Apply builds its result set eagerly, so this forces the battery for every state it reaches. + _ = stratumRule.Apply(input).ToList(); + return scope; + } + + private void AddVerbTemplate() + { + var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; + var dSuffix = new AffixProcessRule + { + Id = "TPAST", + Name = "template_d_suffix", + Gloss = "PAST", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + dSuffix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new CopyFromInput("1"), new InsertSegments(Table3, "+d") }, + } + ); + var verbTemplate = new AffixTemplate + { + Name = "verb_template", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + verbTemplate.Slots.Add(new AffixTemplateSlot(dSuffix) { Optional = true }); + Morphophonemic.AffixTemplates.Add(verbTemplate); + } +} diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs index be195a77f..b72e589b9 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoCorpusVerification.cs @@ -4,29 +4,22 @@ namespace SIL.Machine.Morphology.HermitCrab; /// -/// Stage 5 of memoization.md: the real-data verification the toy-grammar unit tests (stages 2-4) -/// cannot provide. The mrule/template unit tests use 2-3-rule synthetic grammars specifically because -/// they let a specific redundant-order scenario be forced deterministically -- but they cannot exercise -/// whether 's key-completeness audit actually holds against the FULL -/// analysis-side rule set a real grammar invokes (AnalysisAffixProcessRule, -/// AnalysisCompoundingRule, AnalysisRealizationalAffixProcessRule, and every phonological -/// rule feeding into the states those rules read). If the key is missing a field some real rule -/// consults, it manifests as a divergence on some real word, nowhere else -- this harness is that check. -/// -/// [Explicit] and env-var driven, modeled on FstSenaBenchmark.cs's convention (this repo never commits -/// real morphological grammars or word lists -- see the standing grammar-privacy constraint -- so this -/// test ships with zero embedded grammar/word content and writes no output files, only TestContext -/// lines, to avoid leaking derived corpus data such as signature dumps into a committed path): +/// Memo-on/memo-off equality against a real grammar, which is the only way to test +/// 's key-completeness audit against the full analysis-side rule set. The +/// synthetic unit-test grammars can force a specific redundant order deterministically but cannot reach +/// that breadth; a key missing a field some real rule reads shows up here and nowhere else. +/// +/// [Explicit] and env-var driven because this repo never commits real grammars or word lists: the test +/// embeds no grammar content and writes only TestContext lines, so no derived corpus data (signature +/// dumps included) can land in a committed path. +/// +/// /// $env:HC_MEMO_GRAMMAR = "...\sena-hc.xml" /// $env:HC_MEMO_WORDS = "...\sena-words.txt" /// $env:HC_MEMO_MAX_WORDS = "60" # optional, default 60 /// $env:HC_MEMO_TIMEOUT_MS = "5000" # optional, default 5000 (per-word watchdog) /// dotnet test --filter "FullyQualifiedName~MemoCorpusVerification" -/// -/// Deliberately NOT the archive's 900-line parallel-benchmark harness (16-way scheduling, GC heap-limit -/// watchdog): per design, the only new runtime configuration this port introduces is single-threaded -/// (maxDegreeOfParallelism: 1), so there is no new concurrency to stress-test here -- just a -/// per-word timeout so one pathological word can't hang the whole run. +/// /// [TestFixture] [Explicit("Manual corpus verification against a local, uncommitted real grammar; not part of CI.")] @@ -99,11 +92,9 @@ public void MemoOnMatchesMemoOff_AnalysisSetIdentical_OnRealCorpus() TestContext.Out.WriteLine($"words with no parse on both sides: {noParseBoth}"); TestContext.Out.WriteLine($"aggregate wall: {totalMs:F1} ms, p50: {p50:F1} ms, p95: {p95:F1} ms"); - // Count-based vs wall-clock aggregates, reported SEPARATELY on purpose: a corpus is typically - // bimodal (many cheap words the memo makes slightly slower by losing a thread; a few pathological - // words the memo makes drastically faster) -- collapsing that into one ratio hides which regime a - // reader is in. "Most words are a bit slower" and "the corpus finishes much faster in total" are - // both true simultaneously and neither contradicts the other. + // Count-based and wall-clock aggregates stay separate because a corpus is bimodal: many cheap + // words go slightly slower for want of a thread, while a few pathological ones go far faster. One + // combined ratio would hide which regime a reader is in, and both statements are true at once. int fasterCount = perWordTimes.Count(x => x.OnMs < x.OffMs); int slowerCount = perWordTimes.Count(x => x.OnMs > x.OffMs); int tiedCount = perWordTimes.Count - fasterCount - slowerCount; @@ -119,22 +110,17 @@ public void MemoOnMatchesMemoOff_AnalysisSetIdentical_OnRealCorpus() ); if (timedOut.Count > 0) { - // NOT a guaranteed lower bound in either direction: the try block above wraps BOTH the - // memo-on and memo-off calls, so a timeout could come from either side -- this harness - // never records which one actually timed out. If it was memo-on, the word's true - // memo-off time is genuinely unmeasured (could be faster OR slower than the ratio above - // implies); only a longer timeout actually resolves it. Report the fact, don't imply a - // direction the data doesn't support. + // The ratio above is not a bound in either direction: one try block wraps both calls, so which + // side timed out is unrecorded, and if it was memo-on then that word's memo-off time was never + // measured at all. TestContext.Out.WriteLine( $"(the {timedOut.Count} timed-out word(s) above are excluded from both aggregates; " + "re-run with a higher HC_MEMO_TIMEOUT_MS to actually measure them)" ); } - // Per-heavy-word attribution (memoization.md's own methodological rule: an aggregate can be - // dominated by cheap words while hiding what pathological words actually do -- report both). - // memo-on is sequential+memo; memo-off is the untouched parallel default, so this is the - // user-visible claim (sequential-memo vs today's shipped behavior), not an isolated measurement - // of the memo mechanism's own contribution in isolation from single- vs multi-threading. + // Per-word attribution as well, since an aggregate dominated by cheap words hides what the + // pathological ones do. Note memo-on is sequential while memo-off is the parallel default, so + // these times measure the user-visible comparison, not the memo's contribution in isolation. TestContext.Out.WriteLine("heaviest words (by memo-off time), memo-on vs memo-off:"); foreach ((string w, double onMs2, double offMs2) in perWordTimes.OrderByDescending(x => x.OffMs).Take(10)) TestContext.Out.WriteLine($" {w}: memo-on {onMs2:F1} ms, memo-off {offMs2:F1} ms"); @@ -148,11 +134,9 @@ public void MemoOnMatchesMemoOff_AnalysisSetIdentical_OnRealCorpus() ); if (timedOut.Count > 0) { - // Named, not just counted: a timed-out word is EXCLUDED from the equality gate above, so - // "0 divergences" says nothing about it. These are exactly the candidates for a follow-up - // run with a longer HC_MEMO_TIMEOUT_MS (see memoization.md §5's addendum on why this - // mattered -- the heavy words are precisely the ones the memo, and the key-completeness - // audit, most need to be checked against). + // Named rather than counted: these words are excluded from the equality gate, so "0 + // divergences" says nothing about them, and heavy words are exactly what the memo and the + // key-completeness audit most need checking against. TestContext.Out.WriteLine( $"timed-out words (excluded from the equality gate above -- re-run with a higher " + $"HC_MEMO_TIMEOUT_MS to actually check these): {string.Join(", ", timedOut)}" @@ -185,20 +169,17 @@ private static List Signatures(Morpher morpher, string word) } catch (InvalidShapeException) { - // Matches Morpher.AnalyzeWord's own handling: a word list drawn from real text can contain - // strings the grammar's character table doesn't cover (e.g. punctuation) -- not a memo - // concern, both sides would throw identically. + // As Morpher.AnalyzeWord does: a real word list can contain strings the character table does + // not cover, which both sides reject identically and which tells us nothing about the memo. return new List(); } } - // Does NOT cancel `action` on timeout -- Morpher.ParseWord has no cooperative-cancellation - // hook, so a timed-out word's Task keeps running in the background. This can inflate the - // hit/nogood counters and per-word timings reported below with work from an abandoned prior - // word, and piled-up orphaned tasks from several timed-out words in a row can starve the - // thread pool. Acceptable for this [Explicit], never-in-CI diagnostic harness (the equality - // gate itself is unaffected -- a timed-out word is excluded from it either way, see the - // caller), but do not read the reported hit counts/timings as precise when timeouts occurred. + // Cannot cancel `action`: ParseWord has no cooperative-cancellation hook, so a timed-out word keeps + // running in the background, where it can inflate later words' counters and timings, and enough + // orphaned tasks in a row can starve the thread pool. Tolerable only because this harness never runs + // in CI, and the equality gate excludes timed-out words anyway -- but treat any run that reported + // timeouts as having approximate counts. private static T RunWithTimeout(Func action, int timeoutMs) { Task task = Task.Run(action); diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs index 89cd2438b..32a1629e0 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MemoizedCombinationRuleCascadeTests.cs @@ -7,12 +7,9 @@ namespace SIL.Machine.Morphology.HermitCrab; -// Stage 3 of memoization.md: MemoizedCombinationRuleCascade exercised directly, bypassing -// Morpher/AnalysisStratumRule entirely, so the test can force the exact commuting-order redundancy the -// memo targets without depending on whether a particular morphology grammar's search happens to revisit -// a PRODUCTIVE (not just nogood) state -- MorpherTests' end-to-end compounding grammar, checked via the -// real Morpher pipeline, only ever hits the nogood table on this small a grammar (see its own hit-count -// diagnostic output), so this is the test that pins the POSITIVE replay path non-vacuously. +// The cascade exercised directly, bypassing Morpher, so a commuting-order re-arrival at a PRODUCTIVE +// state can be forced. That matters because the end-to-end grammars in MorpherTests are small enough +// that they only ever reach the nogood table, leaving the positive replay path untested. [TestFixture] public class MemoizedCombinationRuleCascadeTests : HermitCrabTestBase { @@ -23,11 +20,9 @@ public void Apply_ReplaysPositiveHit_WhenTwoOrdersReachTheSameKey() var ruleB = new AffixProcessRule { Name = "ruleB" }; var ruleC = new AffixProcessRule { Name = "ruleC" }; - // Each fake rule unapplies its own IMorphologicalRule at most once, so from the initial word, - // trying ruleA-then-ruleB-then-ruleC and ruleB-then-ruleA-then-ruleC reach the SAME - // AnalysisStateKey after the first two steps (multiset {ruleA:1, ruleB:1}) via different orders - // -- exactly the redundancy AnalysisStateKey collapses. Only from that shared state can ruleC - // still apply, so the shared node's own subtree is POSITIVE (one result), not a nogood. + // Each rule unapplies at most once, so A-then-B and B-then-A reach the same key (multiset + // {ruleA:1, ruleB:1}) by different routes. ruleC can still apply from that shared state, making + // its subtree positive rather than a nogood. var cascade = new MemoizedCombinationRuleCascade( new IRule[] { @@ -45,9 +40,6 @@ public void Apply_ReplaysPositiveHit_WhenTwoOrdersReachTheSameKey() long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; List results = new List(cascade.Apply(initial)); - // Every leaf where all 3 rules have been unapplied, in whichever of the two orders explored the - // shared {ruleA,ruleB} state first vs via replay, must appear -- and the replay path must have - // actually fired (memoization.md's non-vacuousness requirement). Assert.That( results, Has.Some.Matches(w => @@ -66,11 +58,9 @@ public void Apply_ReplaysPositiveHit_WhenTwoOrdersReachTheSameKey() [Test] public void Apply_PositiveReplayMatchesUnmemoizedResultSet_IncludingTrailOrder() { - // The previous test proves a replay FIRES; this one proves it produces the RIGHT result. A count - // assertion (3 rules applied) would pass even if ReplayOnto grafted the wrong prefix, since counts - // are order-invariant -- comparing MorphemesInApplicationOrder (which walks the trail ReplayOnto - // rewrites, see WordAnalysisSignature's own doc comment in MorpherTests.cs) catches a wrong-prefix - // graft that silently collapses [ruleB,ruleA,ruleC] into a duplicate of [ruleA,ruleB,ruleC]. + // Compares MorphemesInApplicationOrder rather than rule counts: counts are order-invariant, so + // they would pass even if the graft collapsed [ruleB,ruleA,ruleC] into a duplicate of + // [ruleA,ruleB,ruleC], whereas the trail is exactly what ReplayOnto rewrites. var ruleA = new AffixProcessRule { Id = "RULE_A", Name = "ruleA" }; var ruleB = new AffixProcessRule { Id = "RULE_B", Name = "ruleB" }; var ruleC = new AffixProcessRule { Id = "RULE_C", Name = "ruleC" }; @@ -85,9 +75,8 @@ public void Apply_PositiveReplayMatchesUnmemoizedResultSet_IncludingTrailOrder() memoized.AnalysisScope = new AnalysisScope(); memoized.Freeze(); + // No AnalysisScope: takes the unmemoized fallback, the same path a tracing parse takes. Word unmemoized = NewTestWord(); - // AnalysisScope left null -- exercises MemoizedCombinationRuleCascade's own unmemoized fallback - // (ApplyRulesRaw), the same path a tracing parse takes. unmemoized.Freeze(); var memoizedCascade = new MemoizedCombinationRuleCascade(rules, FreezableEqualityComparer.Default); @@ -121,14 +110,8 @@ public void Apply_PositiveReplayMatchesUnmemoizedResultSet_IncludingTrailOrder() [Test] public void Apply_FallsBackToUnmemoizedExpansion_WhenKeyIsAlreadyInProgress() { - // Pins the in-flight re-entry guard (memoization.md's design rule 3 / the InProgress table): - // a multiApp cascade can reach the same AnalysisStateKey again while its own first expansion is - // still on the call stack (e.g. via a self-loop elsewhere in a real grammar's rule graph). - // Rather than construct that reentrancy organically -- the key is monotonic in rule-application - // count for straightforward single-use rules, so a real cyclic re-arrival is hard to force here - // -- this simulates the in-flight state directly: pre-populate InProgress with the exact key - // `initial` will compute, then call Apply and confirm it falls through to a correct, unmemoized - // expansion (ApplyRulesRaw) instead of reading a nonexistent Memo entry or hanging. + // The in-flight state is simulated by pre-populating InProgress, because single-use rules make the + // key monotonic in application count, so a genuine cyclic re-arrival cannot be forced here. var ruleA = new AffixProcessRule { Id = "RULE_A", Name = "ruleA" }; var cascade = new MemoizedCombinationRuleCascade( new IRule[] { new SingleUseUnapplyRule(ruleA) }, @@ -141,7 +124,7 @@ public void Apply_FallsBackToUnmemoizedExpansion_WhenKeyIsAlreadyInProgress() initial.Freeze(); var key = new AnalysisStateKey(initial); - scope.InProgress.TryAdd(key, 0); + scope.InProgress.Add(key); long hitsBefore = MemoizedCombinationRuleCascade.DiagMemoHits; List results = new List(cascade.Apply(initial)); @@ -195,9 +178,8 @@ private Word NewTestWord() return new Word(Entries["32"].PrimaryAllomorph, FeatureStruct.New().Value) { Stratum = Morphophonemic }; } - // Minimal stand-in for a compiled analysis morphological rule: unapplies `_rule` against the input - // exactly once (per input), independent of Shape/FeatureStruct pattern matching, so the cascade's - // own commuting-order redundancy can be exercised without compiling a real FST-backed rule. + // Stand-in for a compiled analysis rule: unapplies once per input, with no Shape/FeatureStruct + // matching, so commuting orders can be exercised without a real FST-backed rule. private sealed class SingleUseUnapplyRule(IMorphologicalRule rule) : IRule { private readonly IMorphologicalRule _rule = rule; diff --git a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs index c4e0012cd..0197e422e 100644 --- a/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs +++ b/tests/SIL.Machine.Morphology.HermitCrab.Tests/MorpherTests.cs @@ -536,15 +536,10 @@ IList GetNodes(string pattern) return shape.GetNodes(shape.Range).ToList(); } - [Test] - public void ParseWord_SingleThreaded_MatchesParallel_WithCompounding() + // A compounding rule and a commuting PAST prefix as peers in one Unordered cascade, so an equal + // AnalysisStateKey can be re-arrived at by different unapplication orders. + private void AddCompoundingAndPrefixRules() { - // Stage 1 of memoization.md: pins the new runtime MaxDegreeOfParallelism toggle to be a pure - // no-op on results before any memoization is wired up. Compounding specifically (not just plain - // affixes) because it's the mechanism the eventual analysis-cascade memo cares about: an affix - // rule that commutes with a compounding rule -- both peers in the same Unordered - // MorphologicalRules cascade -- can revisit an equal AnalysisStateKey via different arrival - // orders, so this grammar is the one later commits' equivalence tests build on. var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; var crule = new CompoundingRule { Name = "rule1" }; Allophonic.MorphologicalRules.Add(crule); @@ -577,6 +572,13 @@ public void ParseWord_SingleThreaded_MatchesParallel_WithCompounding() Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, } ); + } + + [Test] + public void ParseWord_SingleThreaded_MatchesParallel_WithCompounding() + { + // MaxDegreeOfParallelism must be a pure no-op on results, independent of the memo it gates. + AddCompoundingAndPrefixRules(); var parallel = new Morpher(TraceManager, Language); var singleThreaded = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); @@ -596,43 +598,9 @@ public void ParseWord_SingleThreaded_MatchesParallel_WithCompounding() [Test] public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithCompounding() { - // Stage 3 of memoization.md: the mrule-cascade memo is now actually wired in for - // maxDegreeOfParallelism: 1. This is the standing acceptance gate (memoization.md §5) -- - // analysis-set (signature) equality between the memoized single-threaded cascade and the - // unmemoized parallel default -- made non-vacuous by asserting the memo's hit counter actually - // moved, so a memo that silently stopped firing couldn't pass this test by accident. - var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; - var crule = new CompoundingRule { Name = "rule1" }; - Allophonic.MorphologicalRules.Add(crule); - crule.Subrules.Add( - new CompoundingSubrule - { - HeadLhs = { Pattern.New("head").Annotation(any).OneOrMore.Value }, - NonHeadLhs = { Pattern.New("nonHead").Annotation(any).OneOrMore.Value }, - Rhs = { new CopyFromInput("head"), new InsertSegments(Table3, "+"), new CopyFromInput("nonHead") }, - } - ); - - var prefix = new AffixProcessRule - { - Id = "PREFIX", - Name = "prefix", - Gloss = "PAST", - RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, - OutSyntacticFeatureStruct = FeatureStruct - .New(Language.SyntacticFeatureSystem) - .Feature(Head) - .EqualTo(head => head.Feature("tense").EqualTo("past")) - .Value, - }; - Allophonic.MorphologicalRules.Insert(0, prefix); - prefix.Allomorphs.Add( - new AffixProcessAllomorph - { - Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, - Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, - } - ); + // The standing acceptance gate: analysis-set equality between the memoized sequential cascade and + // the unmemoized parallel default, kept non-vacuous by the hit-counter assertion at the end. + AddCompoundingAndPrefixRules(); var memoOff = new Morpher(TraceManager, Language); var memoOn = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); @@ -664,17 +632,10 @@ public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithCompounding() [Test] public void ParseWord_MemoOnMatchesMemoOff_ForSelfOpaquingSimultaneousEpenthesis() { - // Diagnostic pulled forward per memoization.md §5(a): PanGloss's rust conformance suite documents - // a confirmed C#-oracle nogood-cache bug on this exact fixture shape (a Simultaneous-mode - // epenthesis rule, which AnalysisRewriteRule compiles with ReapplyType.SelfOpaquing -- a - // repeat-until-fixpoint loop -- against root 19's boundary-bearing "b+ubu"). A direct - // reconstruction attempt against this prototype's own AnalysisScope (parse-optimization-archive) - // did not reproduce a divergence for "buibui" under any tracing/order combination tried, and the - // real minimal fixture (rust conformance/rewrite/simultaneous-epenthesis) is not present in this - // checkout to test against directly. Wiring this in as a standing regression case is the - // practical mitigation: the general memo-on/off equality gate (this test) covers it going - // forward, and the ≥2-self-opaquing-iteration case remains an open, documented gap (PanGloss's - // own conclusion too -- no available fixture drives the loop past one iteration). + // Guards the memo against a Simultaneous-mode epenthesis rule, which AnalysisRewriteRule compiles + // as ReapplyType.SelfOpaquing -- a repeat-until-fixpoint loop, and the one rule shape whose + // interaction with the nogood cache has a suspected (never reproduced) bug elsewhere. Known gap: + // no available fixture drives the loop past a single iteration, so two or more remains untested. var highVowel = FeatureStruct .New(Language.PhonologicalFeatureSystem) .Symbol(HCFeatureSystem.Segment) @@ -715,22 +676,17 @@ public void ParseWord_MemoOnMatchesMemoOff_ForSelfOpaquingSimultaneousEpenthesis $"memo-on parse of '{word}' must be analysis-set identical to memo-off" ); } - // The traced/correct oracle value for "buibui" (PanGloss's rust conformance fixture): root 19, - // one epenthesized result. Pinned directly, not just compared on-vs-off, so a bug that happened - // to affect both sides identically (e.g. both wrongly returning empty) would still be caught. + // Pinned as an absolute value, not just on-vs-off, so a bug affecting both sides identically + // (both wrongly returning empty, say) is still caught. Assert.That(memoOn.ParseWord("buibui").Count(), Is.EqualTo(1)); } [Test] public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithAffixTemplate() { - // Stage 4 of memoization.md: exercises the template-battery memo - // (AnalysisStratumRule.ApplyTemplateBattery) specifically -- TWO free prefix rules that commute - // with each other and with a template slot suffix. Unapplying di-then-gu vs gu-then-di reaches - // the same AnalysisStateKey (same shape, same rule MULTISET) via a different trail ORDER, so the - // second arrival replays the first arrival's stored template outputs with its own trail prefix - // grafted on (Word.ReplayOnto). One commuting prefix would NOT be enough -- a single rule can - // only unapply once, so no key would ever be re-arrived at and the memo would never fire. + // Two commuting prefixes, not one: a single rule unapplies only once, so no key would ever be + // re-arrived at and the template memo would never fire. Unapplying di-then-gu or gu-then-di + // reaches the same key by a different trail order, which is what makes the second one replay. var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; var edSuffix = new AffixProcessRule @@ -806,11 +762,9 @@ public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithAffixTemplate() $"template positive hits: {AnalysisStratumRule.DiagTemplateMemoHits - templateHitsBefore}, " + $"template nogood hits: {AnalysisStratumRule.DiagTemplateNogoodHits - templateNogoodHitsBefore}" ); - // Guards against this test going vacuous: the template memo's replay path must actually fire - // for this grammar. (As with the mrule-cascade memo, the ReplayOnto graft's effect on final - // signatures is unobservable through ParseWord's synthesis round-trip -- see memoization.md §5(b) - // -- so this counter, not the equivalence assertions above, is what proves the memoized path is - // actually exercised here, not silently bypassed.) + // The graft's effect on final signatures is invisible through synthesis, which re-derives rule + // orderings anyway, so this counter -- not the equality assertions above -- is what proves the + // memoized path was exercised at all. Assert.That( AnalysisStratumRule.DiagTemplateMemoHits + AnalysisStratumRule.DiagTemplateNogoodHits, Is.GreaterThan(templateHitsBefore + templateNogoodHitsBefore), @@ -819,12 +773,110 @@ public void ParseWord_MemoOnMatchesMemoOff_HitCounterGuarded_WithAffixTemplate() ); } - // Canonical analysis-set signature (memoization.md §5: gates compare this, never byte/object - // equality -- a memo-replayed Word is not guaranteed field-for-field identical to a freshly-computed - // one). AllomorphsInMorphOrder alone would not catch a broken trail/non-head graft (it walks Shape - // annotations, which Word.ReplayOnto never touches) -- MorphemesInApplicationOrder walks - // _mruleApps/_nonHeadApps directly, which is exactly what ReplayOnto rewrites. Root index is included - // so two analyses with the same morpheme sequence but different lexical roots can't collide. + [Test] + public void ParseWord_MemoOnMatchesMemoOff_OnLinearStratumWithAffixTemplate() + { + // Every other memo test runs on Unordered strata, leaving Linear -- MorphologicalRuleOrder's + // default -- with no end-to-end equivalence gate. AnalysisStratumRuleTests covers the exclusion + // that keeps the memo off this path; this covers the results it produces. + var any = FeatureStruct.New().Symbol(HCFeatureSystem.Segment).Value; + + var edSuffix = new AffixProcessRule + { + Id = "TPAST", + Name = "template_ed_suffix", + Gloss = "PAST", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + edSuffix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new CopyFromInput("1"), new InsertSegments(Table3, "+d") }, + } + ); + var verbTemplate = new AffixTemplate + { + Name = "verb_template", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + verbTemplate.Slots.Add(new AffixTemplateSlot(edSuffix) { Optional = true }); + Morphophonemic.AffixTemplates.Add(verbTemplate); + + var diPrefix = new AffixProcessRule + { + Id = "TDI", + Name = "template_di_prefix", + Gloss = "DI", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + diPrefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "di+"), new CopyFromInput("1") }, + } + ); + Morphophonemic.MorphologicalRules.Add(diPrefix); + + var guPrefix = new AffixProcessRule + { + Id = "TGU", + Name = "template_gu_prefix", + Gloss = "GU", + RequiredSyntacticFeatureStruct = FeatureStruct.New(Language.SyntacticFeatureSystem).Symbol("V").Value, + }; + guPrefix.Allomorphs.Add( + new AffixProcessAllomorph + { + Lhs = { Pattern.New("1").Annotation(any).OneOrMore.Value }, + Rhs = { new InsertSegments(Table3, "gu+"), new CopyFromInput("1") }, + } + ); + Morphophonemic.MorphologicalRules.Add(guPrefix); + + SetRuleOrder(MorphologicalRuleOrder.Linear); + var memoOff = new Morpher(TraceManager, Language); + var memoOn = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 1); + + foreach (string word in new[] { "digusagd", "disagd", "gusagd", "sagd", "sag" }) + { + List onResult = memoOn.ParseWord(word).ToList(); + List offResult = memoOff.ParseWord(word).ToList(); + Assert.That( + onResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(offResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"Linear-stratum parse of '{word}' must be analysis-set identical with and without the memo" + ); + } + } + + [Test] + public void ParseWord_HonorsMaxDegreeOfParallelismAsACap_WithoutChangingResults() + { + // The value is a resource knob, never a semantic one, so an intermediate cap must not shift results. + AddCompoundingAndPrefixRules(); + + var unbounded = new Morpher(TraceManager, Language); + var capped = new Morpher(TraceManager, Language, maxDegreeOfParallelism: 2); + + foreach (string word in new[] { "pʰutdidat", "pʰutdat" }) + { + List cappedResult = capped.ParseWord(word).ToList(); + List unboundedResult = unbounded.ParseWord(word).ToList(); + Assert.That( + cappedResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal), + Is.EqualTo(unboundedResult.Select(WordAnalysisSignature).OrderBy(s => s, StringComparer.Ordinal)), + $"a parallelism cap must not change the analysis set for '{word}'" + ); + } + } + + // What the memo gates compare, instead of object equality: a replayed Word is not field-for-field + // identical to a freshly-computed one. MorphemesInApplicationOrder is the load-bearing part, since it + // walks the trail and non-heads that ReplayOnto rewrites; AllomorphsInMorphOrder alone would miss a + // broken graft, walking only Shape annotations that ReplayOnto never touches. The root distinguishes + // analyses that share a morpheme sequence but not a lexical entry. internal static string WordAnalysisSignature(Word word) { return string.Join("+", word.AllomorphsInMorphOrder.Select(a => a.Morpheme.Id))