feat(ai): synthetic-defect harness for critic calibration (AI-044)

[mrviduus]

AI-044 — Synthetic-defect injection harness (Phase 7)

Validates the AI-041 critic's calibration — the calibration AutoPublishCrew/SeoCrew gate publish on. Injects KNOWN defects into clean drafts, runs them through the real nano critic, and measures catch-rate + a clean-control false-positive rate.

How

Mirrors ToolCallEvalRunner: deterministic injection + scoring (pure, no LLM) → CI-testable with a fake critic, no key; the live nano run is admin-triggered (POST /admin/ai-quality/evals/criticdefects/run, ~23 calls, sync) and persists a criticdefects eval_run (no judge, Score=catch-rate, BreakdownJson per-axis + FP). Reuses EvalRun — no schema change.

Defect taxonomy (23 fixtures on a real edition-description brief)

Type	n	Axis	"caught" =
factual_hallucination	6	factual_accuracy	axis ≤2 / factual issue / ParseFailed
banned_phrase	4	banned_phrases	axis ≤2 / banned issue / ParseFailed
length over/under	4	length	length ≤2 / ParseFailed
tone_break	4	tone	tone ≤2 / ParseFailed
clean (control)	5	—	flagged → false positive

ParseFailed (fail-closed verdict) counts as a correct reject for any defect.

Honest gate (hardened per adversarial QA)

• Passed = catchRate ≥ 0.80 AND falsePositiveRate ≤ 0.20. The original catchRate-only gate let a flag-everything critic (FP=1.0) report success — the runner's own test now proves it correctly fails. A useless critic can't masquerade as calibrated.
• Clean controls rewritten meta-phrase-free + grounded + in-bounds, so a legitimately strict critic isn't penalized with false FP.
• Length defects breach by a wide margin (~½·Min / ~1.5·Max) so the breach is actually catchable by an LLM eyeballing prose — not a <1% margin that misses for reasons unrelated to critic quality.

Admin UI

New "Run critic-defect eval" button on the AI-quality Evals tab → catch% / FP% / n + PASS/FAIL badge; result persists into the eval history.

Tests — 15 (full AiEvals suite 41 pass / 5 live-key skip)

Pure injector transforms (breaches are real, deterministic) + runner scoring with fake critics: catches-all → fails (FP guard), catches-none → 0.0, good→pass, FP-just-over→fail, garbage→ParseFailed-caught. StudyBuddy set-equality green; no ITool leaked.

Verify

• dotnet test tests/TextStack.AiEvals → 41 pass / 5 skip (deterministic half runs with no key)
• dotnet test tests/TextStack.UnitTests → 402 pass
• dotnet format --verify-no-changes → clean
• pnpm -C apps/admin exec tsc --noEmit + build → clean

Note: FP-rate enforced now; golden set grows later (per RAG/StudyBuddy golden TODOs). Admin button is build-verified; live click is owner-triggered (needs prod key + admin session).

🤖 Generated with Claude Code