Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 52 additions & 56 deletions .github/workflows/mutation.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# ============================================================
# WorldScript Studio – Mutation Testing (Stryker) — Parallel + Incremental
# QNBS-v3: Matrix-parallel Stryker with per-module incremental caching.
# Run manually via workflow_dispatch. Each matrix job targets one module
# to keep wall-clock time low (~5–15 min per job vs. 30–45 min single job).
# Incremental caching skips unchanged mutants on re-runs (50–80% speedup).
# QNBS-v3: Matrix-parallel Stryker with per-module caches and fail-safe aggregation.
# Run manually via workflow_dispatch. Each matrix job targets one curated module;
# runtime and score remain measured outputs rather than undocumented promises.
# ============================================================

name: 🧬 Mutation Tests (Stryker)
Expand All @@ -19,7 +18,10 @@ on:
options:
- incremental
- force
- force-all-modules
module:
description: 'Scope selector: all, tier-a, or one module name'
required: false
default: 'all'
per_job_concurrency:
description: 'Concurrency per job (default: 2)'
required: false
Expand All @@ -33,31 +35,36 @@ permissions:
contents: read

jobs:
scope:
name: 🧬 Validate Stryker Scope
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.scope.outputs.matrix }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- id: scope
shell: bash
env:
SELECTOR: ${{ github.event.inputs.module || 'all' }}
run: |
node scripts/stryker-scope.mjs --selector "$SELECTOR"
printf 'matrix=%s\n' \
"$(node scripts/stryker-scope.mjs --matrix --selector "$SELECTOR")" \
>> "$GITHUB_OUTPUT"

# ============================================
# PARALLEL MATRIX JOBS (one per module)
# ============================================
stryker:
name: 🧬 Stryker ${{ matrix.name }}
runs-on: ubuntu-latest
timeout-minutes: 45
needs: scope
strategy:
fail-fast: false
matrix:
include:
- name: services-commands
mutate: "services/commands/**/*.ts"
- name: services-core
mutate: "services/{help/helpDocRetrieval,plotBoardService,promptLibrary,deepLinkService}.ts"
- name: services-ai-core
mutate: "services/ai/{modelRecommendations,aiPolicy,aiRetry,fetchAdapter,routingLogger,aiModeService}.ts"
- name: services-ai-providers
mutate: "services/ai/providers/openrouterProvider.ts"
- name: features-project
mutate: "features/project/**/*.ts"
- name: features-misc
mutate: "features/{settings,progressTracker,featureFlags,proForge}/**/*.ts"
- name: copilot
mutate: "services/copilot/**/*.ts"
matrix: ${{ fromJSON(needs.scope.outputs.matrix) }}

permissions:
contents: read
Expand All @@ -83,7 +90,7 @@ jobs:
run: |
mkdir -p reports
MODE="${{ github.event.inputs.mode || 'incremental' }}"
if [ "$MODE" = "force" ] || [ "$MODE" = "force-all-modules" ]; then
if [ "$MODE" = "force" ]; then
echo "🧹 Force mode → clearing incremental cache for ${{ matrix.name }}"
rm -f reports/stryker-incremental-${{ matrix.name }}.json
elif [ -f "reports/stryker-incremental-${{ matrix.name }}.json" ]; then
Expand All @@ -96,7 +103,7 @@ jobs:
run: |
MODE="${{ github.event.inputs.mode || 'incremental' }}"
FORCE_FLAG=""
if [ "$MODE" = "force" ] || [ "$MODE" = "force-all-modules" ]; then
if [ "$MODE" = "force" ]; then
FORCE_FLAG="--force"
fi

Expand All @@ -107,10 +114,13 @@ jobs:
echo " Concurrency: $CONCURRENCY"
echo " Mode: $MODE"

# QNBS-v3: each matrix job uses its own incremental file
export STRYKER_INCREMENTAL_FILE="reports/stryker-incremental-${{ matrix.name }}.json"
# QNBS-v3: pass the supported CLI option; arbitrary env vars are not Stryker config.
INCREMENTAL_FILE="reports/stryker-incremental-${{ matrix.name }}.json"

pnpm exec stryker run \
stryker.config.mjs \
--incremental \
--incrementalFile "$INCREMENTAL_FILE" \
$FORCE_FLAG \
--concurrency "$CONCURRENCY" \
--mutate "${{ matrix.mutate }}" \
Expand All @@ -124,7 +134,7 @@ jobs:
with:
name: stryker-report-${{ matrix.name }}
path: reports/mutation/
if-no-files-found: warn
if-no-files-found: error
retention-days: 30

# ============================================
Expand All @@ -133,50 +143,36 @@ jobs:
aggregate:
name: 📊 Stryker Aggregate Report
runs-on: ubuntu-latest
needs: stryker
needs: [scope, stryker]
if: always()
timeout-minutes: 10
permissions:
contents: read

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Download all Stryker reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
path: all-reports
pattern: stryker-report-*
merge-multiple: true
merge-multiple: false

- name: Build combined summary
env:
SELECTOR: ${{ github.event.inputs.module || 'all' }}
run: |
echo "## 🧬 Stryker Mutation Results (Parallel)" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "| Module | Score | Killed | Survived | Timeout | No Cov |" >> "$GITHUB_STEP_SUMMARY"
echo "|--------|-------|--------|----------|---------|--------|" >> "$GITHUB_STEP_SUMMARY"

for dir in all-reports/stryker-report-*/; do
[ -d "$dir" ] || continue
MODULE=$(basename "$dir" | sed 's/stryker-report-//')
REPORT="${dir}mutation.json"

if [ -f "$REPORT" ]; then
SCORE=$(node -e "
const r = require('./$REPORT');
const m = r.metrics || {};
console.log((m.mutationScore ?? 0).toFixed(1) + '%');
" 2>/dev/null || echo "N/A")
KILLED=$(node -e "const r=require('./$REPORT'); console.log(r.metrics?.killed ?? 'N/A')" 2>/dev/null || echo "N/A")
SURVIVED=$(node -e "const r=require('./$REPORT'); console.log(r.metrics?.survived ?? 'N/A')" 2>/dev/null || echo "N/A")
TIMEOUT=$(node -e "const r=require('./$REPORT'); console.log(r.metrics?.timeout ?? 'N/A')" 2>/dev/null || echo "N/A")
NOCOV=$(node -e "const r=require('./$REPORT'); console.log(r.metrics?.noCoverage ?? 'N/A')" 2>/dev/null || echo "N/A")

echo "| $MODULE | $SCORE | $KILLED | $SURVIVED | $TIMEOUT | $NOCOV |" >> "$GITHUB_STEP_SUMMARY"
else
echo "| $MODULE | N/A | N/A | N/A | N/A | N/A |" >> "$GITHUB_STEP_SUMMARY"
fi
done

echo "" >> "$GITHUB_STEP_SUMMARY"
set -o pipefail
if [ "${{ needs.stryker.result }}" != "success" ]; then
echo "Stryker matrix completed with: ${{ needs.stryker.result }}; the summary remains informational and this job will fail closed." >&2
fi
node scripts/aggregate-stryker-reports.mjs all-reports --selector "$SELECTOR" | tee -a "$GITHUB_STEP_SUMMARY"
if [ "${{ needs.stryker.result }}" != "success" ]; then
exit 1
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.
echo "> Mode: \`${{ github.event.inputs.mode || 'incremental' }}\` | Concurrency: \`${{ github.event.inputs.per_job_concurrency || '2' }}\`" >> "$GITHUB_STEP_SUMMARY"
echo "> Threshold: break ≥ 75 | low ≥ 70 | high ≥ 85" >> "$GITHUB_STEP_SUMMARY"
echo "> Individual HTML/JSON reports are available as artifacts." >> "$GITHUB_STEP_SUMMARY"
19 changes: 11 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The app supports a multi-provider AI stack (Gemini, OpenAI, Claude, Grok, OpenRo
| Type checker | `tsgo` (TypeScript Go port) via `tsconfig.tsgo.json` with 4 checkers (`pnpm run typecheck`) |
| Styling | Tailwind CSS `^4.3.1` via `@tailwindcss/vite` + semantic CSS custom properties (`index.css`) |
| State | Redux Toolkit `^2.12.0` + `redux-undo` (project slice only); Zustand `^5.0.14` for transient UI (`app/transientUiStore.ts`) |
| Testing | Vitest `^4.1.10` (jsdom, `maxWorkers: 1`), Playwright `^1.61.1` (E2E, CI-only), Stryker `^9.2.0` (mutation, manual workflow only) |
| Testing | Vitest `^4.1.10` (jsdom, `maxWorkers: 1`), Playwright `^1.61.1` (E2E, CI-only), Stryker `9.6.1` (mutation, manual workflow only) |
| Lint/Format | Biome `^2.5.4` (`biome.json`) — single toolchain for JS/TS/CSS |
| AI | Multi-provider: Google Gemini (`@google/genai`), OpenAI, Anthropic Claude, Grok, OpenRouter, Ollama, WebLLM, ONNX Runtime Web, Transformers.js |
| Voice | Web Speech API (fallback); WASM engines: Whisper.cpp (STT), Kokoro (TTS), Silero VAD; gated by `featureFlags.enableVoiceWasm` |
Expand Down Expand Up @@ -144,7 +144,7 @@ WorldScript-Studio/
- `playwright.config.ts` — E2E projects: Chromium desktop + Pixel 5 mobile in CI; Firefox + optional mobile locally
- `turbo.json` — task graph for `build`, `dev`, `lint`, `typecheck`, `test`, `mutation`
- `pnpm-workspace.yaml` — workspace packages + pnpm v11 `allowBuilds` default-deny map
- `stryker.conf.json` — ~20 mutation targets (services + features), `break: 75`
- `stryker.config.mjs` + `stryker-scope.json` — 25 curated production targets across 8 risk-tiered modules, `break: 75`
- `.lighthouserc.cjs` — accessibility `error` ≥ 0.95, CLS `error` ≤ 0.1, performance/SEO `warn`
- `src-tauri/tauri.conf.json` / `Cargo.toml` — desktop window config, CSP, updater endpoints, rust-compute feature

Expand Down Expand Up @@ -181,7 +181,9 @@ pnpm run test:e2e # Playwright E2E (CI=true required; CI-only by polic
pnpm run test:e2e:ui # Playwright E2E UI mode (CI=true required)
pnpm run test:e2e:deep # Deep E2E feature-flag matrix (CI=true required)
pnpm run test:vrt # Visual regression (Chromium only)
pnpm run mutation # Stryker mutation testing (CI-only; manual workflow)
pnpm run mutation # Stryker incremental mutation testing (CI-only; manual workflow)
pnpm run mutation:force # Stryker force/no-cache audit (CI-only)
pnpm run mutation:report # Aggregate downloaded reports; fails on missing/invalid shards

# Analysis / budgets
pnpm run analyze # Rollup visualizer → dist/bundle-analysis.html
Expand Down Expand Up @@ -317,11 +319,12 @@ hooks are not installed.

### Mutation Testing (Stryker)

- Config: `stryker.conf.json`
- Targets: ~20 files across `services/`, `features/`, `services/copilot/`, `services/ai/providers/`
- `ignoreStatic: true` drops runtime from ~90 min to ~10 min.
- `break: 75` — score below 75 fails the mutation job.
- **Removed from PR/CI pipeline** (2026-06-02) — now runs only via manual `.github/workflows/mutation.yml` (`workflow_dispatch`).
- Config: `stryker.config.mjs`, with the authoritative target/risk registry in `stryker-scope.json`.
- Scope: 25 production files across 8 modules; Tier A is pure/domain/policy logic, Tier B is bounded adapter/orchestration logic. Generated, type-only, test, and presentation-only glue stays out of scope.
- Incremental mode uses Stryker's supported `--incrementalFile` option and one cache file per module; `vitest.related: true` keeps each mutant on related tests instead of rerunning the full suite. `mutation:force` intentionally bypasses that cache for release, security, or major-refactor audits.
- The manual workflow supports `module: all`, `tier-a`, or one module name. Matrix artifacts retain module identity; aggregation derives canonical metrics from mutant statuses, fails if any expected shard/report is missing or invalid, and exposes killed, survived, timeout, no-coverage, ignored, pending, and error counts separately.
- `break: 75`, `low: 70`, and `high: 85` remain the current operational thresholds; they are not a claimed measured baseline until a trusted force run records one. Do not change them to make a run green.
- **Removed from PR/CI pipeline** (2026-06-02) — mutation runs only via manual `.github/workflows/mutation.yml` (`workflow_dispatch`). Routine local validation must never run broad Stryker; cloud CI owns incremental, force, and report aggregation.

### Storybook

Expand Down
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ pnpm run bench # Vitest perf benchmarks (tests/bench) — baseline gate
pnpm run content:guard # Validate community templates for secrets / eval payloads
pnpm run i18n:check # Locale key parity + bundle rebuild (runs in CI quality job)
pnpm run i18n:bundle # Rebuild public/locales/<lang>/bundle.json from source JSON
pnpm run mutation # Stryker mutation — CI-ONLY; trigger via: gh workflow run mutation.yml
pnpm run mutation # Stryker incremental mutation — CI-ONLY; trigger mutation.yml
pnpm run mutation:force # Stryker force/no-cache audit — CI-ONLY
pnpm run mutation:report # Aggregate downloaded module reports; fails on missing shards
pnpm run test:e2e # Playwright E2E tests (CI=true required; CI-only)
pnpm run test:e2e:deep # Deep coverage suite — feature-flag matrix + error paths (CI-only; non-blocking)
pnpm run test:storybook # Storybook test-runner (CI; needs Storybook running or built)
Expand All @@ -48,7 +50,7 @@ pnpm run token:audit # audit-tokens.mjs — design-token usage gate (CI b
**CI-cloud-first workflow (constrained local hardware only):** On low-end hardware, run only `ci:prepush` locally before pushing. Coverage, E2E, Lighthouse, and Stryker are CI-gate jobs. After each push, update README.md badges and AUDIT.md quality-gate line with CI-reported numbers. Local CI simulation: `act pull_request --job quality` (Docker + `act`; see `infra/low-end-ci/DAILY-DRIVER.md`).

**CI audit & housekeeping policy (ALL CI runs must be fully green):**
- After every commit, monitor ALL CI jobs: security (OSV + CodeQL), quality (Biome + tsc + Vitest), build, e2e, lighthouse, deploy, mutation, storybook.
- After every commit, monitor all jobs for the active workflow. Stryker is a separate manual workflow, not a routine PR check; monitor it when explicitly dispatched for an incremental, Tier-A, module, or force audit.
- **CodeQL scanning**: Check `https://github.com/qnbs/WorldScript-Studio/security/code-scanning` after every push. Fix the root cause — do not just suppress alerts.
- **Token-Permissions**: All GitHub Actions workflows must set top-level `permissions: contents: read`; write permissions belong at the job level, never top-level.
- **OSV vulnerabilities**: Run `pnpm audit` or check the security CI job. Add `pnpm.overrides` with pinned exact versions.
Expand Down
18 changes: 16 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,12 +286,26 @@ Tests live in `tests/e2e/`. Playwright tests verify core user flows:

### Mutation testing (Stryker)

Targets service files (see [`stryker.conf.json`](stryker.conf.json)): `codexService`, `dbMigration`, `fuzzyScore`, `palettePreferences`, `commandBuilder`, `hybridFallback`, `providerFactory`, `helpDocRetrieval`, `listenerMiddleware`. HTML report: `reports/mutation/`. `thresholds.break` is `60` — CI fails if mutation score falls below this value.
Targets are defined once in [`stryker-scope.json`](stryker-scope.json) and consumed by
[`stryker.config.mjs`](stryker.config.mjs): 25 production files across 8 risk-tiered modules.
The manual workflow can run `all`, `tier-a`, or one module. Related-test selection avoids rerunning
the full Vitest suite for every mutant. Reports are kept under `reports/mutation/` and uploaded
with module identity; the aggregate derives metrics from mutant statuses and fails on missing or
invalid shards while showing killed, survived, timeout, no-coverage, ignored, pending, and error counts. Current operational
thresholds are `break: 75`, `low: 70`, and `high: 85`; they are not a measured baseline until a
trusted force run establishes one.

```bash
pnpm run mutation
pnpm run mutation # incremental; CI-only
pnpm run mutation:force # no-cache force audit; CI-only
pnpm run mutation:report # aggregate downloaded reports
```

Do not run broad Stryker locally on constrained hardware. Use targeted tests locally and dispatch
the workflow for PR-related Tier-A/module diagnostics, recurring incremental checks, or release /
security force audits. Treat equivalent mutants as a documented survivor class; never hide
NoCoverage or timeout results and never change thresholds merely to make CI green.

### Storybook

```bash
Expand Down
28 changes: 23 additions & 5 deletions docs/CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,14 +205,32 @@ gh workflow run mutation.yml
```

`mutation.yml` is `workflow_dispatch`-only — it never runs automatically on push or PR. This means
the mutation score is **not continuously tracked**; it's a point-in-time snapshot whenever someone
runs it manually. [`stryker.conf.json`](../stryker.conf.json) still defines the thresholds
(`break: 75`, `high: 85`, `low: 70`) that would apply if it ran.
the mutation score is **not continuously tracked**; it is a point-in-time snapshot whenever someone
runs it manually. [`stryker.config.mjs`](../stryker.config.mjs) consumes the single authoritative
scope in [`stryker-scope.json`](../stryker-scope.json): 25 production targets across 8 modules.
Each module is marked Tier A (pure/domain/policy logic) or Tier B (bounded adapter/orchestration
logic), and the workflow accepts `all`, `tier-a`, or one module name for a bounded diagnostic run.

Incremental runs pass Stryker's supported `--incrementalFile` option and cache one file per module;
the cache is not inferred from an arbitrary environment variable. `force` removes the selected
module cache before running, so it is the explicit no-cache mode for release, security, and major
refactor audits. Reports are uploaded under unique `stryker-report-<module>` artifact directories;
the aggregate job derives metrics from real mutant statuses under `files`, keeps the summary visible
when a shard fails, and fails loudly on any missing, malformed, or incomplete expected report. The
summary keeps killed, survived, timeout, no-coverage, ignored, pending, and error counts visible
instead of treating timeout/no-coverage as equivalent evidence. `vitest.related: true` is enabled
so each mutant uses related tests instead of rerunning the full suite.

The current operational thresholds are `break: 75`, `high: 85`, and `low: 70`. They are preserved
until a trusted force run establishes measured module/global baselines; they must not be changed to
make a run green. The installed TypeScript checker is intentionally not used: Stryker's Vitest
runner and the repository's tsgo gate provide the supported compile/type signal, while enabling the
unused checker would add cost without an accepted compatibility baseline.

**Re-integration criterion:** bring it back into the `quality` job (or a separate required check)
once a manual run demonstrates the flakiness is resolved — concretely, three consecutive manual
`workflow_dispatch` runs on `main` completing without a spurious failure or timeout, at the current
40-target mutate scope. Until then, treat a manual run's score as informational only, not a gate.
25-target mutate scope. Until then, treat a manual run's score as informational only, not a gate.

---

Expand Down Expand Up @@ -374,7 +392,7 @@ act pull_request -j quality -s CODECOV_TOKEN="$CODECOV_TOKEN" -W .github/workflo
| `tests/e2e/collaboration.spec.ts` | Collaboration panel security warning banner pre-connect |
| `services/commands/` | Command registry backing the palette (fuzzy search — regression-sensitive if E2E targets palette copy) |
| `hooks/useGlobalKeyboardShortcuts.ts` | Global shortcut listener — keep in sync with **Settings → Shortcuts** defaults |
| `stryker.conf.json` | Mutation testing targets + thresholds (`break: 75`, `high: 85`, `low: 70`; 40 mutate targets as of v1.19.0) |
| `stryker.config.mjs` / `stryker-scope.json` | Curated 25-file mutation scope across 8 risk-tiered modules; current thresholds are `break: 75`, `high: 85`, `low: 70` |

---

Expand Down
Loading
Loading