Redesign Eval Explorer: family-grouped configs, provider-family pills, headline best-config, category-leader & model-health tables

[justsml]

Problem Statement

As a security researcher using ExploitHunter.app, I open the Eval Explorer to judge which models are worth using for authorized hunting work and how to configure them. The current dashboard exposes thousands of Benchmark Run Records across hundreds of model × thinking-level permutations, but the charts do not tell a cohesive story:

• Models are not grouped with their own thinking/option permutations, so I cannot judge, for one model, whether minimal vs low vs high vs xhigh is worth the time and expense. Configs of the same family are scattered across separate bars/bubbles.
• There is no top-line "best configuration per model" view. Each family appears multiple times, so comparing families at a glance is impossible.
• Provider families (Google, Anthropic, OpenAI, Mistral, Qwen/Alibaba, DeepSeek, Moonshot, …) are not first-class filter pills. Selecting a union of families to compare is fiddly and not obviously reflected in the URL, so I cannot share a view that reproduces my exact selection.
• The dashboard reports scores, cost, and runtime, but does not surface model health: refusals and policy rejections, score variance / inconsistency, timeouts, malformed tool calls, guardrail and approval-following failures, harness failures. A model can post a great average and still be unusable because it times out one in ten runs or refuses guarded actions — and the dashboard hides that.
• There is no curated narrative that explains I do not have to pick one model for everything: a frontier model for memory context compaction, last year's frontier for active probing, a small guardrail-specialized model (e.g. GPT OSS 20B) for permission checks, all configurable via ENV overrides or config, with the system able to recommend per-role defaults.

Solution

Redesign the Eval Explorer (keeping it a standalone static site under evals/explorer/, fed by scripts/eval-explorer/build-data.ts) so it tells a coherent, shareable, health-aware model comparison story:

1. Family-grouped comparison. Every chart and table groups all thinking/option permutations of a model family together, so a user can see all configs of one model in one place and judge the performance / cost / speed tradeoffs themselves.
2. Provider-family pill filters with union semantics. Checkbox-style pills for each canonical provider family (Google, Anthropic, OpenAI, Mistral, Qwen/Alibaba, DeepSeek, Moonshot, …). Selecting multiple families forms a union. All selections are reflected in URL parameters so any shared link reproduces the exact view.
3. Curated headline section. A chart that shows each model family exactly once — its single best configuration — with the winning config (thinking level, cost, speed, score) revealed only in the tooltip. This is the top-line "best-case numbers per model" story.
4. Per-config tradeoff view. A multi-dimensional chart (two-axis score-vs-cost, bubble size = speed, grouped by family) showing all thinking levels of each family together so users can decide whether a higher thinking level justifies the expense and latency.
5. Category-leader tables. Tables of which model is best at each hacking category (Cybench, Webapp, Corpo Net, Network, Offline), with runner-ups and evidence.
6. Model-health columns and table. Build-time-computed health metrics per model/config: score variance, refusal/policy-rejection rate, timeout rate, malformed-tool-call rate, guardrail failure rate, approval-following rate, harness-failure rate, repetition errors. Surfaced as table columns and chart tooltips so real-world model health is readable, not just averages.
7. Multi-model config narrative. A static explainer section on the page (mirrored in a durable docs/model-selection-strategy.md) describing how to mix and match models per role — frontier for memory context compaction, last year's frontier for active probing, a guardrail-specialized model for permission checks — configurable via ENV overrides or config, with the system able to recommend per-role defaults from the existing model role profiles.

User Stories

1. As a security researcher, I want every chart to group all thinking/option permutations of a model family together, so that I can judge one model's tradeoffs in one place instead of chasing scattered bars.
2. As a security researcher, I want a per-family view that plots all thinking levels (minimal/low/medium/high/xhigh) on one chart with score, cost, and speed visible, so that I can decide whether a higher thinking level is worth the time and expense for that model.
3. As a security researcher, I want the chart's bubble size or a third dimension to encode speed/runtime, so that I can read performance, cost, and speed simultaneously without toggling charts.
4. As a security researcher, I want a second y-axis option (e.g. score on one axis, cost-per-task on the other), so that I can compare quality-vs-economy in a single view rather than two separate charts.
5. As a security researcher, I want a curated headline chart showing each model family exactly once — its best configuration — so that I can compare top-line best-case numbers across models without config noise.
6. As a security researcher, I want the winning configuration (thinking level, cost, speed, score) for each headline bubble to appear only in the tooltip, so that the headline stays clean but the detail is one hover away.
7. As a security researcher, I want provider-family filter pills (Google, Anthropic, OpenAI, Mistral, Qwen/Alibaba, DeepSeek, Moonshot, …) rendered as visible toggleable pills, so that I can quickly select which families load into the comparison.
8. As a security researcher, I want each provider-family pill to show checkbox-style on/off state and a row count, so that I can tell at a glance which families are in the comparison and how many records each contributes.
9. As a security researcher, I want selecting multiple provider families to form a union (models matching any selected family), so that I can compare arbitrary groupings like "Anthropic + Google" or "all local models".
10. As a security researcher, I want to combine provider-family pills with the existing category, model-group, runtime-scope, and model filters, so that I can express precise comparison scopes.
11. As a security researcher, I want every filter selection (provider families, categories, groups, runtime, models, metric, color-by, group-by toggle) reflected in the URL query string, so that I can share a link that reproduces my exact view.
12. As a security researcher, I want the URL state to survive a refresh and a copy-link button, so that I can bookmark or hand a colleague the exact comparison I was looking at.
13. As a security researcher, I want a table of which model is best at each hacking category (Cybench, Webapp, Corpo Net, Network, Offline), so that I can pick the right model for the kind of work I am doing.
14. As a security researcher, I want the category-leader table to show runner-ups, not just the winner, so that I can see realistic alternatives.
15. As a security researcher, I want each category-leader entry to carry evidence (score, coverage, sample tasks), so that I can trust the ranking rather than a bare average.
16. As a security researcher, I want a model-health table with columns for score variance, refusal/policy-rejection rate, timeout rate, malformed-tool-call rate, guardrail failure rate, approval-following rate, and harness-failure rate, so that I can spot models that look good on average but fail in the real world.
17. As a security researcher, I want models with high refusal or guardrail-violation rates flagged visibly, so that I do not accidentally pick a model inadequate for authorized security work.
18. As a security researcher, I want models with high variance or inconsistency called out, so that I know a great average may not reproduce on a given run.
19. As a security researcher, I want models that frequently time out or emit malformed tool calls called out, so that I can avoid models that waste a run budget on harness failures.
20. As a security researcher, I want health metrics to appear in chart tooltips, not only in a table, so that an outlier point in a chart is self-explanatory.
21. As a security researcher, I want health metrics computed deterministically at build time from the Benchmark Run Records, so that the numbers are reproducible and auditable rather than an LLM's opinion.
22. As a security researcher, I want the existing KPI cards, composite-index leaderboard, category profile, token, cost, time, and cost-distribution charts preserved (refactored, not removed), so that I keep the views I already rely on.
23. As a security researcher, I want the existing insights panel extended with health-aware findings (e.g. "best cost-quality among stable runners", "most variable model in view", "most guardrail failures in view"), so that the narrative layer reflects real-world model health.
24. As a security researcher, I want an explainer section on the page telling me I do not have to use one model across the board, so that I learn I can mix models per role.
25. As a security researcher, I want the explainer to mention concrete role examples (frontier model for memory context compaction, last year's frontier for active probing, GPT OSS 20B-class guardrail model for permission checks), so that the strategy is concrete, not abstract.
26. As a security researcher, I want the explainer to state these role assignments are configurable via ENV overrides or config, so that I know where to tune the system.
27. As a security researcher, I want the explainer to mention the system can recommend per-role defaults from the model role profiles, so that I know the platform can dial in the mix for my use case.
28. As a security researcher, I want a durable docs/model-selection-strategy.md page mirroring the explainer, so that the multi-model strategy is referenceable outside the dashboard.
29. As a maintainer, I want the redesign to keep the static-site architecture (plain DOM + ECharts via CDN, fed by pre-built JSON), so that I avoid routing a 16.9MB dataset through Next.js and avoid new SSR cost.
30. As a maintainer, I want the 2377-line app.js split into ESM modules (state, filters, aggregations, combobox, provider-pills, per-chart modules, tables, insights, narrative), so that the redesign is navigable and individually testable.
31. As a maintainer, I want the 1677-line build-data.ts augmented with small pure modules (model-health, best-config, category-leaders, provider-families), so that the riskiest derived data is testable in isolation.
32. As a maintainer, I want the existing eval-explorer-ui-metrics.test.ts and eval-explorer-build-data.test.ts to keep passing (or be updated coherently), so that the redesign does not silently regress honest-row handling, comparison-eligibility gating, token chart scaling, or KPI text.
33. As a maintainer, I want the provider-family source of truth shared by both the build script and the browser, so that canonicalization never drifts between build-time and runtime.
34. As a maintainer, I want the URL state contract extended (not replaced), so that existing bookmarked links keep working for the keys they already use.
35. As a security researcher, I want the redesigned dashboard's default view to remain calm and scannable, with detailed diagnostics behind an explicit expand/hover path, so that the forensic-visibility principle is preserved (details are available but not the default).

Implementation Decisions

Architecture

• Stay on the static site. The Eval Explorer remains under evals/explorer/ (plain DOM + ECharts via CDN) fed by scripts/eval-explorer/build-data.ts. No move into src/app/, no React, no render-json. Rationale: the dataset is ~17MB of pre-built JSON loaded on demand; a Next.js route would add SSR cost and break the existing build/CI flow without clear benefit.
• Module split, not rewrite. The 2377-line app.js is decomposed into ESM modules with explicit, narrow interfaces. Existing chart behavior is preserved by extracting, not reimagining.
• Deep pure modules for derived data. The riskiest new data (health metrics, best-config selection, category leaders, provider-family canonicalization) lives in pure functions under scripts/eval-explorer/ imported by build-data.ts, each testable in isolation without DOM or ECharts.

Build-time modules (scripts/eval-explorer/)

• provider-families.ts (new): the single canonical source of truth for provider families (Google, Anthropic, OpenAI, Mistral, Qwen/Alibaba, DeepSeek, Moonshot, z.ai, …) with stable ids, labels, and colors. Consolidates the currently scattered MODEL_COMPANY_LABELS (app.js), providerFromModel (build-data.ts), and modelFamilyFor. Exported to the browser via a small JSON field on eval-explorer-data.json (or a separate provider-families.json) so build and runtime share one map.
• model-health.ts (new): pure rows: EvalExplorerRow[] → ModelHealthSummary[]. Computes per (family × thinking) config: score variance, refusal/policy-rejection rate, timeout rate, malformed-tool-call rate (from commandSyntaxErrors/repeatedCommandSyntaxErrors/certainlyWrongCommands), guardrail failure rate (from guardFailures/pathDomainViolations/escalationAttempts/dangerousCommandsAttempted/dangerousCommandsRun), approval-following rate (from dangerousCommandsRun vs dangerousCommandsAttempted), harness-failure rate (from failureClass/error/timedOut), repetition-error rate (from repeatedSimilarCommandMistakes). All rates are computed over comparison-eligible rows and reported with their denominator so a small-sample caveat is visible.
• best-config.ts (new): pure rows → FamilyBestConfig[]. For each model family, selects the single best config by composite score with deterministic tiebreakers: (1) composite score, (2) lower median cost-per-task, (3) lower median runtime, (4) lower health-risk score, (5) lexicographic config id. Only comparison-eligible rows and families meeting the existing HEADLINE_MIN_TASK_ROWS / HEADLINE_MIN_DISTINCT_TASKS thresholds are eligible; narrow views bypass as today.
• category-leaders.ts (new): pure rows → CategoryLeader[]. For each hacking category in TARGET_CATEGORY_ORDER, ranks model families by mean normalized score (with scored-rows/total-rows coverage), produces a leader plus up to two runner-ups with evidence (score, coverage, sample task labels).
• build-data.ts (modified): orchestrates the above and extends the EvalExplorerData payload with four new top-level fields: providerFamilies, modelHealth, familyBestConfigs, categoryLeaders. Existing fields and summaryRow projection (stripping sourcePath from public rows) are unchanged. Bootstrap JSON is extended with a count of families and health-flagged models so the deferred-load paint can show a headline without the full payload.

Browser modules (evals/explorer/)

• state.js (extracted): owns the state object, URL persistence (persistUrlState/restoreUrlState), localStorage, and a small pub/sub (subscribe/notify). URL contract is extended: new keys families (provider-family pill selection) and groupBy (family vs config toggle) are added; existing keys (categories, providers, groups, runtime, models, metric, colorBy) keep their current names and semantics so old links keep working.
• filters.js (extracted): pure rows + state → filteredRows. No DOM. Mirrors the current applyFilters logic and adds provider-family membership filtering (a row is in scope if its canonical family id is in state.activeFamilies, or if activeFamilies is empty).
• aggregations.js (extracted): pure. Keeps aggregateByModel; adds aggregateByFamily (groups all thinking levels of a family into one summary with per-config breakdowns preserved for tooltips) and aggregateByBestConfig (joins rows to familyBestConfigs so headline views use the precomputed winner).
• combobox.js: the existing reusable renderMultiCombobox. Kept as-is.
• provider-pills.js (new): renders checkbox-style pills into #providerFamilyFilters. Each pill shows family label, on/off checkbox state, and row count; clicking toggles membership in state.activeFamilies. Multi-select is union by default. Pills derive order and color from provider-families.ts.
• charts/headline-best-config.js (new): ECharts chart showing one point/bar per family = its best config. Default visual encodes score (y) and cost-per-task (x); bubble size encodes speed (faster = larger). Tooltip reveals the winning thinking level, exact cost, runtime, score, coverage, and health flags. Only families meeting headline thresholds appear; narrow views fall back to in-scope best config.
• charts/per-config-tradeoff.js (new): ECharts scatter, two y-axes (score on left, cost-per-task on right), x = thinking level (ordinal), bubble size = speed, grouped/colored by family. Shows all configs of each family together so the user can read whether xhigh justifies the cost/time vs minimal.
• charts/category-leader-table.js (new): DOM table; one row per hacking category; columns: category, leader (family + best config), leader score, runner-up, runner-up score, coverage, evidence sample. Sortable.
• charts/model-health-table.js (new): DOM table; one row per family × config (or per family with config breakdowns expandable); columns: variance, refusal/policy-rejection rate, timeout rate, malformed-tool-call rate, guardrail failure rate, approval-following rate, harness-failure rate, repetition-error rate, each with denominator. Visibly flags models exceeding threshold rates.
• existing chart modules (ranking-chart.js, category-profile-chart.js, token-chart.js, cost-scatter-chart.js, time-chart.js, cost-distribution-chart.js): split out from the monolith with behavior preserved. Ranking and cost-scatter gain family-grouping and tooltip health fields. The token chart's square-root scale and accessibility attributes are preserved (pinned by tests).
• tables.js (new): shared DOM table helpers (sortable headers, compact number formatting, expandable rows) used by the two new tables.
• insights.js: existing insights panel, extended with health-aware entries: "best cost-quality among stable runners" (excludes high-variance/high-guardrail-failure), "most variable model in view", "most guardrail failures in view", "most timeouts in view".
• narrative.js (new): renders the static multi-model config explainer section into a #modelSelectionStrategy slot. Content is static prose (no live data dependency).
• app.js (reduced): boot/deferred-load orchestration and module wiring only.

index.html / styles.css

• index.html gains slots for #providerFamilyFilters, #headlineChart, #perConfigTradeoffChart, #categoryLeaderTable, #modelHealthTable, and #modelSelectionStrategy. The existing slots (#kpiGrid, #rankingChart, #radarChart, #tokenChart, #costScatterChart, #timeChart, #costDistributionChart, #insightList, #runTable) remain.
• styles.css adds pill styles (on/off checkbox visual, count badge, hover/focus), table styles (sortable headers, flag chips, expandable breakdowns), and the explainer section. Chart CSS variables stay.

Data model extensions

Extend EvalExplorerData (build-data.ts) with:

type EvalExplorerData = {
  // ...existing fields unchanged...
  providerFamilies: ProviderFamily[];      // canonical source of truth, shared with browser
  modelHealth: ModelHealthSummary[];        // per family × config
  familyBestConfigs: FamilyBestConfig[];    // one winner per family
  categoryLeaders: CategoryLeader[];        // per hacking category
};

The row-level EvalExplorerRow is unchanged; health is aggregated, not per-row, to keep the 17MB payload from growing further. Health summaries carry their denominators so small-sample caveats render.

URL state contract

• Extended, not replaced. New keys: families (comma-separated canonical family ids), groupBy (family default | config). Existing keys keep names and semantics. Empty families means "all families" (no filter), preserving current behavior when the key is absent.
• persistUrlState/restoreUrlState move into state.js but keep producing/consuming the same URLSearchParams shape.

Out-of-scope guardrails

• No live LLM narrative generation. All health metrics are deterministic build-time computations over existing EvalExplorerRow fields. (Eval honesty: a live LLM judging models on the public explorer would risk contamination and non-reproducibility.)
• No real ENV/config plumbing for per-role model assignment in this PRD. That is an app-config change, not an explorer change; the narrative points at the existing role profiles and the existing ENV/config surface.
• No move into src/app/, no React, no render-json.
• No change to comparison-eligibility rules, honest-row handling, or scorer provenance.

Testing Decisions

What makes a good test here

Tests assert external behavior and derived-data correctness, not implementation details or tuning knobs. A test should fail because the product got worse (wrong health rate, wrong best config, wrong family canonicalization, a filter that lets an ineligible row through), not because a label or color changed. Per the AGENTS.md testing guidance, brittle implementation assertions (exact default model id, exact color hex, exact query-string order) are avoided; only narrow mapping tests stay where a broken mapping would create a runtime bug.

Dedicated unit tests for the five deep modules

1. model-health.ts — fixture rows exercising each health column: a clean runner, a refusal-heavy runner, a timeout-heavy runner, a malformed-tool-call runner, a guardrail-violating runner, a small-sample config (denominator caveats). Asserts rates and denominators, not formatting.
2. best-config.ts — fixture rows where the deterministic tiebreakers matter (equal composite score → lower cost wins; equal cost → lower runtime wins; equal runtime → lower health risk wins). Asserts the winner and the runner-ups; asserts headline-threshold gating excludes under-covered families; asserts narrow-view bypass.
3. filters.js + aggregations.js — pure functions tested with the same fixture style as the existing eval-explorer-ui-metrics.test.ts. filters.js: provider-family union, combined with existing category/group/runtime/model filters, empty-families = all. aggregations.js: aggregateByFamily groups all thinking levels under one family and preserves per-config breakdowns; aggregateByBestConfig joins to precomputed winners.
4. provider-families.ts — canonicalization: every provider/modelFamilyId produced by build-data.ts maps to exactly one canonical family; no family overlaps; aliases (e.g. google-gemini- → gemini-, anthropic-claude- → claude-) collapse correctly; local providers (ollama, lmstudio, llamacpp) bucket as local.
5. category-leaders.ts — fixture rows producing a clear leader, a tie, and a category with insufficient coverage (no leader, honest empty). Asserts leader + runner-ups + evidence, not formatting.

Integration tests (updated, not rewritten)

• eval-explorer-build-data.test.ts: extended to assert the four new top-level payload fields exist and are well-formed for existing fixtures; existing assertions unchanged.
• eval-explorer-ui-metrics.test.ts: updated coherently for renamed/new DOM slots and chart series. Preserves the spirit of existing assertions (square-root token scale, accessibility attributes, KPI text, ranking excludes raw Cybench, model-group combobox). Where a series name changes because of family-grouping, the test is updated to assert the new grouping behavior, not the old per-config series name.
• eval-explorer-role-profiles.test.ts: unchanged; the role profiles feed the narrative section.

Out of Scope

• Live LLM-generated narrative or scoring on the public explorer. (Eval honesty: deterministic build-time metrics only.)
• Real ENV/config plumbing for per-role model assignment. (App-config change, separate PRD.)
• Moving the explorer into src/app/ or adopting React/render-json.
• Changing comparison-eligibility rules, honest-row handling, scorer provenance, or cost-accounting signoff.
• Rebuilding the run-logs page (logs.html/logs.js) — it stays as-is.
• Adding new benchmark packs or categories.
• Replacing ECharts with another charting library.

Further Notes

• The redesign intentionally preserves the existing URL contract for old keys so bookmarks keep working. New keys are additive.
• Health metrics are computed over comparison-eligible rows only, to keep eval-honesty invariants (raw/interrupted/harness-smoke rows stay visible but do not skew health rates). Denominators are always carried so small-sample configs are visibly caveated, never silently inflated.
• The multi-model config narrative references the existing model role profiles (evals/explorer/data/model-role-profiles.{json,md}) and the existing ENV/config surface; it does not invent new configuration plumbing.
• The headline best-config chart deliberately keeps config details in tooltips so the top-line story stays "one number per model"; the per-config tradeoff chart is where users who want the full config breakdown go.
• Following the forensic-visibility principle, health flags and raw rates are always available (table + tooltip), but the default headline view stays calm and scannable; detailed diagnostics are an explicit expand/hover away.