Skip to content

feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109

Open
Open
feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing#109
Labels
enhancementNew feature or request

Description

@pacphi
pacphi
opened on Aug 5, 2026

Executive brief

Agentic Kit already indexes machine-local Claude Code, Codex, and OpenCode activity and presents model, provider, token, cost, session, and routing information in the Dashboard. It can currently identify broad patterns such as premium models handling short sessions, but it cannot yet answer the higher-value questions:

  • Which models are demonstrably effective for each operation?
  • Where did a premium model materially improve quality, reduce rework, or prevent escalation?
  • Which operations could have used a cheaper model for materially equivalent benefit?
  • Is a routing change actually saving money or plan capacity after retry and escalation overhead?

This proposal adds Route Intelligence: an evidence-backed, local-first learning and findings layer that derives operation outcomes from cross-host transcripts, structured ak run results, deterministic quality gates, reviewer evidence, and user feedback. It uses Ruflo's real MetaHarness-backed model-routing and outcome-learning surfaces to produce per-activity recommendations, while keeping the Dashboard read-only and refusing to make claims when the evidence is insufficient or confounded.

The desired product outcome is a clear Dashboard finding such as:

Cheaper equivalent found for documentation
The lower-cost route stayed within the configured quality tolerance across comparable operations and reduced API-equivalent cost. Evidence: N matched operations, P paired evaluations, no increase in repair or escalation rate.

All numbers in the shipped interface must come from the user's own evidence. The example above is illustrative, not a measured claim.

Table of contents

Background and current state

The foundation is already substantial:

  • src/lib/usage-index.mjs incrementally indexes Claude Code, Codex, and OpenCode session evidence and aggregates model, host, provider, tokens, observed/API-equivalent cost, duration, and tool usage.
  • src/lib/usage-opencode.mjs preserves OpenCode's observed provider, model, token, and metered-cost evidence instead of repricing unknown providers from assumptions.
  • src/lib/usage-classify.mjs classifies sessions into operational categories with explicit confidence and a first-class Unclassified result.
  • src/lib/usage-insights.mjs emits ranked Dashboard findings, including premium-on-routine and model-routing.
  • src/lib/routing.mjs owns the canonical per-activity routing vocabulary and effective host/model routes.
  • ak run produces structured activity, host, model, duration, status, handoff, and bounded escalation evidence.
  • The Dashboard already has a Usage → Findings destination and a read-only routing view.

The governing plans are living and must remain accurate:

  • ADR-0009 is Implemented (2026-07-25; updated 2026-08-04). It owns local transcript analytics, the offline-first Usage contract, graded findings, cost truthfulness, and the three-area Dashboard structure.
  • ADR-0021 is Accepted (2026-07-31; updated 2026-08-03). It requires execution host, inference provider, provider provenance, and model to remain independent facts.

This feature should extend those contracts. It should not create a fourth Dashboard area, add network activity to ordinary Dashboard reads, infer providers from hosts or model names, or silently turn a heuristic into a learned-routing claim.

Problem statement

The current findings can identify expensive-looking shapes, but session duration and response count are not evidence that a cheaper model would have delivered the same result. Historical routing is also confounded: harder tasks are disproportionately assigned to stronger models, and different hosts expose different prompts, tools, permissions, and harness behavior.

Therefore, transcripts alone can answer what happened, but they cannot automatically establish the counterfactual what would have happened on another model.

The feature needs an evidence model that:

  1. identifies a bounded operation more precisely than an entire interactive session;
  2. separates model, host, provider, reasoning effort, and harness version;
  3. attaches quality outcomes with provenance and confidence;
  4. compares like with like;
  5. verifies high-value recommendations through paired or prospective evidence;
  6. includes retry, repair, escalation, latency, cash cost, and plan pressure in the final result; and
  7. stays silent when a recommendation would be speculative.

Goals

  1. Learn model effectiveness by canonical activity, repository, task shape, complexity, and harness version.
  2. Identify cheaper candidates that are non-inferior within an explicit quality tolerance.
  3. Identify operations where the premium model earns its cost through better quality, fewer repairs, or lower escalation.
  4. Quantify the net effect of cheap-first cascades after fallback overhead.
  5. Surface findings with evidence count, evidence grade, uncertainty, cost basis, and actionable routing commands.
  6. Feed qualified outcomes to Ruflo's real MetaHarness-backed routing loop rather than building a private substitute.
  7. Preserve local-first, rebuildable historical analytics and explicit opt-in for any metered replay or learning write.

Non-goals

  • Producing a universal public model leaderboard.
  • Claiming causal equivalence from one successful session.
  • Treating host, provider, vendor, model, or billing mode as interchangeable.
  • Sending transcript content to a remote service during normal Dashboard reads.
  • Automatically rewriting kit.json routing policy from a finding.
  • Replacing Agentic-QE with transcript sentiment or an assistant's self-reported completion.
  • Treating API-equivalent cost as subscription billing.
  • Joining OpenRouter account-level analytics to sessions when the upstream data has no correlation key.
  • Building another local model router and naming it MetaHarness.

Product principles

  1. No fabricated counterfactuals. A recommendation names the evidence class that supports it.
  2. Quality before price. The selector finds the cheapest candidate that clears the activity's quality bar; it does not merely choose the cheapest model.
  3. Unknown is not zero. Missing cost, provider, quality, or outcome remains unknown.
  4. Model is not harness. Every episode carries a harness fingerprint so a prompt/tooling improvement is not misattributed to the model.
  5. Provider is not host. ADR-0021 provenance remains authoritative.
  6. Read-only findings. The Dashboard explains and offers a copyable command; policy mutation remains an explicit CLI action.
  7. Confidence is visible. Sample size, matching quality, evidence grade, uncertainty, and freshness are part of the finding.
  8. No recommendation is a valid result. Insufficient, stale, or confounded evidence produces a diagnostic, not a guess.

Proposed domain design

Bounded context

Add a Model Intelligence bounded context between Historical Usage and Routing Policy:

Historical Usage ──derived evidence──▶ Model Intelligence ──recommendation──▶ Routing Policy
        │                                      │                                  │
        └──────── transcript/session link ─────┘                                  └── explicit user action

The contexts retain separate ownership:

  • Historical Usage owns transcript discovery, parsing, pricing, session aggregation, and existing findings.
  • Model Intelligence owns operation episodes, outcome provenance, matching, non-inferiority evaluation, recommendation confidence, and the machine-wide read model.
  • Routing Policy remains the only source of truth for configured per-activity routes.
  • Dashboard Delivery renders findings without upgrading provenance or mutating policy.

Operation episode

An OperationEpisode is the minimum comparable unit:

{
  episodeId,
  source: { kind, host, sessionKey, runId, workerId },
  project: { id, worktree, revision },
  activity: { id, provenance, confidence },
  task: { fingerprint, embeddingRef, complexity, featureVersion },
  execution: {
    host, inferenceProvider, providerProvenance,
    model, reasoningEffort, harnessFingerprint,
    startedAt, endedAt, durationMs
  },
  usage: {
    input, output, cacheRead, cacheWrite,
    observedCostUsd, apiEquivalentCostUsd,
    costBasis, quotaPressure
  },
  outcome: {
    status, quality, evidenceGrade,
    signals, retries, repairs, escalation
  }
}

Historical interactive sessions may yield coarse, low-confidence episodes. Structured ak run workers should yield activity-bounded episodes with stronger identity and outcome evidence.

Outcome evidence

Quality is a composition of observed signals, not a single assistant-generated score:

Signal Strength Notes
Deterministic build/test/release gate Strong Artifact-backed and reproducible
Agentic-QE test, coverage, mutation, security, or a11y verdict Strong Measured quality evidence
Independent reviewer verdict Strong Must identify reviewer model/harness separately
Explicit user acceptance or rejection Strong Subjective but authoritative for utility
Later repair, revert, regression, or reopened task Strong negative Requires a bounded attribution window
Retry or model escalation Medium negative Initial route did not complete under the same contract
Successful worker protocol/handoff Medium Completion evidence, not correctness by itself
Session ended without an error Weak Never sufficient for equivalence
Assistant says “done” None Must not train or justify a recommendation alone

Every signal carries source, timestamp, scope, and provenance. Conflicting signals remain visible rather than being silently averaged away.

Evidence grades

Grade Meaning Allowed product language
Insufficient Too little, stale, or conflicting evidence “No recommendation yet”
Observed Descriptive statistics from historical episodes “Model X was used successfully…”
Correlated Matched historical cohorts with bounded confounding “Model X is associated with…”
Verified Paired evaluation against the same fixture and gates “Model X matched the incumbent within tolerance…”
Proven in routing Prospective cheap-first route with measured escalation overhead “This route saved capacity/cost while maintaining the quality bar…”

The UI must never promote an Observed or Correlated result to Verified language.

Cost and capacity model

“Cheaper” has at least three independent meanings:

  1. Observed cash cost — provider-reported metered spend when available.
  2. API-equivalent cost — dated pricing applied to transcript tokens, explicitly not plan billing.
  3. Subscription capacity pressure — effect on independent host limits, resets, and available headroom.

Recommendations must identify the basis they optimize. A subscription-backed model may have zero marginal cash cost while still consuming scarce plan capacity. Unknown cost stays unknown; no fallback price may manufacture a dollar saving.

Proposed architecture

Claude/Codex/OpenCode transcripts       ak run worker results
              │                                  │
              ├──────────────┬───────────────────┤
              ▼              ▼                   ▼
      Historical parser   activity identity   attempts/handoffs
              │              │                   │
              └──────────────┴──────────┬────────┘
                                        ▼
                           OperationEpisode extractor
                                        │
                     ┌──────────────────┴─────────────────┐
                     ▼                                    ▼
          Rebuildable local read model          Qualified outcome events
          (Dashboard/history)                   (Ruflo trajectory/outcome path)
                     │                                    │
                     └──────────────────┬─────────────────┘
                                        ▼
                         Matcher + evidence evaluator
                                        │
                    ┌───────────────────┴──────────────────┐
                    ▼                                      ▼
        MetaHarness-backed quality prediction       Findings projection
        + subscription/headroom overlay                    │
                    │                                      ▼
                    └──────── recommendation ─────▶ Usage → Findings
                                                          │
                                                          ▼
                                               explicit `ak host pick --route …`

Storage responsibilities:

  • Transcript-derived historical episodes are rebuildable projections alongside the existing usage index.
  • Qualified routing decisions and outcomes use Ruflo's real trajectory/outcome-learning surface (including .swarm/model-router-trajectories.jsonl where enabled), not a second private learning store.
  • AgentDB project memory stores accepted decisions and learned project patterns; it is not used as a substitute for the rebuildable Dashboard index.
  • The Dashboard API exposes aggregate evidence and bounded references, never raw task embeddings or unmasked transcript content.

Learning and recommendation method

Historical matched cohorts

Group episodes by:

  • canonical activity;
  • repository and optional project family;
  • task embedding/fingerprint similarity;
  • complexity bucket and relevant features;
  • harness fingerprint and tool availability;
  • time window/model generation; and
  • quality-evidence availability.

Compare models only after matching on these dimensions. Use shrinkage or confidence intervals so thin cohorts regress toward “unknown,” not an extreme recommendation. Report balance diagnostics and exclude cohorts whose model assignment remains strongly confounded by task difficulty.

Historical matching is useful for discovery and prioritizing experiments. It is not sufficient for a high-confidence “same benefit” claim.

Paired evaluation

For eligible read-only, deterministic, or safely sandboxed tasks:

  1. Freeze the task fixture, harness, tools, inputs, and quality gates.
  2. Run the incumbent and candidate model independently.
  3. Grade both with the same deterministic and Agentic-QE evidence.
  4. Compare quality, latency, token use, cash/API-equivalent cost, and repair burden.
  5. Persist the paired provenance and outcome.

Paired replay is explicit opt-in when it consumes a provider API, duplicates meaningful work, or sends repository evidence to another vendor. Subscription execution remains the default where supported, but it still consumes plan allowance and must be described honestly.

Prospective cheap-first cascades

After Verified evidence exists, allow an opt-in route experiment:

  1. select the cheaper candidate for eligible operations;
  2. apply the normal gate;
  3. escalate through the configured bounded ladder on an eligible failure;
  4. include both attempts in the episode; and
  5. compute net value after escalation cost and latency.

The relevant metric is not “cheap-model success rate” alone. It is:

net route value = accepted quality
                  − initial-route cost/capacity
                  − escalation cost/capacity
                  − repair/retry burden
                  − latency penalty

Recommendation rule

Recommend a cheaper candidate only when:

  1. the activity has enough qualified evidence;
  2. the candidate's conservative quality bound is within the configured non-inferiority tolerance of the incumbent;
  3. the result survives matching or paired/prospective validation;
  4. retry, repair, and escalation do not erase the benefit;
  5. cost/capacity basis is known and shown;
  6. the evidence is fresh for the current model and harness versions; and
  7. uncertainty is below the configured ceiling.

Selection should use Ruflo's MetaHarness-backed router semantics: predict candidate quality and choose the cheapest candidate clearing the quality bar, with the active subscription/headroom policy applied as a separate overlay.

User interactions and scenarios

Scenario 1 — Discover a cheaper equivalent

  1. The user opens Dashboard → Usage → Findings.

  2. A “Cheaper equivalent found” card names the activity, incumbent, candidate, estimated effect, cost basis, evidence grade, sample count, and freshness.

  3. Expanding “Why?” shows matched/paired evidence, quality tolerance, escalation rate, and excluded confounders.

  4. The card offers a copyable command such as:

    ak host pick --route 'documentation:codex:gpt-5.3-codex'

  5. The Dashboard does not execute the command or modify policy.

Scenario 2 — Confirm that a premium model earns its place

The user sees that architecture or security-analysis work has a materially better gate/review outcome and lower rework rate on the premium route. The finding recommends keeping the current route and explains the evidence rather than presenting all premium usage as waste.

Scenario 3 — Measure a cheap-first cascade

The user enables a bounded experiment for documentation. The lower tier completes most eligible work, some attempts escalate, and the finding reports the net capacity/cost effect after those escalations.

Scenario 4 — Refuse to guess

The testing activity has many sessions but few artifact-backed outcomes. The Dashboard says “Insufficient quality evidence” and identifies what would improve confidence, such as recording test/QE verdicts from ak run.

Scenario 5 — Separate harness improvement from model improvement

Quality rises after a prompt/tool-policy or harness change across several models. The finding attributes the change to the harness cohort and avoids recommending a model switch.

Scenario 6 — Respect subscription headroom

Claude and Codex candidates have similar measured quality. One host is near its weekly limit while the other has headroom. The recommendation explains that it optimizes capacity pressure, not cash billing, and remains bounded to the current limit window.

Scenario 7 — Inspect evidence from an individual operation

From a finding, the user can navigate to the relevant retained session/operation evidence when available. Transcript access continues to use the existing masked, selected-session content plane; aggregate APIs do not acquire transcript bodies.

Dashboard experience

Keep the existing three primary areas and extend Usage → Findings. Add a “Model fit” filter or finding family, not a fourth primary area.

Recommended finding types:

  • Cheaper equivalent found — candidate clears the quality bar with Correlated, Verified, or Proven evidence.
  • Premium route justified — stronger model materially improves quality or reduces repair/escalation.
  • Cheap-first cascade working — net benefit remains positive after escalation.
  • Escalation tax erased the saving — cheap-first routing costs more or delays work after retries.
  • Harness change drove the gain — improvement is shared across models.
  • Routing blind spot — insufficient outcome coverage.
  • Evidence drift — recommendation was learned on stale model/harness versions.

Each card should show:

  • activity and project scope;
  • incumbent and candidate host/provider/model identities;
  • recommendation status and evidence grade;
  • quality metric and non-inferiority tolerance;
  • matched, paired, and prospective sample counts;
  • cost/capacity basis and estimated net effect;
  • retry, repair, and escalation rates;
  • freshness/model/harness versions;
  • “Why this finding?” evidence disclosure;
  • exact copyable route command where actionable; and
  • “No $ claimed” when cost is unknown or the finding is capacity-only.

A secondary activity × model matrix may show sample count, gate pass rate, rework rate, escalation rate, latency, cost/capacity, and evidence grade. It must not collapse hosts or providers into model names.

Phased technical implementation plan

Phase 0 — Specification, ADR, and domain contract

Deliverables

  • Add a new Proposed ADR for Model Intelligence.
  • Extend docs/ddd/ubiquitous-language.md and docs/ddd/context-map.md with OperationEpisode, OutcomeEvidence, HarnessFingerprint, EvidenceGrade, and RouteRecommendation.
  • Define quality bars, non-inferiority tolerances, minimum evidence, staleness, and attribution windows as explicit policy.
  • Define the privacy boundary and opt-in write/replay modes.
  • Record how the new context amends ADR-0009 and relies on ADR-0021 without changing their current status prematurely.

Exit gate

  • Accepted schema and evidence vocabulary.
  • No contradiction with the implemented Dashboard/Usage contract.
  • Representative episode fixtures for all three hosts.

Phase 1 — Read-only historical episode projection

Likely files

  • src/lib/usage-index.mjs
  • src/lib/usage-opencode.mjs
  • src/lib/usage-classify.mjs
  • new pure modules under src/lib/model-intelligence/
  • transcript fixtures and usage-index tests

Deliverables

  • Derive coarse episodes from completed Claude, Codex, and OpenCode sessions.
  • Carry host, provider provenance, model, reasoning effort when observed, project/revision, activity confidence, usage, duration, and harness fingerprint.
  • Bump the usage-index schema when persisted record shape changes.
  • Produce a rebuildable local projection with coverage and missing-evidence diagnostics.

Exit gate

  • Historical projection is deterministic and idempotent.
  • Unknown fields remain unknown.
  • OpenRouter account analytics remains outside session/model outcomes without a correlation key.
  • No learned recommendations appear yet.

Phase 2 — Prospective ak run outcome instrumentation

Likely files

  • src/commands/run.mjs
  • src/lib/execution/runner.mjs
  • src/lib/execution/schema.mjs
  • src/lib/execution/handoff.mjs
  • Ruflo outcome/trajectory integration seam

Deliverables

  • Emit activity-bounded episodes for each worker and attempt.
  • Record selected route, harness fingerprint, structured result, duration, handoff, retry/escalation trail, and gate references.
  • Attach deterministic and Agentic-QE evidence when available.
  • Write qualified decisions/outcomes through Ruflo's actual trajectory/outcome-learning surface behind an explicit learning-write opt-in.
  • Preserve subscription execution as the default; do not request provider keys for normal runs.

Exit gate

  • One successful, one failed, and one escalated worker produce complete, provenance-correct episodes.
  • Permission/auth/orphaned states never become quality failures or trigger unsafe replay.
  • Duplicate/replayed subagent evidence is not double-counted.

Phase 3 — Evidence evaluator and matched cohorts

Likely files

  • new pure matching/statistics modules under src/lib/model-intelligence/
  • src/lib/usage-insights.mjs
  • focused unit and property tests

Deliverables

  • Match episodes by activity, project/task similarity, complexity, harness, tools, and time/model generation.
  • Calculate outcome coverage, balance/confounding diagnostics, confidence intervals, repair/escalation rates, and cost/capacity effects.
  • Implement evidence-grade promotion rules.
  • Produce internal recommendation candidates, including an explicit insufficient-evidence result.

Exit gate

  • Synthetic confounded corpora do not produce a false cheaper-equivalent recommendation.
  • Thin cohorts regress to unknown.
  • Stable reruns return byte-equivalent results for the same input.

Phase 4 — MetaHarness-backed model prediction and feedback loop

Deliverables

  • Transform qualified episodes into the labelled row shape expected by Ruflo's MetaHarness-backed router.
  • Use the real router's per-candidate predicted quality, quality-bar, alternatives, calibration, and uncertainty surfaces.
  • Keep the agentic-kit subscription/headroom overlay separate from learned model quality.
  • Record accepted/rejected recommendations and subsequent outcomes so the loop learns rather than merely reports.
  • Surface router backend/provenance; never infer that MetaHarness ran because an import was present.

Exit gate

  • The same labelled corpus produces reproducible candidate predictions.
  • The recommended candidate is the cheapest known candidate clearing the quality bar.
  • Backend unavailable/gated paths degrade to “no learned recommendation,” not a renamed heuristic.

Phase 5 — Dashboard findings and evidence disclosure

Likely files

  • src/lib/dashboard-server.mjs
  • src/lib/dashboard/client.mjs
  • src/lib/dashboard/styles.mjs
  • src/lib/usage-insights.mjs
  • tests/dashboard.test.cjs

Deliverables

  • Add the Model fit finding family to Usage → Findings.
  • Add evidence disclosure, activity/model matrix, freshness, cost basis, and copyable route command.
  • Keep transcript bodies out of aggregate routes and continue masking selected-session content.
  • Keep all actions explicit and reversible outside the Dashboard.

Exit gate

  • Keyboard and screen-reader navigation remain intact.
  • No external browser fetches are introduced.
  • Every numerical claim is traceable to an evidence record and cost basis.
  • Insufficient evidence renders clearly without an action command.

Phase 6 — Paired evaluation and bounded route experiments

Deliverables

  • Add explicit opt-in commands to plan and run paired model evaluations on eligible fixtures.
  • Use Agentic-QE to generate/execute measurable gates and assess coverage, security, and regressions.
  • Add bounded cheap-first experiments with configured escalation ladders.
  • Report net benefit after retry/escalation cost, latency, repair, and quota pressure.
  • Require confirmation for any metered provider execution or cross-vendor evidence disclosure.

Exit gate

  • Paired evidence is reproducible from a frozen fixture.
  • A deliberately weaker candidate fails the quality gate and is not recommended.
  • An escalation-heavy route correctly reports that its apparent saving was erased.
  • No route is auto-applied.

Phase 7 — Documentation, ADR reconciliation, and release proof

Deliverables

  • Update ADR-0009's Updated date/note when Findings behavior changes.
  • Update ADR-0021 only if provider/model provenance behavior changes.
  • Move the new ADR through Proposed → Accepted → Implemented only as corresponding code and proof land.
  • Update docs/USAGE-SCORECARD-METRICS.md, docs/TRANSCRIPTS.md, Dashboard documentation, and troubleshooting.
  • Attach exact test, QE, privacy, and release evidence before closure.

Exit gate

  • ADRs describe the shipped code exactly.
  • Full pnpm run check passes.
  • Agentic-QE evidence and immutable release proof are linked.

Security privacy and offline-first constraints

  1. Normal Dashboard reads remain network-silent under ADR-0009.
  2. Transcript bodies remain in the selected-session content plane and retain masking/truncation guards.
  3. Derived stores contain bounded fingerprints/features and evidence references, not raw prompts, tool payloads, secrets, or reasoning traces.
  4. Task embeddings and harness fingerprints are not exposed through aggregate Dashboard APIs.
  5. Provider/model identity preserves field-level provenance and never upgrades inferred/configured evidence to observed.
  6. Learning writes are opt-in and foreground; no hidden daemon is introduced.
  7. Paired replay is opt-in, names the execution host/provider boundary, and requests confirmation before metered or cross-vendor execution.
  8. No credentials are copied into episode records, Router rows, Dashboard payloads, or kit.json.
  9. Retention, rotation, deletion, and rebuild behavior are documented and tested.
  10. Findings are data, never HTML; existing escaping and CSP rules remain in force.

Risks and mitigations

Risk Consequence Mitigation
Selection bias: hard tasks go to strong models Cheap model appears better on easy work Match on task/complexity/harness; require paired/prospective evidence for strong claims
Harness/host confounding Model receives credit for better tools/prompts Carry harness fingerprint and host/tool availability in every episode
Sparse outcomes Unstable recommendations Confidence bounds, shrinkage, minimum evidence, explicit “no recommendation”
Self-reported success contaminates labels Router learns confidence rather than correctness Artifact/QE/reviewer/user evidence hierarchy; “done” has zero label weight
Repair attribution is wrong Later unrelated work penalizes a model Bounded attribution window plus project/task/revision correlation
Escalation hides cheap-route failure Apparent saving is overstated Include every attempt and escalation tax in net value
Subscription cost is misrepresented Dashboard claims fake dollar savings Separate observed cash, API-equivalent cost, and quota pressure
Model/version drift Stale recommendation survives a model change Model/harness version cohorts, freshness badges, automatic evidence expiry
Raw transcript leakage Sensitive content reaches aggregate APIs/stores Derived features only; retain selected-session security boundary
Learned router unavailable Local heuristic is mislabeled MetaHarness Fail closed to no learned recommendation; expose backend provenance
OpenCode/provider identity conflation Invalid vendor/cost comparison Preserve ADR-0021 and issue #95's host/provider separation

Acceptance criteria

Domain and evidence integrity

  • OperationEpisode, OutcomeEvidence, EvidenceGrade, HarnessFingerprint, and RouteRecommendation are defined in DDD/ADR documentation.
  • Host, inference provider, provider provenance, model, reasoning effort, and harness remain separate fields.
  • Every quality and cost claim carries provenance, timestamp/freshness, and evidence grade.
  • Historical, paired, and prospective evidence are distinguishable in the API and UI.
  • Assistant self-reported completion cannot independently create a success label.
  • Insufficient/confounded evidence produces no model recommendation.

Historical and prospective ingestion

  • Claude, Codex, and OpenCode fixtures yield deterministic episodes.
  • Structured ak run workers yield activity-bounded episodes with attempts and escalation.
  • Subagent replay/double-count protections remain intact.
  • Schema-version migration invalidates incompatible cached records.
  • OpenRouter account analytics remains unjoined without a correlation key.

Recommendation correctness

  • Recommendations compare compatible activity/task/complexity/harness cohorts.
  • A cheaper candidate must clear the configured conservative quality bound.
  • Retry, repair, escalation, latency, and capacity/cost are included in net effect.
  • Ruflo's real MetaHarness-backed router supplies learned per-candidate quality.
  • Backend absence or insufficient labels yields no learned recommendation.
  • Accepted/rejected recommendations can feed qualified outcome learning.

Dashboard and UX

  • Route Intelligence appears within Usage → Findings, not as a fourth primary area.
  • Findings show evidence grade, sample counts, uncertainty, freshness, and cost basis.
  • Findings disclose incumbent/candidate host, provider, model, and harness scope.
  • Actionable findings provide a copyable explicit route command.
  • The Dashboard never mutates routing policy.
  • Aggregate routes never expose raw transcript text or embeddings.
  • Keyboard, responsive, and accessibility contracts remain intact.

Security and operations

  • Normal Dashboard operation performs no external fetch.
  • Learning writes and replay remain explicit opt-ins.
  • Metered or cross-vendor evaluation requires confirmation.
  • No secret or credential reaches derived stores or Dashboard payloads.
  • Retention, rotation, rebuild, and deletion behavior is tested.
  • Full project checks and Agentic-QE gates pass before implementation closure.

Documentation and closure evidence

  • New ADR is accepted before learned recommendations ship.
  • ADR-0009 is updated when the Findings contract changes.
  • ADR-0021 remains accurate or is updated in the same change that alters provenance behavior.
  • Metrics, transcript, Dashboard, provider, and troubleshooting docs are updated.
  • Closure links the implementation PR, exact-head CI, Agentic-QE evidence, paired-evaluation proof, privacy proof, and release seal.

Success measures

The feature should report these measures without imposing arbitrary universal targets:

  • percentage of operations with activity identity;
  • percentage with qualified outcome evidence;
  • percentage eligible for matched comparison;
  • number of Verified and Proven recommendations by activity;
  • calibration/error of predicted quality against later outcomes;
  • net API-equivalent cost effect where priced;
  • net observed cash effect where metered;
  • capacity shifted between subscription pools;
  • cheap-first success, escalation, and repair rates;
  • number of recommendations withheld for insufficient/confounded evidence; and
  • recommendation acceptance/rejection and post-change outcome.

The “withheld” count is a trust metric, not a failure metric: a system that never says “I do not know” is not evidence-backed.

Open product decisions

  1. Should quality tolerance be global, per activity, or both?
  2. Which activities are safe for paired replay by default, if any?
  3. What attribution window should connect a later repair/revert to an earlier episode?
  4. Should explicit user feedback be collected from the CLI, Dashboard, or both?
  5. What minimum matched/paired evidence permits each grade transition?
  6. How should reasoning effort be normalized across hosts that expose different controls?
  7. Should recommendations scope first to one project, one machine, or a hierarchical project→machine fallback?
  8. How long should a recommendation remain fresh after a model, host, tool, or harness version changes?
  9. Should capacity-pressure recommendations expire automatically at the next provider limit reset?
  10. Which paired-evaluation artifacts may be retained without preserving sensitive task content?

Related work and source grounding

Agentic Kit

Ruflo / MetaHarness implementation grounding

  • ruflo/v3/@claude-flow/cli/src/ruvector/neural-router.ts — cost-optimal candidate selection, per-model predicted quality, quality bar, alternatives, calibration, uncertainty, and backend provenance.
  • ruflo/v3/@claude-flow/cli/src/commands/neural.ts — router trajectory collection, training from trajectories, cost/counterfactual statistics, and operator inspection surfaces.
  • ruflo/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts — structured task success/quality feedback and learning persistence.

These are the real upstream learning/routing surfaces this feature should integrate. If a required upstream contract is missing, the implementation should expose that dependency rather than silently replacing it with an Agentic Kit heuristic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions