Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Its position does not move when the primary area changes; only the choices insid
| Overview | Hosts & Routing | `#overview/hosts` | Hosts & routing | Enabled execution hosts, activity assignments, primary-host policy, and escalation paths |
| Overview | Providers | `#overview/providers` | Inference providers | Provider bindings, availability, provenance, and configuration health |
| Overview | Runtime | `#overview/runtime` | Runtime health | Local services, MCP connections, processes, and operational readiness |
| Overview | Intelligence | `#overview/intelligence` | Intelligence & learning | Memory, learned patterns, quality feedback, and improvement signals |
| Overview | Intelligence | `#overview/intelligence` | Intelligence & learning | Memory, learned patterns, reasoning-graph growth, and improvement signals, updated near-live while the view is open |
| Usage | Scorecard | `#usage/score` | Usage scorecard | Token consumption, API-equivalent cost, efficiency, and trends |
| Usage | Limits | `#usage/limits` | Provider limits | Current provider windows, reset timing, and available capacity |
| Usage | Findings | `#usage/findings` | Usage findings | Actionable anomalies, efficiency opportunities, and evidence-backed recommendations |
Expand Down Expand Up @@ -61,7 +61,13 @@ Overview keeps status and routing in one health-first area:
of which inference provider served a particular session.
- **Providers** presents inference-provider bindings and their configuration provenance.
- **Runtime** presents operational services, processes, and MCP readiness.
- **Intelligence** presents memory, learning, and quality-improvement signals.
- **Intelligence** presents memory, learning, and quality-improvement signals: the neural pattern
store's current size, its separate lifetime patterns-learned counter, reasoning-graph growth, and
the route-learner's improvement delta. It reads files ruflo/agentic-qe already write under
`.claude-flow/` and updates near-live over its own SSE stream while the view is open, falling back
to the general status poll otherwise. See [Project intelligence](ddd/project-intelligence.md) and
[ADR-0024](adr/0024-project-intelligence-telemetry.md) for the full model and the two learning
metrics' load-bearing distinction.

## Usage

Expand Down
164 changes: 164 additions & 0 deletions docs/adr/0024-project-intelligence-telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# ADR-0024 — Project intelligence: live learning telemetry from ruflo/agentic-qe's own state

- **Status:** Implemented
- **Date:** 2026-08-05
- **Deciders:** agentic-kit maintainers
- **Related:** [ADR-0005](0005-dashboard-in-page-routing-reveal.md),
[ADR-0009](0009-usage-scorecard-local-transcript-analytics.md),
[ADR-0012](0012-observability.md)

## Context

Overview's **Intelligence** destination (`#overview/intelligence`, added by
[ADR-0005](0005-dashboard-in-page-routing-reveal.md)'s 2026-08-04 information-architecture
amendment) has always advertised "memory, learned patterns, quality feedback, and improvement
signals." Until now its "learning over time" strip rendered only two sparklines:

- **patterns learned**, sourced from `.claude-flow/health-history.json` — a ring `dashboard-server.mjs`
itself appended to, and only while a dashboard happened to be running to observe
`.claude-flow/neural/stats.json`. On a machine (or CI checkout) where the dashboard had never
polled long enough to accumulate a ring, this file simply did not exist and the panel showed
`no data`;
- **improvement Δpp**, sourced from the route-learner's existing `.claude-flow/improvement.json`
(unchanged by this decision).

Meanwhile ruflo and agentic-qe were already writing two richer, always-present sources that the
dashboard never read: `.claude-flow/neural/patterns.json` (the neural pattern store — a live JSON
array of pattern entries, each carrying `createdAt`/`type`) and
`.claude-flow/data/intelligence-snapshot.json` (point-in-time samples of the reasoning/knowledge
graph's size: `nodes`, `edges`, `pageRankSum`). `src/commands/status.mjs`'s CLI `learning` row
already reads `.claude-flow/neural/stats.json` for the same lifetime `patternsLearned` counter the
health ring's samples happened to carry — but the *store's own current inventory* (how many
patterns are on disk right now, which shrinks under pruning/compaction even as the lifetime
counter only ever climbs) had no reader at all. The panel's "no data" state was therefore
avoidable, not fundamental: real trend data already existed on disk.

Separately, every value on this panel only ever refreshed on the dashboard's general ~30s
`/api/status` poll (the same cadence used for unrelated subsystem-health cards), so a user
actively watching learning happen — a pattern being stored, the graph growing — waited up to that
long to see it.

This telemetry is a different kind of fact than anything [ADR-0012](0012-observability.md)/
[Observability](../ddd/observability.md) models. It carries no session, actor, host, provider, or
model identity; no lifecycle (`queued → running → completed`); no per-field evidence confidence
(`observed`/`correlated`/`inferred`/`assumed`/`planned`); and no transcript content requiring
redaction. It is four scalar/array-shaped reads over this project's own `.claude-flow/` state —
the same local trust boundary `ak status` already reads directly, with no host-specific
anti-corruption adapter needed because there is only ever one shape, ak's own. The panel that
surfaces it also lives under **Overview**, not under **Observability**'s Live/History scope. The
implementation deliberately did not construct an `ObservedSession`, did not flow through the
canonical event normalizer, and did not extend the replay/snapshot cursor — it reuses only
source-agnostic transport plumbing (`JsonlTailer`, `sseChannel`, `reserveClientSlot`/`clientGone`,
`transcriptSseFrame`) that Dashboard delivery already shares across contexts.

## Decision

### 1. A new bounded context: Project intelligence

Project intelligence is its own bounded context (see the updated
[context map](../ddd/context-map.md) and [Project intelligence](../ddd/project-intelligence.md)),
not an Observability extension and not folded into ADR-0005's navigation-shell amendments. Its
model, invariants, and the reasoning for keeping it separate from `ObservedSession` are specified
in that document; this ADR records the decision and its consequences.

### 2. One read-only history module composes four existing sources

`src/lib/dashboard/intel-history.mjs` adds:

- `readNeuralPatternStoreHistory(cwd)` — every entry currently on disk in
`.claude-flow/neural/patterns.json`, as `{ createdAt, type }`;
- `readGraphHistory(cwd)` — every sample in `.claude-flow/data/intelligence-snapshot.json`, as
`{ timestamp, nodes, edges, pageRankSum }`;
- `readGlobalStats(cwd)` — the current cumulative counters in `.claude-flow/neural/stats.json`
(`patternsLearned`, `trajectoriesRecorded`, `signalsProcessed`, `lastAdaptation`), reading via the
same `readJson` helper and `?? 0` defaulting `status.mjs`'s `learning` row already uses, so the two
call sites cannot drift apart;
- `readHealthRing(cwd)` and `appendHealthSnapshot(cwd, snapshot)` — moved (not duplicated) from
`dashboard-server.mjs`, unchanged behavior, now capped at 500 samples with field-level dedup
(a repeated poll of unchanged stats writes nothing);
- `readIntelHistory(cwd)` — the combinator `collectData()` and the SSE route both call, returning
`{ patternStore, graph, healthRing, globalStats }`.

The **patterns-learned counter** (`globalStats.patternsLearned`, a lifetime total) and the
**pattern-store size** (`patternStore.length`, entries actually present right now) are
independent metrics that may legitimately diverge as the store is pruned or compacted. This
project's own repository demonstrates the divergence today: 28 live pattern-store entries against
a 1,337 lifetime counter. Every reader, doc comment, and rendered label keeps the two separate;
none averages, sums, or substitutes one for the other.

### 3. Push updates over a new SSE route, additive to the existing poll

`src/lib/live/intelligence-watch.mjs` adds `IntelligenceWatch`, which polls the three source files'
`mtime` every second (default), corroborated by a change-only tail of
`.claude-flow/data/pending-insights.jsonl` (via the existing `JsonlTailer`, whose line *contents*
are never read — a record arriving at all is the signal), and flushes a debounced
(2.5s trailing-edge, measured from the most recent detected change) `onUpdate(readIntelHistory(cwd))`
call. `dashboard-server.mjs` exposes this as `GET /api/live/intelligence`: one `event: init` frame
with the current `readIntelHistory(cwd)` on connect, then an `event: update` frame per flush,
fanned out to every connected client. It reuses Dashboard delivery's proven SSE discipline —
reservation-before-await client-cap tracking, the forwarding-cleanup idiom, `sseChannel` backpressure
and heartbeat — but sends `transcriptSseFrame` payloads, not `sseFrame`'s session-privacy-redacted
ones, because this payload was never session or transcript content in need of that redaction
pipeline.

`GET /api/live/intelligence` shares Dashboard delivery's transport primitives with, but is not part
of, [Observability](../ddd/observability.md)'s `/api/live`, `/api/live/events`,
`/api/live/transcripts/:host/:id/events`, and `/api/live/playback/:host/:id` family documented in
[OBSERVABILITY.md](../OBSERVABILITY.md); it is not covered by that document's evidence, privacy, or
capability-coverage contract, and unlike Observability's ruflo/agentic-qe sources, it requires no
`--live-source` registration — the four files it reads are always this project's own.

### 4. `/api/status` keeps working as the fallback path

The SSE route is additive. `collectData()`'s existing return gains `globalStats`, `patternStore`,
and `graph` alongside the unchanged `health` field (still `intel.healthRing`); a client without
`EventSource` support, or one that has not yet opened the stream, still gets the full picture on the
next poll. The `/api/status` error-fallback payload was extended with the same three keys
(`globalStats: null, patternStore: [], graph: null`) so its shape never diverges from the success
path.

## Consequences

### Positive

- The Intelligence panel shows real trend data — pattern-store growth and reasoning-graph growth —
sourced from files that already existed, at zero new collection cost and no new write path beyond
the existing, now-relocated `appendHealthSnapshot`.
- Users watching active learning see updates within the debounce window (≤ ~3.5s) instead of waiting
out the general status poll.
- The lifetime-counter-vs-store-size divergence is now visible and labeled instead of silently
absent or conflated.
- A genuinely different domain got its own bounded context instead of stretching
`ObservedSession`'s Session/Actor/Activity model, or ADR-0005's navigation-shell decision, to cover
facts neither was designed to grade.

### Negative

- A second `/api/live/*`-prefixed SSE endpoint and client-cap surface to operate, alongside
`/api/live/events` and `/api/live/transcripts/...`.
- `mtime`-based polling can in principle miss two rewrites that land on the same filesystem-reported
millisecond; the `pending-insights.jsonl` tail is a second, independent trigger but not a formal
guarantee.
- No per-field confidence grading exists for this data, unlike Observability's evidence model —
accepted because every source is this project's own local file, not another host's evidence
requiring provenance.

### Risks and mitigations

| Risk | Mitigation |
|------|------------|
| Pattern-store size and the lifetime counter get conflated in a future edit or display | Documented domain invariant, shared header comment across both modules, and separate rendered figures with distinct captions |
| Debounce coalesces a burst into a stale window | Trailing-edge debounce measured from the most recently detected change, not a fixed interval |
| A reader is called on this project's own untrusted-shaped JSON (a partially written file, a schema drift) | Malformed or non-array data degrades to `[]`/`null`, matching `readJsonSafe`'s existing null-on-absent convention; individual malformed entries are skipped rather than throwing |
| New SSE route reintroduces a client-cap race | Reuses the same `reserveClientSlot`/forwarding-cleanup idiom already hardened for `/api/live/events` |
| Readers drift from `status.mjs`'s CLI `learning` row over time | `readGlobalStats` calls the same `readJson` helper and `?? 0` default `status.mjs` uses, not a hand-copied reimplementation |

## References

- `src/lib/dashboard/intel-history.mjs`, `tests/kit/intel-history.test.mjs`
- `src/lib/live/intelligence-watch.mjs`, `tests/kit/intelligence-watch.test.mjs`
- `src/lib/dashboard-server.mjs` (`collectData`, `GET /api/live/intelligence`, `lazyIntelWatch`)
- `src/lib/dashboard/client.mjs`, `src/lib/dashboard/page.mjs` (Intelligence panel rendering, SSE
subscription)
- [Project intelligence domain](../ddd/project-intelligence.md)
- [Dashboard guide](../DASHBOARD.md)
10 changes: 10 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Consequences**, and cites the grounded source it rests on where relevant.
| [0021](0021-inference-provider-provenance.md) | Inference-provider provenance for live sessions | Accepted |
| [0022](0022-metaharness-as-optional-assurance-companion.md) | MetaHarness as an optional assurance companion | Proposed |
| [0023](0023-fail-closed-operations-and-explicit-degradation.md) | Fail-closed mutations and explicit degraded operation evidence | Implemented |
| [0024](0024-project-intelligence-telemetry.md) | Project intelligence: live learning telemetry from ruflo/agentic-qe's own state | Implemented |

Theme: ADRs **0001–0006** define **dual-host LLM routing and leadership** — how `ak` lets ruflo route
each development activity (architecture, implementation, testing, review, …) to the right host (Claude
Expand Down Expand Up @@ -121,3 +122,12 @@ managed fallbacks report degradation, SQLite retains classified failure evidence
usage, promised backups fail closed before atomic replacement, status-line failures gain redacted
opt-in diagnostics, process discovery is current-user and argv-minimized, setup discloses and
verifies its project auto-approve manifest, and clean-machine tests isolate every mutable path.

**0024** gives Overview's Intelligence view real trend data instead of a permanently-empty strip:
a new `intel-history.mjs` module reads the neural pattern store, its lifetime learned-pattern
counter, and reasoning-graph size samples that ruflo/agentic-qe already write under
`.claude-flow/`, while a debounced `IntelligenceWatch` pushes near-instant updates over a new
`GET /api/live/intelligence` SSE route additive to the existing status poll. It establishes Project
intelligence as its own bounded context rather than an Observability extension — this telemetry
carries no session, actor, host, provider, or lifecycle identity and needs no per-field evidence
confidence, and the panel it feeds lives under Overview, not Observability's Live/History scope.
1 change: 1 addition & 0 deletions docs/ddd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ describe the current system unless a section is explicitly marked as future work
| [Integration management](integration-management.md) | Hosts, inference providers, bindings, capabilities, lifecycle, facts, and ownership |
| [Routing and orchestration](routing-and-orchestration.md) | Activities, routes, leadership, escalation, projections, and canonical `ak run` execution |
| [Observability](observability.md) | Evidence acquisition, observed-session aggregates, replay, and dashboard delivery |
| [Project intelligence](project-intelligence.md) | Pattern store, learning counters, reasoning-graph size, and live delivery for Overview's Intelligence view |

## Relationship to other documentation

Expand Down
22 changes: 20 additions & 2 deletions docs/ddd/context-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ Native Evidence ----> Evidence Acquisition ----> Canonical Evidence
| +----> Workspace Snapshot Cache |
| | |
+-----------------------+----> Dashboard Delivery <----+
^
Project State (.claude-flow/*) ----> Project Intelligence ┘

Maintainer Administration is a separate, deliberately-egressing context.
```
Expand Down Expand Up @@ -74,11 +76,22 @@ an advisory read-model cache, not the append-only Evidence Archive and not a sou
Restoration supplies inert History context using the original capture time; it cannot query a
current checkout and present that state as historical.

### Project intelligence

Owns read-only trend projections over ruflo/agentic-qe's own project-level learning state: the
neural pattern store, its lifetime learned-pattern counter, reasoning-graph size samples, and the
machine-health sample ring. It reads `.claude-flow/*` files directly — there is only ever one
shape, ak's own, so no anti-corruption adapter is required — and is independent of Evidence
Acquisition and Observability's canonical event model. It carries no session, actor, host,
provider, or lifecycle identity and grades no per-field evidence confidence.

See [Project intelligence](project-intelligence.md).

### Dashboard delivery

Owns protected HTTP/SSE delivery, browser DTOs, filters, presentation, and interaction state. It
may combine read models from Observability, Historical Usage, routing, and integration facts. It
cannot manufacture or strengthen domain facts.
may combine read models from Observability, Historical Usage, Project Intelligence, routing, and
integration facts. It cannot manufacture or strengthen domain facts.

### Maintainer administration

Expand All @@ -99,6 +112,8 @@ and credential policy is distinct from the offline-first dashboard and integrati
| Observability | Workspace snapshot cache | Last safe metadata-only session workspace capture |
| Workspace snapshot cache | Dashboard delivery | Inert last-recorded History context after restart |
| Historical usage | Dashboard delivery | Historical aggregates and findings |
| Project state (`.claude-flow/*`) | Project intelligence | Direct local file reads; no anti-corruption adapter needed |
| Project intelligence | Dashboard delivery | Read-model projection, delivered by poll (`/api/status`) and SSE push (`/api/live/intelligence`) |

## Boundary rules

Expand All @@ -108,3 +123,6 @@ and credential policy is distinct from the offline-first dashboard and integrati
- Dashboard presentation cannot upgrade provenance.
- Historical usage and live topology share identifiers, not aggregate ownership.
- Network egress occurs only in commands and contexts whose contract explicitly permits it.
- Project intelligence reads local project state directly; it never enters Evidence Acquisition's
anti-corruption layer or Observability's canonical event model, and it establishes no session,
actor, host, provider, or lifecycle identity.
Loading
Loading