diff --git a/docs/DASHBOARD.md b/docs/DASHBOARD.md index 73bcd13..f941b0a 100644 --- a/docs/DASHBOARD.md +++ b/docs/DASHBOARD.md @@ -23,7 +23,7 @@ Its position does not move when the primary area changes; only the choices insid | Overview | Hosts & Routing | `#overview/hosts` | Hosts & routing | Enabled execution hosts, activity assignments, primary-host policy, and escalation paths | | Overview | Providers | `#overview/providers` | Inference providers | Provider bindings, availability, provenance, and configuration health | | Overview | Runtime | `#overview/runtime` | Runtime health | Local services, MCP connections, processes, and operational readiness | -| Overview | Intelligence | `#overview/intelligence` | Intelligence & learning | Memory, learned patterns, reasoning-graph growth, and improvement signals, updated near-live while the view is open | +| Overview | Intelligence | `#overview/intelligence` | Intelligence & learning | Machine-wide learning rollup across every ruflo-initialized project, plus near-live detail for one explicitly selected project | | Usage | Scorecard | `#usage/score` | Usage scorecard | Token consumption, API-equivalent cost, efficiency, and trends | | Usage | Limits | `#usage/limits` | Provider limits | Current provider windows, reset timing, and available capacity | | Usage | Findings | `#usage/findings` | Usage findings | Actionable anomalies, efficiency opportunities, and evidence-backed recommendations | @@ -61,13 +61,18 @@ Overview keeps status and routing in one health-first area: of which inference provider served a particular session. - **Providers** presents inference-provider bindings and their configuration provenance. - **Runtime** presents operational services, processes, and MCP readiness. -- **Intelligence** presents memory, learning, and quality-improvement signals: the neural pattern - store's current size, its separate lifetime patterns-learned counter, reasoning-graph growth, and - the route-learner's improvement delta. It reads files ruflo/agentic-qe already write under - `.claude-flow/` and updates near-live over its own SSE stream while the view is open, falling back - to the general status poll otherwise. See [Project intelligence](ddd/project-intelligence.md) and - [ADR-0024](adr/0024-project-intelligence-telemetry.md) for the full model and the two learning - metrics' load-bearing distinction. +- **Intelligence** presents memory, learning, and quality-improvement signals machine-wide: an + always-visible rollup folded across every ruflo-initialized project on this machine, plus detail + for one explicitly selected, explicitly labeled project — the neural pattern store's current + size, its separate lifetime patterns-learned counter, and reasoning-graph growth. Project + selection defaults to whichever discovered project was most recently active; there is no implicit + current-working-directory default. The route-learner's improvement delta remains scoped to the + dashboard's own launching project and is not part of project selection. Detail data reads files + ruflo/agentic-qe already write under `.claude-flow/` and updates near-live over a per-project SSE + stream while the view is open, falling back to the general status poll otherwise. See + [Project intelligence](ddd/project-intelligence.md) and + [ADR-0024](adr/0024-project-intelligence-telemetry.md) for the full model, the project-discovery + mechanism, and the two learning metrics' load-bearing distinction, now also at machine scope. ## Usage diff --git a/docs/adr/0024-project-intelligence-telemetry.md b/docs/adr/0024-project-intelligence-telemetry.md index 8becf7e..3f2e41b 100644 --- a/docs/adr/0024-project-intelligence-telemetry.md +++ b/docs/adr/0024-project-intelligence-telemetry.md @@ -2,11 +2,91 @@ - **Status:** Implemented - **Date:** 2026-08-05 +- **Updated:** 2026-08-05 +- **Update note:** Extended Intelligence from one project's telemetry, implicitly tied to the + dashboard server's own launching cwd, to a machine-wide catalog of every ruflo-initialized + project plus an explicitly selected, explicitly labeled detail project (defaulting to + most-recently-active). This is a deliberate clean-break redesign of `/api/status`'s `intel` + payload shape, made because the project is still on the pre-release `4.0.0-alpha` line with no + external compatibility guarantee — not a shape preserved for, or falling back to, the prior + single-project form. - **Deciders:** agentic-kit maintainers - **Related:** [ADR-0005](0005-dashboard-in-page-routing-reveal.md), [ADR-0009](0009-usage-scorecard-local-transcript-analytics.md), [ADR-0012](0012-observability.md) +**2026-08-05 machine-wide discovery amendment:** `src/lib/dashboard/project-discovery.mjs` adds +`discoverRuvfloProjects()`, unioning three sources into one deduplicated, most-recently-active-first +catalog of every project on this machine ruflo has genuinely initialized — a `.claude-flow/neural/` +subdirectory present, not merely a bare `.claude-flow/`. Source 1 reuses `registryWorkspaces()` from +`daemons.mjs` verbatim (imported, not reimplemented; given a bare `export` so it could be imported +at all — the one change made outside this new module) to walk +`~/.claude-flow/{ai-jobs.json,workspace-leases.json,repo-supervisors.json}`. Source 2 +cross-references Observability's own `WorkspaceSnapshotStore` +(`~/.config/agentic-kit/observability-workspaces.json`, [ADR-0012](0012-observability.md)) for any +record carrying a resolvable absolute path. That store's own privacy sanitizers reject one on +write and on read — `repositoryLabel` rejects any path separator, `directoryLabel` rejects anything +absolute-looking — so source 2 is structurally empty by that store's own design, not a gap; the +defensive per-record check nonetheless stays in place rather than being removed, so it starts +contributing automatically if that schema ever grows a real path field. This cross-reference is +discovery-only: it never supplies evidence, confidence, or any value this domain renders, only a +candidate project path, preserving the reasoning in +["Why this is a separate context, not Observability"](../ddd/project-intelligence.md#why-this-is-a-separate-context-not-observability). +`intel-history.mjs` adds `readMachineWideIntel(projects)`, folding `readIntelHistory()` across +every discovered project into one `{ totals, perProject }` rollup. At machine scope, exactly as at +single-project scope, the lifetime patterns-learned counter and the pattern-store's current size +remain two distinct, never-conflated sums: `totals.patternsLearnedLifetime` sums each project's +cumulative `globalStats.patternsLearned`; `totals.patternStoreEntries` sums each project's current +`patternStore.length`. A project whose read turns up missing or malformed data degrades that +project's row to nulls/zeros rather than aborting the whole machine-wide scan. + +**2026-08-05 discovery correction — real transcript content is source 3, and is the source that +actually matters in practice:** `daemons.mjs`'s own header comment assumes ruflo 3.28+ reliably +writes the central `~/.claude-flow/{ai-jobs,workspace-leases,repo-supervisors}.json` registry +files source 1 depends on. Verified against a real, actively-used development machine (ruflo +3.34.0): none of those three files exist, even though that machine has multiple real, +ruflo-initialized projects with populated `.claude-flow/neural/` trees, including a per-project +`.claude-flow/daemon.pid`/`daemon-state.json` proving a daemon genuinely ran there — so source 1 +returned an empty set, and with source 2 structurally empty by design (above), +`discoverRuvfloProjects()` returned `[]` on a machine where the feature had real, correct data to +find. Source 3 fixes this: it reads real absolute `cwd` values directly out of Claude and Codex +transcript content under `~/.claude/projects/**/*.jsonl` and `~/.codex/sessions/**/*.jsonl`, via +the exact `discoverJsonl()`/`bootstrapRecords()` functions `live-sessions-service.mjs` already +trusts for Observability — flat `record.cwd` for Claude, `record.payload.cwd` for Codex's +`session_meta`/`turn_context` records. Unlike source 2's persisted, privacy-sanitized registry, +raw transcripts are not sanitized and do carry a resolvable path — legitimately readable at this +trust boundary since it is the same user, same machine, and same files Observability itself +already parses. Bounded to the 150 most-recently-modified transcripts per host (`discoverJsonl`'s +own recency sort) so cost stays flat regardless of how many sessions a project has accumulated; +overlap with sources 1/2 dedups for free through the existing resolved-path merge. Verified on the +same real machine: `discoverRuvfloProjects()` now returns all 4 real ruflo-initialized projects in +under a second. + +**2026-08-05 selectable delivery amendment:** This is a deliberate clean-break redesign of +`/api/status`'s `intel` payload, not a shape preserved for, or falling back to, the prior +single-project form — warranted because the project is still on the pre-release `4.0.0-alpha` line +with no external compatibility guarantee yet. The former flat top-level `health`/`globalStats`/ +`patternStore`/`graph` fields are replaced by one nested `intel` object: `{ selectedProjectKey, +selectedProjectLabel, projects, health, globalStats, patternStore, graph, machineWide }`. +`projects` is the full machine-wide catalog (`{ key, label, path, source }` rows, +most-recently-active first); `machineWide` is always the full `readMachineWideIntel()` rollup, +independent of selection. Both `GET /api/status` and `GET /api/live/intelligence` accept the same +optional `?project=` query parameter — `` is the opaque `project:<16-hex>` form +`resolveProjectIdentity(path).key` emits, deliberately not `stableProjectKey`, which reduces its +input to a bare directory-name label before hashing and would collapse two same-named-but-different +projects onto one selection key — and resolve it through the identical shared +`resolveSelectedProject(projects, rawParam)` helper, so the two routes can never disagree about +what an absent or unresolvable key defaults to: `discoverRuvfloProjects()`'s own +most-recently-active-first sort, never the server's launching cwd. `GET /api/live/intelligence` +answers `503` if machine-wide discovery finds zero ruflo-initialized projects at all, rather than +picking anything. The prior server-wide singleton `IntelligenceWatch` is replaced by a small pool +keyed by resolved project path: a project's watcher is created lazily on its first SSE subscriber +and stopped and forgotten the moment its last subscriber for that project disconnects, so two +clients watching different projects never cross-talk and an unwatched project's watcher never runs +unbounded. **"This project" as an implicit, unlabeled, cwd-bound default no longer exists anywhere +in this contract** — the detail strip is always an explicitly selected, explicitly labeled project, +defaulting to whichever discovered project was most recently active. + ## Context Overview's **Intelligence** destination (`#overview/intelligence`, added by @@ -155,10 +235,14 @@ path. ## References -- `src/lib/dashboard/intel-history.mjs`, `tests/kit/intel-history.test.mjs` +- `src/lib/dashboard/intel-history.mjs` (`readIntelHistory`, `readMachineWideIntel`), + `tests/kit/intel-history.test.mjs` +- `src/lib/dashboard/project-discovery.mjs` (`discoverRuvfloProjects`), + `tests/kit/project-discovery.test.mjs` - `src/lib/live/intelligence-watch.mjs`, `tests/kit/intelligence-watch.test.mjs` -- `src/lib/dashboard-server.mjs` (`collectData`, `GET /api/live/intelligence`, `lazyIntelWatch`) -- `src/lib/dashboard/client.mjs`, `src/lib/dashboard/page.mjs` (Intelligence panel rendering, SSE - subscription) +- `src/lib/dashboard-server.mjs` (`collectData`, `buildProjectSnapshotCache`, + `resolveSelectedProject`, `GET /api/live/intelligence`'s `intelPool`) +- `src/lib/dashboard/client.mjs`, `src/lib/dashboard/page.mjs` (Intelligence panel rendering, + machine-wide rollup, project picker, SSE subscription) - [Project intelligence domain](../ddd/project-intelligence.md) - [Dashboard guide](../DASHBOARD.md) diff --git a/docs/ddd/project-intelligence.md b/docs/ddd/project-intelligence.md index a47df66..d0208f3 100644 --- a/docs/ddd/project-intelligence.md +++ b/docs/ddd/project-intelligence.md @@ -1,7 +1,16 @@ # Project Intelligence Domain This document describes the domain implemented by [ADR-0024](../adr/0024-project-intelligence-telemetry.md), -`src/lib/dashboard/intel-history.mjs`, and `src/lib/live/intelligence-watch.mjs`. +`src/lib/dashboard/intel-history.mjs`, `src/lib/dashboard/project-discovery.mjs`, and +`src/lib/live/intelligence-watch.mjs`. + +> **2026-08-05 amendment:** extended from one project's telemetry, implicit and bound to the +> dashboard server's own launching working directory, to a machine-wide catalog of every +> ruflo-initialized project on this machine, a machine-wide aggregate that is always shown, and an +> explicitly selected, explicitly labeled detail project (defaulting to most-recently-active). See +> [ADR-0024](../adr/0024-project-intelligence-telemetry.md)'s update note for the full amendment +> record, including why the `/api/status` payload shape is a clean break rather than a preserved +> fallback. ## Purpose @@ -9,9 +18,13 @@ Project intelligence surfaces trend data about ruflo/agentic-qe's own project-le subsystem — the neural pattern store, its lifetime learned-pattern counter, the reasoning graph's structural growth, and a machine-health sample ring — inside the dashboard's Overview → **Intelligence** view (`#overview/intelligence`). It is a read-only projection over files those -tools already write under `.claude-flow/`. It owns no session, actor, or activity identity; it -grades no per-field evidence confidence; and it cannot steer, retrain, or mutate the learning -subsystem it reads. +tools already write under `.claude-flow/`, discovered across every project on this machine ruflo +has genuinely initialized rather than read from one implicit location. The view always shows a +machine-wide aggregate folded across every discovered project, plus per-project detail for exactly +one explicitly selected, explicitly labeled project — defaulting to whichever discovered project +was most recently active, never the dashboard server's own launching working directory. It owns no +session, actor, or activity identity; it grades no per-field evidence confidence; and it cannot +steer, retrain, or mutate the learning subsystem it reads. The shared terms in [Ubiquitous language](ubiquitous-language.md) are normative. @@ -23,9 +36,10 @@ with per-field confidence (`observed`/`correlated`/`inferred`/`assumed`/`planned state machine, and a protected transcript-content plane. None of that applies here: - Every value in this domain is a scalar count, a timestamp, or a flat historical array read from - this project's own `.claude-flow/` state — the same local trust boundary `ak status` already - reads directly. There is no other host's evidence to normalize through an anti-corruption - adapter, because there is only ever one shape: ak's own. + a discovered project's own `.claude-flow/` state — whether one selected project's detail or the + machine-wide rollup folded across every discovered project — the same local trust boundary + `ak status` already reads directly. There is no other host's evidence to normalize through an + anti-corruption adapter, because there is only ever one shape: ak's own. - There is no session, actor, host, provider, or model identity anywhere in this domain's data, and therefore no capability-coverage matrix, no actor lens, and no court membership. - There is no lifecycle (`queued → running → completed`); sources are either a live inventory @@ -34,17 +48,69 @@ state machine, and a protected transcript-content plane. None of that applies he - The panel is a permanent secondary view under **Overview**, never a mode of **Observability**'s mutually exclusive Live/History scope (see [ADR-0005](../adr/0005-dashboard-in-page-routing-reveal.md)). +[Project discovery](#project-discovery) below cross-references Observability's own +`WorkspaceSnapshotStore` as a secondary source, but only for a candidate project *path* — never for +evidence, confidence, or any value this domain renders. That store's own privacy sanitizers make it +structurally incapable of yielding a resolvable absolute path today, so in practice this domain's +primary registry scan supplies the entire discovered catalog; either way, the boundary above is +unaffected, because a path is not evidence. + The implementation reuses only source-agnostic transport plumbing that Dashboard delivery already shares across contexts — `JsonlTailer`, `sseChannel`, `reserveClientSlot`/`clientGone`, and `transcriptSseFrame` — never Observability's canonical normalizer, `ObservedSession` aggregate, or replay/snapshot cursor. `GET /api/live/intelligence` shares the `/api/live/*` path prefix with Observability's endpoints by transport convention only; it is not covered by [OBSERVABILITY.md](../OBSERVABILITY.md)'s evidence, privacy, or capability-coverage contract, and it -needs no `--live-source` registration because its four sources are always this project's own. +needs no `--live-source` registration because its sources are always a discovered project's own, +never a remote or unregistered one. + +## Project discovery + +`discoverRuvfloProjects()` (`src/lib/dashboard/project-discovery.mjs`) returns every project on +this machine ruflo has genuinely initialized — a `.claude-flow/neural/` subdirectory present, not +merely a bare `.claude-flow/` (a project that only ever ran, say, `ruflo daemon start` without ever +training or learning anything is correctly excluded) — deduplicated by resolved absolute path and +sorted most-recently-active first by `.claude-flow/neural/stats.json`'s `lastAdaptation`. + +Three sources are unioned: + +1. **Registry.** `registryWorkspaces()`, reused verbatim from `daemons.mjs` (imported, not + reimplemented), walks `~/.claude-flow/{ai-jobs.json,workspace-leases.json,repo-supervisors.json}` + for every workspace path recorded there that carries a `.claude-flow` directory. `daemons.mjs`'s + own header comment assumes ruflo 3.28+ reliably writes these; verified false on a real ruflo + 3.34.0 machine with real, populated ruflo projects — none of the three files existed. Kept as a + source (cheap, and correct wherever those files do exist), but no longer described as the + guaranteed-correct primary. +2. **Observability cross-reference.** `WorkspaceSnapshotStore` (`src/lib/live/workspace-store.mjs`, + [Observability](observability.md)) is checked for any record whose workspace carries a + genuinely resolvable absolute path. That store's own privacy sanitizers — `repositoryLabel` + rejects any path separator, `directoryLabel` rejects anything absolute-looking — mean a real + record never carries one, so this source is structurally empty by that store's own design, not + a gap. The check remains a real, defensive one rather than being skipped outright, so it starts + contributing automatically if that schema ever grows a genuine path field. +3. **Transcript content (the source that matters in practice).** Real absolute `cwd` values read + directly out of Claude and Codex transcript content under `~/.claude/projects/**/*.jsonl` and + `~/.codex/sessions/**/*.jsonl`, via the same `discoverJsonl()`/`bootstrapRecords()` functions + `live-sessions-service.mjs` already trusts for Observability's own live session tracking — flat + `record.cwd` for Claude, `record.payload.cwd` for Codex's `session_meta`/`turn_context` records. + Unlike source 2's sanitized, persisted registry, raw transcripts are not sanitized and do carry + a resolvable path — legitimately readable at this trust boundary since it's the same user, same + machine, same files Observability already parses. Bounded to the 150 most-recently-modified + transcripts per host so cost stays flat regardless of session count. On the real machine where + source 1 returned nothing, this source alone found all 4 real ruflo-initialized projects. + +Each discovered row is `{ path, label, source }`, where `label` reuses Observability's own +`resolveProjectLabel` for the same path (falling back to the bare directory name) so a project +reads identically wherever it is named, and `source` is `'registry'`, `'observability'`, +`'transcript'`, or `'both'` when a project was found by two or more sources (not necessarily +exactly two). ## Model ```text +discoverRuvfloProjects() -> ProjectRow[] { path, label, source } (every ruflo-initialized + project on this machine) + .claude-flow/neural/patterns.json -> PatternStoreEntry[] { createdAt, type } .claude-flow/neural/stats.json -> GlobalLearningStats { patternsLearned, trajectoriesRecorded, signalsProcessed, lastAdaptation } @@ -53,14 +119,22 @@ needs no `--live-source` registration because its four sources are always this p .claude-flow/data/pending-insights.jsonl -> change signal only (line contents never read) .claude-flow/improvement.json -> ImprovementEval (pre-existing; unchanged by this domain) -readIntelHistory(cwd) -> { patternStore, graph, healthRing, globalStats } - | - +--> collectData() (Dashboard delivery) --> GET /api/status (poll, ~30s) +readIntelHistory(cwd) -> { patternStore, graph, healthRing, globalStats } (one project) +readMachineWideIntel(ProjectRow[]) -> { totals, perProject } (every project, folds readIntelHistory) + +resolveSelectedProject(ProjectRow[], ?project=) -> selected project + (explicit key match, else most-recently-active; shared by BOTH routes below) | - +--> IntelligenceWatch --> broadcastIntel --> GET /api/live/intelligence (SSE push, debounced) + +--> readMachineWideIntel(ProjectRow[]) -- always the full machine-wide rollup + +--> readIntelHistory(selected.path) -- detail for the selected project only + | + +--> collectData() (Dashboard delivery) --> GET /api/status?project= (poll, ~30s) | - v - Overview -> Intelligence (#overview/intelligence): five sparklines + improvement verdict badge + +--> per-project IntelligenceWatch pool --> GET /api/live/intelligence?project= (SSE push, debounced) + | + v + Overview -> Intelligence (#overview/intelligence): machine-wide rollup (always visible) + + project picker + five sparklines/improvement verdict badge for the selection ``` ### Pattern-store size vs. patterns-learned counter @@ -78,6 +152,25 @@ This repository's own `.claude-flow/` state demonstrates it directly: 28 pattern against a 1,337 lifetime counter. No reader, computation, or rendered label treats one as a substitute display for the other. +### Machine-wide rollup + +`readMachineWideIntel(projects)` folds `readIntelHistory()` across every project +`discoverRuvfloProjects()` returns into one `{ totals, perProject }` view. Exactly as at +single-project scope above, the lifetime counter and the current store size are never conflated — +now at machine scope too: + +- `totals.patternsLearnedLifetime` sums every project's cumulative `globalStats.patternsLearned`. +- `totals.patternStoreEntries` sums every project's current `patternStore.length`. + +`totals.mostActiveProject` is the label of whichever project has the highest positive +`globalStats.lastAdaptation` among projects that have one; it is `null` when no project has +adaptation data, matching `readGlobalStats`'s own `?? 0` "never adapted" default. A project whose +`readIntelHistory()` call turns up missing or malformed data degrades that project's `perProject` +row to nulls/zeros rather than aborting the whole scan — one bad project never hides every other +project's data. This is a plain on-demand scan with no aggregation-layer caching of its own; +caching the catalog and the rollup together is a Dashboard delivery concern (see +[Live delivery](#live-delivery) below). + ### Reasoning graph size `graph` is a point-in-time series of the reasoning/knowledge graph's structural size @@ -101,30 +194,55 @@ verbatim by `collectData()` and rendered as the existing Δpp sparkline and verd ## Live delivery -`IntelligenceWatch` polls the three source files' `mtime` on an interval (default 1,000 ms), -corroborated by a change-only tail of `pending-insights.jsonl` via the existing `JsonlTailer` — -line *contents* are never read; a record arriving at all is the signal. Detected changes accumulate -against a trailing-edge debounce (default 2,500 ms, measured from the most recently detected -change) so a burst of writes during an active session collapses into one flush. A flush re-reads -`readGlobalStats(cwd)`; if it differs from the watcher's own last-seen value, it appends a health -snapshot (independent of, and in addition to, `intel-history.mjs`'s own on-disk dedup) and then -calls `readIntelHistory(cwd)` and forwards the combined result to every connected -`GET /api/live/intelligence` client. - -The route follows Dashboard delivery's existing SSE discipline exactly: reservation-before-await -client-cap tracking (default cap 32, clamped to 256), the forwarding-cleanup idiom for a close that -races connection setup, and `sseChannel`'s bounded per-client queue (default 256 frames, clamped to -4,096) with heartbeat. `/api/status` remains a fully sufficient fallback for a client that has not -opened, or does not support, the stream — both paths return the identical `readIntelHistory(cwd)` -shape. +Each `IntelligenceWatch` instance still polls its one project's three source files' `mtime` on an +interval (default 1,000 ms), corroborated by a change-only tail of that project's own +`pending-insights.jsonl` via the existing `JsonlTailer` — line *contents* are never read; a record +arriving at all is the signal. Detected changes accumulate against a trailing-edge debounce +(default 2,500 ms, measured from the most recently detected change) so a burst of writes during an +active session collapses into one flush. A flush re-reads `readGlobalStats(cwd)` for that project; +if it differs from the watcher's own last-seen value, it appends a health snapshot (independent of, +and in addition to, `intel-history.mjs`'s own on-disk dedup) and then calls `readIntelHistory(cwd)` +and forwards the combined result to every `GET /api/live/intelligence` client currently subscribed +to that project. + +What changed with machine-wide, selectable scope is the *lifecycle*, not the per-project polling +above: a single server-wide watcher tied to the dashboard's launching working directory is replaced +by a small pool keyed by resolved project path — at most one `IntelligenceWatch` per project +actually being watched by at least one client. A project's watcher is created lazily on that +project's first SSE subscriber (however many other projects already have their own watcher +running) and is stopped and removed from the pool the moment its last subscriber for that project +disconnects. An unwatched project's watcher therefore never runs indefinitely, and two clients +watching two different projects never cross-talk, because each pool entry owns its own subscriber +set and its own broadcast closure. + +`GET /api/status` and `GET /api/live/intelligence` accept the identical optional `?project=` +query parameter and resolve it through the same shared `resolveSelectedProject(projects, rawParam)` +helper — an explicit key match if present and valid, else the first entry in +`discoverRuvfloProjects()`'s own most-recently-active-first order — so the two routes can never +disagree about what an absent or unresolvable key defaults to. `GET /api/live/intelligence` +answers `503` if discovery finds zero ruflo-initialized projects on this machine at all, rather +than picking anything. The machine-wide project catalog and the `readMachineWideIntel()` rollup +themselves are shared, TTL-cached (~60s) state — one scan per dashboard instance per window, reused +by every poll and every new connection within it, rather than re-walking the machine per request. + +The route otherwise follows Dashboard delivery's existing SSE discipline exactly: +reservation-before-await client-cap tracking (default cap 32, clamped to 256, shared across all +projects), the forwarding-cleanup idiom for a close that races connection setup, and +`sseChannel`'s bounded per-client queue (default 256 frames, clamped to 4,096) with heartbeat. +`/api/status` remains a fully sufficient fallback for a client that has not opened, or does not +support, the stream — both paths resolve the same selected project and return the identical +`readIntelHistory(cwd)` shape for its detail fields. ## Invariants 1. Pattern-store size and the patterns-learned counter are computed, labeled, and rendered - independently; neither substitutes for the other. + independently; neither substitutes for the other — at single-project scope + (`patternStore.length` vs. `globalStats.patternsLearned`) and at machine scope + (`totals.patternStoreEntries` vs. `totals.patternsLearnedLifetime`) alike. 2. Every reader degrades to an honest empty/`null` result on a missing, unreadable, or wrong-shaped source file rather than throwing or fabricating a value; individual malformed - entries are skipped rather than corrupting a bucket. + entries are skipped rather than corrupting a bucket. One project's missing or malformed data + degrades only that project's row in a machine-wide scan; it never aborts the whole scan. 3. `readGlobalStats` and `status.mjs`'s CLI `learning` row read the same file through the same helper and default logic, so the two cannot silently drift apart. 4. The health-history ring is capped and deduplicated; unchanged repeated snapshots do not grow it. @@ -132,10 +250,24 @@ shape. arrived" is a signal. 6. This domain introduces no session, actor, host, provider, model, or lifecycle identity, and no per-field evidence-confidence grading. -7. `GET /api/live/intelligence` requires no `--live-source` registration; its sources are always - this project's own `.claude-flow/` state. -8. `/api/status` and `GET /api/live/intelligence` return the same `readIntelHistory(cwd)` shape, so - client rendering has one code path regardless of delivery route. +7. `GET /api/live/intelligence` requires no `--live-source` registration; its sources are always a + discovered project's own `.claude-flow/` state, never a remote or unregistered source. +8. `/api/status` and `GET /api/live/intelligence` resolve `?project=` identically and return + the same `readIntelHistory(selected.path)` shape for their detail fields, so client rendering + has one code path regardless of delivery route or which project is selected. +9. A project appears in the discovered catalog only when it has genuine neural state + (`.claude-flow/neural/` present) — a bare `.claude-flow/` directory is not enough. +10. Selection defaults to the most-recently-active discovered project by `lastAdaptation`; it is + never implicitly the dashboard server's own launching working directory. There is no unlabeled + "this project" default anywhere in this domain's delivery contract. +11. `machineWide`/`totals` are always computed across every discovered project, independent of + which project is currently selected for detail; switching the selection never changes the + machine-wide figures. +12. A project's `IntelligenceWatch` exists in the delivery pool only while at least one client is + subscribed to it; that project's last disconnect stops and removes it. +13. Discovery's cross-reference into Observability's `WorkspaceSnapshotStore` supplies only a + candidate project path; it is never treated as evidence, confidence, or a rendered value in + this domain. ## References diff --git a/docs/ddd/ubiquitous-language.md b/docs/ddd/ubiquitous-language.md index 46e7ebb..6c4e7eb 100644 --- a/docs/ddd/ubiquitous-language.md +++ b/docs/ddd/ubiquitous-language.md @@ -69,10 +69,17 @@ inference vendors served a workflow. Generalized execution belongs to `ak run`. | Reasoning graph sample | A point-in-time structural-size measurement (`nodes`, `edges`, `pageRankSum`) of the reasoning/knowledge graph | | Health-history ring | The capped, deduplicated sample ring recording learning-stat snapshots over time | | Project intelligence | Read-only trend telemetry over ruflo/agentic-qe's own local learning state, distinct from Observability evidence | +| Discovered project | A project on this machine ruflo has genuinely initialized (`.claude-flow/neural/` present); found by project discovery and eligible for Intelligence selection | +| Project discovery | The registry-plus-Observability-cross-reference scan (`discoverRuvfloProjects()`) that produces the machine-wide, deduplicated, most-recently-active-first catalog of discovered projects | +| Selected project | The one discovered project whose detail the Intelligence panel currently shows; defaults to the most-recently-active discovered project, never an implicit cwd default | +| Machine-wide rollup | The `{ totals, perProject }` aggregate (`readMachineWideIntel()`) folded across every discovered project; always shown regardless of which project is selected | +| Intelligence watcher pool | The per-discovered-project pool of `IntelligenceWatch` instances backing `GET /api/live/intelligence`; a project's watcher is created on its first SSE subscriber and torn down on its last disconnect | Pattern-store size and the patterns-learned counter are never interchangeable displays of "how many -patterns exist" — the store can be pruned while the counter keeps climbing. See -[Project intelligence](project-intelligence.md). +patterns exist" — the store can be pruned while the counter keeps climbing — at single-project +scope and at machine-wide-rollup scope alike. There is no unlabeled "this project" default in +Intelligence: the panel always shows an explicitly selected, explicitly labeled project alongside +the always-visible machine-wide rollup. See [Project intelligence](project-intelligence.md). ## Usage rules diff --git a/src/lib/daemons.mjs b/src/lib/daemons.mjs index fe4840a..f82fd39 100644 --- a/src/lib/daemons.mjs +++ b/src/lib/daemons.mjs @@ -14,8 +14,11 @@ const alive = (pid) => { /** Known-workspace discovery from ruflo's machine-level registries * (~/.claude-flow/*.json record workspaces; each workspace has - * .claude-flow/daemon.pid + daemon-state.json with startedAt). */ -function registryWorkspaces() { + * .claude-flow/daemon.pid + daemon-state.json with startedAt). Exported + * (visibility-only change, behavior unchanged) so project-discovery.mjs can + * reuse it verbatim as its guaranteed-correct primary source instead of + * reimplementing the same registry walk. */ +export function registryWorkspaces() { const out = new Set(); const reg = path.join(home, '.claude-flow'); for (const f of ['ai-jobs.json', 'workspace-leases.json', 'repo-supervisors.json']) { diff --git a/src/lib/dashboard-server.mjs b/src/lib/dashboard-server.mjs index 2a4d115..d4f20fe 100644 --- a/src/lib/dashboard-server.mjs +++ b/src/lib/dashboard-server.mjs @@ -4,16 +4,30 @@ // GET / → one self-contained HTML document (all CSS + JS inline, // no external fetches — offline-first, matches the kit ethos) // GET /api/status → JSON: the same subsystem rows `ak status --json` emits, -// PLUS version drift, the project's .claude-flow/improvement.json -// (if present), the health-history ring, and the intel-history -// (pattern store / graph / global stats) series — see -// dashboard/intel-history.mjs and collectData()'s own field -// comments for the exact, frozen shape of each. +// PLUS version drift, the LAUNCHING project's own +// .claude-flow/improvement.json (if present), and an +// `intel` object covering the machine-wide Intelligence +// feature — see collectData()'s own field comments for +// the exact shape. `intel` is keyed off a machine-wide +// project catalog (dashboard/project-discovery.mjs), +// NOT the server's launching cwd: pass ?project= +// (a key from intel.projects[].key, e.g. the previous +// response's own intel.selectedProjectKey) to pick which +// discovered project's detail is shown; omit it (or pass +// an unresolvable key) to default to the most-recently- +// active discovered project. intel.machineWide is always +// the full machine-wide rollup, independent of ?project. // GET /api/live → bounded, privacy-safe live session projection // GET /api/live/events → resumable Server-Sent Events stream -// GET /api/live/intelligence → SSE stream of intel-history.mjs's combined -// read; one initial frame on connect, then a fresh frame -// whenever IntelligenceWatch detects a real change +// GET /api/live/intelligence → SSE stream of one discovered project's +// intel-history.mjs combined read; one initial frame on +// connect, then a fresh frame whenever that project's +// IntelligenceWatch detects a real change. Accepts the +// SAME optional ?project= param as /api/status, +// resolved identically (so both endpoints agree on what +// an absent/unresolvable key defaults to); a small pool +// keyed by resolved project path keeps at most one +// watcher per distinct project actually being watched. // GET /api/usage → the usage Aggregate MINUS sessions[] (ADR-0009) // GET /api/sessions → the session list, filtered + paginated // GET /api/session/:id → one transcript, secrets masked SERVER-side @@ -44,7 +58,9 @@ import { sseChannel, reserveClientSlot, clientGone } from './dashboard/sse.mjs'; // dashboard-server.mjs no longer defines it locally. It isn't called directly // here because readIntelHistory() already composes it (as `.healthRing`, // forwarded verbatim below); a bare `readHealthRing` import would be unused. -import { readIntelHistory } from './dashboard/intel-history.mjs'; +import { readIntelHistory, readMachineWideIntel } from './dashboard/intel-history.mjs'; +import { discoverRuvfloProjects } from './dashboard/project-discovery.mjs'; +import { resolveProjectIdentity, safeProjectKey } from './live/project-label.mjs'; import { TRANSCRIPT_ROOTS, maskMeta, @@ -103,8 +119,104 @@ function shellOutStatus(cwd) { }); } +// ───────────────────────────────────────────────────────────────────────────── +// Machine-wide Intelligence: project catalog + selection (ADR pending). +// +// The dashboard used to hardcode Intelligence data to the server's own +// launching cwd. This is a deliberate clean break: Intelligence is now +// machine-wide, backed by dashboard/project-discovery.mjs's catalog of every +// ruflo-initialized project on this machine, with one of them "selected" for +// detail display. Both /api/status and /api/live/intelligence resolve that +// selection identically (resolveSelectedProject below) so they can never +// disagree about what an absent/unresolvable ?project= defaults to. +// ───────────────────────────────────────────────────────────────────────────── + +const PROJECT_SNAPSHOT_TTL_MS = 60_000; + +/** intel-history.mjs's readIntelHistory() shape, for a selection of "no + * project" (discovery found zero ruflo-initialized projects on this + * machine) — same null/[]-on-absent conventions readIntelHistory itself + * uses for a single missing file, applied here for "no project at all". */ +const EMPTY_SELECTED_HISTORY = { patternStore: [], graph: null, healthRing: null, globalStats: null }; + +/** A stable, opaque key for a discovered project, derived from its resolved + * absolute path — NOT from its display label, which two different projects + * can share (e.g. two repos both named "backend"). resolveProjectIdentity + * (src/lib/live/project-label.mjs, already used machine-wide for live- + * session project identity) hashes the project's own canonical git root + * path when one is found — a distinct, collision-resistant key per real + * project on disk. stableProjectKey alone was checked and does NOT fit + * here: it reduces its input to a bare directory-name label before hashing, + * so keying on it directly would collapse two same-named-but-different + * projects onto one key — exactly the ambiguity a *selection* key (unlike a + * *display* label, where that collision is merely cosmetic) cannot afford. + * resolveProjectIdentity falls back to that same label-hash only for a + * genuinely non-git directory, which is an acceptable degraded case. */ +function keyForProject(project) { + return resolveProjectIdentity(project.path).key; +} + +/** Accept an incoming ?project= value only if it is ALREADY shaped like a + * genuine opaque project key (the exact `project:<16-hex>` form + * safeProjectKey emits) — reusing safeProjectKey's own shape check rather + * than duplicating its regex. Passing the raw value as both arguments means + * a well-formed key round-trips unchanged, while anything else (garbage, + * a raw path, empty) resolves to some OTHER key that will simply fail to + * match any discovered project below — never a fabricated key that could be + * echoed back as if it were valid. */ +function validProjectKeyParam(raw) { + if (typeof raw !== 'string' || !raw) return null; + const sanitized = safeProjectKey(raw, raw); + return sanitized === raw ? raw : null; +} + +/** Resolve which discovered (and key-tagged) project a request means: an + * explicit ?project= match if present and valid, else the first entry + * — discoverRuvfloProjects() itself already sorts most-recently-active + * first, so "no explicit selection" naturally means "whatever this machine + * used most recently", never the server's launching cwd. Returns null only + * when discovery found zero ruflo-initialized projects on this machine. + * Shared verbatim by /api/status and /api/live/intelligence. */ +function resolveSelectedProject(projects, rawParam) { + const requested = validProjectKeyParam(rawParam); + if (requested) { + const match = projects.find((p) => p.key === requested); + if (match) return match; + } + return projects[0] ?? null; +} + +/** Machine-wide discovery + the machine-wide intel rollup are the SAME data + * for every client polling /api/status or connecting to + * /api/live/intelligence — nothing about either is per-client — and + * discovery walks the registry files plus every discovered project's own + * .claude-flow tree, so it is not free. Cached in-memory with a short + * (~60s) TTL, keyed by nothing (one snapshot per dashboard instance is + * correct for machine-wide data), so a burst of polls/connections within + * the window reuses one scan instead of re-walking the machine on every + * request. `discoverProjectsFn`/`machineWideIntelFn` are injected by + * startDashboard (defaulting to the real imports) purely for testability — + * discovery reads real machine-global state (~/.claude-flow/*.json, + * ~/.config/agentic-kit/observability-workspaces.json) that a test must + * never depend on. */ +function buildProjectSnapshotCache(discoverProjectsFn, machineWideIntelFn) { + let snapshot = null; + let fetchedAt = 0; + return () => { + const now = Date.now(); + if (!snapshot || now - fetchedAt > PROJECT_SNAPSHOT_TTL_MS) { + const projects = discoverProjectsFn().map((project) => ( + { ...project, key: keyForProject(project) } + )); + snapshot = { projects, machineWide: machineWideIntelFn(projects) }; + fetchedAt = now; + } + return snapshot; + }; +} + /** Assemble the full /api/status payload. */ -async function collectData({ cwd, fetchStatus }) { +async function collectData({ cwd, fetchStatus, projectParam, getProjectSnapshot }) { let status; try { status = await fetchStatus(); } catch (e) { status = { overall: 'unknown', rows: [], error: String(e && e.message || e) }; } const rows = Array.isArray(status?.rows) ? status.rows : []; @@ -134,12 +246,14 @@ async function collectData({ cwd, fetchStatus }) { } catch { /* banner is best-effort — the subsystem card still carries the ruvector row */ } } - // Learning/intelligence history (src/lib/dashboard/intel-history.mjs) — one - // combined read powering the four DISTINCT series exposed below. See that - // module's header for the authoritative statement of why patternStore and - // globalStats.patternsLearned must never be conflated; the payload shape - // here preserves that separation instead of collapsing it. - const intel = readIntelHistory(cwd); + // Machine-wide Intelligence (clean break from the old cwd-hardcoded, + // single-project shape — see the block comment above buildProjectSnapshotCache). + // `getProjectSnapshot()` is the shared, TTL-cached {projects, machineWide} + // read; `projectParam` is the raw ?project= query value, resolved the exact + // same way /api/live/intelligence resolves it (resolveSelectedProject). + const { projects, machineWide } = getProjectSnapshot(); + const selected = resolveSelectedProject(projects, projectParam); + const selectedHistory = selected ? readIntelHistory(selected.path) : EMPTY_SELECTED_HISTORY; return { generatedAt: new Date().toISOString(), @@ -148,38 +262,51 @@ async function collectData({ cwd, fetchStatus }) { error: status?.error ?? null, rows, drift, + // improvement — UNCHANGED contract: still the LAUNCHING project's own + // .claude-flow/improvement.json (server cwd), independent of Intelligence + // project selection below. Not part of the Intelligence redesign — this + // is a different subsystem ("is this project's own status stale") that + // never had a "which project" ambiguity to begin with. improvement: readJsonSafe(path.join(cwd, '.claude-flow', 'improvement.json')), - // health — UNCHANGED contract: the machine-health snapshot RING from - // .claude-flow/health-history.json (an array of point samples, each - // typically carrying a `patternsLearned` counter value and/or an - // `improvement`/`deltaPP` field — see appendHealthSnapshot's callers), or - // null if that file is absent. This is intel.healthRing forwarded - // verbatim; client.mjs's renderHistory() keeps reading it exactly as - // before this integration. - health: intel.healthRing, - // globalStats — the CURRENT (not historical) cumulative counters read - // straight from .claude-flow/neural/stats.json right now: - // { patternsLearned, trajectoriesRecorded, signalsProcessed, - // lastAdaptation }, or null if that file is absent. patternsLearned here - // is a LIFETIME counter and can legitimately be higher than - // patternStore.length below — the store gets pruned/compacted over time - // while this counter only ever climbs. Do not treat the two as - // interchangeable displays of "how many patterns exist". - globalStats: intel.globalStats, - // patternStore — every entry CURRENTLY PRESENT in the neural pattern - // store (.claude-flow/neural/patterns.json), as `{ createdAt, type }` - // pairs, NOT pre-bucketed by day (the client buckets). A point-in-time - // inventory of the store's live contents — distinct from - // globalStats.patternsLearned (a lifetime counter, see above) and - // distinct from health[] (machine-health samples, not pattern-store - // entries). - patternStore: intel.patternStore, - // graph — point-in-time samples of the reasoning graph's size over time - // (.claude-flow/data/intelligence-snapshot.json), as - // `{ timestamp, nodes, edges, pageRankSum }`, or null if that file is - // absent. A structural-growth series, independent of the three - // learning/pattern-count metrics above. - graph: intel.graph, + // intel — the machine-wide Intelligence feature's ENTIRE payload, newly + // designed from scratch (no relation to the prior alpha's flat + // health/globalStats/patternStore/graph top-level fields, which this + // supersedes and removes). Shape: + // selectedProjectKey/selectedProjectLabel — which discovered project's + // detail the four fields below describe; both null only when + // discovery found zero ruflo-initialized projects on this machine. + // Always present so the client can label the detail strip correctly + // regardless of whether ?project was supplied. + // projects — the full machine-wide catalog (most-recently-active + // first, matching discoverRuvfloProjects()'s own sort), each row + // `{ key, label, path, source }` — the key a client echoes back as + // ?project= to select a different project's detail. Without + // this catalog, ?project= would be undiscoverable from the API + // alone. + // health/globalStats/patternStore/graph — readIntelHistory(selected + // project's path)'s four series, UNCHANGED individual shapes from + // the prior alpha (see intel-history.mjs's own header for why + // patternStore and globalStats.patternsLearned must never be + // conflated) — only their home moved, from top-level to nested here, + // and their source moved, from the server's launching cwd to + // whichever project is selected. + // machineWide — intel-history.mjs's readMachineWideIntel() rollup + // across EVERY discovered project, `{ totals, perProject }`. ALWAYS + // the full machine-wide figure, independent of selectedProjectKey — + // switching the selected project changes the four detail series + // above, never this. + intel: { + selectedProjectKey: selected?.key ?? null, + selectedProjectLabel: selected?.label ?? null, + projects: projects.map(({ key, label, path: projectPath, source }) => ( + { key, label, path: projectPath, source } + )), + health: selectedHistory.healthRing, + globalStats: selectedHistory.globalStats, + patternStore: selectedHistory.patternStore, + graph: selectedHistory.graph, + machineWide, + }, routing: routingPayload(), }; } @@ -373,21 +500,6 @@ function lazyLive(liveOptions = {}) { }; } -/** Load the intelligence watcher only when /api/live/intelligence is first - * requested — same "pay only when used" rationale as lazyLive. Unlike - * LiveSessionsService's subscribe/replay pub-sub, IntelligenceWatch wires its - * `onUpdate` once, at construction; the route below fans that single callback - * out to every currently-connected client itself (see `broadcastIntel`). */ -function lazyIntelWatch(cwd, onUpdate) { - let instancePromise; - return async () => { - instancePromise ||= import('./live/intelligence-watch.mjs').then(({ IntelligenceWatch }) => ( - new IntelligenceWatch({ cwd, onUpdate }) - )); - return instancePromise; - }; -} - /** * Start the dashboard HTTP server, bound to loopback only. * @param {{ port?: number, cwd?: string, fetchStatus?: () => Promise, usage?: any, @@ -395,7 +507,11 @@ function lazyIntelWatch(cwd, onUpdate) { * liveClientBuffer?: number, liveMaxClients?: number, liveOptions?: any, * liveIdleMs?: number, transcripts?: any, transcriptOptions?: any, * transcriptClientBuffer?: number, transcriptMaxClients?: number, - * intelWatch?: any, intelClientBuffer?: number, intelMaxClients?: number }} [opts] + * intelWatch?: ((projectPath: string, onUpdate: (combined: any) => void) => + * any|Promise)|any, + * intelClientBuffer?: number, intelMaxClients?: number, + * discoverProjects?: () => Array<{ path: string, label: string, source?: string }>, + * machineWideIntel?: (projects: Array) => any }} [opts] * @returns {Promise<{ url: string, urlWithToken: string, port: number, token: string, close: () => Promise }>} */ export function startDashboard({ @@ -404,6 +520,7 @@ export function startDashboard({ liveOptions = {}, liveIdleMs = 30_000, transcripts, transcriptOptions = {}, transcriptClientBuffer = 64, transcriptMaxClients = 16, intelWatch, intelClientBuffer = 256, intelMaxClients = 32, + discoverProjects, machineWideIntel, } = {}) { const provide = fetchStatus || shellOutStatus(cwd); const usageApi = usage || lazyUsage(); @@ -483,38 +600,86 @@ export function startDashboard({ return service; }; - // ── /api/live/intelligence singleton plumbing ──────────────────────────── - // `intelClients` mirrors `liveClients`/`transcriptClients`: cap-tracking - // AND the set force-closed on shutdown. `intelWriters` is separate — the - // subset of those clients' raw `write` functions the watcher's single - // `onUpdate` callback broadcasts to, since (unlike LiveSessionsService) - // IntelligenceWatch has no built-in per-subscriber pub-sub of its own. + // Shared, TTL-cached machine-wide project catalog + rollup — see the block + // comment above buildProjectSnapshotCache. One instance per dashboard + // server, used by BOTH /api/status and /api/live/intelligence so they can + // never scan the machine independently or disagree on results within the + // same TTL window. + const getProjectSnapshot = buildProjectSnapshotCache( + typeof discoverProjects === 'function' ? discoverProjects : discoverRuvfloProjects, + typeof machineWideIntel === 'function' ? machineWideIntel : readMachineWideIntel, + ); + + // ── /api/live/intelligence pool plumbing ────────────────────────────────── + // Was a single server-wide IntelligenceWatch hardcoded to the server's own + // cwd; now Intelligence is machine-wide and project-selectable, so this is + // a small POOL keyed by resolved project path instead — at most one + // watcher per DISTINCT project actually being watched by at least one SSE + // client, however many clients (on the same or different projects) are + // connected. `intelClients` mirrors `liveClients`/`transcriptClients`: + // cap-tracking AND the set force-closed on shutdown, across every project. + // `intelPool` is the per-project state: each entry owns its own `writers` + // set (only THIS project's SSE clients) and its own lazily-started, + // memoized IntelligenceWatch, since — unlike LiveSessionsService — that + // class has no built-in per-subscriber pub-sub of its own; each entry's + // `broadcast` fans its watch's single `onUpdate` out to only its own + // writers, so two clients watching two different projects never cross-talk. const intelClients = new Set(); - const intelWriters = new Set(); - const broadcastIntel = (combined) => { - const frame = transcriptSseFrame('update', combined); - for (const write of intelWriters) { - try { write(frame); } catch { /* a dead writer is reaped by its own cleanup */ } - } - }; - const provideIntelWatch = typeof intelWatch === 'function' - ? intelWatch : intelWatch ? async () => intelWatch : lazyIntelWatch(cwd, broadcastIntel); - let intelWatchPromise; - let intelWatchStartPromise; - let intelWatchStarted = false; - const getIntelWatch = async () => { - if (shuttingDown) throw new Error('dashboard is closing'); - const watch = await (intelWatchPromise ||= Promise.resolve().then(provideIntelWatch)); - if (shuttingDown) throw new Error('dashboard is closing'); - if (!watch || typeof watch.start !== 'function' || typeof watch.stop !== 'function') { - throw new TypeError('intelligence watch must implement start and stop'); + const intelPool = new Map(); // resolved project path -> pool entry + + // intelWatch injection contract for tests/embedders: a FUNCTION is called + // as (projectPath, onUpdate) -> watch|Promise, mirroring the pool's + // real per-path shape (a deliberate signature change from the old + // singleton's `() => watch`, part of this clean break). A non-function + // value is reused verbatim for EVERY project path — a single-project test + // convenience; the pool never re-wires that instance's own onUpdate, so a + // test using this form should only ever have one project selected. + const provideIntelWatchFor = typeof intelWatch === 'function' + ? intelWatch + : intelWatch + ? () => intelWatch + : (projectPath, onUpdate) => import('./live/intelligence-watch.mjs').then( + ({ IntelligenceWatch }) => new IntelligenceWatch({ cwd: projectPath, onUpdate }), + ); + + /** One pool entry per resolved project path. */ + function createIntelPoolEntry(projectPath) { + const writers = new Set(); + const broadcast = (combined) => { + const frame = transcriptSseFrame('update', combined); + for (const write of writers) { + try { write(frame); } catch { /* a dead writer is reaped by its own cleanup */ } + } + }; + let watchPromise; + let startPromise; + let started = false; + const getWatch = async () => { + if (shuttingDown) throw new Error('dashboard is closing'); + const watch = await (watchPromise ||= Promise.resolve() + .then(() => provideIntelWatchFor(projectPath, broadcast))); + if (shuttingDown) throw new Error('dashboard is closing'); + if (!watch || typeof watch.start !== 'function' || typeof watch.stop !== 'function') { + throw new TypeError('intelligence watch must implement start and stop'); + } + if (!started) await (startPromise ||= Promise.resolve() + .then(() => watch.start()) + .then(() => { started = true; }) + .catch((error) => { startPromise = null; throw error; })); + return watch; + }; + const stop = async () => { try { (await watchPromise)?.stop?.(); } catch { /* best-effort */ } }; + return { writers, getWatch, stop }; + } + + function getOrCreateIntelPoolEntry(projectPath) { + let entry = intelPool.get(projectPath); + if (!entry) { + entry = createIntelPoolEntry(projectPath); + intelPool.set(projectPath, entry); } - if (!intelWatchStarted) await (intelWatchStartPromise ||= Promise.resolve() - .then(() => watch.start()) - .then(() => { intelWatchStarted = true; }) - .catch((error) => { intelWatchStartPromise = null; throw error; })); - return watch; - }; + return entry; + } const html = renderPage({ name: '@pacphi/agentic-kit', version: kitVersion() }); @@ -562,8 +727,28 @@ export function startDashboard({ if (url === '/api/status') { let payload; - try { payload = await collectData({ cwd, fetchStatus: provide }); } - catch (e) { payload = { generatedAt: new Date().toISOString(), overall: 'unknown', rows: [], drift: null, improvement: null, health: null, globalStats: null, patternStore: [], graph: null, error: String(e && e.message || e) }; } + try { + payload = await collectData({ + cwd, fetchStatus: provide, projectParam: query.get('project'), getProjectSnapshot, + }); + } catch (e) { + payload = { + generatedAt: new Date().toISOString(), overall: 'unknown', rows: [], drift: null, improvement: null, + intel: { + selectedProjectKey: null, selectedProjectLabel: null, projects: [], + health: null, globalStats: null, patternStore: [], graph: null, + machineWide: { + totals: { + patternsLearnedLifetime: 0, patternStoreEntries: 0, trajectoriesRecorded: 0, + projectCount: 0, mostActiveProject: null, + }, + perProject: [], + }, + }, + routing: null, + error: String(e && e.message || e), + }; + } sendJson(res, 200, payload); return; } @@ -718,10 +903,11 @@ export function startDashboard({ if (url === '/api/live/intelligence') { // Same reservation-before-await discipline as /api/live/events above - // (sse.mjs's reserveClientSlot doc comment): getIntelWatch() may await a - // dynamic import() + watch.start() on the very first connection, and - // concurrent requests arriving during that gap must not all observe the - // same pre-reservation size and all pass the cap. + // (sse.mjs's reserveClientSlot doc comment): resolving + starting a + // project's pool entry may await a dynamic import() + watch.start() on + // that project's very first connection, and concurrent requests + // arriving during that gap must not all observe the same + // pre-reservation size and all pass the cap. const maxClients = Math.max(1, Math.min(256, Number(intelMaxClients) || 32)); const slot = reserveClientSlot(intelClients, maxClients); if (!slot) { @@ -740,12 +926,31 @@ export function startDashboard({ req.once('close', cleanup); res.once('close', cleanup); - try { await getIntelWatch(); } catch { + // Same ?project= resolution as /api/status, off the SAME cached + // discovery snapshot — the two endpoints can never disagree about + // which project an absent/unresolvable key defaults to. + const { projects } = getProjectSnapshot(); + const selected = resolveSelectedProject(projects, query.get('project')); + if (!selected) { intelClients.delete(slot); + sendJson(res, 503, { error: 'no ruflo-initialized project found on this machine' }); + return; + } + const poolEntry = getOrCreateIntelPoolEntry(selected.path); + + try { await poolEntry.getWatch(); } catch { + intelClients.delete(slot); + // Nobody else is (yet) watching this path — don't leave a dead entry + // behind for the next request to trip over. + if (poolEntry.writers.size === 0) intelPool.delete(selected.path); sendJson(res, 503, { error: 'intelligence telemetry unavailable' }); return; } - if (earlyClosed || clientGone(req, res)) { intelClients.delete(slot); return; } + if (earlyClosed || clientGone(req, res)) { + intelClients.delete(slot); + if (poolEntry.writers.size === 0) intelPool.delete(selected.path); + return; + } res.writeHead(200, { 'content-type': 'text/event-stream; charset=utf-8', @@ -762,26 +967,35 @@ export function startDashboard({ // scrubbing sseFrame applies is not the right tool here. const channel = sseChannel(res, { limit, heartbeatMs: liveHeartbeatMs, - onOverflow: () => transcriptSseFrame('init', readIntelHistory(cwd)), + onOverflow: () => transcriptSseFrame('init', readIntelHistory(selected.path)), }); const write = channel.write; realCleanup = (terminate = false) => { if (channel.isClosed()) return; channel.cleanup(terminate); - intelWriters.delete(write); + poolEntry.writers.delete(write); intelClients.delete(cleanup); + // Last writer for this project gone — stop its watcher and forget + // the pool entry entirely, so an unwatched project's watcher does + // not run forever (the exact leak this pool replaces the old + // singleton to avoid). + if (poolEntry.writers.size === 0) { + intelPool.delete(selected.path); + void poolEntry.stop(); + } }; intelClients.delete(slot); intelClients.add(cleanup); - intelWriters.add(write); + poolEntry.writers.add(write); if (earlyClosed) { cleanup(true); return; } - // One initial frame with the current combined read so a fresh page load - // doesn't have to wait out the watcher's own debounce window; every - // frame after this is pushed by IntelligenceWatch's onUpdate via - // broadcastIntel. - write(transcriptSseFrame('init', readIntelHistory(cwd))); + // One initial frame with the current combined read for the SELECTED + // project so a fresh page load doesn't have to wait out the watcher's + // own debounce window; every frame after this is pushed by that + // project's IntelligenceWatch onUpdate, fanned out to every writer + // currently watching this same path (and only this path). + write(transcriptSseFrame('init', readIntelHistory(selected.path))); channel.startHeartbeat(); return; } @@ -1063,7 +1277,14 @@ export function startDashboard({ await new Promise((res) => server.close(() => res(undefined))); await stopLive({ force: true }); try { await (await transcriptServicePromise)?.close?.(); } catch {} - try { (await intelWatchPromise)?.stop?.(); } catch {} + // Backstop: each intel client's own cleanup above already stops + + // deletes its pool entry the instant its last writer disconnects, + // so this is normally a no-op — but stop every REMAINING entry + // (e.g. one still mid-start with zero writers) rather than assume + // that cascade always wins the race, so every pool watcher is + // stopped on shutdown, not just one. + for (const entry of [...intelPool.values()]) { try { await entry.stop(); } catch {} } + intelPool.clear(); }, }); }); diff --git a/src/lib/dashboard/client.mjs b/src/lib/dashboard/client.mjs index f96a6ee..e16a06b 100644 --- a/src/lib/dashboard/client.mjs +++ b/src/lib/dashboard/client.mjs @@ -71,6 +71,15 @@ export const JS = ` var LEVEL_WORD={ok:"all systems nominal",warn:"attention advised",fail:"action required",unknown:"status unknown"}; var LAST=null, lastUpdated=0; + // Intelligence project selection (machine-wide redesign, see server-side + // dashboard-server.mjs's collectData()). selectedProjectKey starts null so + // the very first /api/status and /api/live/intelligence requests omit + // ?project= entirely and let the server apply ITS OWN default (the + // most-recently-active discovered project) rather than the client guessing + // at one — there is no "current project"/cwd concept on this side anymore. + // Every later request carries the current selection explicitly. + var intelProjects=[], selectedProjectKey=null, selectedProjectLabel=null, intelRequestSeq=0; + ${esc.toString()} // ── tabs (segmented control) ── @@ -395,6 +404,111 @@ export const JS = ` document.getElementById("spark-curve").innerHTML=curveVals.length>1?sparkline(curveVals):flat(curveVals.length?(curveVals[0]*100).toFixed(0)+"% (one sample)":"no data"); } + // ── machine-wide Intelligence (ALWAYS visible) + project selection ── + // intel.machineWide (from /api/status) is the full machine-wide rollup, + // independent of whichever project is selected in the picker below it — + // see render()'s wiring. Reuses the shared kpi()/esc()/fmtNum()/ago() + // helpers (defined further down, but function declarations hoist) rather + // than inventing a second vocabulary for stat tiles. + function buildHistoryView(data){ + var intel=(data&&data.intel)||{}; + return { + health:intel.health, globalStats:intel.globalStats, + patternStore:intel.patternStore, graph:intel.graph, + // "improvement" is a DIFFERENT, unchanged contract (the dashboard's own + // launching-project .claude-flow/improvement.json — see + // dashboard-server.mjs's own comment on this) — never project-selectable, + // so it rides along unchanged rather than being pulled from intel.*. + improvement:data&&data.improvement, + }; + } + + function renderMachineWide(mw){ + var totals=(mw&&mw.totals)||{}; + var perProject=Array.isArray(mw&&mw.perProject)?mw.perProject.slice():[]; + var note=document.getElementById("mw-note"); + if(note)note.textContent=totals.projectCount + ?(fmtNum(totals.projectCount)+" project"+(totals.projectCount===1?"":"s")+" tracked on this machine") + :"no ruflo-initialized projects discovered on this machine"; + var hero=document.getElementById("mw-hero"); + if(hero)hero.innerHTML= + kpi("patterns learned",fmtNum(totals.patternsLearnedLifetime),"lifetime · every tracked project","") + +kpi("projects tracked",fmtNum(totals.projectCount),"ruflo-initialized on this machine","") + +kpi("most active project",totals.mostActiveProject||"—","by most recent learning adaptation","accent"); + perProject.sort(function(a,b){return (Number(b&&b.patternsLearned)||0)-(Number(a&&a.patternsLearned)||0);}); + var table=document.getElementById("mw-table"); + if(!table)return; + if(!perProject.length){table.innerHTML='
no projects discovered on this machine.
';return;} + var html='
projectpatterns learned' + +'pattern storelast active
'; + for(var i=0;i'+esc(p.label||"(unlabeled)")+"" + +''+esc(fmtNum(p.patternsLearned))+"" + +''+esc(fmtNum(p.patternStoreCount))+"" + +''+esc(lastTxt)+"" + +""; + } + table.innerHTML=html; + } + + // The picker's option list AND its default selection come from the SAME + // /api/status response as the machine-wide rollup above — no separate + // fetch for the option list. selectedProjectKey/Label are only ADOPTED + // from a response's own intel.selectedProjectKey/Label when they differ + // from the current selection — a manual pick (wireIntelPicker) updates + // both synchronously and intelRequestSeq (pollStatus) keeps a slow, + // now-stale response from a previously selected project clobbering it. + function renderProjectPicker(intel){ + intelProjects=Array.isArray(intel.projects)?intel.projects:[]; + if(intel.selectedProjectKey!==selectedProjectKey){ + selectedProjectKey=intel.selectedProjectKey; + selectedProjectLabel=intel.selectedProjectLabel; + } + var note=document.getElementById("intel-picker-note"); + if(note)note.textContent=intelProjects.length + ?(intelProjects.length+" project"+(intelProjects.length===1?"":"s")+" available") + :"no projects discovered"; + var nameEl=document.getElementById("history-project-name"); + if(nameEl)nameEl.textContent=selectedProjectLabel||"—"; + var sel=document.getElementById("intel-project-select"); + if(!sel)return; + if(!intelProjects.length){ + sel.innerHTML=''; + sel.disabled=true; + return; + } + sel.disabled=false; + sel.innerHTML=intelProjects.map(function(p){ + return '"; + }).join(""); + } + + function wireIntelPicker(){ + var sel=document.getElementById("intel-project-select"); + if(!sel)return; + sel.addEventListener("change",function(){ + var key=sel.value; + if(!key||key===selectedProjectKey)return; + var proj=null; + for(var i=0;i URL (openIntelStream). + if(!d||typeof d!=="object"||!LAST||!LAST.intel)return; + LAST.intel.health=d.healthRing; + LAST.intel.globalStats=d.globalStats; + LAST.intel.patternStore=d.patternStore; + LAST.intel.graph=d.graph; + renderHistory(buildHistoryView(LAST)); } function openIntelStream(){ if(intelSource||!window.EventSource)return; - var src=new EventSource(dashSseUrl("/api/live/intelligence")); + // Same ?project= contract as /api/status, resolved identically + // server-side (resolveSelectedProject) — omitted entirely when no + // selection is known yet, which defers to the server's own default + // rather than the client guessing at one. + var url="/api/live/intelligence"+(selectedProjectKey?"?project="+encodeURIComponent(selectedProjectKey):""); + var src=new EventSource(dashSseUrl(url)); intelSource=src; src.addEventListener("init",function(e){try{receiveIntel(JSON.parse(e.data));}catch(e){}}); src.addEventListener("update",function(e){try{receiveIntel(JSON.parse(e.data));}catch(e){}}); @@ -492,7 +614,9 @@ export const JS = ` renderVerdict(data.overall); renderNotice(data.drift); renderPanels(data.rows); - renderHistory(data); + renderMachineWide(data.intel&&data.intel.machineWide); + renderProjectPicker(data.intel||{}); + renderHistory(buildHistoryView(data)); renderRouting(data.routing); renderModels(data.routing); positionThumb(); // badges can change segment widths @@ -542,12 +666,21 @@ export const JS = ` } function pollStatus(){ - return fetch("/api/status",{cache:"no-store",headers:authHeaders()}).then(function(r){ + // Carries the CURRENT project selection on every request (omitted only + // before the very first response has told us the server's own default — + // see the intelProjects/selectedProjectKey block above). intelRequestSeq + // guards against a slow response for a project the picker has since + // moved away from (wireIntelPicker) clobbering a newer one's render. + var seq=++intelRequestSeq; + var url="/api/status"+(selectedProjectKey?"?project="+encodeURIComponent(selectedProjectKey):""); + return fetch(url,{cache:"no-store",headers:authHeaders()}).then(function(r){ if(r.status===401){try{localStorage.removeItem(DASH_TOKEN_KEY);}catch(e){}showGate("Wrong or missing dashboard token.");throw Error("unauthorized");} return r.json(); }).then(function(d){ hideGate(); - lastUpdated=Date.now(); render(d); tickClock(); + lastUpdated=Date.now(); + if(seq===intelRequestSeq)render(d); + tickClock(); }).catch(function(){ var t=document.getElementById("verdict-text"); if(t)t.textContent="server unreachable"; }); @@ -1352,6 +1485,7 @@ export const JS = ` setUsageView(usageView); wirePoll(); wireUsage(); + wireIntelPicker(); schedulePoll(); lastAttempt=Date.now(); inflight=true; Promise.all([pollStatus()].concat(activeTab==="usage"?[loadUsage()]:[])) diff --git a/src/lib/dashboard/intel-history.mjs b/src/lib/dashboard/intel-history.mjs index b62288f..262c38d 100644 --- a/src/lib/dashboard/intel-history.mjs +++ b/src/lib/dashboard/intel-history.mjs @@ -154,3 +154,133 @@ export function readIntelHistory(cwd) { globalStats: readGlobalStats(cwd), }; } + +/** + * Machine-wide rollup — folds readIntelHistory() across every ruflo- + * initialized project on this machine into one totals/perProject view. + * `projects` is the frozen output shape of discoverRuvfloProjects() + * (src/lib/dashboard/project-discovery.mjs, built against this exact + * contract in parallel with this function): Array<{ path: string, label: + * string, source?: 'registry'|'observability'|'both' }>. Only `path` and + * `label` are read here; `source` is accepted but ignored. + * + * Investigative note on how that array is expected to be assembled (recorded + * here because this function is the actual consumer of its output): the + * PRIMARY source is daemons.mjs's (unexported) registryWorkspaces(), which + * walks ~/.claude-flow/{ai-jobs.json,workspace-leases.json,repo-supervisors.json} + * for real absolute workspace paths. A SECONDARY source was investigated — + * Observability's WorkspaceSnapshotStore (src/lib/live/workspace-store.mjs), + * persisted at ~/.config/agentic-kit/observability-workspaces.json — and + * checked against the real file on this machine. Its records never carry an + * absolute filesystem path: `directoryLabel` is validated with + * `workspaceText(..., { pathLike: true })`, which rejects anything shaped + * like an absolute path (leading `/`, a drive letter, or a `..` segment) on + * both write AND read; `repositoryLabel` is validated with `{ leaf: true }`, + * which rejects any value containing a path separator at all; and + * `project`/`projectKey` are a sanitized basename and an irreversible + * SHA-256 hash, respectively — never a path. So by that store's own privacy + * design, the secondary source can never contribute a resolvable absolute + * path for any entry; on this machine's real observability-workspaces.json + * (multiple sessions across two projects at inspection time), zero entries + * yielded one. discoverRuvfloProjects() is expected to skip every such entry + * rather than fabricate a path from a label, so in practice it is source 1 + * (the registry scan) that supplies the projects array, today. + * + * Like readIntelHistory()'s own patternsLearned-vs-patternStore distinction, + * this rollup keeps two DIFFERENT sums distinct at machine scope: + * - totals.patternsLearnedLifetime sums each project's globalStats + * .patternsLearned — a cumulative counter, includes patterns since + * pruned/compacted/replaced. + * - totals.patternStoreEntries sums each project's patternStore.length — + * entries actually present on disk right now, a smaller number for the + * exact same reason. Do not conflate the two. + * + * A project whose readIntelHistory() call turns up missing/malformed data + * degrades that project's row to nulls/zeros (matching readIntelHistory's + * own null-on-absent conventions) rather than throwing — one bad project + * never aborts the whole machine-wide scan. mostActiveProject is the label + * of whichever project has the highest globalStats.lastAdaptation among + * projects that actually have one (a positive timestamp; 0/absent is + * "never adapted", matching readGlobalStats' own `?? 0` default), or null + * when no project has adaptation data. This is a plain on-demand scan with + * no caching/TTL of its own — a later caller adds that at the server layer. + * @param {Array<{ path: string, label: string }>} projects + * @returns {{ + * totals: { patternsLearnedLifetime: number, patternStoreEntries: number, + * trajectoriesRecorded: number, projectCount: number, + * mostActiveProject: string|null }, + * perProject: Array<{ path: string, label: string, + * patternsLearned: number|null, patternStoreCount: number, + * trajectoriesRecorded: number|null, + * graphLatest: { nodes: number, edges: number }|null, + * lastAdaptation: number|null }> + * }} + */ +export function readMachineWideIntel(projects) { + const list = Array.isArray(projects) ? projects : []; + const perProject = []; + let patternsLearnedLifetime = 0; + let patternStoreEntries = 0; + let trajectoriesRecorded = 0; + let mostActiveProject = null; + let mostActiveAdaptation = 0; + + for (const entry of list) { + const cwd = typeof entry?.path === 'string' && entry.path ? entry.path : null; + const label = typeof entry?.label === 'string' && entry.label.trim() + ? entry.label + : cwd + ? path.basename(cwd) + : 'unknown'; + + let history = null; + if (cwd) { + try { + history = readIntelHistory(cwd); + } catch { + history = null; + } + } + + const globalStats = history?.globalStats ?? null; + const patternStore = Array.isArray(history?.patternStore) ? history.patternStore : []; + const graph = Array.isArray(history?.graph) ? history.graph : null; + + const patternsLearned = globalStats ? globalStats.patternsLearned : null; + const trajectories = globalStats ? globalStats.trajectoriesRecorded : null; + const lastAdaptation = globalStats ? globalStats.lastAdaptation : null; + const graphLatest = graph && graph.length + ? { nodes: graph[graph.length - 1].nodes, edges: graph[graph.length - 1].edges } + : null; + + patternsLearnedLifetime += patternsLearned ?? 0; + patternStoreEntries += patternStore.length; + trajectoriesRecorded += trajectories ?? 0; + + if (lastAdaptation != null && lastAdaptation > mostActiveAdaptation) { + mostActiveAdaptation = lastAdaptation; + mostActiveProject = label; + } + + perProject.push({ + path: cwd, + label, + patternsLearned, + patternStoreCount: patternStore.length, + trajectoriesRecorded: trajectories, + graphLatest, + lastAdaptation, + }); + } + + return { + totals: { + patternsLearnedLifetime, + patternStoreEntries, + trajectoriesRecorded, + projectCount: list.length, + mostActiveProject, + }, + perProject, + }; +} diff --git a/src/lib/dashboard/page.mjs b/src/lib/dashboard/page.mjs index 7b29a17..aabb2b5 100644 --- a/src/lib/dashboard/page.mjs +++ b/src/lib/dashboard/page.mjs @@ -2,6 +2,31 @@ import { CSS } from './styles.mjs'; import { JS } from './client.mjs'; import { LIVE_CSS, LIVE_HTML, LIVE_JS } from './live-view.mjs'; +// ── Intelligence: machine-wide rollup + project picker ────────────────────── +// Scoped to this file (not styles.mjs) since page.mjs is the only owner of +// this markup. Reuses styles.mjs's existing design tokens (--panel, --line, +// --ink*, --r-sm, --accent) so the new rows/picker match the Apple system +// motif everywhere else, without duplicating any of styles.mjs's own rules. +const INTEL_CSS = ` +.mw-table{display:flex; flex-direction:column; gap:1px; background:var(--line); border:1px solid var(--line); border-radius:var(--r-sm); overflow:hidden; margin-top:14px} +.mw-row{display:grid; grid-template-columns:minmax(140px,1.6fr) repeat(3,minmax(96px,1fr)); gap:10px; align-items:center; padding:8px 14px; background:var(--panel); font-size:12.5px} +.mw-row.mw-head{background:var(--panel-2); color:var(--ink-dim); font-size:10.5px; font-weight:600; text-transform:uppercase; letter-spacing:.06em} +.mw-row:not(.mw-head):hover{background:var(--panel-2)} +.mw-name{color:var(--ink); overflow:hidden; text-overflow:ellipsis; white-space:nowrap} +.mw-val{color:var(--ink-2); text-align:right} +.mw-row.mw-head .mw-val{color:var(--ink-dim)} +@media(max-width:560px){.mw-row{grid-template-columns:1fr repeat(3,minmax(60px,1fr)); gap:6px}} +.mw-picker{display:flex; align-items:center; gap:9px; flex-wrap:wrap} +.mw-picker label{color:var(--ink-dim); font-size:11.5px} +.mw-picker select{ + background:var(--panel-2); border:1px solid var(--line); color:var(--ink); + font-family:inherit; font-size:12.5px; padding:6px 12px; border-radius:100px; + cursor:pointer; max-width:100%; +} +.mw-picker select:focus-visible{outline:2px solid var(--accent); outline-offset:1px} +.mw-picker select:disabled{opacity:.5; cursor:not-allowed} +`; + // ───────────────────────────────────────────────────────────────────────────── // The page. One document, everything inline. Only `name` and `version` are // interpolated server-side; the client fetches /api/status and renders live. @@ -20,7 +45,7 @@ export function renderPage({ name, version }) { agentic-kit · dashboard - +