From 2805b3610e43f50e786c3d32c01f943fc077d605 Mon Sep 17 00:00:00 2001 From: Phillip Cunliffe Date: Tue, 7 Jul 2026 13:39:15 -0700 Subject: [PATCH] Add LLP 0086-0092: automatic projection of derived datasets (design) Design set making automatic graph projection the default (opt-out via config), superseding LLP 0023 section on-demand-projection for the default trigger only. The rest of LLP 0023 (contracts, ids, merge, dedup, compaction) stands and is what makes automatic re-projection idempotent. - 0086 spec: automatic derivation requirements (default-on, opt-out, bounded, eventually fresh, general; cost/exfiltration derivations stay opt-in) - 0087 decision: automatic projection is the default - 0088 decision: scheduled daemon tick as the trigger (not a commit hook) - 0089 decision: bounded, isolated execution (answers 0023's OOM objection) - 0090 decision: general derivation registration seam - 0091 design: registry seam, daemon tick loop, derive config block - 0092 plan: implementation steps - 0023: superseded-by note on the on-demand-projection default Docs only; no code changes. Implementation is LLP 0092. Co-Authored-By: Claude Opus 4.8 (1M context) --- llp/0023-context-graph-projection.decision.md | 10 ++ llp/0086-automatic-derivation.spec.md | 90 ++++++++++ ...7-automatic-projection-default.decision.md | 50 ++++++ llp/0088-scheduled-daemon-tick.decision.md | 50 ++++++ ...89-bounded-isolated-derivation.decision.md | 52 ++++++ llp/0090-general-derivation-seam.decision.md | 63 +++++++ llp/0091-automatic-derivation.design.md | 161 ++++++++++++++++++ llp/0092-automatic-derivation.plan.md | 66 +++++++ 8 files changed, 542 insertions(+) create mode 100644 llp/0086-automatic-derivation.spec.md create mode 100644 llp/0087-automatic-projection-default.decision.md create mode 100644 llp/0088-scheduled-daemon-tick.decision.md create mode 100644 llp/0089-bounded-isolated-derivation.decision.md create mode 100644 llp/0090-general-derivation-seam.decision.md create mode 100644 llp/0091-automatic-derivation.design.md create mode 100644 llp/0092-automatic-derivation.plan.md diff --git a/llp/0023-context-graph-projection.decision.md b/llp/0023-context-graph-projection.decision.md index 6a1854a2..52043084 100644 --- a/llp/0023-context-graph-projection.decision.md +++ b/llp/0023-context-graph-projection.decision.md @@ -134,6 +134,16 @@ have to define preference/cleanup rules across `(source_dataset, projector)`. ## On-demand projection +> **Superseded (default only) by [LLP 0087](./0087-automatic-projection-default.decision.md).** +> Automatic projection is now the default: the daemon projects on a bounded +> scheduled tick, opt-out via config ([LLP 0086](./0086-automatic-derivation.spec.md)). +> The section below stands as the record of why on-demand was originally chosen, +> and the manual `hyp graph project` / `hyp graph compact` commands remain. The +> two premises here were reversed by a scheduled tick that needs no commit hook +> ([LLP 0088](./0088-scheduled-daemon-tick.decision.md)) and by the daemon's +> existing bounded-background-work guards +> ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)). + The projection runs only via `hyp graph project` (and compaction via `hyp graph compact`). There is no snapshot/commit hook in the kernel to chain from, and **eventual freshness is acceptable** for an activity graph — the diff --git a/llp/0086-automatic-derivation.spec.md b/llp/0086-automatic-derivation.spec.md new file mode 100644 index 00000000..c794496a --- /dev/null +++ b/llp/0086-automatic-derivation.spec.md @@ -0,0 +1,90 @@ +# LLP 0086: automatic projection of derived datasets + +**Type:** spec +**Status:** Draft +**Systems:** Graph, Daemon, Config, Plugins +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0023, LLP 0017, LLP 0013, LLP 0028, LLP 0024, LLP 0031, LLP 0010, LLP 0006 + +> The activity graph (and derived datasets like it) must stay fresh on its own. +> Today the T0 projection runs only when an operator types `hyp graph project` +> ([LLP 0023](./0023-context-graph-projection.decision.md#on-demand-projection)), +> so a running install accumulates capture while its graph silently drifts stale +> until someone remembers to re-project. This spec makes the running daemon keep +> derived datasets projected automatically, by default, safely, with a config +> opt-out. It reverses the *default* of LLP 0023 §on-demand-projection while +> keeping the rest of that decision (contracts, ids, merge, dedup, compaction) +> intact: those are exactly what make automatic re-projection safe. + +## Motivation + +A derived dataset that only updates on a manual command is stale the moment +capture continues. For the activity graph this is a real gap: it is a query +substrate a user hits interactively (`hyp graph neighbors`, SQL over `node`/`edge`), +so "silently stale until a human runs a command" is a poor default. LLP 0023 +accepted that staleness because projecting from the daemon risked blocking or +OOMing it. That objection is now answerable (see [R4](#requirements)): the daemon +already hosts bounded background work safely (the cache-maintenance loop, +[LLP 0013](./0013-local-query-cache.decision.md) / [LLP 0017](./0017-daemon-runtime.decision.md)). + +The user framed this as "things like the graph," so the requirement is a reusable +mechanism, not a graph-only patch. + +## Requirements + +- **R1 - Automatic freshness.** A running daemon keeps registered derived datasets + projected without an operator invoking a command. The T0 activity graph is the + first such dataset. +- **R2 - On by default.** Automatic derivation is enabled by default for + deterministic, on-machine, free derivations. This is the reversal of LLP 0023's + on-demand-only default. +- **R3 - Config opt-out.** A user can disable automatic derivation globally and + per-derivation, mirroring `query.cache.maintenance.enabled: false` + ([LLP 0013](./0013-local-query-cache.decision.md)). +- **R4 - Bounded, never harmful.** Automatic derivation must never block, starve, + or OOM the daemon or its capture path. This is the hard constraint LLP 0023 + raised; here it is a requirement, not a nicety. +- **R5 - Eventual freshness, not real-time.** The graph answers "what happened," + so bounded staleness (one tick interval) is acceptable. No per-commit trigger is + required. +- **R6 - Idempotent and skip-safe.** A tick cut short by its budget, skipped + because one is in flight, or failed, must leave committed derived data correct + and must never double-write. LLP 0023's content-addressed ids and pre-write + dedup already provide this; this spec depends on it. +- **R7 - General seam.** The mechanism is a reusable registration any plugin can + contribute a derivation into. The graph is the first consumer, not a special + case. +- **R8 - Cost/exfiltration-gated derivations stay opt-in.** A derivation that + spends money or sends captured content off-machine (for example the LLP 0028 + T1/T2 enrichment, which runs through the explicit-opt-in `hypaware.completion` + capability) must NOT be default-on. The default-on policy of R2 is for + deterministic, on-machine, free derivations only. This mirrors the + embedder/completion explicit-opt-in rule + ([LLP 0024](./0024-vector-search-plugin.decision.md#embedding-is-a-separate-capability)). +- **R9 - Manual command stays.** `hyp graph project` / `hyp graph compact` remain + for on-demand and forced runs, one-off backfills, and non-daemon installs. + Automatic derivation is additive. +- **R10 - Observable.** Each automatic tick emits a span/log recording which + derivations ran, rows written, duration, and whether it was skipped or budgeted, + so a stale graph is diagnosable from telemetry, not guesswork + (log-driven-development). + +## Non-goals + +- Real-time or per-commit projection. R5 accepts eventual freshness; a kernel + commit/settle hook (which LLP 0023 noted does not exist) is out of scope. +- Auto-running model-backed enrichment (R8). Its ongoing regime + ([LLP 0028](./0028-context-graph-enrichment.decision.md#two-regimes)) is a + related mechanism the seam may later absorb; this slice does not move it. +- A declarative contract compiler. LLP 0023 already defers that as a later slice. + +## Settled by + +The choices this spec implies are settled in narrow decisions: +[LLP 0087](./0087-automatic-projection-default.decision.md) (default-on reversal), +[LLP 0088](./0088-scheduled-daemon-tick.decision.md) (trigger), +[LLP 0089](./0089-bounded-isolated-derivation.decision.md) (safety), +[LLP 0090](./0090-general-derivation-seam.decision.md) (general seam). The +buildable design is [LLP 0091](./0091-automatic-derivation.design.md); steps are +[LLP 0092](./0092-automatic-derivation.plan.md). diff --git a/llp/0087-automatic-projection-default.decision.md b/llp/0087-automatic-projection-default.decision.md new file mode 100644 index 00000000..19f04235 --- /dev/null +++ b/llp/0087-automatic-projection-default.decision.md @@ -0,0 +1,50 @@ +# LLP 0087: automatic projection is the default + +**Type:** decision +**Status:** Draft +**Systems:** Graph, Daemon +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0086, LLP 0023, LLP 0013, LLP 0017, LLP 0088, LLP 0089 + +> Supersedes [LLP 0023 §on-demand-projection](./0023-context-graph-projection.decision.md#on-demand-projection). +> The rest of LLP 0023 stands. + +## Decision + +The running daemon projects the activity graph automatically, on by default. The +on-demand-only stance of LLP 0023 is no longer the default. Automatic projection +is opt-out through config ([LLP 0086 R3](./0086-automatic-derivation.spec.md#requirements)), +not opt-in. + +## Why the original default no longer holds + +LLP 0023 chose on-demand-only for two reasons, both now weaker: + +1. *"There is no snapshot/commit hook in the kernel to chain from."* True, but a + commit hook was never required. Eventual freshness is acceptable for an + activity graph ([LLP 0086 R5](./0086-automatic-derivation.spec.md#requirements)), + and a scheduled tick needs no such hook + ([LLP 0088](./0088-scheduled-daemon-tick.decision.md)). +2. *"Registering the command keeps the plugin out of the daemon loop entirely; + nothing here can block or OOM the daemon."* The daemon already hosts bounded + background work safely: the cache-maintenance loop + ([LLP 0013](./0013-local-query-cache.decision.md) / [LLP 0017](./0017-daemon-runtime.decision.md)) + is default-on, interval-driven, budget-capped, single-flight, and + error-isolated. So "in the daemon loop" is no longer synonymous with "can block + or OOM it." The same guards make projection safe in-loop + ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)). + +With both premises addressable, the user-facing cost of on-demand-only decides it: +a graph that is silently stale until someone re-projects is a worse default than a +graph the daemon keeps fresh within a bounded interval. + +## Scope of the supersession + +This decision replaces only the *default trigger* of LLP 0023. Everything else in +LLP 0023 is unchanged and is load-bearing here: content-addressed ids, the merge +policy, and pre-write dedup are exactly what make repeated automatic projection +idempotent ([LLP 0086 R6](./0086-automatic-derivation.spec.md#requirements)). The +manual `hyp graph project` / `hyp graph compact` commands also stay +([LLP 0086 R9](./0086-automatic-derivation.spec.md#requirements)); automatic +projection is additive, not a replacement. diff --git a/llp/0088-scheduled-daemon-tick.decision.md b/llp/0088-scheduled-daemon-tick.decision.md new file mode 100644 index 00000000..c51a0052 --- /dev/null +++ b/llp/0088-scheduled-daemon-tick.decision.md @@ -0,0 +1,50 @@ +# LLP 0088: scheduled daemon tick as the projection trigger + +**Type:** decision +**Status:** Draft +**Systems:** Daemon, Graph +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0086, LLP 0087, LLP 0089, LLP 0013, LLP 0017, LLP 0028 + +## Decision + +Automatic projection fires on a **scheduled interval inside the daemon**, reusing +the shape of the existing cache-maintenance loop +([`runtime.js` maintenance tick](../src/core/daemon/runtime.js)): a config-driven +interval, an `unref()`'d timer, a single-flight guard, a per-tick time budget, and +error isolation. It does not fire on a per-commit or post-settle event. + +## Alternatives considered + +**Post-settle / commit event hook (rejected).** Projecting incrementally the +moment new source rows commit gives the freshest possible graph. But it requires a +kernel commit/settle seam that LLP 0023 explicitly noted does not exist, and it +couples derivation latency to the capture hot path, reintroducing the exact +blocking risk LLP 0023 feared. The graph answers "what happened," so real-time +freshness buys little ([LLP 0086 R5](./0086-automatic-derivation.spec.md#requirements)). + +**Subprocess per tick (rejected as the default, kept as an escape hatch).** The +daemon could spawn a `hyp graph project` child on the interval, the way the +client-action reconciler spawns `hyp backfill` +([LLP 0037](./0037-backfill-on-join.decision.md)), so a heavy projection can never +grow the daemon's own heap. This is full isolation but adds process-spawn overhead +every interval, and the T0 projection is deterministic in-process cache I/O, not +the parquet encoder whose heap behavior motivates subprocess isolation elsewhere. +A derivation that proves heap-heavy can still opt into subprocess execution +([LLP 0089](./0089-bounded-isolated-derivation.decision.md)); the default stays +in-process. + +**Scheduled in-process tick (chosen).** It matches a pattern already proven safe +in the daemon, needs no new kernel seam, and is bounded at any interval by its +budget ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)). Eventual +freshness is exactly what the graph needs. + +## Cadence + +The interval is config-driven (see the `derive` config block in +[LLP 0091](./0091-automatic-derivation.design.md)). Because each tick is +budget-capped and single-flight, a short interval is cheap: the design picks a +default short enough that the graph feels fresh for interactive queries, while any +value stays safe. The specific default number is a design detail, not part of this +decision. diff --git a/llp/0089-bounded-isolated-derivation.decision.md b/llp/0089-bounded-isolated-derivation.decision.md new file mode 100644 index 00000000..6cd4c189 --- /dev/null +++ b/llp/0089-bounded-isolated-derivation.decision.md @@ -0,0 +1,52 @@ +# LLP 0089: bounded, isolated derivation execution + +**Type:** decision +**Status:** Draft +**Systems:** Daemon +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0086, LLP 0087, LLP 0088, LLP 0023, LLP 0013, LLP 0017 + +> This decision is the answer to LLP 0023's objection that projecting from the +> daemon could block or OOM it. It names the guards that make automatic +> projection safe in-loop, and they are the guards the cache-maintenance loop +> already runs under. + +## Decision + +Every automatic derivation tick runs under four guards, matching the daemon's +cache-maintenance loop ([`runtime.js`](../src/core/daemon/runtime.js), the +`maintenance.tick` block): + +1. **Time budget.** Each tick carries a `max_tick_ms`-style cap. A tick that + reaches its budget yields and resumes next interval. Partial progress is safe + because projection is idempotent: LLP 0023 content-addressed ids plus pre-write + dedup mean an interrupted tick commits a correct prefix and the next tick picks + up the remainder with no double-write. +2. **Single-flight.** A new tick never starts while one is in flight (mirrors the + maintenance loop's `maintenanceInFlight` skip). Overruns coalesce instead of + piling up. +3. **Error isolation.** The tick runs in its own span with catch-and-log (mirrors + `daemon.maintenance_failed`). A derivation failure degrades freshness only; it + never propagates into the daemon or the capture path. +4. **Non-blocking lifecycle.** The interval timer is `unref()`'d so it never holds + the process open, and shutdown drains any in-flight tick before exit (mirrors + the maintenance handle's clear-and-await on shutdown). + +## Escape hatch for heap-heavy derivations + +In-process budgeting caps wall-time but not peak heap. If a derivation is heavy +enough that a single tick's peak memory threatens the daemon (the parquet-encoder +failure mode), that derivation may declare **subprocess execution**; the seam +supports it ([LLP 0090](./0090-general-derivation-seam.decision.md)). In-process is +the default because the T0 projection is deterministic cache I/O, not that. + +## The growing-scan follow-on + +LLP 0023's projection re-scans all source rows each run and filters against the +committed id set (pre-write dedup). Under a periodic tick that scan grows with +history. The time budget caps each tick regardless, so this is a cost concern, not +a correctness one. The efficient fix, **incremental projection** (advance a +per-source cursor and project only newly-settled rows), is a follow-on named in +[LLP 0091](./0091-automatic-derivation.design.md), not required for this slice +because budget plus idempotence already make repeated full re-scan safe. diff --git a/llp/0090-general-derivation-seam.decision.md b/llp/0090-general-derivation-seam.decision.md new file mode 100644 index 00000000..0c442394 --- /dev/null +++ b/llp/0090-general-derivation-seam.decision.md @@ -0,0 +1,63 @@ +# LLP 0090: general derivation registration seam + +**Type:** decision +**Status:** Draft +**Systems:** Plugins, Daemon, Graph +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0086, LLP 0087, LLP 0088, LLP 0089, LLP 0023, LLP 0028, LLP 0024, LLP 0006 + +## Decision + +Core exposes a **derivation registration** on the plugin activation context. A +plugin registers a named derivation during `activate()`: a runnable plus +scheduling metadata and a default-on flag. The daemon ticks every registered +derivation under the LLP 0089 guards. The mechanism is not graph-specific; +`@hypaware/context-graph` is the first consumer. + +``` +ctx.derivations.register({ + name, // stable id, e.g. 'graph project' + plugin, // owning plugin + defaultOn, // see below + run, // ({ query, storage, config, budgetMs, signal }) => Promise +}) +``` + +This mirrors the registrations plugins already contribute and the daemon already +drives: sources ([LLP 0012](./0012-sources.spec.md)), sinks +([LLP 0014](./0014-sinks.spec.md)), commands, verbs, datasets, capabilities. Core +owns the registry and the tick; each plugin owns its derivation's runnable, id, +and default-on declaration. Same ownership split as LLP 0023's contract +contribution: a plugin maps *into* a fixed seam core owns, it does not fork it. + +The runnable shape is exactly what the graph engine already consumes +(`projectGraph({ query, storage, config })`, +[`project.js`](../hypaware-core/plugins-workspace/context-graph/src/project.js)), so +the graph plugin wraps its existing engine rather than rewriting it. + +## Default-on is a per-derivation property, not a global truth + +A derivation declares whether it is default-on. This keeps the "automatic by +default" policy ([LLP 0087](./0087-automatic-projection-default.decision.md)) from +silently switching on paid or off-machine work: + +- **Deterministic, on-machine, free** derivations declare `defaultOn: true`. The + T0 activity graph is one. +- **Cost- or exfiltration-bearing** derivations declare `defaultOn: false` and run + only when the user enables them explicitly. LLP 0028 T1/T2 enrichment is the + motivating case: it spends model tokens and can send captured content + off-machine, and it already depends on the explicit-opt-in `hypaware.completion` + capability ([LLP 0024](./0024-vector-search-plugin.decision.md#embedding-is-a-separate-capability), + the same rule that excludes the completion providers from default activation in + the bundled allowlist). A default-off derivation requires an explicit + per-derivation enable in config; it never rides the global default-on. + +## Relationship to enrichment's ongoing regime + +LLP 0028 already defines an "ongoing" automatic regime for enrichment (a daily +batch over newly-settled sessions). That is a related but distinct mechanism +(model-backed, batch-API-driven). This seam does not absorb it in this slice; +enrichment keeps its own regime and stays `defaultOn: false` here. Folding the +ongoing regime into this seam is a later slice +([LLP 0086 non-goals](./0086-automatic-derivation.spec.md#non-goals)). diff --git a/llp/0091-automatic-derivation.design.md b/llp/0091-automatic-derivation.design.md new file mode 100644 index 00000000..2e00df8f --- /dev/null +++ b/llp/0091-automatic-derivation.design.md @@ -0,0 +1,161 @@ +# LLP 0091: automatic derivation - technical design + +**Type:** design +**Status:** Draft +**Systems:** Daemon, Plugins, Graph, Config +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0086, LLP 0087, LLP 0088, LLP 0089, LLP 0090, LLP 0017, LLP 0013, LLP 0031, LLP 0010 + +> Buildable design for automatic projection of derived datasets. +> `@ref LLP 0086 [implements]` - realizes the spec (R1-R10). +> `@ref LLP 0090 [constrained-by]` - a general derivation registration, not +> graph-specific. +> `@ref LLP 0089 [constrained-by]` - every tick runs under budget, single-flight, +> error isolation, and non-blocking lifecycle. +> `@ref LLP 0088 [constrained-by]` - scheduled in-process tick, reusing the +> cache-maintenance loop shape. + +## Shape at a glance + +Three parts, each independently mergeable: + +1. **A core derivation registry** contributed through the activation context, + alongside the existing `ctx.commands` / `ctx.verbs` / `ctx.skills` / + `ctx.query.registerDataset` registrations. +2. **A daemon derivation-tick loop** in + [`runtime.js`](../src/core/daemon/runtime.js), a sibling of the existing + `maintenance.tick` block, copying its guards. +3. **A `derive` config block** ([LLP 0010](./0010-config-model.spec.md) / + [LLP 0031](./0031-layered-config.decision.md)) with a `normalizeDeriveConfig` + mirroring `normalizeMaintenanceConfig`. + +Then the graph plugin registers its projection as the first, default-on +derivation. + +## 1. The registry seam + +New activation-context registrar `ctx.derivations.register(entry)`: + +``` +DerivationEntry { + name: string // stable id, e.g. 'graph project' + plugin: PluginName + defaultOn: boolean // LLP 0090: per-derivation, not global + isolation?: 'in-process' | 'subprocess' // default 'in-process' (LLP 0089 escape hatch) + run(args: { + query: QueryRegistry, + storage: ExtendedQueryStorageService, + config?: HypAwareV2Config, + budgetMs: number, + signal: AbortSignal, + }): Promise<{ label: string, wrote: number }> +} +``` + +The kernel keeps a `DerivationRegistry` (list + lookup) built at boot, exposed on +the runtime the way `runtime.sources` / `runtime.sinks` are, so the daemon can +enumerate `runtime.derivations.list()`. Registration only; the registry never +runs anything. A plain `hyp` CLI boot builds the registry but nothing ticks it. + +## 2. The daemon tick loop + +A new block in `runDaemon`, structured exactly like the maintenance block it sits +next to (interval from config, `unref()`'d `setInterval`, single-flight +`derivationInFlight` guard, `derivation.tick` span, `.catch` to +`daemon.derivation_failed`, and shutdown that clears the handle and awaits any +in-flight tick). Per tick: + +``` +for (const d of runtime.derivations.list()) { + if (!derivationEnabled(d, deriveCfg)) continue // R3 + LLP 0090 resolution + await d.run({ query, storage, config, budgetMs, signal }) // LLP 0089 budget/abort +} +``` + +`derivationEnabled(d, cfg)` is the resolution rule: + +- global off (`cfg.enabled === false`) - nothing runs; +- else a per-derivation override in `cfg.derivations[d.name].enabled` wins if + present; +- else fall back to `d.defaultOn`. + +So a default-on derivation (graph) needs an explicit `enabled: false` to stop, and +a default-off derivation (enrich) needs an explicit `enabled: true` to start +([LLP 0090](./0090-general-derivation-seam.decision.md)). A `subprocess` +derivation is dispatched by spawning its command child instead of calling `run` +in-process (LLP 0089 escape hatch); the graph is `in-process`. + +Daemon-only by construction: the loop lives in `runDaemon`, so `hyp status` and +other plain CLI boots never tick derivations, exactly as the client-action +reconciler is daemon-only ([LLP 0041](./0041-central-config-client-actions.design.md)). +Automatic projection therefore requires the installed daemon; the manual command +([LLP 0086 R9](./0086-automatic-derivation.spec.md#requirements)) covers non-daemon +installs. + +## 3. The `derive` config block + +A new top-level section (peer of `query`, `plugins`, `sinks`), because a derivation +is plugin/graph-level, not a property of the query cache: + +``` +derive: { + enabled: true, // global opt-out (R3), default true + interval_minutes: 15, // default cadence; fresher than the hourly cache loop + max_tick_ms: 30000, // per-tick budget (LLP 0089), matches maintenance default + derivations: { // per-derivation overrides (optional) + "graph project": { enabled: true }, + // model-backed derivations stay off unless explicitly turned on: + // "context-graph-enrich": { enabled: true } + } +} +``` + +Defaults mirror `query.cache.maintenance` (`enabled: true`, `max_tick_ms: 30000`), +with a shorter default interval (15 min vs 60) because the graph is user-facing +query data and each tick is cheap under budget + single-flight +([LLP 0088](./0088-scheduled-daemon-tick.decision.md#cadence)). +`normalizeDeriveConfig` supplies these defaults; validation lives with the config +layer ([LLP 0031](./0031-layered-config.decision.md)) and is section-scoped so a +malformed `derive` block is rejected/rolled back like any other. + +## 4. What the graph plugin registers + +In `@hypaware/context-graph`'s `activate()`, alongside the existing `graph project` +command registration, add: + +``` +ctx.derivations.register({ + name: 'graph project', + plugin: PLUGIN_NAME, + defaultOn: true, + run: async ({ query, storage, config }) => + projectGraph({ query, storage, contracts: requireGraphRuntime().registry.list(), config }), +}) +``` + +It wraps the same engine the command already calls +([`project.js`](../hypaware-core/plugins-workspace/context-graph/src/project.js) / +[`command.js`](../hypaware-core/plugins-workspace/context-graph/src/command.js)), so +there is one projection code path, driven either by the command or by the tick. The +`graph compact` step stays manual in this slice (duplicates are benign and +mergeable per LLP 0023); auto-compaction is a follow-on. + +## Observability (R10) + +The `derivation.tick` span carries `component=daemon`, `operation=derivation.tick`, +and per-derivation attributes: `derivation` (name), `rows_written`, `duration_ms`, +plus `skipped_in_flight` / `budget_exceeded` flags. A `daemon.derivation_ran` +info log lists which derivations ran and their row counts; failures log +`daemon.derivation_failed` with `error_kind`. A stale graph is then diagnosable +from telemetry (did the tick run? was it budgeted out? did it error?), not by +guesswork. + +## Follow-ons (out of this slice) + +- **Incremental projection**: a per-source cursor so each tick projects only + newly-settled rows, bounding tick cost as history grows + ([LLP 0089](./0089-bounded-isolated-derivation.decision.md#the-growing-scan-follow-on)). +- **Auto-compaction** as a lower-cadence default-on derivation. +- **Absorbing enrichment's ongoing regime** ([LLP 0028](./0028-context-graph-enrichment.decision.md#two-regimes)) + into this seam as a default-off derivation. diff --git a/llp/0092-automatic-derivation.plan.md b/llp/0092-automatic-derivation.plan.md new file mode 100644 index 00000000..c7e16fa0 --- /dev/null +++ b/llp/0092-automatic-derivation.plan.md @@ -0,0 +1,66 @@ +# LLP 0092: automatic derivation - implementation plan + +**Type:** plan +**Status:** Draft +**Systems:** Daemon, Plugins, Graph, Config +**Author:** Phil / Claude +**Date:** 2026-07-07 +**Related:** LLP 0091, LLP 0086, LLP 0087, LLP 0088, LLP 0089, LLP 0090 + +> Implementation steps for the automatic derivation designed in +> [LLP 0091](./0091-automatic-derivation.design.md) (spec +> [LLP 0086](./0086-automatic-derivation.spec.md)). Small, +> independently-mergeable tasks: the core registry and the config block first +> (parallel), then the daemon tick loop (the core), then the graph registration, +> closed by a hermetic smoke and the doc/annotation pass. No task touches the +> capture path or the cache schema. + +## Tasks + +- **T1 - Core derivation registry + activation-context seam.** Add + `DerivationRegistry` (list + lookup) and `ctx.derivations.register`, wired onto + the runtime the way `sources`/`sinks` are. Types in a `.d.ts`, imported via + `@import`. Unit tests: registration, duplicate-name rejection, and the + `derivationEnabled` resolution rule (global-off, per-derivation override, + `defaultOn` fallback). No daemon dependency, so it merges alone. + +- **T2 - `derive` config block + validation.** Add the top-level `derive` section, + `normalizeDeriveConfig` (mirroring `normalizeMaintenanceConfig`: `enabled`, + `interval_minutes`, `max_tick_ms`, `derivations`), and section-scoped validation + in the layered-config path ([LLP 0031](./0031-layered-config.decision.md)). Unit + tests: defaults applied, opt-out honored, malformed block rejected. Parallel with + T1. + +- **T3 - Daemon derivation-tick loop.** Add the tick block to + [`runtime.js`](../src/core/daemon/runtime.js) next to `maintenance.tick`, + copying its guards (interval, `unref()`, single-flight, budget, span, + catch-to-log, shutdown drain). Depends on T1 + T2. Unit test with a fake + derivation and fake timers (mirror the maintenance-loop tests): fires on + interval, single-flights an overrun, isolates a throwing derivation, drains on + shutdown, and respects opt-out. + +- **T4 - Graph plugin registers its projection.** In `@hypaware/context-graph` + `activate()`, register `graph project` as a `defaultOn: true` derivation wrapping + `projectGraph` ([LLP 0091 §4](./0091-automatic-derivation.design.md)). Keep the + `hyp graph project` command intact (one engine, two drivers). Unit test: the + registered `run` projects the same rows the command does. + +- **T5 - Hermetic smoke.** Boot the daemon with the graph derivation and captured + fixture rows; assert `node`/`edge` rows appear with no manual `graph project`; + assert `derive.enabled: false` suppresses it; assert the `derivation.tick` span / + `daemon.derivation_ran` log fired with row counts (log-driven, R10). Stable + `DEV_RUN_ID` / `smoke_name` / `smoke_step`. + +- **T6 - Docs + annotations.** Land the LLP 0023 §on-demand-projection + superseded-by note (already staged with this set); flip this doc set from Draft + as it lands; add `@ref LLP 0087/0088/0089/0090/0091` annotations on the daemon + tick loop, the registry seam, and the graph registration; refresh the + [LLP 0000](./0000-hypaware.explainer.md) subsystem map if the Graph/Daemon entry + needs it. + +## Ordering + +T1 and T2 in parallel, then T3 (the core), then T4. T5 and T6 close the set once +projection runs automatically end to end. Follow-ons (incremental cursor, +auto-compaction, enrichment ongoing regime) are explicitly deferred by +[LLP 0091](./0091-automatic-derivation.design.md#follow-ons-out-of-this-slice).