LLP 0086-0092: automatic projection of derived datasets (design)

llp/0023-context-graph-projection.decision.md

@@ -134,6 +134,16 @@ have to define preference/cleanup rules across `(source_dataset, projector)`.
 
 ## On-demand projection
 
+> **Superseded (default only) by [LLP 0087](./0087-automatic-projection-default.decision.md).**
+> Automatic projection is now the default: the daemon projects on a bounded
+> scheduled tick, opt-out via config ([LLP 0086](./0086-automatic-derivation.spec.md)).
+> The section below stands as the record of why on-demand was originally chosen,
+> and the manual `hyp graph project` / `hyp graph compact` commands remain. The
+> two premises here were reversed by a scheduled tick that needs no commit hook
+> ([LLP 0088](./0088-scheduled-daemon-tick.decision.md)) and by the daemon's
+> existing bounded-background-work guards
+> ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)).
+
 The projection runs only via `hyp graph project` (and compaction via
 `hyp graph compact`). There is no snapshot/commit hook in the kernel to chain
 from, and **eventual freshness is acceptable** for an activity graph — the

llp/0086-automatic-derivation.spec.md

@@ -0,0 +1,90 @@
+# LLP 0086: automatic projection of derived datasets
+
+**Type:** spec
+**Status:** Draft
+**Systems:** Graph, Daemon, Config, Plugins
+**Author:** Phil / Claude
+**Date:** 2026-07-07
+**Related:** LLP 0023, LLP 0017, LLP 0013, LLP 0028, LLP 0024, LLP 0031, LLP 0010, LLP 0006
+
+> The activity graph (and derived datasets like it) must stay fresh on its own.
+> Today the T0 projection runs only when an operator types `hyp graph project`
+> ([LLP 0023](./0023-context-graph-projection.decision.md#on-demand-projection)),
+> so a running install accumulates capture while its graph silently drifts stale
+> until someone remembers to re-project. This spec makes the running daemon keep
+> derived datasets projected automatically, by default, safely, with a config
+> opt-out. It reverses the *default* of LLP 0023 §on-demand-projection while
+> keeping the rest of that decision (contracts, ids, merge, dedup, compaction)
+> intact: those are exactly what make automatic re-projection safe.
+
+## Motivation
+
+A derived dataset that only updates on a manual command is stale the moment
+capture continues. For the activity graph this is a real gap: it is a query
+substrate a user hits interactively (`hyp graph neighbors`, SQL over `node`/`edge`),
+so "silently stale until a human runs a command" is a poor default. LLP 0023
+accepted that staleness because projecting from the daemon risked blocking or
+OOMing it. That objection is now answerable (see [R4](#requirements)): the daemon
+already hosts bounded background work safely (the cache-maintenance loop,
+[LLP 0013](./0013-local-query-cache.decision.md) / [LLP 0017](./0017-daemon-runtime.decision.md)).
+
+The user framed this as "things like the graph," so the requirement is a reusable
+mechanism, not a graph-only patch.
+
+## Requirements
+
+- **R1 - Automatic freshness.** A running daemon keeps registered derived datasets
+  projected without an operator invoking a command. The T0 activity graph is the
+  first such dataset.
+- **R2 - On by default.** Automatic derivation is enabled by default for
+  deterministic, on-machine, free derivations. This is the reversal of LLP 0023's
+  on-demand-only default.
+- **R3 - Config opt-out.** A user can disable automatic derivation globally and
+  per-derivation, mirroring `query.cache.maintenance.enabled: false`
+  ([LLP 0013](./0013-local-query-cache.decision.md)).
+- **R4 - Bounded, never harmful.** Automatic derivation must never block, starve,
+  or OOM the daemon or its capture path. This is the hard constraint LLP 0023
+  raised; here it is a requirement, not a nicety.
+- **R5 - Eventual freshness, not real-time.** The graph answers "what happened,"
+  so bounded staleness (one tick interval) is acceptable. No per-commit trigger is
+  required.
+- **R6 - Idempotent and skip-safe.** A tick cut short by its budget, skipped
+  because one is in flight, or failed, must leave committed derived data correct
+  and must never double-write. LLP 0023's content-addressed ids and pre-write
+  dedup already provide this; this spec depends on it.
+- **R7 - General seam.** The mechanism is a reusable registration any plugin can
+  contribute a derivation into. The graph is the first consumer, not a special
+  case.
+- **R8 - Cost/exfiltration-gated derivations stay opt-in.** A derivation that
+  spends money or sends captured content off-machine (for example the LLP 0028
+  T1/T2 enrichment, which runs through the explicit-opt-in `hypaware.completion`
+  capability) must NOT be default-on. The default-on policy of R2 is for
+  deterministic, on-machine, free derivations only. This mirrors the
+  embedder/completion explicit-opt-in rule
+  ([LLP 0024](./0024-vector-search-plugin.decision.md#embedding-is-a-separate-capability)).
+- **R9 - Manual command stays.** `hyp graph project` / `hyp graph compact` remain
+  for on-demand and forced runs, one-off backfills, and non-daemon installs.
+  Automatic derivation is additive.
+- **R10 - Observable.** Each automatic tick emits a span/log recording which
+  derivations ran, rows written, duration, and whether it was skipped or budgeted,
+  so a stale graph is diagnosable from telemetry, not guesswork
+  (log-driven-development).
+
+## Non-goals
+
+- Real-time or per-commit projection. R5 accepts eventual freshness; a kernel
+  commit/settle hook (which LLP 0023 noted does not exist) is out of scope.
+- Auto-running model-backed enrichment (R8). Its ongoing regime
+  ([LLP 0028](./0028-context-graph-enrichment.decision.md#two-regimes)) is a
+  related mechanism the seam may later absorb; this slice does not move it.
+- A declarative contract compiler. LLP 0023 already defers that as a later slice.
+
+## Settled by
+
+The choices this spec implies are settled in narrow decisions:
+[LLP 0087](./0087-automatic-projection-default.decision.md) (default-on reversal),
+[LLP 0088](./0088-scheduled-daemon-tick.decision.md) (trigger),
+[LLP 0089](./0089-bounded-isolated-derivation.decision.md) (safety),
+[LLP 0090](./0090-general-derivation-seam.decision.md) (general seam). The
+buildable design is [LLP 0091](./0091-automatic-derivation.design.md); steps are
+[LLP 0092](./0092-automatic-derivation.plan.md).

llp/0087-automatic-projection-default.decision.md

@@ -0,0 +1,50 @@
+# LLP 0087: automatic projection is the default
+
+**Type:** decision
+**Status:** Draft
+**Systems:** Graph, Daemon
+**Author:** Phil / Claude
+**Date:** 2026-07-07
+**Related:** LLP 0086, LLP 0023, LLP 0013, LLP 0017, LLP 0088, LLP 0089
+
+> Supersedes [LLP 0023 §on-demand-projection](./0023-context-graph-projection.decision.md#on-demand-projection).
+> The rest of LLP 0023 stands.
+
+## Decision
+
+The running daemon projects the activity graph automatically, on by default. The
+on-demand-only stance of LLP 0023 is no longer the default. Automatic projection
+is opt-out through config ([LLP 0086 R3](./0086-automatic-derivation.spec.md#requirements)),
+not opt-in.
+
+## Why the original default no longer holds
+
+LLP 0023 chose on-demand-only for two reasons, both now weaker:
+
+1. *"There is no snapshot/commit hook in the kernel to chain from."* True, but a
+   commit hook was never required. Eventual freshness is acceptable for an
+   activity graph ([LLP 0086 R5](./0086-automatic-derivation.spec.md#requirements)),
+   and a scheduled tick needs no such hook
+   ([LLP 0088](./0088-scheduled-daemon-tick.decision.md)).
+2. *"Registering the command keeps the plugin out of the daemon loop entirely;
+   nothing here can block or OOM the daemon."* The daemon already hosts bounded
+   background work safely: the cache-maintenance loop
+   ([LLP 0013](./0013-local-query-cache.decision.md) / [LLP 0017](./0017-daemon-runtime.decision.md))
+   is default-on, interval-driven, budget-capped, single-flight, and
+   error-isolated. So "in the daemon loop" is no longer synonymous with "can block
+   or OOM it." The same guards make projection safe in-loop
+   ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)).
+
+With both premises addressable, the user-facing cost of on-demand-only decides it:
+a graph that is silently stale until someone re-projects is a worse default than a
+graph the daemon keeps fresh within a bounded interval.
+
+## Scope of the supersession
+
+This decision replaces only the *default trigger* of LLP 0023. Everything else in
+LLP 0023 is unchanged and is load-bearing here: content-addressed ids, the merge
+policy, and pre-write dedup are exactly what make repeated automatic projection
+idempotent ([LLP 0086 R6](./0086-automatic-derivation.spec.md#requirements)). The
+manual `hyp graph project` / `hyp graph compact` commands also stay
+([LLP 0086 R9](./0086-automatic-derivation.spec.md#requirements)); automatic
+projection is additive, not a replacement.

llp/0088-scheduled-daemon-tick.decision.md

@@ -0,0 +1,50 @@
+# LLP 0088: scheduled daemon tick as the projection trigger
+
+**Type:** decision
+**Status:** Draft
+**Systems:** Daemon, Graph
+**Author:** Phil / Claude
+**Date:** 2026-07-07
+**Related:** LLP 0086, LLP 0087, LLP 0089, LLP 0013, LLP 0017, LLP 0028
+
+## Decision
+
+Automatic projection fires on a **scheduled interval inside the daemon**, reusing
+the shape of the existing cache-maintenance loop
+([`runtime.js` maintenance tick](../src/core/daemon/runtime.js)): a config-driven
+interval, an `unref()`'d timer, a single-flight guard, a per-tick time budget, and
+error isolation. It does not fire on a per-commit or post-settle event.
+
+## Alternatives considered
+
+**Post-settle / commit event hook (rejected).** Projecting incrementally the
+moment new source rows commit gives the freshest possible graph. But it requires a
+kernel commit/settle seam that LLP 0023 explicitly noted does not exist, and it
+couples derivation latency to the capture hot path, reintroducing the exact
+blocking risk LLP 0023 feared. The graph answers "what happened," so real-time
+freshness buys little ([LLP 0086 R5](./0086-automatic-derivation.spec.md#requirements)).
+
+**Subprocess per tick (rejected as the default, kept as an escape hatch).** The
+daemon could spawn a `hyp graph project` child on the interval, the way the
+client-action reconciler spawns `hyp backfill`
+([LLP 0037](./0037-backfill-on-join.decision.md)), so a heavy projection can never
+grow the daemon's own heap. This is full isolation but adds process-spawn overhead
+every interval, and the T0 projection is deterministic in-process cache I/O, not
+the parquet encoder whose heap behavior motivates subprocess isolation elsewhere.
+A derivation that proves heap-heavy can still opt into subprocess execution
+([LLP 0089](./0089-bounded-isolated-derivation.decision.md)); the default stays
+in-process.
+
+**Scheduled in-process tick (chosen).** It matches a pattern already proven safe
+in the daemon, needs no new kernel seam, and is bounded at any interval by its
+budget ([LLP 0089](./0089-bounded-isolated-derivation.decision.md)). Eventual
+freshness is exactly what the graph needs.
+
+## Cadence
+
+The interval is config-driven (see the `derive` config block in
+[LLP 0091](./0091-automatic-derivation.design.md)). Because each tick is
+budget-capped and single-flight, a short interval is cheap: the design picks a
+default short enough that the graph feels fresh for interactive queries, while any
+value stays safe. The specific default number is a design detail, not part of this
+decision.

llp/0089-bounded-isolated-derivation.decision.md

@@ -0,0 +1,52 @@
+# LLP 0089: bounded, isolated derivation execution
+
+**Type:** decision
+**Status:** Draft
+**Systems:** Daemon
+**Author:** Phil / Claude
+**Date:** 2026-07-07
+**Related:** LLP 0086, LLP 0087, LLP 0088, LLP 0023, LLP 0013, LLP 0017
+
+> This decision is the answer to LLP 0023's objection that projecting from the
+> daemon could block or OOM it. It names the guards that make automatic
+> projection safe in-loop, and they are the guards the cache-maintenance loop
+> already runs under.
+
+## Decision
+
+Every automatic derivation tick runs under four guards, matching the daemon's
+cache-maintenance loop ([`runtime.js`](../src/core/daemon/runtime.js), the
+`maintenance.tick` block):
+
+1. **Time budget.** Each tick carries a `max_tick_ms`-style cap. A tick that
+   reaches its budget yields and resumes next interval. Partial progress is safe
+   because projection is idempotent: LLP 0023 content-addressed ids plus pre-write
+   dedup mean an interrupted tick commits a correct prefix and the next tick picks
+   up the remainder with no double-write.
+2. **Single-flight.** A new tick never starts while one is in flight (mirrors the
+   maintenance loop's `maintenanceInFlight` skip). Overruns coalesce instead of
+   piling up.
+3. **Error isolation.** The tick runs in its own span with catch-and-log (mirrors
+   `daemon.maintenance_failed`). A derivation failure degrades freshness only; it
+   never propagates into the daemon or the capture path.
+4. **Non-blocking lifecycle.** The interval timer is `unref()`'d so it never holds
+   the process open, and shutdown drains any in-flight tick before exit (mirrors
+   the maintenance handle's clear-and-await on shutdown).
+
+## Escape hatch for heap-heavy derivations
+
+In-process budgeting caps wall-time but not peak heap. If a derivation is heavy
+enough that a single tick's peak memory threatens the daemon (the parquet-encoder
+failure mode), that derivation may declare **subprocess execution**; the seam
+supports it ([LLP 0090](./0090-general-derivation-seam.decision.md)). In-process is
+the default because the T0 projection is deterministic cache I/O, not that.
+
+## The growing-scan follow-on
+
+LLP 0023's projection re-scans all source rows each run and filters against the
+committed id set (pre-write dedup). Under a periodic tick that scan grows with
+history. The time budget caps each tick regardless, so this is a cost concern, not
+a correctness one. The efficient fix, **incremental projection** (advance a
+per-source cursor and project only newly-settled rows), is a follow-on named in
+[LLP 0091](./0091-automatic-derivation.design.md), not required for this slice
+because budget plus idempotence already make repeated full re-scan safe.

llp/0090-general-derivation-seam.decision.md

@@ -0,0 +1,63 @@
+# LLP 0090: general derivation registration seam
+
+**Type:** decision
+**Status:** Draft
+**Systems:** Plugins, Daemon, Graph
+**Author:** Phil / Claude
+**Date:** 2026-07-07
+**Related:** LLP 0086, LLP 0087, LLP 0088, LLP 0089, LLP 0023, LLP 0028, LLP 0024, LLP 0006
+
+## Decision
+
+Core exposes a **derivation registration** on the plugin activation context. A
+plugin registers a named derivation during `activate()`: a runnable plus
+scheduling metadata and a default-on flag. The daemon ticks every registered
+derivation under the LLP 0089 guards. The mechanism is not graph-specific;
+`@hypaware/context-graph` is the first consumer.
+
+```
+ctx.derivations.register({
+  name,        // stable id, e.g. 'graph project'
+  plugin,      // owning plugin
+  defaultOn,   // see below
+  run,         // ({ query, storage, config, budgetMs, signal }) => Promise<result>
+})
+```
+
+This mirrors the registrations plugins already contribute and the daemon already
+drives: sources ([LLP 0012](./0012-sources.spec.md)), sinks
+([LLP 0014](./0014-sinks.spec.md)), commands, verbs, datasets, capabilities. Core
+owns the registry and the tick; each plugin owns its derivation's runnable, id,
+and default-on declaration. Same ownership split as LLP 0023's contract
+contribution: a plugin maps *into* a fixed seam core owns, it does not fork it.
+
+The runnable shape is exactly what the graph engine already consumes
+(`projectGraph({ query, storage, config })`,
+[`project.js`](../hypaware-core/plugins-workspace/context-graph/src/project.js)), so
+the graph plugin wraps its existing engine rather than rewriting it.
+
+## Default-on is a per-derivation property, not a global truth
+
+A derivation declares whether it is default-on. This keeps the "automatic by
+default" policy ([LLP 0087](./0087-automatic-projection-default.decision.md)) from
+silently switching on paid or off-machine work:
+
+- **Deterministic, on-machine, free** derivations declare `defaultOn: true`. The
+  T0 activity graph is one.
+- **Cost- or exfiltration-bearing** derivations declare `defaultOn: false` and run
+  only when the user enables them explicitly. LLP 0028 T1/T2 enrichment is the
+  motivating case: it spends model tokens and can send captured content
+  off-machine, and it already depends on the explicit-opt-in `hypaware.completion`
+  capability ([LLP 0024](./0024-vector-search-plugin.decision.md#embedding-is-a-separate-capability),
+  the same rule that excludes the completion providers from default activation in
+  the bundled allowlist). A default-off derivation requires an explicit
+  per-derivation enable in config; it never rides the global default-on.
+
+## Relationship to enrichment's ongoing regime
+
+LLP 0028 already defines an "ongoing" automatic regime for enrichment (a daily
+batch over newly-settled sessions). That is a related but distinct mechanism
+(model-backed, batch-API-driven). This seam does not absorb it in this slice;
+enrichment keeps its own regime and stays `defaultOn: false` here. Folding the
+ongoing regime into this seam is a later slice
+([LLP 0086 non-goals](./0086-automatic-derivation.spec.md#non-goals)).