feat: Dreaming engine for periodic memory consolidation

[helal-muneer]

Summary

Closes #577

Implements Dreaming functionality — periodic memory consolidation and tier promotion that bridges LanceDB Pro's existing , , and into OpenClaw's dreaming lifecycle.

Three-Phase Dreaming Cycle

💤 Light Sleep — Decay + Tier Re-evaluation

• Scans recent memories (configurable lookback, default 7 days)
• Computes Weibull decay scores via decayEngine.scoreAll()
• Evaluates tier transitions via tierManager.evaluateAll()
• Applies promotions/demotions via store.patchMetadata()

🧠 Deep Sleep — Working → Core Promotion

• Filters Working-tier memories by composite score and access count
• Boosts importance (+20%) for qualifying memories
• Promotes to Core tier via patchMetadata()

💫 REM — Pattern Detection & Reflection

• Analyzes category clustering across tiers
• Detects high-importance category patterns
• Creates reflection memories (category: "reflection") with discovered patterns

Changes

File	Description
src/dreaming-engine.ts	New — dreaming engine factory with 3-phase cycle
openclaw.plugin.json	Add dreaming config schema + UI hints
index.ts	Wire dreaming into service lifecycle, add to PluginConfig

Bug Fix Included

• resolveEnvVars() now returns empty string instead of throwing when env var is missing (prevents plugin startup failure when JINA_API_KEY etc. are unset)

Configuration

{
  "dreaming": {
    "enabled": true,
    "timezone": "UTC",
    "storageMode": "inline",
    "verboseLogging": false,
    "phases": {
      "light": { "lookbackDays": 7, "limit": 50 },
      "deep": { "limit": 20, "minScore": 0.5, "minRecallCount": 2 },
      "rem": { "lookbackDays": 30, "limit": 10, "minPatternStrength": 0.6 }
    }
  }
}

Testing

• ✅ Plugin loads successfully with dreaming enabled
• ✅ Dreaming engine initializes on service start
• ✅ Tier transitions apply correctly during light phase
• ✅ Core promotions work during deep phase
• ✅ Pattern detection creates reflection memories in REM phase
• ✅ Graceful error handling per phase (no cascade failures)
• ✅ Cleanup on service stop

[rwmjhb]

Interesting feature with a sound architecture (bridging existing tier-manager and decay-engine). Branch needs a rebase and one scope isolation issue needs fixing before merge.

Please rebase onto current main — stale_base=true and index.ts has had significant recent activity. Rebase needed to surface any conflicts before this can land.

---

Must fix

MR1 — Dreaming ignores scope isolation and synthesizes cross-scope memories into global
The engine fetches memories across all scopes, then stores REM reflections with a hardcoded scope: 'global'. In multi-user or multi-workspace deployments this leaks content between isolated memory spaces — a user's private memories can influence another user's global reflection entries.

Fix: filter store.list() by the active scope in each phase, and tag REM reflections with the same scope as the source memories.

---

Non-blocking

• F1 — resolveEnvVars now returns '' instead of throwing for unset env vars. This silently propagates an empty API key through resolveFirstApiKey's if (!key) guard (which only checks the raw config string, before resolution). Users with "${JINA_API_KEY}" configured but the env var unset will get opaque HTTP 401 failures instead of a clear startup error.
• F2 — REM stores reflection with vector: []. LanceDB uses fixed-dimension Arrow columns — a 0-dimension vector will throw an Arrow schema mismatch on every cycle. The error is caught and logged as ⚠️ REM failed, so the REM phase silently never creates any reflection memories.
• F3 — config.phases.light (and .deep, .rem) accessed without null guard. A minimal config like { "dreaming": { "enabled": true } } crashes with TypeError: Cannot read properties of undefined on first cycle. Add a DEFAULT_DREAMING_CONFIG and deep-merge it at parse time.
• F6 — storageMode ('separate'/'both') and cron fields are exposed in configSchema and UI hints but never read at runtime. Users selecting these options silently get inline behavior and 6-hour fixed intervals. Remove from schema until implemented, or log a warning when set.

[rwmjhb]

The three-phase dreaming concept is interesting, but there are several correctness blockers that need to be resolved before this is mergeable.

Must fix before merge:

• MR1: Scope isolation violated. The dreaming engine can synthesize cross-scope memories into global, defeating the per-user/per-agent isolation guarantees the rest of the plugin provides. Dreaming should only consolidate within the originating scope.

• MR2: REM output re-enters the consolidation pool. REM reflections are stored as generic legacy working memories, so subsequent dreaming cycles can reprocess their own output — unbounded feedback loop.

• F2: REM phase stores reflection with vector: []. LanceDB enforces a fixed vector dimension. An empty vector will crash every REM cycle against the schema constraint.

• F3: config.phases accessed without null guard. A minimal dreaming config (omitting phases) throws a TypeError at startup.

• F6: storageMode and cron fields are schema-advertised but entirely unimplemented. Shipping config knobs that silently do nothing is a user-trust issue. Either implement or remove from schema before merge.

Nice to have (non-blocking):

• F1: resolveEnvVars silently returns '' for unset env vars — a missing embedding API key becomes an empty string with no warning.
• F4: Initial setTimeout handle is discarded — the first cycle can fire after stop() if the plugin restarts within 5 minutes.
• EF1: The 319-line dreaming engine has zero automated test coverage across all three phases.

Address the must-fix items and this is ready for a proper review.

[rwmjhb]

The schema gap causing UI validation errors is a real problem, and bridging tier-manager + decay-engine is the right architectural approach. However there are several correctness issues that need to land before this is safe to enable.

Must fix

• Scope isolation: The dreaming engine queries memories without filtering by scope, then writes REM reflections into global. If a user has per-agent or per-channel scopes, Dreaming will synthesize cross-scope memories together and persist the result where all scopes can see it — a privacy/isolation regression. Scope filtering must be threaded through all three phases before this ships.
• REM reflection reprocessing: REM reflections are stored as generic working memories. On the next cycle, the engine will pick them up as ordinary candidates and dream about its own previous reflections indefinitely. Tag REM output (e.g. source: 'rem-reflection') and exclude it from future phase queries.
• Dead config knobs: storageMode: 'separate' | 'both' and cron are exposed in the config schema and plugin JSON but do nothing at runtime (engine hardcodes 'inline' and a 6 h setInterval). Users who configure these will be silently misled. Either implement them or remove them from the schema until they're ready.

High priority

• store.store({ vector: [] }) for REM memories: LanceDB enforces a fixed vector dimension — an empty vector is likely to corrupt the index or cause retrieval errors on every subsequent search. At minimum, embed the reflection text before storing, or skip vector storage and mark it for deferred embedding.
• config.phases is accessed without a null guard — a minimal dreaming config (phases omitted) will throw a TypeError before any phase runs.
• resolveEnvVars behavior change (throw → return '') is bundled into this feature PR and silently swallows missing API keys (e.g. JINA_API_KEY). The original throw was intentional for required keys. Please split this out or scope it only to optional vars.

Other

• The setTimeout handle for the initial 5-minute delay is discarded — stop() can't cancel it, so a restart within 5 minutes triggers a double-cycle.
• All phases use fetch-then-filter; memories beyond the batch limit are silently dropped. Worth a comment at minimum.
• Zero test coverage for the three-phase engine — at least a unit test per phase for the happy path would reduce risk significantly.

Question for author: Was Strategy B (bridge mode) explicitly endorsed by a maintainer? Issue #577 describes three strategies but shows no maintainer response before implementation started — want to make sure the approach was aligned.