feat(memory): add native SQLite and MCAP recorder stores

[TomCC7]

Contribution path

• No linked issue; draft implementation for native Mem2 recording.

Problem

The Python recorder performs encoding and durable writes in its module process. High-throughput image and point-cloud streams need a native record path with configurable encoding concurrency, while recorded artifacts must remain readable through the existing Python memory and replay APIs.

Solution

Add RustRecorder, a NativeModule wrapper backed by dimos-memory-recorder, and put native persistence behind a record-only Rust RecordingStore trait.

• SQLite store: preserves the Mem2 schema and Python registry, supports lcm, jpeg, and lz4+lcm storage codecs, splits TF batches, writes JSONB tags, and remains directly replayable through SqliteStore.
• MCAP store: records original LCM wire payloads into indexed Zstd chunks. MCAP publish time carries source time and log time carries recorder reception time. Channel metadata declares the DimOS payload type, allowing McapStore to discover and decode native artifacts without a caller-supplied codec registry.
• Recorder pipeline: uses a bounded encoder pool sized by encoding_threads; one writer restores arrival order, batches writes, and finalizes the selected artifact during graceful shutdown.
• Configuration: selects a typed RustSqliteStoreConfig or RustMcapStoreConfig. Native launch configuration is sent only as stdin JSON; the obsolete generated config CLI-argument exclusion list is removed.

MCAP deliberately rejects storage-codec overrides and append mode because it represents a transport capture. This PR remains record-only: no Rust replayer, Python binding, or standalone user-facing recorder CLI is added.

How to test

uv run pytest dimos/memory -q
# 470 passed, 1 optional-MCAP test skipped when the Python mcap package is absent

cd native/rust
cargo test -p dimos-memory-recorder
cargo clippy -p dimos-memory-recorder --all-targets -- -D warnings
# 9 Rust tests passed; Clippy passed

The optional Python MCAP test passes in an environment with mcap installed. A release-binary LCM interoperability check recorded 30 typed IMU messages to each store and replayed 30/30 from both artifacts through Python with exact LCM payload equality and source timestamps. A fresh reliable-Zenoh harness run captured and replayed 1,500/1,500 observations at roughly 65 MiB/s offered input. Ruff, mypy, the optimized release build, and all 17 tests in dim-recorder-perf also pass.

AI assistance

OpenAI Codex with GPT-5 implemented the Rust recorder stores, Python integration, tests, documentation, performance-harness migration, PR draft, and verification under user direction.

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 85.19737% with 45 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/memory/rust_recorder.py	83.21%	16 Missing and 8 partials ⚠️
dimos/memory/store/test_mcap.py	32.00%	17 Missing ⚠️
dimos/memory/store/mcap.py	80.00%	1 Missing and 2 partials ⚠️
dimos/memory/codecs/lcm.py	66.66%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3615      +/-   ##
==========================================
+ Coverage   77.19%   77.28%   +0.09%     
==========================================
  Files        1260     1265       +5     
  Lines      119881   120598     +717     
  Branches    10581    10667      +86     
==========================================
+ Hits        92539    93202     +663     
- Misses      24276    24323      +47     
- Partials     3066     3073       +7     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	72.19% <81.90%> (+0.12%)	⬆️
OS-ubuntu-latest	74.08% <81.90%> (+0.10%)	⬆️
Py-3.10	74.07% <81.90%> (+0.10%)	⬆️
Py-3.11	74.08% <81.90%> (+0.10%)	⬆️
Py-3.12	74.07% <81.90%> (+0.09%)	⬆️
Py-3.13	74.06% <81.90%> (+0.09%)	⬆️
Py-3.14	74.08% <81.90%> (+0.09%)	⬆️
Py-3.14t	74.07% <81.90%> (+0.10%)	⬆️
SelfHosted-Large	30.03% <40.13%> (+<0.01%)	⬆️
SelfHosted-Linux	35.15% <43.09%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/memory/test_rust_recorder.py	100.00% <100.00%> (ø)
dimos/robot/all_blueprints.py	100.00% <ø> (ø)
dimos/memory/codecs/lcm.py	91.66% <66.66%> (-8.34%)	⬇️
dimos/memory/store/mcap.py	80.00% <80.00%> (-0.59%)	⬇️
dimos/memory/store/test_mcap.py	32.00% <32.00%> (ø)
dimos/memory/rust_recorder.py	83.21% <83.21%> (ø)

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 24, 2026, 4:35 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.