Opt-in historical revision indexing for the Basic Memory data lake

[phernandez]

Summary

Add an opt-in historical revision index for Basic Memory: a policy-aware “BM data lake” that can retain, query, and search immutable note/document revisions without changing the default current-head experience.

This starts where #1156 intentionally stops. #1156 defines backend-neutral history, historical reads, compare, and restore. This issue adds a durable revision catalog plus optional historical metadata/text/vector indexing.

Product model

By default, Basic Memory continues to index and search only the current note head.

Eligible Cloud/enterprise projects can enable revision indexing so users or agents can ask:

• what versions existed for this note or stable entity?
• find content that was present in an older revision;
• search raw extraction and later agent-enriched revisions separately;
• query by revision kind, actor/source, extractor, model, prompt, checksum, or time range;
• search “as of” a time without pretending the current graph describes the past;
• preserve selected revisions under legal hold or an explicit retention class.

“Save everything forever” is not a safe universal default. Retention, deletion, tenant isolation, quotas, and legal hold are product policy, not storage implementation details.

Architecture

revision sources
  - Tigris object versions / snapshots
  - local Git or JJ revisions
          |
          v
Core RevisionRecord contract
          |
          +--> revision catalog and lineage
          |      identity, path, checksums, timestamps,
          |      provenance, permissions, retention
          |
          +--> current-head index       # always/default
          |
          +--> historical revision index # opt-in/metered

Tigris remains the immutable byte store for Cloud. Postgres holds the queryable revision catalog and stable lineage. Search backends hold derived text/vector indexes and may be rebuilt from authoritative bytes plus catalog metadata.

Do not turn the current NoteContent head table into an append-only history store.

Identity model

Keep these layers explicit:

• entity_external_id — stable Basic Memory identity across revisions and, eventually, moves;
• revision_id — provider-neutral opaque public revision reference;
• db_version — logical accepted-head ordering;
• db_checksum — canonical content identity and CAS precondition;
• Tigris version_id — exact stored object revision for one key;
• storage ETag — transport/storage precondition, not assumed to be a content SHA;
• snapshot/commit reference — point-in-time project or tree reference;
• parent_revision_id / parent_checksum — lineage where known.

A write may be accepted in Postgres before its Tigris object version is materialized. Catalog state must represent that explicitly instead of fabricating a storage version ID.

Proposed Core contract

Define a backend-neutral revision record conceptually containing:

revision_id: <opaque>
entity_external_id: <stable entity>
project_id: <project>
path: notes/report.md
content_type: text/markdown
created_at: <RFC3339>
is_current: true
materialization_status: synced
db_version: 2
db_checksum: sha256:<...>
provider_version_id: <opaque optional>
parent_revision_id: <opaque optional>
revision_kind: agent_enriched
actor:
  kind: service
  id: document-ingester
source:
  kind: pdf_extractor
ingestion:
  run_id: <uuid>
  source_entity_id: <PDF entity>
  source_checksum: sha256:<...>
  extractor: firecrawl/pdf-inspector
  extractor_version: <version>
  pipeline_version: bm-document-ingest-v1
  prompt_version: document-normalize-v1
retention:
  class: standard
  retain_until:
  legal_hold: false
deleted_at:

The exact schema needs contract review. Core owns terminology, public response models, current-vs-historical search semantics, and capability errors. Cloud/local adapters own provider details.

Query semantics

Current behavior remains the safe default:

search_notes(...) -> current heads only
build_context(...) -> current heads only
graph(...) -> current heads only

Historical access is explicit:

search_revisions(
  query=...,
  entity_external_id=...,
  revision_kind=...,
  after=...,
  before=...,
  as_of=...,
  include_deleted=false,
)

Possible later shorthand:

search_notes(..., revision_scope="current|historical|all")

Historical results must always identify their revision and whether they are current. They must not silently hydrate current observations/relations onto historical bodies.

Metadata indexing

Index catalog metadata independently from body/vector content so operators can query lineage and policy without fetching or embedding every historical body.

Text/vector indexing

• Deduplicate chunks by canonical chunk/content hash.
• Reuse embeddings for unchanged chunks across revisions.
• Keep revision_id, entity_external_id, is_current, timestamps, and access scope on each historical hit.
• Make indexing asynchronous, replayable, and idempotent.
• Allow retention policy to remove historical index entries while authoritative bytes remain retained, or remove both when policy requires deletion.

Moves and deletes

Tigris history is path/key based. Stable entity history across moves requires an explicit catalog; rename inference is not sufficient.

• Record old and new paths against the same stable entity when identity is proven.
• Represent delete markers/tombstones explicitly.
• Enforce project/tenant authorization on every historical read and search result.
• Deletion/GDPR workflows must cover catalog rows, search/vector indexes, cached content, raw extraction artifacts, and provider versions according to policy.

Delivery phases

Phase 1 — revision catalog

• Core RevisionRecord models and repository contracts
• Cloud catalog rows written from accepted/materialized note lifecycle
• explicit pending/synced/failed/deleted states
• stable entity/path lineage
• metadata-only history queries
• backfill/reconciliation from current Tigris versions where possible

Phase 2 — historical text search

• explicit historical search API/MCP contract
• per-revision body extraction on index
• current vs historical result labeling
• tenant-safe pagination and authorization
• replay/backfill tooling and observability

Phase 3 — vector and temporal search

• chunk-hash embedding reuse
• as_of/time-range semantics
• cost accounting, quotas, and metering
• result grouping by stable entity/revision lineage

Phase 4 — retention and enterprise policy

• configurable retention classes
• legal hold
• tenant export and audit
• deletion/GDPR enforcement
• storage/index reconciliation
• admin controls and capacity reporting

Local Git/JJ indexing can follow the same Core contract after #1156 selects and implements the local revision source.

Acceptance criteria

Catalog foundation

• Every accepted note revision can be represented without conflating DB, content, provider, or snapshot identities.
• Accepted-but-not-materialized revisions have an honest pending state.
• Stable entity lineage survives path changes when Basic Memory identity is available.
• Catalog writes and lifecycle reconciliation are idempotent.
• Metadata history is queryable without fetching every historical body.
• Tenant/project authorization applies to catalog reads.

Historical search

• Default search_notes, build_context, and graph behavior remain current-only.
• Historical search requires an explicit capability/scope.
• Every result returns revision identity, stable entity identity, timestamp, path, and is_current.
• Historical bodies never inherit current observations/relations.
• Repeated content/chunks do not require repeated embeddings.
• Indexing is replayable from authoritative revision bytes and catalog metadata.

Policy and operations

• Retention class, retain-until, deletion, and legal-hold state are modeled explicitly.
• Deletion covers bytes, catalog, text/vector indexes, and derived artifacts according to policy.
• Reconciliation detects missing bytes, orphan catalog rows, and orphan index entries.
• Quotas/cost accounting prevent unbounded tenant growth.
• Audit events identify who or what requested historical content.

Non-goals

• Making every Basic Memory project retain and index all revisions forever.
• Treating Tigris ETags as canonical SHA-256 checksums.
• Using the current head database row as revision history.
• Reconstructing a historical graph in the first implementation.
• Implicitly mixing current and historical results.
• Duplicating immutable file bodies in Postgres.

Related work

• #1156 — revision-source contract, historical reads, compare, restore
• #1006 — PDF/document extraction; raw and enriched revisions are an early motivating case
• basic-memory-cloud#1636 — version churn demonstrates why dedup, retention, and metering are required

[claude]

Claude finished @phernandez's task in 2m 16s —— View job

---

Triage complete

• Gather context (issue body, related issues Native note revision history across cloud and local projects #1156, Parse PDFs into searchable LiteParse sidecar notes #1006)
• Classify type, priority, component, complexity
• Check for duplicate/related issues
• Apply labels
• Post triage summary

Classification

• Type: Feature request / architectural proposal (large enhancement spanning Core, Cloud, Postgres, and search indexing — not a bug or docs task).
• Priority: Low/strategic. This is explicitly opt-in and forward-looking (phased, Cloud/enterprise-only), with no impact on the default current-head experience. No urgency to act on immediately.
• Component: Spans cloud (Postgres catalog, Tigris versions, retention/legal hold), database/search indexing, and MCP (new search_revisions surface). Repo doesn't have granular database/mcp/priority/complexity labels, so I've kept it to what actually exists in this repo's label set.
• Complexity: Complex — a 4-phase roadmap (revision catalog → historical text search → vector/temporal search → retention & enterprise policy) touching Core contracts, Cloud adapters, Postgres schema, search backends, and GDPR/retention policy.

Labels applied: arch-review (in addition to the enhancement and cloud labels already on the issue). Note: the repo's actual label set (arch-review, bug, cloud, documentation, duplicate, enhancement, needs investigation, python, question, v1.0, validated) doesn't include the granular feature/priority: low/complexity: complex/component: database|mcp labels listed in my triage instructions, so those were skipped rather than silently applied as no-ops. arch-review was the best fit given the issue itself calls for schema/contract review before implementation.

Duplicate check: Searched issues for "revision index", "historical revision", "data lake", and "revision catalog" — no duplicates found. The issue's own "Related work" section already correctly cross-references the relevant issues:

• Native note revision history across cloud and local projects #1156 (open) — "Native note revision history across cloud and local projects": defines the backend-neutral history/read/compare/restore contract this issue explicitly builds on top of.
• Parse PDFs into searchable LiteParse sidecar notes #1006 (open) — PDF/document extraction, cited as an early motivating case for raw vs. agent-enriched revisions.
• basic-memory-cloud#1636 (external repo, not independently verifiable from here).

This issue is a well-scoped continuation of #1156 rather than an overlapping proposal — it explicitly says "This starts where #1156 intentionally stops."

Notes for maintainers:

• This is a design/RFC-level issue with detailed acceptance criteria already included; given its scope (new Core contract, Postgres schema, retention/GDPR policy, quotas/metering), it likely warrants a written spec (see the spec skill/process referenced in this repo) and explicit sign-off before Phase 1 implementation begins.
• No clarifying questions needed — the proposal is thorough and self-contained (identity model, query semantics, delivery phases, non-goals, and acceptance criteria are all spelled out).

Per my triage scope, no implementation, file edits, or branch/PR work was performed.