Skip to content

memory api: read through the record log (DO NOT MERGE -- wedges retrieve) - #110

Draft
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/memory-api-record-log-read
Draft

memory api: read through the record log (DO NOT MERGE -- wedges retrieve)#110
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/memory-api-record-log-read

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

memory api: read through the record log (DO NOT MERGE -- wedges retrieve)

Correct on the memory surface, wrong on the system. Opened for review and to hold the work; it
must not land until the retrieve wedge below is explained.

What it fixes

Every memory method except ingest and retrieve misbehaves on the record-log backends -- the ones
that ship -- while looking correct on the JSONL one. Five problems, one shape: a method either
resolves to the JSONL adapter's implementation on a backend with no JSONL log, or the native read
skips a stage of the serving pipeline the JSONL read runs.

  • read_all resolved to the JSONL implementation. MatrixArkLocalAdapter precedes the direct mixins
    in the adapter's MRO and also defines read_all, so on a native backend it returned ZERO records
    and every reader built on it -- get, get_all, update, history, keyed recall -- read empty while
    the records sat durably in the record log.

  • A bundled append stores several records as ONE hash field. The wrapper carries no record_type,
    which is what every reader filters on, so bundled records were skipped even once they were read.

  • The native read ran only the LAST of the serving pipeline's three stages:

    • apply_memory_tombstones is what makes forget / delete / reset remove anything. Without it a
      forget wrote its tombstone, reported an accurate removed_count, and served every one of
      those records right back.
    • compact_latest_value_records collapses rows sharing a latest-value key, which for a
      context_event is the event id. Ingest persists an event row twice (hot path, then again when
      extraction commits), so without it both rows serve and one memory lists as two.
  • history reads the RAW log on purpose -- a memory's change history IS the tombstones and
    superseded rows the live view hides -- but called the inherited reader, which returns empty as
    soon as the JSONL log is disabled. The delete-before-extract guard in session_commit reads
    through the same method and was inert on native backends for the same reason.

  • Keyed upsert never ran: three memory paths read the compacted seam directly, so overriding only
    read_all leaves a second ingest under the same identity_key finding nothing to supersede.

  • The native writer skipped the per-ingestion stamp, so identity_key / truth_class and
    expires_at / ephemeral never reached the store; and extraction, rewriting an event row that
    already exists, rebuilt it from the extraction result alone and dropped them again.

Measured over a single-worker gateway, 20-operation sweep, 5 memories: get 404 -> the memory;
get_all 0 then 10-for-5 -> 5; update 500 -> updated; get_all after forget every record -> 0;
reset did not wipe -> wipes; history empty -> ingested/superseded/deleted; keyed recall 404 ->
the current value with upsert superseding; ttl never expired -> expires. 16/20 -> 19/20.
105 tests green, 10 of them new and every one failing without this.

Why it must not merge yet

With a 16-memory store behind a single-worker gateway, the sixth retrieve stops responding for
120s and every retrieve after it is rejected on the proxy pack lane after 40s. The gateway's
whole data path goes with it -- a get_all for a user with NO memories does not return in 60s
while /v1/healthz answers in 3ms -- and it never recovers. Idle host, freshly built matching
binary, no orphaned proxy workers, arms back to back, 16 ingests then 12 retrieves:

without this change    12/12 retrieves, 165-322 ms
with this change       5 fast, then 120s, then 40s rejections to the end of the run

Refuted, so nobody spends the time again

  • A stale proxy binary -- rebuilt from the same tree; reproduces.
  • Host CPU contention -- idle host, load 1.0; reproduces.
  • Orphaned proxy workers holding a store open -- this DID contaminate one round; cleaned;
    reproduces.
  • The pipeline's own cost -- 0.12 -> 0.64 ms per read at 160 records, 1.37 -> 4.73 ms at 1200,
    and only 3-7% of an ingest. Orders of magnitude too small.
  • The retrieval candidate set -- second commit here moves the two added stages off the shared
    choke point onto the memory-API read alone. Same failure, same retrieve.
  • A lock-order inversion on the raw read -- third commit here drops a records_lock held across a
    proxy call. Same failure, same retrieve. (Kept anyway: the lock buys nothing there.)
  • It does NOT reproduce in-process: 10 retrieves against the same adapter without the gateway run
    46-855 ms.

The proxy worker sits at ~3% CPU with zero sockets open while a retrieve hangs -- blocked, not
working. Next place to look is what the gateway does around a retrieve that an in-process call
does not: the audit and summary-dirty writes land on the same single proxy lane.

Note on ingest latency

Ingest looks ~2x slower with this change (16 sequential into one scope: 414 -> 2884 ms before,
347 -> 5584 ms after). That is not overhead. Pointing both trees at the same store, the current
tree's read_all() returns 0 records from a store holding 197; this one returns 112. Ingest was
cheap because it read nothing -- the same defect that made forget serve everything back.

…eve)

Correct on the memory surface, wrong on the system. Opened for review and to hold the work; it
must not land until the retrieve wedge below is explained.

## What it fixes

Every memory method except ingest and retrieve misbehaves on the record-log backends -- the ones
that ship -- while looking correct on the JSONL one. Five problems, one shape: a method either
resolves to the JSONL adapter's implementation on a backend with no JSONL log, or the native read
skips a stage of the serving pipeline the JSONL read runs.

* read_all resolved to the JSONL implementation. MatrixArkLocalAdapter precedes the direct mixins
  in the adapter's MRO and also defines read_all, so on a native backend it returned ZERO records
  and every reader built on it -- get, get_all, update, history, keyed recall -- read empty while
  the records sat durably in the record log.

* A bundled append stores several records as ONE hash field. The wrapper carries no record_type,
  which is what every reader filters on, so bundled records were skipped even once they were read.

* The native read ran only the LAST of the serving pipeline's three stages:
    - apply_memory_tombstones is what makes forget / delete / reset remove anything. Without it a
      forget wrote its tombstone, reported an accurate removed_count, and served every one of
      those records right back.
    - compact_latest_value_records collapses rows sharing a latest-value key, which for a
      context_event is the event id. Ingest persists an event row twice (hot path, then again when
      extraction commits), so without it both rows serve and one memory lists as two.

* history reads the RAW log on purpose -- a memory's change history IS the tombstones and
  superseded rows the live view hides -- but called the inherited reader, which returns empty as
  soon as the JSONL log is disabled. The delete-before-extract guard in session_commit reads
  through the same method and was inert on native backends for the same reason.

* Keyed upsert never ran: three memory paths read the compacted seam directly, so overriding only
  read_all leaves a second ingest under the same identity_key finding nothing to supersede.

* The native writer skipped the per-ingestion stamp, so identity_key / truth_class and
  expires_at / ephemeral never reached the store; and extraction, rewriting an event row that
  already exists, rebuilt it from the extraction result alone and dropped them again.

Measured over a single-worker gateway, 20-operation sweep, 5 memories: get 404 -> the memory;
get_all 0 then 10-for-5 -> 5; update 500 -> updated; get_all after forget every record -> 0;
reset did not wipe -> wipes; history empty -> ingested/superseded/deleted; keyed recall 404 ->
the current value with upsert superseding; ttl never expired -> expires. 16/20 -> 19/20.
105 tests green, 10 of them new and every one failing without this.

## Why it must not merge yet

With a 16-memory store behind a single-worker gateway, the sixth retrieve stops responding for
120s and every retrieve after it is rejected on the proxy pack lane after 40s. The gateway's
whole data path goes with it -- a get_all for a user with NO memories does not return in 60s
while /v1/healthz answers in 3ms -- and it never recovers. Idle host, freshly built matching
binary, no orphaned proxy workers, arms back to back, 16 ingests then 12 retrieves:

    without this change    12/12 retrieves, 165-322 ms
    with this change       5 fast, then 120s, then 40s rejections to the end of the run

## Refuted, so nobody spends the time again

* A stale proxy binary -- rebuilt from the same tree; reproduces.
* Host CPU contention -- idle host, load 1.0; reproduces.
* Orphaned proxy workers holding a store open -- this DID contaminate one round; cleaned;
  reproduces.
* The pipeline's own cost -- 0.12 -> 0.64 ms per read at 160 records, 1.37 -> 4.73 ms at 1200,
  and only 3-7% of an ingest. Orders of magnitude too small.
* The retrieval candidate set -- second commit here moves the two added stages off the shared
  choke point onto the memory-API read alone. Same failure, same retrieve.
* A lock-order inversion on the raw read -- third commit here drops a records_lock held across a
  proxy call. Same failure, same retrieve. (Kept anyway: the lock buys nothing there.)
* It does NOT reproduce in-process: 10 retrieves against the same adapter without the gateway run
  46-855 ms.

The proxy worker sits at ~3% CPU with zero sockets open while a retrieve hangs -- blocked, not
working. Next place to look is what the gateway does around a retrieve that an in-process call
does not: the audit and summary-dirty writes land on the same single proxy lane.

## Note on ingest latency

Ingest looks ~2x slower with this change (16 sequential into one scope: 414 -> 2884 ms before,
347 -> 5584 ms after). That is not overhead. Pointing both trees at the same store, the current
tree's read_all() returns 0 records from a store holding 197; this one returns 112. Ingest was
cheap because it read nothing -- the same defect that made forget serve everything back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant