memory api: read through the record log (DO NOT MERGE -- wedges retrieve) - #110
Draft
bjmeetsfo wants to merge 1 commit into
Draft
memory api: read through the record log (DO NOT MERGE -- wedges retrieve)#110bjmeetsfo wants to merge 1 commit into
bjmeetsfo wants to merge 1 commit into
Conversation
…eve)
Correct on the memory surface, wrong on the system. Opened for review and to hold the work; it
must not land until the retrieve wedge below is explained.
## What it fixes
Every memory method except ingest and retrieve misbehaves on the record-log backends -- the ones
that ship -- while looking correct on the JSONL one. Five problems, one shape: a method either
resolves to the JSONL adapter's implementation on a backend with no JSONL log, or the native read
skips a stage of the serving pipeline the JSONL read runs.
* read_all resolved to the JSONL implementation. MatrixArkLocalAdapter precedes the direct mixins
in the adapter's MRO and also defines read_all, so on a native backend it returned ZERO records
and every reader built on it -- get, get_all, update, history, keyed recall -- read empty while
the records sat durably in the record log.
* A bundled append stores several records as ONE hash field. The wrapper carries no record_type,
which is what every reader filters on, so bundled records were skipped even once they were read.
* The native read ran only the LAST of the serving pipeline's three stages:
- apply_memory_tombstones is what makes forget / delete / reset remove anything. Without it a
forget wrote its tombstone, reported an accurate removed_count, and served every one of
those records right back.
- compact_latest_value_records collapses rows sharing a latest-value key, which for a
context_event is the event id. Ingest persists an event row twice (hot path, then again when
extraction commits), so without it both rows serve and one memory lists as two.
* history reads the RAW log on purpose -- a memory's change history IS the tombstones and
superseded rows the live view hides -- but called the inherited reader, which returns empty as
soon as the JSONL log is disabled. The delete-before-extract guard in session_commit reads
through the same method and was inert on native backends for the same reason.
* Keyed upsert never ran: three memory paths read the compacted seam directly, so overriding only
read_all leaves a second ingest under the same identity_key finding nothing to supersede.
* The native writer skipped the per-ingestion stamp, so identity_key / truth_class and
expires_at / ephemeral never reached the store; and extraction, rewriting an event row that
already exists, rebuilt it from the extraction result alone and dropped them again.
Measured over a single-worker gateway, 20-operation sweep, 5 memories: get 404 -> the memory;
get_all 0 then 10-for-5 -> 5; update 500 -> updated; get_all after forget every record -> 0;
reset did not wipe -> wipes; history empty -> ingested/superseded/deleted; keyed recall 404 ->
the current value with upsert superseding; ttl never expired -> expires. 16/20 -> 19/20.
105 tests green, 10 of them new and every one failing without this.
## Why it must not merge yet
With a 16-memory store behind a single-worker gateway, the sixth retrieve stops responding for
120s and every retrieve after it is rejected on the proxy pack lane after 40s. The gateway's
whole data path goes with it -- a get_all for a user with NO memories does not return in 60s
while /v1/healthz answers in 3ms -- and it never recovers. Idle host, freshly built matching
binary, no orphaned proxy workers, arms back to back, 16 ingests then 12 retrieves:
without this change 12/12 retrieves, 165-322 ms
with this change 5 fast, then 120s, then 40s rejections to the end of the run
## Refuted, so nobody spends the time again
* A stale proxy binary -- rebuilt from the same tree; reproduces.
* Host CPU contention -- idle host, load 1.0; reproduces.
* Orphaned proxy workers holding a store open -- this DID contaminate one round; cleaned;
reproduces.
* The pipeline's own cost -- 0.12 -> 0.64 ms per read at 160 records, 1.37 -> 4.73 ms at 1200,
and only 3-7% of an ingest. Orders of magnitude too small.
* The retrieval candidate set -- second commit here moves the two added stages off the shared
choke point onto the memory-API read alone. Same failure, same retrieve.
* A lock-order inversion on the raw read -- third commit here drops a records_lock held across a
proxy call. Same failure, same retrieve. (Kept anyway: the lock buys nothing there.)
* It does NOT reproduce in-process: 10 retrieves against the same adapter without the gateway run
46-855 ms.
The proxy worker sits at ~3% CPU with zero sockets open while a retrieve hangs -- blocked, not
working. Next place to look is what the gateway does around a retrieve that an in-process call
does not: the audit and summary-dirty writes land on the same single proxy lane.
## Note on ingest latency
Ingest looks ~2x slower with this change (16 sequential into one scope: 414 -> 2884 ms before,
347 -> 5584 ms after). That is not overhead. Pointing both trees at the same store, the current
tree's read_all() returns 0 records from a store holding 197; this one returns 112. Ingest was
cheap because it read nothing -- the same defect that made forget serve everything back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
memory api: read through the record log (DO NOT MERGE -- wedges retrieve)
Correct on the memory surface, wrong on the system. Opened for review and to hold the work; it
must not land until the retrieve wedge below is explained.
What it fixes
Every memory method except ingest and retrieve misbehaves on the record-log backends -- the ones
that ship -- while looking correct on the JSONL one. Five problems, one shape: a method either
resolves to the JSONL adapter's implementation on a backend with no JSONL log, or the native read
skips a stage of the serving pipeline the JSONL read runs.
read_all resolved to the JSONL implementation. MatrixArkLocalAdapter precedes the direct mixins
in the adapter's MRO and also defines read_all, so on a native backend it returned ZERO records
and every reader built on it -- get, get_all, update, history, keyed recall -- read empty while
the records sat durably in the record log.
A bundled append stores several records as ONE hash field. The wrapper carries no record_type,
which is what every reader filters on, so bundled records were skipped even once they were read.
The native read ran only the LAST of the serving pipeline's three stages:
forget wrote its tombstone, reported an accurate removed_count, and served every one of
those records right back.
context_event is the event id. Ingest persists an event row twice (hot path, then again when
extraction commits), so without it both rows serve and one memory lists as two.
history reads the RAW log on purpose -- a memory's change history IS the tombstones and
superseded rows the live view hides -- but called the inherited reader, which returns empty as
soon as the JSONL log is disabled. The delete-before-extract guard in session_commit reads
through the same method and was inert on native backends for the same reason.
Keyed upsert never ran: three memory paths read the compacted seam directly, so overriding only
read_all leaves a second ingest under the same identity_key finding nothing to supersede.
The native writer skipped the per-ingestion stamp, so identity_key / truth_class and
expires_at / ephemeral never reached the store; and extraction, rewriting an event row that
already exists, rebuilt it from the extraction result alone and dropped them again.
Measured over a single-worker gateway, 20-operation sweep, 5 memories: get 404 -> the memory;
get_all 0 then 10-for-5 -> 5; update 500 -> updated; get_all after forget every record -> 0;
reset did not wipe -> wipes; history empty -> ingested/superseded/deleted; keyed recall 404 ->
the current value with upsert superseding; ttl never expired -> expires. 16/20 -> 19/20.
105 tests green, 10 of them new and every one failing without this.
Why it must not merge yet
With a 16-memory store behind a single-worker gateway, the sixth retrieve stops responding for
120s and every retrieve after it is rejected on the proxy pack lane after 40s. The gateway's
whole data path goes with it -- a get_all for a user with NO memories does not return in 60s
while /v1/healthz answers in 3ms -- and it never recovers. Idle host, freshly built matching
binary, no orphaned proxy workers, arms back to back, 16 ingests then 12 retrieves:
Refuted, so nobody spends the time again
reproduces.
and only 3-7% of an ingest. Orders of magnitude too small.
choke point onto the memory-API read alone. Same failure, same retrieve.
proxy call. Same failure, same retrieve. (Kept anyway: the lock buys nothing there.)
46-855 ms.
The proxy worker sits at ~3% CPU with zero sockets open while a retrieve hangs -- blocked, not
working. Next place to look is what the gateway does around a retrieve that an in-process call
does not: the audit and summary-dirty writes land on the same single proxy lane.
Note on ingest latency
Ingest looks ~2x slower with this change (16 sequential into one scope: 414 -> 2884 ms before,
347 -> 5584 ms after). That is not overhead. Pointing both trees at the same store, the current
tree's read_all() returns 0 records from a store holding 197; this one returns 112. Ingest was
cheap because it read nothing -- the same defect that made forget serve everything back.