Keep every message of a batched ingest, and fix three extraction defects - #148
Open
bjmeetsfo wants to merge 3 commits into
Open
Keep every message of a batched ingest, and fix three extraction defects#148bjmeetsfo wants to merge 3 commits into
bjmeetsfo wants to merge 3 commits into
Conversation
A ten-message batch ingest reported success and kept ONE message. The other nine
were unreachable through retrieval, on the write path, silently.
The sync-accept path writes a single pending context_event for the WHOLE
envelope. The commit path re-emits one event PER MESSAGE, taking each id from
source_event_ids -- and skipped any message whose index ran past that list:
for index, message in enumerate(envelope["messages"]):
if index >= len(source_event_ids):
continue
A batch therefore arrived with one source id for ten messages, so nine were
dropped. Worse, the one re-emitted event reused source_event_ids[0], so
latest-value compaction replaced the full 359-char pending event with a 35-char
event holding only the first message. Nothing errored and the call returned
accepted.
before commit id=8185658655269157160 status=observed len=359 tokens=[0..9]
after commit id=8185658655269157160 status=extraction_committed len=35 tokens=[0]
Messages past the supplied ids now get an id derived exactly as the non-derived
branch does -- stable for the same batch and message position -- so a batch
leaves one event per message with distinct ids:
after the fix 10 events, tokens=[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]
The text was never lost from the store; it survived in the session summary. But
retrieval returns EVENTS, so a fact ingested in a batch could not be recalled --
which is what the end-to-end check caught: "batch-ingested event" was the only
failing case of fifteen, and now passes.
Single-message ingest is untouched: it always supplied one id for one message.
Four tests added; two fail on the unfixed code. They assert every message
survives commit, that a batch leaves one event per message, that event ids are
distinct (reusing one id is what let compaction hide the loss), and that the
single-message path is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e ingest
A single bundled ingest of eight messages was enough to surface all three: a
warning that now lies, a location that swallows the clause after it, and one
preference stored twice.
It told callers a batch "retains ~1 raw context_event; the rest survive only as
lossy extractions and their exact facts become unretrievable", and to ingest
per-turn instead. That was true until the commit loop stopped skipping messages
past source_event_ids. Measured now, bundled and per-turn are identical on every
context record type, and bundled is CHEAPER on per-call bookkeeping:
record_type BUNDLED ONE-BY-1
context_event 6 6
context_entity 6 6
context_embedding 18 18
context_summary 2 2
matrixark_idempotency 2 7
So the warning was pushing callers onto the more expensive path to avoid a loss
that cannot occur. It is removed rather than reworded, and the tests that
asserted it are replaced by ones asserting the equivalence -- a regression that
reintroduced the loss would now fail on the facts, not on the warning text.
"I live in Seattle and prefer metric units" -> "Seattle and prefer metric units"
The capitalised pattern above it already stopped at the place; this fallback
exists for lowercase locations ("the downtown office") and had no such bound. It
now stops at the clause boundary, which also keeps the bad value out of the
profile summary text that carries cross-session memory. 8/8 on a battery
covering conjunctions, trailing purpose clauses, and "Austin, Texas".
Two extractors fire on one clause: the directive scan produces state "user
directive: prefer metric units" named "preference:prefer metric units", the
pattern table produces state "metric units" named "preference". Different names,
so the name-keyed dedupe never saw them as one -- and promotion to the profile
node doubled both again.
The PLAIN entity is the one kept. Its name is the bare entity type, which is the
stable identity cross-session supersession matches on; the directive name embeds
its own text and cannot supersede a later correction. Keeping the directive
instead reads better -- it retains the verb -- and breaks lineage, which is what
test_profile_preference_correction_supersedes_stale_cross_session_state caught
when I had the rule the wrong way round. Only directive-prefixed entities are
ever dropped, and only when a same-type plain entity is literally contained in
them, so project "Aurora" cannot absorb "Aurora v2".
Entities for the demo conversation: 10 -> 8, location now "Seattle".
Full codex pipeline suite: failing names identical to baseline.
This test, added in 1f0eb99f, failed roughly one run in three -- in any tree, at any commit -- and its failures have been polluting baseline comparisons all session. Several names I attributed to order-dependence were this. Two independent causes, both mine: ## 1. It asserted equality on a count that legitimately differs context_summary and context_node were compared between the bundled and per-turn arms. node_l1_generation_policy gates L1 on event_count, so a single bundled call and a sequence of per-turn calls evaluate that gate at different counts and emit a different number of summaries. That is correct behaviour; the assertion was wrong. Equality now covers the retention-critical types only -- context_event, context_entity, context_embedding -- which is what the test exists to defend: batching loses no MESSAGE. ## 2. It raced the embedding worker Reading immediately after commit catches a varying amount of in-flight background work. Measured across runs, the batched arm returned 19, 20, 21 or 22 context_embedding records against a steady 18 for per-turn -- the harness racing the worker, not a difference between ingest shapes. It now waits for the record count to settle before comparing, the same remedy the attachment-policy test needed for the same reason. ## How it was found A full-suite run flagged it, and bisecting across all seven commits of this branch showed it passing at EVERY one -- while failing in the live tree at the same commit. Not a regression: an isolated worktree reproduced OK/FAILED/OK and the live tree FAILED/FAILED/OK. The commit-by-commit result was the tell; a test that passes at every commit and fails on the tip is nondeterministic, not broken by a change. 6/6 on the test and 5/5 on the whole file after the fix, against ~1-in-3 before. Recorded also because the first bisect attempt was run in the live worktree with git stash and git checkout per commit. That left a conflicted wal.rs and tried to pop a stash belonging to ANOTHER session. The stash survived (the pop failed cleanly) and only the one file needed restoring, but a shared tree is the wrong place to bisect -- the second attempt used a throwaway worktree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
maincurrently loses messages on a batched ingest, and its own test says so.test_batch_event_integrityfails on cleanmainwith 2 failures — the test wasmerged but the fix behind it never was.
Measured on this branch's base: a six-message batch retained one message.
Three commits, in the order they have to land.
1. Keep every message of a batch ingest
The sync-accept path writes ONE pending event for the whole envelope; the commit path
re-emits one event PER MESSAGE from
source_event_ids, and skipped any message whoseindex ran past that list:
A batch arrives with one source id for N messages, so N−1 were dropped. Worse, the one
re-emitted event reused
source_event_ids[0], so latest-value compaction replaced thefull pending event with a short one holding only the first message. Nothing errored and
the call returned accepted.
Messages past the supplied ids now derive an id exactly as the non-derived branch does —
stable for the same batch and message position — so a batch leaves one event per message
with distinct ids. Distinct ids matter independently: reusing one id is what let
compaction hide the loss.
test_batch_event_integritygoes from 2 failures to green.2. Three extraction and ingest defects
The location pattern ran to the end of the sentence.
"I live in Seattle and prefer metric units"stored the location as"Seattle and prefer metric units", which thenpolluted the profile summary text carrying cross-session memory. It now stops at the
clause boundary. Verified on
main's own extractor: the value isSeattle.One preference became two entities, then four. Two extractors fire on one clause and
produce different
entity_names, so the name-keyed dedupe never saw them as one — andpromotion to the profile node doubled both again. The PLAIN entity is the one kept: its
name is the bare entity type, which is the stable identity cross-session supersession
matches on, where the directive name embeds its own text and cannot supersede a later
correction.
The batched-ingest warning is removed — but only because commit 1 above makes it
false. It told callers a batch retains ~1 raw event and loses the rest, which is
accurate on
maintoday. Removing it without the fix would have deleted a true warning;that ordering is why these ship together.
3. Make the equivalence test deterministic
The test replacing that warning failed roughly one run in three, for two reasons. It
asserted equality on a summary count that legitimately varies —
node_l1_generation_policygates L1 on
event_count, so a bundled call and per-turn calls evaluate that gate atdifferent counts. And it raced the background embedding worker, which returned 19–22
embedding records against a steady 18. It now compares the retention-critical types only
and waits for the record count to settle.
Verification
test_batch_event_integrity— 2 failures → greentest_matrixark_profile_scope_warning+test_batch_event_integrity— 4/4 runs green, checked repeatedly because one commit is a flake fixmain: one name fixed, one appears —test_retrieval_audit_is_off_by_default, verified to fail identically on cleanmain(a run-context artifact, not caused here)Conflict resolution notes
Two hunks needed resolving, both kept minimal on purpose:
matrixark_local_adapter_ingest.py— keptCALLER_SUPPLIED_EVENT_FIELDS, whichmaingained independently, and dropped only the warning block.
matrixark_mcp_core.py— boundedmain's existing location pattern. The incoming sidealso carried a capitalised-location and a relationship pattern from earlier commits that
were never mirrored; bringing those in would have smuggled unrelated changes into a
bug-fix PR, so they were left out.