Skip to content

Keep every message of a batched ingest, and fix three extraction defects - #148

Open
bjmeetsfo wants to merge 3 commits into
mainfrom
oss-pr/batch-message-loss
Open

Keep every message of a batched ingest, and fix three extraction defects#148
bjmeetsfo wants to merge 3 commits into
mainfrom
oss-pr/batch-message-loss

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

main currently loses messages on a batched ingest, and its own test says so.
test_batch_event_integrity fails on clean main with 2 failures — the test was
merged but the fix behind it never was.

Measured on this branch's base: a six-message batch retained one message.

kept:  'user: I am a robotics engineer at Acme.'
lost:  the other five

Three commits, in the order they have to land.

1. Keep every message of a batch ingest

The sync-accept path writes ONE pending event for the whole envelope; the commit path
re-emits one event PER MESSAGE from source_event_ids, and skipped any message whose
index ran past that list:

for index, message in enumerate(envelope["messages"]):
    if index >= len(source_event_ids):
        continue

A batch arrives with one source id for N messages, so N−1 were dropped. Worse, the one
re-emitted event reused source_event_ids[0], so latest-value compaction replaced the
full pending event with a short one holding only the first message. Nothing errored and
the call returned accepted.

Messages past the supplied ids now derive an id exactly as the non-derived branch does —
stable for the same batch and message position — so a batch leaves one event per message
with distinct ids. Distinct ids matter independently: reusing one id is what let
compaction hide the loss.

test_batch_event_integrity goes from 2 failures to green.

2. Three extraction and ingest defects

The location pattern ran to the end of the sentence. "I live in Seattle and prefer metric units" stored the location as "Seattle and prefer metric units", which then
polluted the profile summary text carrying cross-session memory. It now stops at the
clause boundary. Verified on main's own extractor: the value is Seattle.

One preference became two entities, then four. Two extractors fire on one clause and
produce different entity_names, so the name-keyed dedupe never saw them as one — and
promotion to the profile node doubled both again. The PLAIN entity is the one kept: its
name is the bare entity type, which is the stable identity cross-session supersession
matches on, where the directive name embeds its own text and cannot supersede a later
correction.

The batched-ingest warning is removed — but only because commit 1 above makes it
false. It told callers a batch retains ~1 raw event and loses the rest, which is
accurate on main today. Removing it without the fix would have deleted a true warning;
that ordering is why these ship together.

3. Make the equivalence test deterministic

The test replacing that warning failed roughly one run in three, for two reasons. It
asserted equality on a summary count that legitimately varies — node_l1_generation_policy
gates L1 on event_count, so a bundled call and per-turn calls evaluate that gate at
different counts. And it raced the background embedding worker, which returned 19–22
embedding records against a steady 18. It now compares the retention-critical types only
and waits for the record count to settle.

Verification

  • test_batch_event_integrity2 failures → green
  • test_matrixark_profile_scope_warning + test_batch_event_integrity4/4 runs green, checked repeatedly because one commit is a flake fix
  • Full suite failing-name diff against clean main: one name fixed, one appearstest_retrieval_audit_is_off_by_default, verified to fail identically on clean main (a run-context artifact, not caused here)

Conflict resolution notes

Two hunks needed resolving, both kept minimal on purpose:

  • matrixark_local_adapter_ingest.py — kept CALLER_SUPPLIED_EVENT_FIELDS, which main
    gained independently, and dropped only the warning block.
  • matrixark_mcp_core.py — bounded main's existing location pattern. The incoming side
    also carried a capitalised-location and a relationship pattern from earlier commits that
    were never mirrored; bringing those in would have smuggled unrelated changes into a
    bug-fix PR, so they were left out.

superhaiou and others added 3 commits August 22, 2026 17:21
A ten-message batch ingest reported success and kept ONE message. The other nine
were unreachable through retrieval, on the write path, silently.

The sync-accept path writes a single pending context_event for the WHOLE
envelope. The commit path re-emits one event PER MESSAGE, taking each id from
source_event_ids -- and skipped any message whose index ran past that list:

    for index, message in enumerate(envelope["messages"]):
        if index >= len(source_event_ids):
            continue

A batch therefore arrived with one source id for ten messages, so nine were
dropped. Worse, the one re-emitted event reused source_event_ids[0], so
latest-value compaction replaced the full 359-char pending event with a 35-char
event holding only the first message. Nothing errored and the call returned
accepted.

    before commit   id=8185658655269157160 status=observed              len=359 tokens=[0..9]
    after  commit   id=8185658655269157160 status=extraction_committed  len=35  tokens=[0]

Messages past the supplied ids now get an id derived exactly as the non-derived
branch does -- stable for the same batch and message position -- so a batch
leaves one event per message with distinct ids:

    after the fix   10 events, tokens=[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]

The text was never lost from the store; it survived in the session summary. But
retrieval returns EVENTS, so a fact ingested in a batch could not be recalled --
which is what the end-to-end check caught: "batch-ingested event" was the only
failing case of fifteen, and now passes.

Single-message ingest is untouched: it always supplied one id for one message.

Four tests added; two fail on the unfixed code. They assert every message
survives commit, that a batch leaves one event per message, that event ids are
distinct (reusing one id is what let compaction hide the loss), and that the
single-message path is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e ingest

A single bundled ingest of eight messages was enough to surface all three: a
warning that now lies, a location that swallows the clause after it, and one
preference stored twice.

It told callers a batch "retains ~1 raw context_event; the rest survive only as
lossy extractions and their exact facts become unretrievable", and to ingest
per-turn instead. That was true until the commit loop stopped skipping messages
past source_event_ids. Measured now, bundled and per-turn are identical on every
context record type, and bundled is CHEAPER on per-call bookkeeping:

    record_type          BUNDLED  ONE-BY-1
    context_event              6         6
    context_entity             6         6
    context_embedding         18        18
    context_summary            2         2
    matrixark_idempotency      2         7

So the warning was pushing callers onto the more expensive path to avoid a loss
that cannot occur. It is removed rather than reworded, and the tests that
asserted it are replaced by ones asserting the equivalence -- a regression that
reintroduced the loss would now fail on the facts, not on the warning text.

    "I live in Seattle and prefer metric units"  ->  "Seattle and prefer metric units"

The capitalised pattern above it already stopped at the place; this fallback
exists for lowercase locations ("the downtown office") and had no such bound. It
now stops at the clause boundary, which also keeps the bad value out of the
profile summary text that carries cross-session memory. 8/8 on a battery
covering conjunctions, trailing purpose clauses, and "Austin, Texas".

Two extractors fire on one clause: the directive scan produces state "user
directive: prefer metric units" named "preference:prefer metric units", the
pattern table produces state "metric units" named "preference". Different names,
so the name-keyed dedupe never saw them as one -- and promotion to the profile
node doubled both again.

The PLAIN entity is the one kept. Its name is the bare entity type, which is the
stable identity cross-session supersession matches on; the directive name embeds
its own text and cannot supersede a later correction. Keeping the directive
instead reads better -- it retains the verb -- and breaks lineage, which is what
test_profile_preference_correction_supersedes_stale_cross_session_state caught
when I had the rule the wrong way round. Only directive-prefixed entities are
ever dropped, and only when a same-type plain entity is literally contained in
them, so project "Aurora" cannot absorb "Aurora v2".

Entities for the demo conversation: 10 -> 8, location now "Seattle".

Full codex pipeline suite: failing names identical to baseline.
This test, added in 1f0eb99f, failed roughly one run in three -- in any tree, at
any commit -- and its failures have been polluting baseline comparisons all
session. Several names I attributed to order-dependence were this.

Two independent causes, both mine:

## 1. It asserted equality on a count that legitimately differs

context_summary and context_node were compared between the bundled and per-turn
arms. node_l1_generation_policy gates L1 on event_count, so a single bundled call
and a sequence of per-turn calls evaluate that gate at different counts and emit
a different number of summaries. That is correct behaviour; the assertion was
wrong. Equality now covers the retention-critical types only --
context_event, context_entity, context_embedding -- which is what the test exists
to defend: batching loses no MESSAGE.

## 2. It raced the embedding worker

Reading immediately after commit catches a varying amount of in-flight
background work. Measured across runs, the batched arm returned 19, 20, 21 or 22
context_embedding records against a steady 18 for per-turn -- the harness racing
the worker, not a difference between ingest shapes. It now waits for the record
count to settle before comparing, the same remedy the attachment-policy test
needed for the same reason.

## How it was found

A full-suite run flagged it, and bisecting across all seven commits of this
branch showed it passing at EVERY one -- while failing in the live tree at the
same commit. Not a regression: an isolated worktree reproduced OK/FAILED/OK and
the live tree FAILED/FAILED/OK. The commit-by-commit result was the tell; a test
that passes at every commit and fails on the tip is nondeterministic, not broken
by a change.

6/6 on the test and 5/5 on the whole file after the fix, against ~1-in-3 before.

Recorded also because the first bisect attempt was run in the live worktree with
git stash and git checkout per commit. That left a conflicted wal.rs and tried to
pop a stash belonging to ANOTHER session. The stash survived (the pop failed
cleanly) and only the one file needed restoring, but a shared tree is the wrong
place to bisect -- the second attempt used a throwaway worktree.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 23, 2026 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants