Skip to content

test: add replay consistency framework for session and memory#222

Open
FSJGDJ wants to merge 2 commits into
trpc-group:mainfrom
FSJGDJ:feat/replay-consistency-89
Open

test: add replay consistency framework for session and memory#222
FSJGDJ wants to merge 2 commits into
trpc-group:mainfrom
FSJGDJ:feat/replay-consistency-89

Conversation

@FSJGDJ

@FSJGDJ FSJGDJ commented Jul 22, 2026

Copy link
Copy Markdown

Summary

  • add 10 replay consistency scenarios for Session, Memory, and Summary
  • compare events, state, memory, and summary snapshots between InMemory and SQLite backends
  • add field-level diff reporting and intentional inconsistency detection
  • fix SQL Session summary anchor ordering
  • fix SQL Memory limit=0 semantics

Testing

  • 240 related tests passed
  • replay consistency tests completed in approximately 4.4 seconds
  • flake8 passed
  • git diff --check passed

Closes #89

@helloopenworld

Copy link
Copy Markdown
Contributor

AI Code Review

我已经完成了审查,以下是审查结论。

发现的问题

🚨 Critical

  • tests/sessions/test_replay_consistency.py:438test_replay_cases_are_consistent_across_backends 用例 duplicate_or_recovery 在跨后端比较时存在不稳定性风险,可能误报或漏报。
    • 该用例连续 append 两条文本完全相同的 agent 事件("recover answer")。事件 idEvent.new_id() 随机生成、各自唯一,时间戳为 REPLAY_EPOCH + step_index(互不相同),因此排序本身稳定;但问题在于 InMemory 后端 append_event 把事件直接 append 到存储 session 的 events 列表,而 SQL 后端 get_session 读取时按 timestamp.desc 排序后 reversed,当两个事件时间戳非常接近(整数偏移相同则不可能,但若未来 step 复用同一时间戳)会依赖数据库次级排序。当前用 REPLAY_EPOCH + step_index 可规避,但测试对"时间戳必须严格单调且唯一"有隐性强依赖却未断言。建议在用例中对同作者同文本事件显式校验顺序,或改用更可区分的文本,避免后续修改 step_index 赋值逻辑时引入隐蔽 flaky。
    • 实际更确定的风险:duplicate_or_recovery 注入逻辑(:399)执行 broken["events"].append(copy.deepcopy(broken["events"][-1])),只是复制最后一条,确实会产生 diff,注入测试可过;但跨后端一致性测试本身并不验证"重复事件被正确去重/保留"这一语义,用例名与实际断言内容不匹配,属于测试有效性问题。

⚠️ Warning

  • trpc_agent_sdk/memory/_sql_memory_service.py:232limit or None 使 limit=0 时 SQL 查询不再加 LIMIT,返回全部行;但后续 Python 循环没有任何 limit 截断(:238-254count >= limit 判断)。

    • 这与 InMemoryMemoryService:130if limit > 0 and count >= limit: return)行为不一致:当调用方传入正数 limit(如默认 10)时,SQL 后端只在 DB 层做 limit 截断,而 DB limit 作用在"所有命中 save_key 的事件"上、再在 Python 里做关键词过滤,意味着若前 N 条(按 timestamp desc)不命中关键词,SQL 会返回空,而 InMemory 会遍历全部事件找到 N 条命中。两个后端对正数 limit 的语义本就不同;本次改动只修正了 limit=0,并未修复正数 limit 的跨后端不一致。本次为测试一致性而改,可接受,但建议在注释中说明 SQL 的 limit 是"原始事件数"而非"命中数",避免后续误用。
  • tests/sessions/test_replay_consistency.py:89REPLAY_EPOCH = time.time() 在模块导入时求值,测试数据与"当前时间"耦合。

    • 若 CI 机器时钟异常回拨,或测试运行极慢导致 REPLAY_EPOCH 远早于 SQL func.now()_get_sessionfunc.now() 刷新 update_timeappend_eventtime_diff > 1.0 判定 stale),可能触发 append_event 的 stale-session 重载分支(_sql_session_service.py:557),使事件窗口被 DB 状态覆盖,导致跨后端 diff。当前 TTL 已 clean_ttl_config 关闭、时间戳用 REPLAY_EPOCH + step_index,通常不会触发;但 time_diff 比较的是 storage_session.update_timestamp_tz - session.last_update_timeupdate_timestamp_tz 来自 func.now()(DB 服务器时间),与 REPLAY_EPOCH(应用进程时间)可能存在时钟差。建议 REPLAY_EPOCH 用固定常量或 time.time() 在每个用例内重新取值,并断言其与 DB 时间偏差可控。

💡 Suggestion

  • tests/sessions/test_replay_consistency.py:334-366_compare_snapshots 在列表长度不一致时用 "<missing>" 字符串占位,该占位会被写进 diff 的 expected/actual 字段并参与递归比较。由于它是一个普通字符串,与对端 dict/list 比较时会走到叶子分支产生一条 diff,语义可用;但 "<missing>" 作为魔法值容易与真实文本混淆。建议改用哨兵对象并在叶子分支单独产出 missing 类 diff,提升可读性。

总结

生产代码改动(_sql_session_service.py 的 summary anchor 重排、_sql_memory_service.pylimit or None)方向正确、未引入明确逻辑错误;新增测试覆盖了回放一致性框架,但 duplicate_or_recovery 用例的命名与实际断言不匹配、且测试对时间戳单调性存在隐性强依赖,属于测试有效性问题而非阻塞性缺陷。不存在必须修复才能合入的安全或核心功能问题。

测试建议

  • 补充一条断言:在 duplicate_or_recovery 中显式验证两条相同 agent 事件均被保留且顺序正确(events[1] 和 events[2] 文本相同但 id 不同),使用例名与断言语义一致。
  • 建议增加一个 num_recent_events > 0 的回放用例,覆盖 get_session 的 SQL limit 截断与 summary anchor 重排叠加时的顺序正确性(当前所有用例 num_recent_events=0,未触发该路径)。

)


async def test_replay_cases_are_consistent_across_backends():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate_or_recovery 用例跨后端不稳定且命名与断言不匹配

duplicate_or_recovery 用例对时间戳严格单调且唯一存在隐性强依赖但未断言,存在 flaky 风险;且该用例实际不验证“重复事件被正确去重/保留”语义,用例名与断言内容不匹配,属于测试有效性问题。建议显式校验同作者同文本事件顺序,或改用更可区分的文本。

@github-actions

Copy link
Copy Markdown

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@f8b05ba). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #222   +/-   ##
==========================================
  Coverage        ?   87.91873%           
==========================================
  Files           ?         479           
  Lines           ?       44987           
  Branches        ?           0           
==========================================
  Hits            ?       39552           
  Misses          ?        5435           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Session / Memory 多后端回放一致性测试框架

2 participants