feat(eval): add batch-evaluation simulate (dataset simulations) - #2000

Draft

jariy17 wants to merge 8 commits into

aws:refactorfrom

jariy17:feat/eval-batch-simulate

Commits on Aug 13, 2026

feat(eval): add ondemand evaluate (synchronous, client-side)
jariy17
committed
59e81dd
Merge remote-tracking branch 'origin/refactor' into feat/eval-ondemand-evaluate-clean
jariy17
committed
9b57081
feat(eval): add batch-evaluation simulate (dataset replay)
jariy17
committed
e265d99
fix(eval): omit empty ground-truth arrays (service rejects length 0)
jariy17
committed
95d341a
fix(eval): drop --session-id from simulate, add ingestion wait + reject multi-turn
jariy17
committed
80291f9

Commits on Aug 14, 2026

feat(eval): support multi-turn scenarios (invoke each turn on one session, in order)
jariy17
committed
028bd0c
feat(eval): thread AbortSignal through simulate (Ctrl-C cancels invokes + ingestion wait)
jariy17
committed
6ba1994
fix(eval): unlink temp dataset file after simulate (dataset-id path)
jariy17
committed
a8675a2