docs: refresh README as evidence-generating eval infra#271
Merged
Conversation
Reframe the README around what this repo actually is: evaluation and benchmarking infrastructure that produces evidence for OpenAdapt's governed-demonstration-compiler claims, not an end-user product. - Keep and clean the research-status banner (no em dashes anywhere) - Add the openadapt-flow evaluation section (replay + hybrid on WAA, scored by WAA's own verifier, dry-run-by-default and cost-capped) - Document the meta-benchmark harness, the unified oa VM CLI, the openadapt-eval-flow entry point, and the committed evidence reports - Update the architecture tree to match the current package layout (flow/, harness/, openenv/, workflow/, analysis/, cli/) - Frame substrates as first-class in the product with honest maturity; keep private crown-jewel data and recipes out of scope - Cross-link docs.openadapt.ai and the OpenAdaptAI org Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refreshes the
openadapt-evalsREADME to reflect what the repo actually is today: evaluation and benchmarking infrastructure that produces the evidence behind OpenAdapt's governed-demonstration-compiler claims, not an end-user product.Verified against the current package layout,
pyproject.tomlentry points, and recent commits (#265, #266, #269, #270) before writing.Key changes
scripts/eval_flow_on_waa.py/openadapt-eval-flow:replay(compile one demo, replay against WAA with ~0 model calls) andhybrid(compiled-first, agent fallback on halt), both scored by WAA's own verifier. Notes it is dry-run by default and hard cost-capped.eval_results/reports and their generator scripts.harness/), the unifiedoaVM CLI,openenv/,workflow/,analysis/.openadapt_evals/layout.openadaptlauncher, andopenadapt-flow.Docs-only change; no code touched.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM