Thanks for releasing TestExplora, it's a really useful benchmark and I'm keen to build on it.
I'm trying to run the full pipeline end-to-end (generation -> execution on base/head -> the F2P/HP/EC/CFG metrics) and want to make sure I'm following the intended path rather than reinventing something you already provide - quite possibly I've just missed an entrypoint.
A couple of things I wasn't sure about:
-
I can see the generation side (inference.py) and the execution building blocks (test_runner.py, eval_docker.py, eval_ces.py, repo_manager.py), plus load_data4eva in data_manager.py. Is there a top-level driver that ties generation -> run on base_commit -> apply code_patch -> run on head -> compute/aggregate the metrics, or is that meant to be assembled by the user?
-
eval_ces.py / ces_api.py look like they target an internal execution service (CES). For running outside your environment, is eval_docker.py the intended path?
Thanks for releasing TestExplora, it's a really useful benchmark and I'm keen to build on it.
I'm trying to run the full pipeline end-to-end (generation -> execution on base/head -> the F2P/HP/EC/CFG metrics) and want to make sure I'm following the intended path rather than reinventing something you already provide - quite possibly I've just missed an entrypoint.
A couple of things I wasn't sure about:
I can see the generation side (
inference.py) and the execution building blocks (test_runner.py,eval_docker.py,eval_ces.py,repo_manager.py), plusload_data4evaindata_manager.py. Is there a top-level driver that ties generation -> run onbase_commit-> applycode_patch-> run on head -> compute/aggregate the metrics, or is that meant to be assembled by the user?eval_ces.py/ces_api.pylook like they target an internal execution service (CES). For running outside your environment, iseval_docker.pythe intended path?