fix(benchmark): don't reject an accuracy-only run for listing many endpoints - #453
Open
leopck wants to merge 1 commit into
Open
fix(benchmark): don't reject an accuracy-only run for listing many endpoints#453leopck wants to merge 1 commit into
leopck wants to merge 1 commit into
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
leopck
force-pushed
the
fix/acc-only-multi-endpoint
branch
from
August 19, 2026 13:41
9acf127 to
8f5247f
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #453 +/- ##
=======================================
Coverage ? 80.98%
=======================================
Files ? 150
Lines ? 20210
Branches ? 0
=======================================
Hits ? 16367
Misses ? 3843
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…dpoints
An accuracy-only run that lists more than one endpoint fails during setup:
Failed to connect to endpoint: 1 validation error for HTTPClientConfig
Value error, num_workers (1) must be a multiple of the number of
endpoint URLs (4) ... Got remainder 1.
and exits 3 before any dataset is planned or scored.
Two forced choices collide. setup_benchmark pins num_workers=1 and
max_connections=1 for every TestMode.ACC run, deliberately, so the compliance
gate's single_stream assertion holds. HTTPClientConfig separately requires
num_workers to divide the endpoint count so each endpoint gets equal workers.
One worker cannot divide four endpoints, so the run is refused.
What makes this a defect rather than a tight constraint is that the rejected
client does no work. Both SWE-bench scorers set SKIP_ENDPOINT_PHASE, so no
sample is ever issued through it -- the same run logs "Expected samples: 0"
moments earlier. A validator is rejecting a configuration on behalf of a
component that never runs, and it takes the whole run down with it.
Give the idle client a single endpoint when the run will issue nothing, so the
divisibility invariant still means what it says for runs that do issue.
Scorers that fan work out across endpoints themselves read the endpoint list
from the run's config.yaml rather than from this client, so this does not
narrow the run.
The proper fix is to skip building an issuer at all when nothing will be
issued. That requires a null issuer type, because BenchmarkSession takes a
non-None issuer, and is a larger change than this defect warrants on its own;
endpoints[:1] is the narrow form of it, not the intended end state.
Only reachable with more than one endpoint in accuracy-only mode: a
single-endpoint run divides exactly and never surfaces it.
leopck
force-pushed
the
fix/acc-only-multi-endpoint
branch
from
August 20, 2026 16:12
8f5247f to
ae125e7
Compare
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An accuracy-only run that lists more than one endpoint fails during setup:
and exits 3 before any dataset is planned or scored.
Two forced choices collide. setup_benchmark pins num_workers=1 and max_connections=1 for every TestMode.ACC run, deliberately, so the compliance gate's single_stream assertion holds. HTTPClientConfig separately requires num_workers to divide the endpoint count so each endpoint gets equal workers. One worker cannot divide four endpoints, so the run is refused.
What makes this a defect rather than a tight constraint is that the rejected client does no work. Both SWE-bench scorers set SKIP_ENDPOINT_PHASE, so no sample is ever issued through it -- the same run logs "Expected samples: 0" moments earlier. A validator is rejecting a configuration on behalf of a component that never runs, and it takes the whole run down with it.
Give the idle client a single endpoint when the run will issue nothing, so the divisibility invariant still means what it says for runs that do issue. Scorers that fan work out across endpoints themselves read the endpoint list from the run's config.yaml rather than from this client, so this does not narrow the run.
The proper fix is to skip building an issuer at all when nothing will be issued. That requires a null issuer type, because BenchmarkSession takes a non-None issuer, and is a larger change than this defect warrants on its own; endpoints[:1] is the narrow form of it, not the intended end state.
Only reachable with more than one endpoint in accuracy-only mode: a single-endpoint run divides exactly and never surfaces it.
What does this PR do?
Type of change