docs: steady-state metrics reporting design proposal - #447
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| a run killed by `SIGKILL`/OOM before the log is flushed yields a truncated log; | ||
| the step must degrade to a best-effort result with a status flag rather than | ||
| fail (§5.6). |
There was a problem hiding this comment.
Maybe I am not reading it in the right way - but I thought it should fail if the original run fails?
There was a problem hiding this comment.
What does "failing" mean here — a request timeout or a process crash? My intent is that a timeout should still yield a best-effort (partial) result over whatever was logged, while a hard crash of the post-process can just fail. Could you clarify which case you had in mind so I word the edge-case handling to match?
| - **Relative active concurrency.** In-flight count as a percentage of the | ||
| max-concurrency budget (100% at saturation). | ||
| - **Issue-time window.** A contiguous range of super-passes `[start, end)`. Its | ||
| measured set is every request _issued_ in that range. Throughput uses the |
There was a problem hiding this comment.
This might requires some clarification - as far as i understand from below a sample issued but whose response is not received has no logged TTFT and is naturally excluded. So either it is issue only or issue with at least one event to be included in a window.
| client. The window measures the sustained behavior of a healthy server; a | ||
| genuinely unhealthy server produces bad-but-real numbers, which the drift | ||
| detector (§5.5) distinguishes from transient pollution. | ||
| - **Super-pass.** One full **dataset pass** — `dataset_size` consecutive requests |
There was a problem hiding this comment.
minor - would it be easier to just call it a pass instead of a super-pass since it is a single pass over the entire dataset?
There was a problem hiding this comment.
Kept super-pass deliberately rather than pass: it is not always one full dataset pass. A low-concurrency run may not issue even a single full pass, so reusing pass would create a terminology conflict. It's also a tunable hyperparameter — e.g. 2 dataset passes per super-pass to reduce per-super-pass variance. Reworded the §5.1 definition in bad5e83 to make this explicit.
| (everything issued at `t=0`), so there is no client-side issue-time window and no | ||
| client-side drain boundary. The step detects the mode and finds the steady region |
There was a problem hiding this comment.
for TPOT calculations, we could potentially use the completion-first_token time, but that would require the server to not evict requests in decode phase.
| ## 5.7 Complementary: staggered ("feathered") issuance | ||
|
|
||
| The ramp exists because the target concurrency `C` is filled as a burst. A | ||
| staggered fill flattens the ramp-up spike: issue in steps of `ceil(C/k)` and, between |
There was a problem hiding this comment.
add a small definition for k.
| - **ISL / OSL** are reported for analytics, not as validation numbers: once the | ||
| window is allowed to drop samples (not enforcing full dataset-pass boundaries), | ||
| the input/output-length distribution is skewed relative to the constructed | ||
| dataset and is no longer a meaningful validation quantity, though it remains | ||
| useful for analysis. |
There was a problem hiding this comment.
If the window is allowed to drop samples, should the ISL/OSL statistics be computed for the samples included in the measurement?
|
|
||
| # 6 Open questions {#6-open-questions} | ||
|
|
||
| - **Default parameters.** The warmup band, trailing-window length, per-percentile |
There was a problem hiding this comment.
Can we ignore the first pass/super-pass over the dataset as a warmup pass and the other parameters can be benchmark specific decided by TF?
| - **Guard distance measure.** The exact two-sample statistic and acceptance bound | ||
| for the per-token-invariance guard (§5.4) need to be fixed. | ||
| - **No-steady-state runs.** When no steady state is found (a tracked metric is | ||
| Drifting Up), should the run be reported _invalid_ — analogous to legacy |
There was a problem hiding this comment.
We should declare the gates for each benchmark and also define whether a submission should be invalid if it fails a certain gate. It should be a benchmark TF decision on what gates to include.
5e936f6 to
bad5e83
Compare
| client. The window measures the sustained behavior of a healthy server; a | ||
| genuinely unhealthy server produces bad-but-real numbers, which the drift | ||
| detector (§5.5) distinguishes from transient pollution. | ||
| - **Super-pass.** The atomic unit of the analysis: a contiguous block of requests |
There was a problem hiding this comment.
Is there an assumption here that we need more than one super-pass? How many would be needed to get a good estimate?
| [ ingest -> per-super-pass series ] issue-order bucketing | ||
| | | ||
| v | ||
| [ adaptive warmup crop ] remove the ramp |
| **Ensemble and window selection.** CoV alone is insufficient — a window is only | ||
| meaningful if the metric is in Plateau. In practice, over a large set of runs, no | ||
| single `(window, bound)` fits all metrics and workloads, so the rule is run as an | ||
| **ensemble** of preset `(window, bound)` settings. Among the settings that report |
There was a problem hiding this comment.
Can you explain what does "ensemble of (window, bound)" mean exactly? How are the values of window and bound decided?
Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bad5e83 to
158d294
Compare
github-actions
Bot
added
the
size/very-large
4 participants
Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling.
What does this PR do?
Adds a design doc for a new steady-state window detection and metrics reporting feature
Type of change
Related issues
cc @arekay-nv to tag any related issues
Testing
Checklist