Fix partial metric re-runs#59
Draft
fanny-riols wants to merge 4 commits intopr/fr/response_speed_decompositionfrom
Draft
Fix partial metric re-runs#59fanny-riols wants to merge 4 commits intopr/fr/response_speed_decompositionfrom
fanny-riols wants to merge 4 commits intopr/fr/response_speed_decompositionfrom
Conversation
When re-running a subset of metrics (e.g. --metrics response_speed --force-rerun-metrics), the summary now aggregates per_metric for all metrics found across records rather than only the re-run ones. Also merges metric_errors and pass_at_k_config from the existing file so unrelated fields are not lost.
add strict_llm param to apply_env_overrides; pass strict_llm=False when --force-rerun-metrics is set so metrics-only re-runs on runs whose LLM deployment is no longer in EVA_MODEL_LIST don't fail
Turns with missing audio timestamps store None in per_turn_latency; guard against this in _compute_speed_stats and the main latency loop. Also rename section header to "Diagnostic & Validation Metrics".
fanny-riols
changed the title
fanny-riols
changed the base branch from
main
to
pr/fr/response_speed_decomposition
fanny-riols
changed the title
from_existing_run now loads the saved config using only init_settings (no env vars / .env file), preventing the saved model config from being contaminated by the current environment's pipeline mode vars. Also skip the pipeline mode conflict check in _strip_other_mode_fields when --force-rerun-metrics is set, as the model config is unused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes four issues that made partial metric re-runs (e.g.
--metrics response_speed --force-rerun-metrics) unreliable.Changes
metrics_summary.jsonno longer overwritten on partial re-runWhen re-running a subset of metrics,
per_metricnow covers all metrics found across records rather than only the ones being re-run.metric_errorsandpass_at_k_configare merged from the existing file so unrelated fields are not lost.LLM deployment check skipped for metrics-only re-runs
apply_env_overridesgains astrict_llmparam. Passingstrict_llm=Falsewhen--force-rerun-metricsis set lets re-runs succeed on runs whose LLM deployment is no longer inEVA_MODEL_LIST— no simulation is needed, so the check is unnecessary.Nonelatency values handled gracefully inresponse_speedTurns with missing audio timestamps store
Noneinper_turn_latency.response_speednow skips them instead of crashing with aTypeError.RunConfig loading no longer conflicts with current env pipeline mode
from_existing_runusedmodel_validate_jsonwhich, in pydantic-settings v2, merges env vars and.envon top of the saved JSON — causing a conflict if e.g.EVA_MODEL__LLMis set in the environment but the saved run used S2S. Fixed by loading with a localsettings_customise_sourcesoverride that reads only from init kwargs. Also skips the pipeline mode conflict check in_strip_other_mode_fieldswhen--force-rerun-metricsis set.