fix: honor every eval.defaults key and restore per-dataset stop / min_new_tokens

[keepkeen]

What

Two related holes in eval config resolution: eval.defaults silently drops most keys, and per-dataset stop / stop_token_ids / min_new_tokens are accepted from the YAML but never read.

Bug 1: eval.defaults only works for 14 fields

build_eval_dataset_configs consults defaults exclusively through _pick_from_mapping(defaults, spec["default_keys"]) for the names in DATASET_RUNTIME_SPECS + DATASET_SAMPLE_SPECS. Every other EvalDatasetConfig field placed under eval.defaults — rm_type, repetition_penalty, skip_special_tokens, no_stop_trim, custom_generate_function_path, app_service, eval_task_timeout, eval_early_stop_*, metadata_overrides — is silently discarded. With a config in the shipped examples/eval_multi_task style:

eval:
  defaults: {max_response_len: 16384, top_p: 0.7, rm_type: deepscaler, repetition_penalty: 1.05}
  datasets: [{name: aime, path: /root/aime.jsonl}]

max_response_len and top_p apply, but rm_type and repetition_penalty come out None — the eval runs with no reward model, silently. The README documents no subset restriction: "eval.defaults defines inference parameters shared by every dataset entry. Override them inside an individual dataset block if needed."

The typo handling is also asymmetric: a misspelled key in defaults is silently accepted, while the same misspelling inside a dataset entry raises TypeError from the dataclass constructor.

Bug 2: stop / stop_token_ids / min_new_tokens are dead config

EvalDatasetConfig accepts all three from the YAML, but the sampling params hardcode the training-time values:

stop=args.rollout_stop,
stop_token_ids=args.rollout_stop_token_ids,

and min_new_tokens is never set — --eval-min-new-tokens has zero readers in the repo. This is a regression: #1005 ("Fix evaluation parameter parsing") moved the sibling fields (temperature, top_p, top_k, max_response_len) into the spec tables and deleted the resolution lines for these three without a replacement (later #8f732538 re-added skip_special_tokens / no_stop_trim / repetition_penalty, but not these). An eval config that sets stop: ["</answer>"] for one dataset silently evaluates with the training stop strings instead — changing eval scores with no warning.

Fix

• After the spec pass, merge the remaining defaults keys with setdefault (dataset entry still wins), and reject unknown keys with a ValueError listing the valid names — defaults and dataset entries now get the same strictness.
• Restore the pre-Fix evaluation parameter parsing #1005 resolution for stop / stop_token_ids / min_new_tokens in eval_rollout_single_dataset, and add the three fields to DATASET_RUNTIME_SPECS (arg fallbacks rollout_stop / rollout_stop_token_ids / eval_min_new_tokens) so they also flow through eval.defaults like their siblings — which revives --eval-min-new-tokens.

Resolution order for every field is now uniformly: dataset entry > eval.defaults > args.

Test

New tests/test_eval_config.py in the cpu-unittest job: non-spec defaults reaching every dataset, dataset-entry override priority, the three restored fields resolving dataset > defaults > args, unknown-key rejection, and the existing spec-field arg fallback. Four of five cases fail on main.