Remove hardcoded /scratch nuPlan paths from shipped drive.ini

[eugenevinitsky]

What

• pufferlib/config/ocean/drive.ini:
  • [eval.validation_replay]: now enabled = false with a placeholder env.map_dir = "path/to/nuplan_bins" and a comment pointing at docs/nuplan_data.md for preparing local nuPlan bins.
  • [eval.behaviors_full_dir]: dropped its enabled = true override (inherits enabled = false from behaviors_defaults) and got the same placeholder map_dir treatment.
  • Deleted the 11 [eval.behaviors_<category>] sections (hard_stop … unprotected_right) that pointed at the undocumented, irreproducible categories_v021 split. The behaviors block header now documents the pattern: prepare a folder of labelled nuPlan bins per category and add one [eval.behaviors_<category>] section per folder, following behaviors_full_dir.
• scripts/cluster_configs/nightly_best.yaml and scripts/cluster_configs/single_agent_speed_run.yaml: removed the eval.behaviors_<category>.enabled: 0 lines for the deleted sections (their flags would no longer be registered by the ini-driven parser, aborting the launch with "unrecognized arguments") plus the now-redundant validation_replay / behaviors_full_dir enabled: 0 overrides.
• docs/evaluation.md: built-in evaluator table now lists behaviors_full_dir instead of behaviors_*; documents that nuPlan bins are external user-supplied data, that validation_replay/behaviors_full_dir ship disabled until env.map_dir points at a local bin directory, and that per-category behavior sections are user-defined.
• AGENTIC_PR: per-repo instruction for agent-created PRs.

Why

The shipped default config enabled 13 evaluators whose map_dir pointed at the maintainer's personal NYU cluster tree (/scratch/ev2237/data/nuplan/...). Every external user got 13 FileNotFoundError tracebacks per eval pass and no metrics from those evaluators. The 11 per-category sections additionally referenced a categories_v021 split nobody else can reproduce, so disabling them in place had no value — the pattern is documented instead.

Notes

Verification (venv sourced, run from the PR worktree):

• load_config("puffer_drive") parses the modified ini; EvalManager.from_config yields validation_replay enabled=False, behaviors_full_dir enabled=False, validation_gigaflow enabled=True. Disabled evaluators are skipped by the manager before env construction, so placeholder paths are never resolved.
• Simulated submit_cluster.py's yaml→flag conversion for nightly_best.yaml (84 keys), single_agent_speed_run.yaml (23) and train_base.yaml (5); all emitted flags are registered and parse cleanly. Negative control: a deleted-section flag (--eval.behaviors-hard-stop.enabled 0) now aborts with unrecognized arguments, confirming the yaml cleanup was required.
• pytest tests/unit_tests/test_eval_manager.py: 31 passed (its behaviors_hard_stop fixtures are self-contained dicts, independent of drive.ini).

Risks / intentionally untouched:

• Anyone with a private workflow relying on the previously-enabled nuPlan evaluators must now set env.map_dir and enabled = true locally (or re-add per-category sections). Ad-hoc puffer eval --eval_simulation replay still works — run_one_by_name ignores enabled.
• scripts/baseline_10G/config-*.yaml and weights/06_02_checkpoint/config.yaml still contain /scratch/ev2237 paths; they are nested full-config/checkpoint snapshots that never pass through the ini-driven flag parser, so they were left as historical artifacts.

Merge-order dependency: the ini and docs comments reference docs/nuplan_data.md, added by a sibling PR — merge that PR first (or accept a briefly dangling doc link).

🤖 Generated with Claude Code

🤖 Generated with Claude Code

[eugenevinitsky]

Folded into #525 — the drive.ini fix and the S3 fetch infrastructure land together so the placeholder paths and the default fetch location stay consistent.