fix(swebench-service): give Pyxis container creation its own deadline

[leopck]

Under Pyxis, creating the container is its own piece of infrastructure work: --container-image makes enroot import a multi-GB SWE-bench image and slurmstepd launch a step for it. That was charged against environment.timeout -- the per-command budget, 300s in both templates, sized for pytest-scale work inside an already-running container -- because PyxisSweBenchRunner._configure_environment dropped the template's pull_timeout: 3600 as a docker-only key and PyxisEnvironment.__init__ had nothing else to use.

A create budget must be separate from a per-command budget because the two scale with completely different things. A command's cost depends on the task; a create's cost depends on how much other work is contending for the node. Measured on an idle node, one create is ~35s and eight concurrent creates finish in 55s wall -- so 300s looks generous right up until it isn't. In the run that exposed this, four SWE-bench services drove 40 concurrent agents across 5,148 srun steps in 78 minutes, every step requesting all 144 CPUs, on a node also running four vLLM engines. Creation slowed by an order of magnitude, subprocess.run(timeout=timeout_s + 30) SIGKILLed the step, and 96 steps died at a uniform 5m47s-5m56s -- 330s plus step-accounting skew, against 3-11s for every ordinary command step. 17 of 20 units were lost and the run produced no accuracy number at all. The registry was never the bottleneck; step contention was.

Carry pull_timeout through to a distinct create_timeout_s (default 3600) and use it for the create step only. Command steps keep timeout.

Also stop discarding srun's own output. Both infrastructure-failure paths in run_srun_step() raised a fixed string and threw away the captured stream, so an import failure, an out-of-space enroot, and a step that never got resources were one indistinguishable message -- the 17 lost units above could not be told apart from their artifacts. The failure now carries srun's last 2000 characters, names the deadline it blew, and reports srun's exit code.

Finally, make creation measurable while it happens rather than only afterwards in sacct: with SWEBENCH_PYXIS_CREATE_TIMING_PATH set, each create appends one JSONL record with its duration and outcome. Off by default, and a sink that cannot be written degrades to nothing -- a create that succeeded and could not be logged is still a create that succeeded.

What does this PR do?

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.17647% with 3 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@c30860a). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...ench_service/swebench_service/pyxis_environment.py	90.90%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #456   +/-   ##
=======================================
  Coverage        ?   81.02%           
=======================================
  Files           ?      150           
  Lines           ?    20240           
  Branches        ?        0           
=======================================
  Hits            ?    16399           
  Misses          ?     3841           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.