Add plain train.sbatch and best_train.sbatch launchers

[eugenevinitsky]

What

Adds a copy-pasteable single-GPU cluster path alongside the submitit pipeline:

• scripts/train.sbatch — plain default training job (1 GPU, 16 CPU, 96 GB, 48 h). TRAIN_CMD is a bash array in the exact puffer train CLI dash-flag format, so it copy-pastes to and from a console command.
• scripts/best_train.sbatch — same skeleton (192 GB, 30 h) carrying the best-known hyperparameters translated from scripts/cluster_configs/nightly_best.yaml, one flag per line with the yaml's section comments preserved.
• Both echo a launch record (hostname, date, git commit, full command) into the job log before running, and support an optional singularity wrap when SINGULARITY_IMAGE + SINGULARITY_OVERLAY are set (required on NYU Greene). The wrap cds into the repo before sourcing the venv and %q-quotes every argument so the command survives the bash -c round-trip.
• docs/cluster_training.md gains a short "Simple path — plain sbatch" section and a pointer from the submit_cluster.py section. The submit snippet includes the one-time mkdir -p slurm_logs — slurmd opens the stdout file before the job script runs, so the directory must exist at submit time.
• slurm_logs/ added to .gitignore.
• AGENTIC_PR poem per repo convention.

Why

The only documented cluster path was the ~500-line submitit pipeline, whose yaml/underscore and sweep-override formats can't be copy-pasted to or from the real puffer train CLI, and whose launched args end up buried in submitit stdout. These two files give a plain sbatch default where the exact command and code version are greppable from the job log.

Notes

Verification performed

• bash -n on both scripts.
• Both TRAIN_CMD flag sets parsed through pufferlib.pufferl.load_config (pufferlib resolved in-worktree) with zero unrecognized arguments; spot values match nightly_best.yaml (num_agents=4096, backbone_num_layers=3, total_timesteps=10B, reward_conditioning=False, num_goals=3).
• Singularity branch dry-run with a stub singularity and SEED="1 2": payload arrives cwd-first, venv-sourced, with whitespace-containing args intact (--train.seed 1\ 2).
• Non-singularity branch dry-run with a stub puffer and fake venv: full launch record echoed, command invoked verbatim, exit 0.
• Commit contains exactly 5 files, sbatch scripts mode 100755, no build artifacts.

Risks

• best_train.sbatch duplicates nightly_best.yaml values by design ("keep in sync" header); drift is possible until one becomes canonical.
• Jobs run from the live checkout (no code isolation) — documented in the script headers; use submit_cluster.py when isolation, sweeps, DDP, or the Greene heartbeat is needed.
• date -Is in the launch record requires GNU date (fine on Linux clusters; fails harmlessly on macOS).

Merge-order dependencies

• Deliberately emits no --eval.* flags: the sibling PR fix/drive-ini-remove-hardcoded-scratch-paths removes/disables the [eval.behaviors_*] / [eval.validation_replay] drive.ini sections, and such flags would become unrecognized. This PR is safe to merge before or after it.
• Another PR touches the quick-overview block of docs/cluster_training.md (~lines 8–13 pre-change); this PR only inserts a new section above it and one pointer sentence at the submit_cluster.py heading, so conflicts should be trivial.

🤖 Generated with Claude Code

🤖 Generated with Claude Code