Make a klone YOLO checkpoint resumable on Tillicum, and move y11x_pano there to finish its schedule (#51)

[jonfroehlich]

Adds scripts/model_comparison/retarget_yolo_checkpoint.py, the tool that let y11x_pano continue on Tillicum as one continuous training trajectory rather than a restart. Job 207774 is running; full context and numbers are in the #51 status comment.

Why

resume=True reuses every training arg saved in the checkpoint. That is what keeps the LR schedule and epoch counter continuous across a preemption — and it is exactly what makes a checkpoint non-portable, because data / project / name / save_dir / model / resume are all absolute paths on the cluster that wrote them.

Two failure modes, one of them silent:

1. the saved data path is missing on the new host → the run dies at dataset load (obvious); or
2. a copied save_dir does exist → Ultralytics writes results back into the original run directory and corrupts the source run.

(2) is the trap already recorded in the #51 weight-snapshot MANIFEST. This replaces a warning that describes it with a tool that avoids it.

What it does and does not touch

Rewrites exactly those six path keys. Every hyperparameter is deliberately left alone — epochs, patience, lr0/lrf, batch, imgsz, workers, close_mosaic, seed, optimizer define the pre-registered #71 schedule, and changing one mid-run would make the resumed arm a different config rather than a continuation.

Worth stating for epochs: 60 specifically: it is the denominator of the LR decay, not a label. Rewriting it would not shorten the experiment — it would change what every remaining epoch does. That is also why a run cannot be retroactively redeclared as a shorter one.

Dry run by default. --apply writes, keeps a .preretarget backup, and reloads the written file to assert the keys persisted rather than trusting torch.save.

Verified in use

• checkpoint sha256 identical klone → workstation → Tillicum (8f85791587dd…729c, matching .sha256.verified)
• pano val labels and JPEGs md5-match byte-for-byte across clusters — stronger than the sorted-filename-list md5 recorded in docs/tillicum.md
• resume confirmed: Resuming training … from epoch 22 to 60 total epochs, with optimizer=auto → MuSGD(lr=0.01, momentum=0.9) identical to the klone runs

Dependency and a follow-up for #91

The run uses scripts/model_comparison/run_yolo_train_tillicum.slurm, which lives on docs/tillicum-onboarding (#91) and not on main — so reproducing this run today needs both branches. Merging #91 removes that.

Also for #91: its claim "The YOLO stack, however, is VERIFIED (2026-07-31)" should be narrowed. It was established by a data-prep job; no training job had ever run there. The first training submit (job 207761) failed at 4 min with RuntimeError: operator torchvision::nms does not exist — Tillicum had the generic torchvision 0.28.0 wheel against torch 2.13.0+cu126 where klone has 0.28.0+cu126. Prep never calls NMS, so prep could not have caught it. Cost of the failure: $0.06. numpy also differs (2.4.6 vs 2.4.4).

Test plan

No automated test: the script's input is a multi-hundred-MB Ultralytics checkpoint, which does not belong in fixtures. It self-verifies instead — reloads after writing and asserts all six keys persisted, exiting non-zero otherwise — and was exercised end to end on the real checkpoint (dry run, then --apply, then a resumed job that reported the correct epoch).

🤖 Generated with Claude Code (claude-opus-5[1m])