Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) by jonfroehlich · Pull Request #87 · ProjectSidewalk/RampNet

jonfroehlich · 2026-07-29T22:27:45Z

Refreshes scripts/model_comparison/yolo_baseline/README.md against the live runs. Issue #51.
Documentation only — no code, no behaviour change, no new numbers in the reported tables.

Why

The committed record had drifted from reality in three ways, one of them actively false.

1. A false statement. "Where the weights live" still read:

TODO — stage to Hugging Face (projectsidewalk/…) or lab storage and record the URL here.

The weights are now staged and verified:

/gscratch/makelab/jonf/rampnet_yolo_baseline_51/
  <arm>/{best.pt,last.pt,results.csv,args.yaml}   # 5 arms, 1.8 GB
  MANIFEST.md                                     # job IDs, per-arm metrics, sha256 of every weight

All 10 weight files sha256-verified against source (10 ok, 0 mismatched). They had lived
only on /gscratch/scrubbed, which auto-purges after ~21 idle days, and the run dirs date
from 07-26 — so anyone reading the old text would reasonably conclude the weights were still
one purge from gone, and either redo the staging or worry.

2. Stale numbers, and one that mattered. The status table was the 07-28 snapshot. Beyond
the refresh, the record claimed every checkpoint is undertrained and every benchmark number a
lower bound. That is no longer uniform:

y26_pano's best.pt is epoch 18 — proven, not inferred: best.pt and last.pt are
byte-identical (both sha256 aac93e74…50bf). mAP50-95 0.460 vs 0.413 at ep1.
The other four arms' best.pt are frozen at epoch 1 (mtimes 07-26 20:22/21:42/23:35/23:58,
each ~one epoch after the 18:10 start, while last.pt is current). For y26_tiles this is
exact: best_fitness = 0.35579 = its ep1 mAP50-95.

Reporting all five identically would understate y26_pano and overstate the rest, so the
caveat is now per-arm and says "one-epoch model" in those words where that is what it is.

3. Two findings were missing entirely.

The ckpt slice ceiling — a limitation distinct from the LR instability already documented.
The instability says scores got worse; this says an arm cannot advance at all. Across 158
scheduling segments over 3.5 days, the longest slice ever granted was 8.24 h and none
reached 9.5 h. Tiles epoch time rose from ~5.2 h to ~9.5 h under filesystem contention — the
logs warn read: 2.1±1.3 MB/s across 557,413 tiles, so it is I/O-bound, not compute-bound. Since
Ultralytics checkpoints only at epoch boundaries, an epoch longer than the slice can never
complete: both tiles arms consumed ~19 GPU-hours each and logged zero epochs. Same mechanism
that dropped y11x_tiles on 07-27. This is a structural gap, not a slow one, and calling it
"undertrained" would misdescribe it.

The fork. y26_tiles resumed on one of makelab's two gpu-l40s GPUs (job 37889646, 72 h
cap, one GPU left free for students); the ckpt original was not cancelled and still runs. Its
provenance is recorded in full because a fork here is easy to get silently wrong: resume=True
restores every arg from the checkpoint including save_dir, so the copied last.pt needs
project/name/save_dir/model/resume rewritten, or the fork resumes into the original
run directory and corrupts a live training arm. Everything governing the schedule (epochs=60,
lr0, lrf, warmup_epochs, batch, imgsz, seed, patience, data, epoch,
best_fitness) was asserted unchanged on read-back.

Deliberately deferred

Not in this PR, by agreement — all three are worth doing once, against final curves rather
than a moving target:

re-pulling runs/*/results.csv from scratch,
regenerating figures/*.png via the committed plot_training_curves.py,
updating the "Supervised baseline in progress" paragraph in docs/model_comparison.md.

So the committed CSVs still hold the 07-28 snapshot while the new table shows live epochs. The
file now says this explicitly in two places, so the staleness reads as a decision rather than an
oversight — including a note that docs/model_comparison.md is behind this file by design.

Verification

y26_pano best.pt == last.pt byte-identical: sha256 compared on the cluster.
Snapshot integrity: 10/10 sha256 match, recorded in MANIFEST.md.
Fork isolation: original y26_tiles results.csv and last.pt still Jul 28 07:44 / 3 epochs
after the fork started; the fork resumed from its own path, Resuming training … from epoch 4 to 60 total epochs.
Segment statistics from sacct -X -D on jobs 37745358,37745360,37745361,37745362,37745363.
Docs-only change; pytest -q unaffected.

Live status continues in #51. Bearing on #70 (any tiles arm in the stabilized rerun needs a
non-preemptable partition or a shorter epoch, independent of the LR fix) and #72 (the ceiling
belongs alongside the instability caveat).

🤖 Generated with Claude Code (claude-opus-5[1m])

…#51) The committed record had drifted from the runs in three ways, one of them an actively false statement. "Where the weights live" said `_TODO -- stage to Hugging Face or lab storage_`. The weights are now staged: /gscratch/makelab/jonf/rampnet_yolo_baseline_51/, five arms, best.pt + last.pt + curves + a MANIFEST.md, all ten weight files sha256-verified against source. They had existed only on /gscratch/scrubbed, which auto-purges after ~21 idle days, and the run dirs date from 07-26. A reader of the old text would reasonably conclude the weights were still at risk. The status table was the 07-28 snapshot (epochs 14/10/3/3/7) and is now live (18/13/9/3/3 plus the fork). More consequentially, "every benchmark number from these checkpoints is a lower bound" is no longer uniform: y26_pano's best.pt is epoch 18 -- proven byte-identical to last.pt, sha256 aac93e74..50bf -- while the other four are frozen at epoch 1. Reporting all five the same way would both understate y26_pano and overstate the rest, so the caveat is now per-arm and says "one-epoch model" in those words where that is what it is. Two findings were missing entirely. The ckpt slice ceiling is a limitation distinct from the LR instability already documented: across 158 scheduling segments the longest ever granted was 8.24 h and none reached 9.5 h, while tiles epoch time rose from ~5.2 h to ~9.5 h under filesystem contention (the logs warn `read: 2.1+-1.3 MB/s` over 557,413 tiles). Since Ultralytics checkpoints only at epoch boundaries, an epoch longer than the slice can never complete -- so both tiles arms burned ~19 GPU-hours each and logged zero epochs. That is a structural gap, not a slow one, and it is the same mechanism that dropped y11x_tiles. The response was additive: y26_tiles forked onto one of makelab's two gpu-l40s GPUs (job 37889646, 72 h cap), original left running on ckpt. The fork's provenance is recorded in full because it is easy to get silently wrong -- resume=True restores every arg from the checkpoint including save_dir, so the copied last.pt needs project/name/save_dir/model/resume rewritten or the fork resumes into the live original directory and corrupts a training arm. Everything governing the schedule was asserted unchanged on read-back. Deferred deliberately: re-pulling runs/*/results.csv, regenerating figures/, and updating the docs/model_comparison.md paragraph -- all worth doing once against final curves rather than a moving target. The file now says so explicitly so the staleness reads as a decision rather than an oversight. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The provenance note said the mail flags were "worth omitting" for the #70 rerun. Jon's call is the opposite -- the emails are fine and easy to delete -- so the record now says the volume is deliberate and explains what it is useful for (reading the run history), while keeping the actionable detail: the flags come from the sbatch command line, not the launcher, so a quieter rerun has to set them per job with scontrol update. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The 4.1 h/epoch figure for the fork was written as if measured. It is an extrapolation from 6.2 it/s at 12,123 of 92,903 iterations -- 13% of the fork'"'"'s first epoch -- and no epoch has completed on that partition yet. In a record whose whole point is that numbers be re-derivable, a projection stated as a measurement is the wrong kind of claim, and this one is load-bearing: it is the basis for "~17 epochs in 72 h." Says where the real number will come from (the time column of y26_tiles_l40s/results.csv) so it gets replaced rather than inherited. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…time measured (#51) Three substantive changes, all from live data six hours after the previous check. y11l_pano unfroze its best.pt at ep14 (mAP50-95 0.437 > ep1 0.423 after 12 epochs unbeaten), and y26_pano advanced to ep20. So two of five arms now hold genuinely trained checkpoints and only three are one-epoch models. The per-arm caveat table is re-sorted accordingly, and now says outright that it has already moved once so the split must be re-derived at eval time rather than inherited. The fork'"'"'s epoch time is measured rather than projected: 4.94 h from the time column of its completed ep4, against the 4.1 h I extrapolated from 13% of that epoch -- about 20% low, the usual direction. Its ep5 is tracking ~4.0 h, so the dedicated-node figure is a ~4-5 h range and the 72 h budget buys ~14 epochs rather than ~17. The fork'"'"'s first new epoch also landed: ep4 is slightly below ep3, which is where the LR schedule puts it, so the recovery question stays open until ~ep9. Refines the ceiling finding in a way that matters for planning: y11x_pano has now also stopped advancing -- no epoch in ~15 h -- despite a ~3.4 h epoch that fits the 8.24 h nominal cap with room to spare. Its segments that evening were 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the constraint is the effective preemption interval, not the slice cap, and "epoch < 8.24 h is safe" is the wrong rule. The arm is not wedged; its counter is live at ep10. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…gh (#51) The train-stage caveat told the reader to run y11x_tiles "only on a non-preemptable partition" without saying how, and the only account-ish knob sub() exposes is -A. But run_yolo_train.slurm hardcodes "#SBATCH -p ckpt-g2 / -q ckpt-gpu", so an account-only override leaves the job on ckpt and faithfully reproduces the 8.24 h slice ceiling the move exists to escape -- silently, as a mysterious repeat of the original failure. Spell out the full hand-submit line (-p / -A / -q / --time) and note that the launcher sets no MailType, so a hand-submitted arm is silent; the 2026-07-26 arms' mail came from their sbatch CLI, not the script. Comment-only; no change to what the six-arm train stage submits. Verified against the live fork (job 37889646: Partition=gpu-l40s, Account= gpu-l40s-makelab, QOS=normal), which is the only arm submitted this way. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jonfroehlich · 2026-07-31T19:06:55Z

Added one commit (ac0b4dc) that is not part of the README refresh, so the file list has grown by one — flagging it here rather than letting a reviewer wonder.

While recording the y11x_tiles restart decision on #51 I found that the runbook's train-stage caveat says to run that arm "only on a non-preemptable partition" but never says how, and the only account-ish knob sub() exposes is -A. Since run_yolo_train.slurm hardcodes #SBATCH -p ckpt-g2 / -q ckpt-gpu, following that advice by changing the account alone leaves the job on ckpt — silently reproducing the exact 8.24 h slice ceiling the move exists to escape, as what looks like a mysterious repeat of the original failure. I nearly walked into it myself: the submit line I had been carrying in my own notes was missing -p gpu-l40s.

The commit is comment-only (11 added lines, bash -n clean, no change to what the six-arm train stage submits). It spells out the full hand-submit line and notes that the launcher sets no MailType, so a hand-submitted arm is silent — the 2026-07-26 arms' mail came from their sbatch CLI, not the script. Verified against the live fork (job 37889646: Partition=gpu-l40s, Account=gpu-l40s-makelab, QOS=normal), the only arm actually submitted this way.

It rides here because #87 is the open YOLO-record PR and the caveat concerns the very arm this PR documents; it touches no file #87 touches, so it can equally be dropped into its own PR if you'd rather keep this one README-only.

Unchanged: the deferred items (re-pull runs/*/results.csv, regenerate figures/, update the docs/model_comparison.md paragraph) still wait for the arms to stop. The fork ends 2026-08-01T14:09:47 PT.

🤖 Generated with Claude Code (claude-opus-5[1m])

…hot staleness (#51) - The status-table blockquote still said "18 epochs below" from the 14:45 PT check; the 20:45 table it introduces says 20. Stale remnant, now consistent. - Record the y11x_tiles restart (fresh, gpu-l40s, job 38063498, batch 6) in the grid section and its status row - the 2026-07-31 decision comment in #51 explicitly earmarked this status change for this file once the arm submitted, which happened while the PR was open. - Blockquote pointer to the other post-snapshot developments (fork TIMEOUT at ep18 on 08-01, y11x_pano to Tillicum on 08-04) so the merged snapshot says where its own successors live instead of implying it is current. - "Where the weights live": the durable pack is a point-in-time copy, not a mirror - per #51's 08-02 check it predates every checkpoint now worth keeping, and the completed fork's weights exist only on scrubbed with the idle-purge clock running (~08-22). - Fork provenance: bridge the epoch=2 (0-based) / "epoch-3 checkpoint" / "epoch 4" numbering so the three numbers read as one fact; and mark the shared-lineage caveat as now binding (ckpt original reached ep7 on 08-02). - Runbook: say why the hand-submit example uses BATCH=6 when the grid line above it says 12 (the #51 decision rationale, condensed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jonfroehlich · 2026-08-04T18:06:03Z

Review 2026-08-04 — ready for merge; fixes pushed as `3dacca6` + `906b454` + `01e6dc5`

Reviewed the full diff against the sources rather than just reading it:

Runbook claims verified against the launcher: run_yolo_train.slurm does hardcode #SBATCH -p ckpt-g2 / -q ckpt-gpu and sets no MailType, so both the partition-override warning and the silent-arm warning were correct. (The runbook block itself was later dropped from this PR — see the correction below.)
Internal consistency of the dated tables checked: the frozen-arm mtimes reconcile with the 18:10 start plus each arm's epoch time; the fitness rule (0.9·mAP50-95 + 0.1·mAP50) really does select ep20/ep14 over ep1 for the two graduated arms; the 14:45→20:45 PT layering is coherent.
The five-key fork rewrite list is not in conflict with retarget_yolo_checkpoint.py's six keys — the klone→klone fork needed no data rewrite; the Tillicum migration did.

Found and fixed:

Stale remnant: the status-table blockquote still said y26_pano had "18 epochs below" from the 14:45 PT draft; the 20:45 table says 20.
The y11x_tiles restart is now recorded in the grid section and status row — the Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51 decision comment explicitly earmarked that status change for this file "when this arm is submitted", which happened (2026-08-03, job 38063498) while this PR was open. As-run: batch 12, --time=7-00:00:00, fresh start, no shared-lineage caveat; the deviation from the decision comment's batch 6 is named explicitly, with the run's args.yaml as the authority at eval time.
Snapshot lifecycle in "Where the weights live": the durable pack is a point-in-time copy — Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51's 08-02 check caught it stale (07-29 state, the converged fork absent entirely), and it was refreshed 2026-08-03 to all six arms (24/24 sha256-verified; superseded manifest preserved as MANIFEST-2026-07-29.md). The README now records that lifecycle and tells readers to check MANIFEST.md's date before trusting any copy.
Shared-lineage caveat is now binding: the ckpt original reached ep7 on 08-02 while the fork ran to ep18, so "they may never diverge" got a dated correction.
Epoch-numbering bridge in the fork provenance: epoch=2 (0-based last-completed) = the epoch-3 checkpoint = resumes as 1-based "epoch 4" — three numbers, one fact, now stated as such.
A dated blockquote pointer to the other post-snapshot developments (fork TIMEOUT 08-01 at ep18; y11x_pano → Tillicum 08-04, job 207774) so the merged snapshot names its successors instead of implying it is current.

Corrections along the way, same day. My first fix commit (3dacca6) wrote the restart up at batch 6, taking the 2026-07-31 decision comment at face value — the committed as-run record on exp/y11x-tiles-restart-51 says batch 12; corrected in 906b454. It also warned the fork's weights had "no durable copy" — true only for 08-01→08-03, since the pack was refreshed before this review; corrected in 01e6dc5. And 906b454 drops the runbook hand-submit block (ac0b4dc plus my rider), reverting hyak_yolo_runbook.sh to main: exp/y11x-tiles-restart-51 rewrites that exact train-stage region with the richer as-run submission block, so keeping a second copy here guaranteed a merge conflict and preserved a batch-6/72 h example that contradicts what actually ran. This PR is README-only again, as its body describes. The dropped block's two unique nuggets survive elsewhere: the MailType note is in this README's provenance section, and the partition-override trap is recorded in #51's decision comment (worth grafting into the restart branch's block at the site of use).

One follow-up this review surfaced, not blocking merge: exp/y11x-tiles-restart-51 (4 commits, pushed) has no PR yet — it carries the as-run restart record, the MODE: RESUME/MODE: FRESH log-header fix, and the WORKERS knob, and it is the canonical home of the off-ckpt submission guidance.

Deliberately not touched, per the PR's stated deferral: no curve numbers refreshed, runs/*/results.csv and figures/ untouched, docs/model_comparison.md untouched — all wait for final curves.

🤖 Generated with Claude Code (claude-fable-5)

…ock (#51) Two corrections to the previous review-fix commit, both against the committed as-run record on exp/y11x-tiles-restart-51: - The y11x_tiles restart ran at BATCH=12 on 2026-08-03 (job 38063498, g3103, --time=7-00:00:00) - not the batch 6 the 2026-07-31 decision comment specified. The README now records the as-run value, names the deviation explicitly, and defers to args.yaml as the authority at eval time. - The runbook hand-submit comment (ac0b4dc + its review rider) is dropped, reverting hyak_yolo_runbook.sh to main: exp/y11x-tiles-restart-51 rewrites the same train-stage region with the richer as-run submission block, so keeping a second copy here guaranteed a merge conflict and preserved a batch-6/72h example that contradicts what actually ran. Its two unique nuggets survive elsewhere: the MailType note is already in this README's provenance section, and the partition-override trap is recorded in #51's 2026-07-31 decision comment. This PR is now README-only again, as its body describes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…026-08-03 (#51) The previous commit warned the completed fork weights existed only on scrubbed with the purge clock running. True only for 08-01 -> 08-03: the durable pack was refreshed on 2026-08-03 to all six arms (24/24 sha256-verified, superseded manifest preserved as MANIFEST-2026-07-29.md), and the 08-04 Tillicum migration already resumed from it, sha256-matched. The caveat now records that lifecycle - staleness caught, refresh done, check MANIFEST.md dates before trusting a copy - instead of re-raising a solved problem. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jonfroehlich and others added 4 commits July 29, 2026 15:26

This was referenced Jul 30, 2026

Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51

Open

Record where the YOLO baseline weights actually live (#51) #92

Closed

jonfroehlich and others added 2 commits August 4, 2026 11:10

Navigation Menu

Uh oh!

Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) - #87

Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51)#87
jonfroehlich wants to merge 8 commits into
mainfrom
docs/yolo-record-refresh-51

jonfroehlich commented Jul 29, 2026

Uh oh!

jonfroehlich commented Jul 31, 2026

Uh oh!

jonfroehlich commented Aug 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jonfroehlich commented Jul 29, 2026

Why

Deliberately deferred

Verification

Uh oh!

jonfroehlich commented Jul 31, 2026

Uh oh!

jonfroehlich commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review 2026-08-04 — ready for merge; fixes pushed as 3dacca6 + 906b454 + 01e6dc5

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jonfroehlich commented Aug 4, 2026 •

edited

Loading

Review 2026-08-04 — ready for merge; fixes pushed as `3dacca6` + `906b454` + `01e6dc5`