Cut markerless post-processing: give stage 4 the cores it has, drop an overlay nobody reads - #138
Cut markerless post-processing: give stage 4 the cores it has, drop an overlay nobody reads#138amburger66 wants to merge 2 commits into
Conversation
The shard-8 failure is pytest-split regrouping, not this PR's code
Mechanism. CI runs
For example So Why it is not this PR's change. Nothing here touches the NSRT or GNN path; Locally, this branch is 1578 passed, 0 failed against base A trap for anyone re-checking this. Comparing the two shards at file Two further environment notes, neither related to this PR but both capable of
Not fixed here. The isolation bug deserves its own change; #137 adds 10 tests |
Follow-up:
|
…lay nobody reads run_20260818_092302 recorded a good episode and then could not use it. Post- processing took 1008 s against a 900 s fit deadline, so the fit skipped the take it had just paid a robot to produce and fell back to per-step scoring -- the twin-against-twin case the whole open-loop design exists to avoid. It missed by 108 s. The pipeline's own estimate says "about 3x the length of its take". Measured from the stage artifacts' timestamps it was 7.3x, and the discrepancy is not noise: the 3x figure was calibrated at 30 fps, and these takes record at 60 to resolve the propagation intervals. Cost scales with FRAMES, so doubling the frame rate doubles the multiple. 138 s of video, 7773 frames, 1008 s to a track: stage 1 replay -> frames.mp4 116 s stage 2 boxes (given) 0.5 crop -> frames_crop.mp4 40 stage 3 SAM-2 propagation 326 masks_overlay.mp4 render 163 <- nothing reads this stage 4 fit + emit 364 stage 5 poses overlay 60 (after the track; free) Two of those are addressed here. JOBS. Stage 4 fans out and ran 16 workers on a 32-core machine, which is the driver's default rather than a decision about this box. The arithmetic says it was cleanly core-limited -- 486 frames per worker over 364 s is 16.0 cores busy for the entire stage -- so it is the rare case where more workers really do translate. real_robot_track_jobs=0 keeps the driver's own choice; the domino config asks for 30, sized for this machine and deliberately not 32, since the pipeline runs in the background while the next episode is driving the robot. TRACK_VIZ. masks_overlay.mp4 is a debugging aid nothing downstream opens, and it is rendered BEFORE stage 4 -- so its 163 s is not tacked onto the end, it is paid straight out of time-to-track. Default off for automated runs, and easy to put back: that same run turned up a real id-matching problem, and the overlay is how id swaps get spotted. Needs BabyRobotPredicator#79; a driver that predates it renders the overlay as it always did rather than failing, which is what lets this land first. 0 means "leave the driver alone" for JOBS, and viz=True means saying nothing at all rather than passing TRACK_VIZ=1 -- an empty or contradictory env var is worse than an absent one when the far side is a shell. Both are asserted, and both assertions were checked against a mutant: dropping either guard fails its test. Expected effect, not yet measured on hardware: ~165 s from JOBS and 163 s from the overlay, so roughly 1008 s -> 11 min. That still exceeds no deadline by itself -- code_sim_learning_track_wait_s is untouched here and is the next thing to fix, along with reading a track only once it is completely written.
…ething BabyRobotPredicator 603e4d4 -> ba4ee81 (#79). The previous commit already sets TRACK_VIZ=0 on every launch; against a driver without it that was simply ignored, which is what let the two land in either order. With this bump the 163 s masks_overlay.mp4 render is genuinely skipped and real_robot_track_viz stops being a no-op.
13b1b3b to
901696d
Compare
yichao-liang
left a comment
There was a problem hiding this comment.
It's good to go when all checks pass!
Cuts markerless post-processing by roughly 5 minutes, so a recorded episode has
a chance of being ready before the fit gives up on it.
Why
run_20260818_092302recorded a good episode and then could not use it.Post-processing took 1008 s against a 900 s deadline, so the fit skipped the
take it had just paid a robot to produce and fell back to per-step scoring --
the twin-against-twin case the whole open-loop design exists to avoid. It missed
by 108 s.
The pipeline's own estimate says "about 3x the length of its take". Measured
from the stage artifacts' timestamps it was 7.3x, and that gap is not noise:
the 3x figure was calibrated at 30 fps, and these takes record at 60 to resolve
the propagation intervals. Cost scales with frames, so doubling the frame rate
doubles the multiple.
138 s of video, 7773 frames, 1008 s to a track:
frames.mp4frames_crop.mp4masks_overlay.mp4renderWhat changed
real_robot_track_jobs->JOBS. Stage 4 fans out and ran 16 workers on a32-core machine -- the driver's default, not a decision about this box. The
arithmetic says it was cleanly core-limited: 486 frames per worker over 364 s is
16.0 cores busy for the entire stage.
0keeps the driver's own choice; thedomino config asks for 30, deliberately not 32, because the pipeline runs in
the background while the next episode is driving the robot.
real_robot_track_viz->TRACK_VIZ.masks_overlay.mp4is a debugging aidnothing downstream opens, and it is rendered before stage 4 -- so its 163 s is
not tacked onto the end, it is paid straight out of time-to-track. Off by
default for automated runs, and easy to put back: that same run turned up a real
id-matching problem, and the overlay is how id swaps get spotted.
Third commit bumps the submodule to pick up BasisResearch/BabyRobotPredicator#79,
which added
TRACK_VIZ. The two were written to land in either order -- a driverwithout the flag ignores it rather than failing.
Notes
0means "leave the driver alone" forJOBS, andviz=Truemeans sayingnothing at all rather than passing
TRACK_VIZ=1: an empty or contradictory envvar is worse than an absent one when the far side is a shell. Both are asserted,
and both assertions were checked against a mutant -- dropping either guard
fails its test.
Verification
Full suite 1578 passed, 0 failed, against the base commit's 1576 passed, 0
failed in the same worktree under the same conditions -- the +2 are this PR's
two tests. mypy clean, pylint 10.00/10, yapf/docformatter/isort applied.
Both new assertions were checked against a mutant: dropping either guard fails
its test.
A note for anyone who runs the suite here and sees red.
install_ikfast_moduleshells out to
os.system("python setup.py"), and a machine with onlypython3on PATH fails it at fixture setup with exit 32512 (127 << 8), which looks like a
code regression and is not one. Separately, the suite has genuine test-ordering
dependence -- with
pytest-randomlyactive, 9 GNN/NSRT tests fail that pass inisolation. Both reproduce identically at the base commit, and #137's author saw
the same 9 independently. Neither is touched here.
This does not fix the deadline
Expected saving is ~165 s from
JOBSand 163 s from the overlay: about1008 s -> 11 min, not yet measured on hardware.
code_sim_learning_track_wait_sis untouched at 900 s, so a take this long would still miss it.
That wait is now the only remaining defect from run_20260818_092302. The
torn read this section previously also listed --
_await_trackstreating fileexistence as completion, which read a 28 MB track mid-write -- is fixed by #137,
whose
track_is_completeparses instead.I am taking the wait, built on top of #137 rather than on master: that PR
rewrites
_await_tracks, and putting this on master would conflict in thefunction it just fixed. The bound should scale by FRAMES rather than take
length, derived from the take's
meta.json(host_elapsed_s x fps_requested).Note the per-frame cost moves once THIS PR lands, so 0.130 s/frame is not a
stable constant -- which is fine, because the wait is a timeout and not a sleep:
it returns as soon as the track is ready, so a generous bound costs nothing
except when the pipeline has actually died.
The largest remaining win is structural: the take is 131 s but the interval
residual uses a ~2 s window (onsets at 107.09-107.31 s). Recording only from the
push is written and parked on
domino-record-from-push, held until #137 lands-- it moves what the track's first frame shows, which the id matching anchors
on.