Skip to content

Cut markerless post-processing: give stage 4 the cores it has, drop an overlay nobody reads - #138

Open
amburger66 wants to merge 2 commits into
masterfrom
worktree-pipeline-speedup
Open

Cut markerless post-processing: give stage 4 the cores it has, drop an overlay nobody reads#138
amburger66 wants to merge 2 commits into
masterfrom
worktree-pipeline-speedup

Conversation

@amburger66

@amburger66 amburger66 commented Aug 18, 2026

Copy link
Copy Markdown

Cuts markerless post-processing by roughly 5 minutes, so a recorded episode has
a chance of being ready before the fit gives up on it.

Why

run_20260818_092302 recorded a good episode and then could not use it.
Post-processing took 1008 s against a 900 s deadline, so the fit skipped the
take it had just paid a robot to produce and fell back to per-step scoring --
the twin-against-twin case the whole open-loop design exists to avoid. It missed
by 108 s.

The pipeline's own estimate says "about 3x the length of its take". Measured
from the stage artifacts' timestamps it was 7.3x, and that gap is not noise:
the 3x figure was calibrated at 30 fps, and these takes record at 60 to resolve
the propagation intervals. Cost scales with frames, so doubling the frame rate
doubles the multiple.

138 s of video, 7773 frames, 1008 s to a track:

stage seconds
1: replay -> frames.mp4 116
2: boxes (given) 0.5
crop -> frames_crop.mp4 40
3: SAM-2 propagation 326
masks_overlay.mp4 render 163
4: fit + emit 364
5: poses overlay (after the track) 60

What changed

real_robot_track_jobs -> JOBS. Stage 4 fans out and ran 16 workers on a
32-core machine -- the driver's default, not a decision about this box. The
arithmetic says it was cleanly core-limited: 486 frames per worker over 364 s is
16.0 cores busy for the entire stage. 0 keeps the driver's own choice; the
domino config asks for 30, deliberately not 32, because the pipeline runs in
the background while the next episode is driving the robot.

real_robot_track_viz -> TRACK_VIZ. masks_overlay.mp4 is a debugging aid
nothing downstream opens, and it is rendered before stage 4 -- so its 163 s is
not tacked onto the end, it is paid straight out of time-to-track. Off by
default for automated runs, and easy to put back: that same run turned up a real
id-matching problem, and the overlay is how id swaps get spotted.

Third commit bumps the submodule to pick up BasisResearch/BabyRobotPredicator#79,
which added TRACK_VIZ. The two were written to land in either order -- a driver
without the flag ignores it rather than failing.

Notes

0 means "leave the driver alone" for JOBS, and viz=True means saying
nothing at all rather than passing TRACK_VIZ=1: an empty or contradictory env
var is worse than an absent one when the far side is a shell. Both are asserted,
and both assertions were checked against a mutant -- dropping either guard
fails its test.

Verification

Full suite 1578 passed, 0 failed, against the base commit's 1576 passed, 0
failed
in the same worktree under the same conditions -- the +2 are this PR's
two tests. mypy clean, pylint 10.00/10, yapf/docformatter/isort applied.

Both new assertions were checked against a mutant: dropping either guard fails
its test.

A note for anyone who runs the suite here and sees red. install_ikfast_module
shells out to os.system("python setup.py"), and a machine with only python3
on PATH fails it at fixture setup with exit 32512 (127 << 8), which looks like a
code regression and is not one. Separately, the suite has genuine test-ordering
dependence -- with pytest-randomly active, 9 GNN/NSRT tests fail that pass in
isolation. Both reproduce identically at the base commit, and #137's author saw
the same 9 independently. Neither is touched here.

This does not fix the deadline

Expected saving is ~165 s from JOBS and 163 s from the overlay: about
1008 s -> 11 min, not yet measured on hardware. code_sim_learning_track_wait_s
is untouched at 900 s, so a take this long would still miss it.

That wait is now the only remaining defect from run_20260818_092302. The
torn read this section previously also listed -- _await_tracks treating file
existence as completion, which read a 28 MB track mid-write -- is fixed by #137,
whose track_is_complete parses instead.

I am taking the wait, built on top of #137 rather than on master: that PR
rewrites _await_tracks, and putting this on master would conflict in the
function it just fixed. The bound should scale by FRAMES rather than take
length, derived from the take's meta.json (host_elapsed_s x fps_requested).
Note the per-frame cost moves once THIS PR lands, so 0.130 s/frame is not a
stable constant -- which is fine, because the wait is a timeout and not a sleep:
it returns as soon as the track is ready, so a generous bound costs nothing
except when the pipeline has actually died.

The largest remaining win is structural: the take is 131 s but the interval
residual uses a ~2 s window (onsets at 107.09-107.31 s). Recording only from the
push is written and parked on domino-record-from-push, held until #137 lands
-- it moves what the track's first frame shows, which the id matching anchors
on.

@amburger66 amburger66 self-assigned this Aug 18, 2026
@amburger66
amburger66 marked this pull request as ready for review August 18, 2026 15:05
@amburger66
amburger66 changed the base branch from master to domino-track-id-mapping August 18, 2026 15:16
@amburger66

Copy link
Copy Markdown
Author

The shard-8 failure is pytest-split regrouping, not this PR's code

unit-tests (3.10.14, 8) fails on test_gnn_action_policy_approach with
TypeError: cannot pickle '_abc._abc_data' object, raised by dill inside
nsrt_learning_approach._learn_nsrts. It failed twice on the same test,
including a re-run of the untouched job, so it is not simple flakiness.

Mechanism. CI runs pytest tests/ --splits 8 --group N --splitting-algorithm least_duration, and .test_durations is committed. The two tests this PR adds
have no entry there, so the bin-packing changes and individual tests move
between groups. Comparing master's shard 8 (job 95710164386) with this PR's
(job 95764606521), from the -q progress lines:

master shard 8 this PR shard 8
tests 910 913
file-lines 86 89
file-lines whose test count differs 95 of ~175

For example test_bilevel_sketch_info_seeking.py runs 1 test on master and 3
here; test_capture_decision.py 3 vs 4; test_docker_agent_runner.py 2 vs 1;
test_bilevel_sketch_near_miss.py appears in this PR's shard 8 and not in
master's.

So test_gnn_action_policy_approach sits in shard 8 in both, but with a
substantially different set of tests running before it in the same process.
Because the split is deterministic given (test set, .test_durations), the same
regrouping recurs on every run -- which is exactly why it reproduces rather than
flaps.

Why it is not this PR's change. Nothing here touches the NSRT or GNN path;
the diff is track_pipeline.py, two settings.py attributes, a config, a
submodule pointer, and two tests. The underlying fault is a pre-existing
test-isolation bug: the same 9-test dill cluster reproduces on unmodified
origin/master locally (1565 passed, 9 failed), independently confirmed on
#137. Any PR that adds tests can trip it by shifting the packing.

Locally, this branch is 1578 passed, 0 failed against base 1d809f3's
1576 passed, 0 failed in the same worktree -- the +2 being this PR's tests.

A trap for anyone re-checking this. Comparing the two shards at file
granularity shows byte-identical lists (170 files each, comm empty both ways),
which looks like proof that regrouping did not happen. pytest-split splits at
test granularity, so that comparison is meaningless. Count tests, not files.

Two further environment notes, neither related to this PR but both capable of
producing a red local run that looks like a regression:

  • install_ikfast_module shells out to os.system("python setup.py"). A machine
    with only python3 on PATH fails at fixture setup with exit 32512 (127 << 8).
  • CI does not check out submodules (actions/checkout@v2, no submodules key).
    A local checkout that has initialised submodules/BabyRobotPredicator makes
    babyrobot importable, and test_make_real_robot_dry_constructs_an_armless_robot
    then fails on base as well as on any branch.

Not fixed here. The isolation bug deserves its own change; #137 adds 10 tests
and will likely hit the same cluster, possibly on a different test.

@amburger66

Copy link
Copy Markdown
Author

Follow-up: .test_durations is stale, which is why the packing is so easy to swing

Checked on #137, and it sharpens the diagnosis above. .test_durations is dated
Aug 6 with 4707 entries, and contains zero entries for anything under
predicators/code_sim_learning -- that module arrived with #135, which merged
today. So master's shard balance is already computed from a file blind to 33
tests, before any PR adds one.

pytest-split gives unlisted tests a fallback duration rather than skipping
them, so a block of identically-estimated tests is exactly the input that makes
least_duration bin-packing swing hard. That is consistent with the 95-of-175
file-lines difference measured above, and it predicts #137 (which takes the
unlisted block to 39) will swing further still.

The real remediation is regenerating the file (pytest --store-durations),
not chasing whichever test pollutes the process. But regenerating reshuffles
groups for everyone the moment it lands, so it belongs in its own maintenance PR
rather than folded into a feature branch. Flagging it here rather than doing it.

Base automatically changed from domino-track-id-mapping to master August 18, 2026 18:01
…lay nobody reads

run_20260818_092302 recorded a good episode and then could not use it. Post-
processing took 1008 s against a 900 s fit deadline, so the fit skipped the take
it had just paid a robot to produce and fell back to per-step scoring -- the
twin-against-twin case the whole open-loop design exists to avoid. It missed by
108 s.

The pipeline's own estimate says "about 3x the length of its take". Measured
from the stage artifacts' timestamps it was 7.3x, and the discrepancy is not
noise: the 3x figure was calibrated at 30 fps, and these takes record at 60 to
resolve the propagation intervals. Cost scales with FRAMES, so doubling the
frame rate doubles the multiple. 138 s of video, 7773 frames, 1008 s to a track:

  stage 1 replay -> frames.mp4     116 s
  stage 2 boxes (given)              0.5
  crop -> frames_crop.mp4           40
  stage 3 SAM-2 propagation        326
  masks_overlay.mp4 render         163     <- nothing reads this
  stage 4 fit + emit               364
  stage 5 poses overlay             60     (after the track; free)

Two of those are addressed here.

JOBS. Stage 4 fans out and ran 16 workers on a 32-core machine, which is the
driver's default rather than a decision about this box. The arithmetic says it
was cleanly core-limited -- 486 frames per worker over 364 s is 16.0 cores busy
for the entire stage -- so it is the rare case where more workers really do
translate. real_robot_track_jobs=0 keeps the driver's own choice; the domino
config asks for 30, sized for this machine and deliberately not 32, since the
pipeline runs in the background while the next episode is driving the robot.

TRACK_VIZ. masks_overlay.mp4 is a debugging aid nothing downstream opens, and it
is rendered BEFORE stage 4 -- so its 163 s is not tacked onto the end, it is
paid straight out of time-to-track. Default off for automated runs, and easy to
put back: that same run turned up a real id-matching problem, and the overlay is
how id swaps get spotted. Needs BabyRobotPredicator#79; a driver that predates
it renders the overlay as it always did rather than failing, which is what lets
this land first.

0 means "leave the driver alone" for JOBS, and viz=True means saying nothing at
all rather than passing TRACK_VIZ=1 -- an empty or contradictory env var is
worse than an absent one when the far side is a shell. Both are asserted, and
both assertions were checked against a mutant: dropping either guard fails its
test.

Expected effect, not yet measured on hardware: ~165 s from JOBS and 163 s from
the overlay, so roughly 1008 s -> 11 min. That still exceeds no deadline by
itself -- code_sim_learning_track_wait_s is untouched here and is the next
thing to fix, along with reading a track only once it is completely written.
…ething

BabyRobotPredicator 603e4d4 -> ba4ee81 (#79). The previous commit already sets
TRACK_VIZ=0 on every launch; against a driver without it that was simply
ignored, which is what let the two land in either order. With this bump the
163 s masks_overlay.mp4 render is genuinely skipped and real_robot_track_viz
stops being a no-op.
@amburger66
amburger66 force-pushed the worktree-pipeline-speedup branch from 13b1b3b to 901696d Compare August 18, 2026 18:01

@yichao-liang yichao-liang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to go when all checks pass!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants