Open the ZED take in front of the push, not the whole episode - #139
Open
amburger66 wants to merge 2 commits into
Open
Open the ZED take in front of the push, not the whole episode#139amburger66 wants to merge 2 commits into
amburger66 wants to merge 2 commits into
Conversation
amburger66
marked this pull request as ready for review
August 18, 2026 16:07
Only the cascade is scored. On run_20260818_092302 the first onset landed 107 s into a 131 s track, so roughly 80% of the video was the pick-and-place that arranges the row -- none of it evidence, all of it post-processing, which scales with frames rather than seconds. real_robot_record_from_option names the option the take opens in front of. _ship_episode sends the prologue as one unrecorded request, opens the take, then sends the rest. That costs one controller round trip and NOT a planner call: the twin has already simulated every option by the time _ship_episode runs, so this does not give back what open-loop batching bought. Splitting is safe for the same reason per-boundary shipping was -- _split_actions restarts its gripper tracking per call and RealRobot dedups session-wide -- and both requests still go out with observe=False, so the write-only side-effect argument that makes the twin trajectory bit-identical is untouched. The arm coming to rest at the boundary is a small gain of its own: the free-run is anchored at the last rest state before the push, and now there really is one. Empty, or a name the episode never runs, records everything and warns. Too much video is slow; too little is an episode whose first topple happened off camera, and the first onset is what every interval is measured against. STACKED ON #137. This moves the track's first frame from the episode's initial arrangement to the post-place one, and the id matching anchors on a position set. Against #137 as originally written -- anchored on the episode's first state -- that was fatal: the two placed dominoes sit ~150-200 mm from the anchor and drop out, taking the match from 4/4 to 2/4. 98e24cd moved both sides to the settled arrangement immediately BEFORE the cascade, which is independent of when recording starts, and push-only measures 4/4 again. This commit is only correct with that fix underneath it. An earlier version of this branch claimed recording from the push FIXES the id matching. That was wrong. The dominant cause was the base-frame/world-frame rotation #137 identified -- 144-307 mm against a 40 mm tolerance -- and starting the take later does not touch it. No lead-in, as a deliberate first cut. Two known hazards ride on that, neither fixed here and both likelier on a short take: load_track takes each domino's first OBSERVED centre, so a start domino occluded by the gripper for the whole pre-cascade window contributes no position and drops out of the match; and the offset vote's tie-break does not settle exact aliasing, so an evenly spaced row missing one domino can match completely and be shifted by one. If either bites, splitting one option earlier is the fix. Tests: five, plus three mutants -- no prologue split, recording opened before the prologue, and the unknown-option fallback recording nothing. The third initially PASSED, because asserting one shipment of three chunks cannot tell "everything recorded" from "everything shipped, then the take opened"; the ordering probe was added to pin it, and it fails under that mutant now.
amburger66
force-pushed
the
domino-record-from-push
branch
from
August 18, 2026 18:01
84baa0e to
4e1b76c
Compare
yichao-liang
approved these changes
Aug 18, 2026
yichao-liang
left a comment
Collaborator
There was a problem hiding this comment.
LTGM! Let's see how well this works!
The boxes are SAM-2 prompts applied to the take's FIRST FRAME, so they have to describe the arrangement that frame shows. While the take began at the reset those were the same thing and the draw sat at run start. Recording from a later option makes them different, and nothing detected it: stage 2 does not report an empty box, it fits a mask to whatever is inside the one it was given. Observed on run_20260818_140211. boxes.json was written at 14:02:37, before any motion; the take opened at 14:05:37, after the prologue had picked and placed two dominoes. Two of the four prompts pointed at bare table. A human noticed it in the overlay, which is not a detection mechanism. So the draw moves to the recording boundary: after the prologue ships, before _start_recording. The arm is at rest and the row is in its final arrangement, and it is also the only moment a human can draw boxes that match the take. Where the take still opens at the reset -- no record_from_option -- the draw stays at run start, which was already right there. ensure_boxes is now idempotent. The boundary call site runs every episode and the draw must not: a fixed-plan replay arranges the same row each time, and a drag window opening per take is exactly what pick_boxes_at_start exists to avoid. This makes a real trade rather than removing a cost. The human is now needed ~50 s into the first episode instead of before it starts. The unattended alternative is projecting the twin's geometry into image space, which needs no human at all and is considerably more work. Three tests, on the ordering rather than the fact of the draw -- asserting that boxes were drawn cannot tell the right moment from the wrong one, which is the whole defect. The probe records what had shipped and whether the take was open at draw time, and asserts one shipment and no open take. Both mutations fail it: drawing at run start regardless, and deleting the boundary call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update: second commit, and the lead-in question is settled
f4873c4draws the prompt boxes at the recording boundary instead of at runstart. It belongs here because THIS PR is what makes it necessary: the boxes are
SAM-2 prompts applied to the take's first frame, and while the take began at the
reset, "the arrangement at run start" and "the arrangement the first frame shows"
were the same thing. Recording from the Push separates them.
Observed on
run_20260818_140211:boxes.jsonwritten at 14:02:37 before anymotion, take opened at 14:05:37 after the prologue had picked and placed two
dominoes. Two of the four prompts pointed at bare table. Nothing detected it
-- stage 2 does not report an empty box, it fits a mask to whatever is inside
the one it was given. A human caught it in the overlay, which is not a detection
mechanism.
The draw now happens after the prologue ships and before the take opens: the arm
is at rest, the row is final, and it is the only moment a human can draw boxes
that match the take. With no
record_from_optionthe draw stays at run start,which was already correct there.
ensure_boxesis idempotent, since the newcall site runs every episode and a drag window per take is what
pick_boxes_at_startexists to avoid.It trades rather than removes a cost: the human is needed ~50 s into the first
episode instead of before it. The unattended alternative -- projecting the
twin's geometry into image space -- needs no human and is considerably more work.
The lead-in: not needed, and for a better reason than "we chose to skip it"
The body below records two hazards taken deliberately, and says a lead-in would
fix the first. That is now known to be wrong, and the correction is
evidence-based rather than a change of mind.
Re-running stage 4 with visibility gates from 0.6 down to 0.2 closed the pushed
domino's occlusion gap from 1.4 s to 4 frames -- and its onset still swung
780 ms non-monotonically, against propagation intervals of 117-283 ms. The
reference onset is noise-limited, not occlusion-limited, so buying more
visibility with a lead-in would not have bought a usable onset.
The fix being pursued instead is excluding the pushed domino from interval
scoring, which is not on any branch yet.
The aliasing hazard is untouched by any of this and still stands.
#137 has merged
The "do not merge ahead of #137" caveat below is resolved -- it is on master as
b802ae3, and this branch sits above it.tests/pybullet_helpers170 passed on the current head.Third in the stack: #137 (id matching) -> #138 (pipeline speed) -> this.
Records only the part of the episode that is actually scored.
Why
On
run_20260818_092302the first topple onset landed 107 s into a 131 strack. Everything before it was the pick-and-place that arranges the row --
roughly 80% of the video, none of it evidence. Post-processing scales with
frames, so that 80% is paid twice: once recording it, once fitting poses to it.
Push+Waitis about 25-30 s against 138 s, so this is ~4.5-5x fewer frames-- a larger lever than everything in #138 combined.
What it does
real_robot_record_from_optionnames the option the take opens in front of._ship_episodesends the prologue as one unrecorded request, opens the take,then sends the rest.
This does not give back what open-loop batching bought. The twin has already
simulated every option by the time
_ship_episoderuns, so splitting theshipment costs one controller round trip and not a planner call. Splitting is
safe for the same reason per-boundary shipping was:
_split_actionsrestarts itsgripper tracking per call and
RealRobotdedups session-wide. Both requestsstill go out with
observe=False, so the write-only side-effect argument thatmakes the twin trajectory bit-identical is untouched -- there is a test asserting
the split commands the same motion as one batch.
The arm coming to rest at the boundary is a small gain of its own: the free-run
is anchored at the last rest state before the push, and now there really is one.
Empty, or a name the episode never runs, records everything and warns. Too much
video is slow; too little is an episode whose first topple happened off camera,
and the first onset is what every interval is measured against.
Why it must sit on top of #137
This moves the track's first frame from the episode's initial arrangement to the
post-place one, and the id matching anchors on a position set.
Against #137 as first written -- anchored on the episode's first state -- that
was fatal: the two placed dominoes sit ~150-200 mm from the anchor and drop out,
taking the match from 4/4 to 2/4.
98e24cdmoved both sides to the settledarrangement immediately before the cascade, which is independent of when
recording starts, and push-only measures 4/4 again.
An earlier revision of this branch claimed recording from the push fixes the id
matching. That was wrong, and is corrected in the commit message. The dominant
cause was the base-frame/world-frame rotation #137 identified (144-307 mm against
a 40 mm tolerance); starting the take later does not touch it.
No lead-in, deliberately
Two known hazards ride on that, neither fixed here and both likelier on a short
take:
load_tracktakes each domino's first observed centre, so a start dominooccluded by the gripper for the whole pre-cascade window contributes no
position and drops out of the match. On a push-only take that is the domino
whose onset every interval is measured against.
row missing one domino can match completely and be shifted by one.
If either bites, splitting one option earlier is the fix. Called deliberately by
the user as a first cut, to be revisited on evidence rather than pre-emptively.
Tests
Five, plus three mutants: no prologue split, recording opened before the
prologue, and the unknown-option fallback recording nothing.
The third mutant initially passed, which is worth recording. Asserting "one
shipment of three chunks" cannot distinguish everything recorded from
everything shipped, then the take opened -- the latter being an episode with
no motion in the take at all. An ordering probe was added to pin it, and it
fails under that mutant now.
test_real_robot_executor.py84 passed, mypy clean, pylint 10.00/10.CI note
Expect the shard-8
test_gnn_action_policy_approach/dillfailure documentedon #138. It is pre-existing
pytest-splitregrouping, not from this branch --.test_durationsis a two-week-old snapshot that already omits an entire module,so any PR adding tests re-packs the groups and changes what runs before what in
a given process.