Skip to content

domino_real: bring-up Stages 2 and 3 -- a perception check and a launcher - #113

Draft
amburger66 wants to merge 3 commits into
masterfrom
stage2-perception-check
Draft

domino_real: bring-up Stages 2 and 3 -- a perception check and a launcher#113
amburger66 wants to merge 3 commits into
masterfrom
stage2-perception-check

Conversation

@amburger66

Copy link
Copy Markdown

Why

Bring-up Stages 2 (live cameras, no motion) and 3 (human resets) had no vehicle.

replay_plan structurally cannot do either — it forces perception and human resets off so a fixed-plan replay stays reproducible. The only alternative was the active-learning launcher, which spends a full predicate-invention episode with LLM calls to take three camera readings, and buries the numbers in its log. Stage 2's checks are a tight edit-run loop (read, nudge a domino, read again, topple one, read again), so the tool needs to take seconds.

What's here

scripts/domino_debug/check_perception.py — measures the live ZED scene against the simulated twin. Nothing moves: RealRobot is built with no arm, so it runs with the robot powered down and needs no polymetis controller.

It goes through make_real_robot and the env's own state_from_observation — the exact conversion RealRobotExecutor uses at an option boundary — so a disagreement here is a real disagreement rather than an artifact of a parallel code path.

Two reporting choices that matter:

  • Per-domino, not just the worst number. _max_position_divergence returns only the maximum, but Stage 2's moved-domino check has to identify which domino moved. A different one moving means the capture-id → slot mapping is wrong, which is the failure this stage exists to catch.
  • Mean dz and its spread, separate from position error. A constant z offset across every domino is a table-height disagreement (domino_real_table_z), not a perception error — different fix entirely.

--repeat samples the noise floor across captures, which is what real_robot_divergence_atol should be set from; it currently ships as a guessed 0.02.

observe_scene in real_robot_bridge.py — the bare look, without reset_env's arm homing and human prompt or execute_chunks' motion.

scripts/configs/predicatorv3/stage3_domino_real.yaml — Stage 2's flags plus real_robot_human_reset, driven by oracle rather than the agent. This stage tests reset plumbing; running the learner alongside it would make a failure ambiguous between the two, which is the attribution the staged ladder exists to preserve. Oracle needs no LLM.

It runs one episode per invocation, not the two the doc asks for: oracle is not learning-based, so main.py takes the non-learning branch — a single _run_testing over this env's single generated task, and num_test_tasks is ignored. The header says so and points at tests/envs/test_domino_real_online.py::test_each_episode_prompts_exactly_once, which already covers the across-episode cadence in simulation.

Testing

Both stages verified on hardware.

Locally on the changed files: pylint (with .predicators_pylintrc), yapf, isort, docformatter and mypy all clean. The Stage 3 config resolves to pybullet_domino_real + oracle_process_planning with all four real-robot flags intact.

Note

Rebased onto master after #112. The scene-path edits this branch originally carried were dropped — #112 landed them identically.

Touches real_robot_bridge.py, as does probe-lazy-grounding, but in a different function (observe_scene vs _split_actions); whichever merges second will want a rebase.

…cher

Stage 2 (cameras, no motion) and Stage 3 (human resets) had no vehicle.
replay_plan structurally cannot do either: it forces perception off and
human_reset off so a fixed-plan replay stays reproducible. The only other
option was the AL launcher, which spends a full predicate-invention episode
to take three camera readings.

check_perception.py measures the live ZED scene against the twin and moves
nothing -- RealRobot is built with no arm, so it runs with the robot powered
down and needs no polymetis controller. It goes through make_real_robot and
the env's own state_from_observation, i.e. the exact conversion the executor
uses at an option boundary, so a disagreement here is real rather than an
artifact of a parallel path. It reports per domino rather than just the worst
number, because Stage 2's moved-domino check has to identify WHICH domino
moved -- a different one moving means the capture-id -> slot mapping is wrong,
which is the failure the stage exists to catch. Mean dz and its spread are
reported apart from the position error, since a constant offset across every
domino is a table-height disagreement rather than a perception error.

observe_scene is the bare look: no arm homing and human prompt like
reset_env, no motion like execute_chunks.

stage3_domino_real.yaml adds human resets on top, driven by oracle rather
than the agent -- this stage tests reset plumbing, and running the learner
alongside would make a failure ambiguous between the two. It runs one episode
per invocation, not the two the doc asks for, because oracle is not
learning-based and this env generates a single task; the header says so and
points at the sim test that already covers the across-episode cadence.

Both verified on hardware.
The bring-up doc folded its full-recorded-plan stage into Stage 4, so active
learning is Stage 6 now, not 7.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant