domino_real: bring-up Stages 2 and 3 -- a perception check and a launcher - #113
Draft
amburger66 wants to merge 3 commits into
Draft
domino_real: bring-up Stages 2 and 3 -- a perception check and a launcher#113amburger66 wants to merge 3 commits into
amburger66 wants to merge 3 commits into
Conversation
…cher Stage 2 (cameras, no motion) and Stage 3 (human resets) had no vehicle. replay_plan structurally cannot do either: it forces perception off and human_reset off so a fixed-plan replay stays reproducible. The only other option was the AL launcher, which spends a full predicate-invention episode to take three camera readings. check_perception.py measures the live ZED scene against the twin and moves nothing -- RealRobot is built with no arm, so it runs with the robot powered down and needs no polymetis controller. It goes through make_real_robot and the env's own state_from_observation, i.e. the exact conversion the executor uses at an option boundary, so a disagreement here is real rather than an artifact of a parallel path. It reports per domino rather than just the worst number, because Stage 2's moved-domino check has to identify WHICH domino moved -- a different one moving means the capture-id -> slot mapping is wrong, which is the failure the stage exists to catch. Mean dz and its spread are reported apart from the position error, since a constant offset across every domino is a table-height disagreement rather than a perception error. observe_scene is the bare look: no arm homing and human prompt like reset_env, no motion like execute_chunks. stage3_domino_real.yaml adds human resets on top, driven by oracle rather than the agent -- this stage tests reset plumbing, and running the learner alongside would make a failure ambiguous between the two. It runs one episode per invocation, not the two the doc asks for, because oracle is not learning-based and this env generates a single task; the header says so and points at the sim test that already covers the across-episode cadence. Both verified on hardware.
The bring-up doc folded its full-recorded-plan stage into Stage 4, so active learning is Stage 6 now, not 7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bring-up Stages 2 (live cameras, no motion) and 3 (human resets) had no vehicle.
replay_planstructurally cannot do either — it forces perception and human resets off so a fixed-plan replay stays reproducible. The only alternative was the active-learning launcher, which spends a full predicate-invention episode with LLM calls to take three camera readings, and buries the numbers in its log. Stage 2's checks are a tight edit-run loop (read, nudge a domino, read again, topple one, read again), so the tool needs to take seconds.What's here
scripts/domino_debug/check_perception.py— measures the live ZED scene against the simulated twin. Nothing moves:RealRobotis built with no arm, so it runs with the robot powered down and needs no polymetis controller.It goes through
make_real_robotand the env's ownstate_from_observation— the exact conversionRealRobotExecutoruses at an option boundary — so a disagreement here is a real disagreement rather than an artifact of a parallel code path.Two reporting choices that matter:
_max_position_divergencereturns only the maximum, but Stage 2's moved-domino check has to identify which domino moved. A different one moving means the capture-id → slot mapping is wrong, which is the failure this stage exists to catch.domino_real_table_z), not a perception error — different fix entirely.--repeatsamples the noise floor across captures, which is whatreal_robot_divergence_atolshould be set from; it currently ships as a guessed 0.02.observe_sceneinreal_robot_bridge.py— the bare look, withoutreset_env's arm homing and human prompt orexecute_chunks' motion.scripts/configs/predicatorv3/stage3_domino_real.yaml— Stage 2's flags plusreal_robot_human_reset, driven byoraclerather than the agent. This stage tests reset plumbing; running the learner alongside it would make a failure ambiguous between the two, which is the attribution the staged ladder exists to preserve. Oracle needs no LLM.It runs one episode per invocation, not the two the doc asks for: oracle is not learning-based, so
main.pytakes the non-learning branch — a single_run_testingover this env's single generated task, andnum_test_tasksis ignored. The header says so and points attests/envs/test_domino_real_online.py::test_each_episode_prompts_exactly_once, which already covers the across-episode cadence in simulation.Testing
Both stages verified on hardware.
Locally on the changed files: pylint (with
.predicators_pylintrc), yapf, isort, docformatter and mypy all clean. The Stage 3 config resolves topybullet_domino_real+oracle_process_planningwith all four real-robot flags intact.Note
Rebased onto master after #112. The scene-path edits this branch originally carried were dropped — #112 landed them identically.
Touches
real_robot_bridge.py, as doesprobe-lazy-grounding, but in a different function (observe_scenevs_split_actions); whichever merges second will want a rebase.