Skip to content

Add task-first GenSim action and orchestration pipeline - #531

Open
skywhite1024 wants to merge 60 commits into
mainfrom
ljd/action_engine_refactor
Open

Add task-first GenSim action and orchestration pipeline#531
skywhite1024 wants to merge 60 commits into
mainfrom
ljd/action_engine_refactor

Conversation

@skywhite1024

@skywhite1024 skywhite1024 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Note

Superseded by native GitHub Stack #539 (PRs #532 through #538). Review and merge the incremental stack instead of this closed aggregate PR.

Description

This PR introduces a task-first GenSim pipeline that connects scene preparation, symbolic task planning, direct AtomicAction graph execution, and auditable workflow recovery.

The main changes are:

  • add Action Engine v2 contracts, capability registration, SeedGraph compilation, planning, runtime execution, recovery, recording, and A/B evaluation;
  • add Task Engine orchestration with unified prepare, run, and run-all CLI flows across image, image-edit, scene, and scene-edit inputs;
  • extend Scene Engine generation/editing, semantic scene graphs, layout, scale/orientation handling, import/export, and final inspection;
  • integrate motion planning, coordinated dual-arm behavior, grasp collision filtering, task-level validation, and deterministic workflow recovery;
  • add architecture documentation and broad Action Engine, Task Engine, and Scene Engine regression coverage.

The change consolidates cross-engine ownership under Task Engine while keeping Scene Engine responsible for scene materialization and Action Engine responsible for graph compilation and execution.

Issue: none linked.

Dependencies: no new third-party Python dependency is introduced. Runtime workflows continue to use the existing optional model, image/geometry service, simulator, and motion-planning integrations.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which improves an existing functionality)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Validation

  • git diff --check origin/main...origin/ljd/action_engine_refactor
  • black --check --diff --color ./ — one file requires reformatting: tests/gen_sim/action_engine/config/test_runtime_policy.py; 845 files pass.
  • pytest -q tests/gen_sim/action_engine tests/gen_sim/task_engine tests/gen_sim/scene_engine — 703 passed, 3 failed, 9 warnings. The failures are stale expectations for CoordinatedPickment.middle_empty_ratio (0.4 expected, 0.7 configured) and camera recording interval_step (5 expected, 1 configured).
  • Full simulator/GPU end-to-end validation was not run locally.

The source branch is currently four commits behind main; it should be synchronized before merge so CI validates the final integration state.

Screenshots

Not included. Generated SeedGraph PNGs and runtime artifacts are covered by the visualization and generation tests.

Checklist

  • I have run the black . command to format the code base.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove my fix is effective or that my feature works.
  • Dependencies have been updated, if applicable (no new dependency required).

MuziWong and others added 30 commits August 14, 2026 19:32
i
1. modify cli/start logic
2. add xy position info in scene object data sturcture
3. add some test files
4. modify scene export and scene import, thus the scene graph will be export and import automatically
Copilot AI lite review requested due to automatic review settings August 20, 2026 15:45
@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown

Too many files changed for review (211 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Features related to agentic system atomic action atomic action related functionality enhancement New feature or request refactor task A task written in openai gym format for imitation learning or reinforcement learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants