[Task Clean-up][Assets] Serve the Shadow Hand from one asset with native PhysX tendons - #7161
Draft
hujc7 wants to merge 16 commits into
Draft
[Task Clean-up][Assets] Serve the Shadow Hand from one asset with native PhysX tendons#7161hujc7 wants to merge 16 commits into
hujc7 wants to merge 16 commits into
Conversation
The published asset's physx variant carries neither the four fixed tendons the mujoco variant expresses as MjcTendon prims nor a world attachment PhysX honours, so the hand spawned floating with no tendons and Isaac Lab had to synthesise both at spawn time. physx.usda gains the four tendons: PhysxTendonAxisRootAPI on each finger's J3 with the gains Menagerie gives the equivalent MuJoCo actuator, and PhysxTendonAxisAPI on J2 and J1 with gearing pi/180, which cancels PhysX's radians-to-degrees conversion so the tendon length is q_J1 + q_J2 -- the quantity MjcTendon computes with coef [1, 1]. physics.usda moves PhysicsArticulationRootAPI from the forearm link to the enclosing prim and replaces the in-link weld with a rootJoint whose body0 is empty, which is how UsdPhysics spells "attach to world"; the previous joint named an Xform rather than a rigid body, so PhysX reported fixed_base False. A .gitattributes opts the text layers out of LFS, which otherwise reduces them to pointer files.
A fixed tendon's length is a weighted sum of the joint angles it spans, and each engine actuates it through a different native mechanism: PhysX shifts the tendon's length offset, MuJoCo drives the actuator whose transmission is the tendon. Without a shared entry point every caller has to branch on the backend. set_fixed_tendon_position_target_index is declared once on BaseArticulation and implemented by both engines, so a task commands a target length and never names a mechanism. PhysX buffers the negated offset and flushes it from write_data_to_sim, the same shared step that flushes joint targets, rather than writing per-call. Newton has no tendon accessor on its ArticulationView -- MuJoCo's tendons and their actuators reach the model only as flat mujoco:* arrays spanning every world -- so MjcTendonActuatorView resolves the per-articulation indexing that implementation needs. It indexes eligible actuators once by (world, target label); scanning them per tendon instead cost 174 s for one hand and 344 s per articulation view for two. The MuJoCo custom attributes carrying that data are registered on the model builder before the stage is traversed, without which every mjc:* attribute is dropped on import and joint armature, joint friction and actuator definitions silently fall back to schema defaults.
Tendons have their own index space, so a joint action term cannot reach them: a hand whose fingers are tendon-coupled had no way to command those joints from a manager-based task at all. The term maps a normalised action onto the tendon's commandable span and clips to [-1, 1] first, as the joint terms do -- a policy samples from an unbounded Gaussian, so early actions otherwise leave the tendon's reachable range and yank the fingers. It buffers through the asset and reaches the simulation from write_data_to_sim.
Applying a schema rebuilt the prim's applied-schema list, which dropped the fragments composed from other layers. A hand's tendon APIs live in the asset's physx layer, so spawning it through the schema path silently removed them, and the articulation came up reporting no tendons for the action term to bind to.
The hand had a configuration per engine, so the two could and did drift: different spawn rotations, different gains, and a Newton-only variant that tasks had to select by name. ShadowHand.cfg(physics) returns one asset whose Physics USD variant selects the engine, and everything a task needs to spawn or address the hand -- the asset path, the sixteen joint motors, the four tendon motors, the fingertip bodies -- is stated once on the class. joint_ordering and body_ordering are declared so both engines expose the hand's joints in one index space. Left unset each backend uses its own enumeration order, and MEASURED they differ: Newton walks depth-first per finger, PhysX breadth-first by joint level, so index 3 is rh_FFJ3 on one and rh_MFJ4 on the other. Every observation past index 2 then names a different joint, and a policy trained on one engine scored 0.0005 on the other against a 0.6165 native baseline. articulation_root_prim_path is stated rather than searched for: the search demands exactly one prim carrying an articulation-root API, and the asset's root link also carries NewtonArticulationRootAPI, which satisfies the same check. usd_path defaults to the copy vendored in this package until the fixed asset is published; SHADOW_HAND_USD overrides it.
The task took its hand from the per-engine configurations and re-stated the hand's own joint, tendon and fingertip names, so a change to the hand had to be mirrored in the task. It now takes them from ShadowHand. The tendon command is a Shadow-specific concern, so the tendon write and the index resolution it needs live in a Shadow subclass of the reorientation environment rather than in the shared base, which the Allegro hand also uses and which has no tendons. The manager configuration declares its tendon action term in an actions class of its own for the same reason.
Both hands took their configuration from the per-engine Shadow Hand configs and raised every actuator's gains, including the tendon-coupled joints. Those joints take no position command, and MEASURED, driving them costs the tendon most of its travel: 11.1 rad falls to 1.0 rad. The hands now take the gains the asset supplies. The reward summed one squared-action penalty over both agents' actions, so each hand paid for the other's. Once a hand releases the ball the shared distance term stops responding to its pose, and nothing else discourages it from saturating its motors -- which is what the released hand's over-curling was. Each agent now pays for its own actions.
Both command modules defined a term and its configuration in one file, where the rest of the repository keeps configurations in a module of their own. Each configuration moves to commands_cfg.py beside its implementation, which imports it only for typing. The goal orientation is now drawn from the shared uniform SO(3) sampler. Composing two axis-angle rotations, as both terms did, reaches only a two-axis subset of orientations and needs a unit-axis buffer per axis to do it -- so success rates measured after this change are measured on a harder goal distribution than those before it.
set_fixed_tendon_position_target_index is abstract on BaseArticulation, and OVPhysX was the one backend without an implementation -- so every OVPhysX articulation raised TypeError on construction, not only those with tendons. OVPhysX carries the same tendon model as PhysX, so it commands the target the same way: the schema documents offset as the value added to the accumulated length, so a target length is applied as its negation and delegated to the offset setter this backend already implements natively.
Six call sites each searched for a prim carrying ArticulationRootAPI and demanded exactly one match. More than one prim can answer that query without the asset being ambiguous: a backend-specific root such as NewtonArticulationRootAPI includes ArticulationRootAPI among its built-in schemas, so a link carrying the former also answers for the latter, while the enclosing prim carries the root PhysX needs to treat the articulation as fixed-base. Both are required, and an asset carrying both could not be spawned without naming its root in the configuration. The search moves to one helper beside the generic resolver it wraps, and drops any root nested inside another: the outer prim describes the articulation the inner one belongs to. Sibling roots remain an error, since they are separate articulations and nothing says which was meant, and the message now names the paths it found and the field that disambiguates. The Shadow Hand no longer has to state articulation_root_prim_path, which existed only to step around this.
…tendon-codex-base # Conflicts: # source/isaaclab_tasks/test/core/test_handover_physics_presets.py
The tendon target is commanded every step, and OVPhysX applied it through set_fixed_tendon_offset_index, which pushes the buffer to the simulation as it fills it. That is right for a property edited once at setup and wrong for a per-step command: it made the action term the only sim write outside write_data_to_sim, the shared step that flushes joint targets. Both setters now fill the buffer through one helper. The property setter still pushes immediately; the target command leaves the write to write_data_to_sim, matching PhysX and Newton, so a step has exactly one place where data reaches the simulation.
The handover environment expanded {ENV_REGEX_NS} itself before building its assets,
against a failure whose cause was never identified. AssetBase does that expansion for
assets a direct environment builds itself, and MEASURED, the training entry point the
workaround was written for now reaches three learning iterations with zero "is not
global" errors without it.
Both hand environments resolved their tendon indices and built the same two limit tensors
from the same source. That moves to one helper beside the other shared task utilities.
Also dropped: a goal-command unit vector left behind when the sampler moved to uniform
SO(3), a factory wrapping one action-term constructor used twice, three module constants
computing two adjacent values, and a five-line layout helper called once.
Pairing a tendon with the actuator that drives it read as one 98-line function. It is four steps: read the mujoco:* arrays, index the eligible actuators by (world, target label), walk the instances pairing them up, then require every instance to agree. Each is now named and carries the reasoning that belongs to it -- in particular why the indexing happens once, which is the difference between 0.1 s and minutes of start-up per articulation view. No behaviour changes: the Newton articulation suite returns the same 319 passed / 4 skipped.
…tendon-codex-base
The repository sends *.usd and *.usda to LFS. This asset opted its text layers out so the tendon and root-weld edits would diff as text during review, which made it the only *.usda in the tree stored as a plain blob -- and `git lfs fsck --pointers`, which the pre-commit LFS hook falls back to, rejects exactly that. The layers now follow the same rule as every other .usda here. The asset is vendored only until the fixed one is published, at which point these files leave the repository.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PhysicsUSD variant selects the engine, so the two configurations can no longer drift apart.set_fixed_tendon_position_target_indexis declared once onBaseArticulationand implemented by PhysX, Newton and OVPhysX, so a task commands a tendon target without naming a backend.The asset
physx.usdagains four fixed tendons:PhysxTendonAxisRootAPIon each finger'sJ3carrying the gains Menagerie gives the equivalent MuJoCo actuator, andPhysxTendonAxisAPIonJ2/J1withgearing = 0.017453292. That constant is pi/180, which cancels PhysX's radians-to-degrees conversion so tendon length isq_J1 + q_J2— the quantityMjcTendoncomputes withcoef [1, 1]. That equality is what lets one command mean the same thing on both engines.physics.usdamovesPhysicsArticulationRootAPIonto the enclosing prim and replaces the in-link weld with arootJointwhosebody0is empty, which is howUsdPhysicsspells "attach to world". The previous joint named an Xform rather than a rigid body, so PhysX reportedfixed_base=False.The asset is vendored in-repo and is load-bearing. With the spawn-time patch removed, nothing fixes up the published tendon-less asset at runtime, so
usd_pathdefaults to the copy inisaaclab_assets/data/. This reverts toISAAC_NUCLEUS_DIRonce the fixed asset is published.Framework
ArticulationRootAPI; a backend root such asNewtonArticulationRootAPIincludes it as a built-in, so an asset carrying both could not spawn without naming its root. Sibling roots remain an error, now naming the paths it found.mjc:*attribute was dropped on import and armature/friction fell back to schema defaults.Validation
24 handover runs x 4000 iterations on PhysX (4096 envs, 12 seeds x 2 asset variants) completed with 0 NaN. Goals are sampled uniformly over SO(3), a harder distribution than the two-axis subset these numbers were previously measured on.
Suites:
isaaclab_physxarticulation 241,isaaclab_newtonarticulation 319,isaaclab_ovarticulation 119, joint-wrench sensors 16 + 11, tendon fragments 22. All 32 registered entry points resolve.Known-open
mjc:dampingon the tendon-coupled joints is not what reaches the solver; both backends agree on the value they do use, so this is a fidelity question rather than a cross-backend divergence.