[Fix] Automatically work around OVPhysX multi-GPU device selection - #7142
[Fix] Automatically work around OVPhysX multi-GPU device selection#7142hujc7 wants to merge 5 commits into
Conversation
Multi-GPU training with presets=ovphysx,ovrtx hangs on any --num_gpus >= 2: OVPhysX does not apply the requested active_cuda_gpus before PhysX creates its CUDA context, so with OVRTX in the process PhysX auto-selects the already current device. Rank 1 gets a device-0 binding for device-1 tensors and dies, and rank 0 then blocks forever in the NCCL bootstrap. The flag forwards torchrun's --virtual_local_rank, which gives each worker one GPU as cuda:0, making PhysX's default correct for every rank. Device selection needed no change: AppLauncher already derives the device from LOCAL_RANK bounded by the visible device count, which is the invariant virtual mode preserves. This is a workaround for nvbug 6573426, fixed in ovphysx 0.5.11; remove it once the pin moves past 0.5.10.
The multi-GPU launchers now give every worker a single GPU as cuda:0 whenever the
forwarded presets select OVPhysX, instead of requiring the caller to know about
--virtual_local_rank and pass it.
Detection matches the physics backend alone rather than an OVPhysX/OVRTX pair,
because the renderer has no stable name at launcher time: renderer=rtx is a
placeholder the worker resolves to OVRTX only once it knows no Kit runtime is
needed. A pair match reads {ovphysx, rtx}, misses, and lets that configuration
hang. The cost is engaging the workaround for OVPhysX with a renderer that does
not need it, where it is inert: device selection is bounded by the visible device
count, and rank identity keys on the global rank.
--no_virtual_local_rank opts out and --virtual_local_rank forces it on. Both
launcher preamble messages are flushed so they precede the workers' inherited-fd
output in a redirected log.
Remove once the ovphysx pin moves past 0.5.10, where nvbug 6573426 is fixed.
hujc7
changed the title
The workaround is now inferred from the presets alone; --virtual_local_rank and --no_virtual_local_rank are gone. A flag would be public CLI surface that outlives the defect it exists for: removing it after the ovphysx pin moves past 0.5.10 would need a deprecation and migration path, for an option whose only correct value is the one the launcher can already infer. With no flag, retiring the workaround is a deletion. The two detection helpers collapse into one, since with no explicit choice there is nothing to prefer over the preset scan.
Typed selectors are validated against their cfg type, so renderer= can never carry a physics backend and scanning it was dead. The match is on the raw name, which a future ovphysx alias would bypass; noted rather than resolved, since the alias table lives in isaaclab_tasks and this code is deleted at ovphysx 0.5.11.
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
Isaac Lab Review Bot
The automatic OVPhysX detection and torchrun flag placement are supported by focused tests, but the new detection helper writes an informational banner to stdout while building the command, breaking OVPhysX dry-run output.
- Design and architecture: Inferring the temporary workaround from forwarded OVPhysX preset tokens avoids adding a launcher option. However,
_use_virtual_local_rankmixes detection with output, making the shared command builder observably impure. - API: Preset tokens remain forwarded verbatim and no launcher option is added. The
--dry_runoutput contract regresses for OVPhysX runs because stdout no longer contains only the shell-parsable command. - Implementation: The bare
--virtual_local_rankflag is correctly placed in torchrun's arguments rather than the worker arguments. Move the informational banner outside command construction and emit it only after the dry-run path, or send it to stderr, so dry-run stdout remains machine-readable.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
| selected.update(name.strip() for name in arg[len(prefix) :].split(",")) | ||
| if VIRTUAL_LOCAL_RANK_PRESET not in selected: | ||
| return False | ||
| print( |
There was a problem hiding this comment.
🟡 Warning · Implementation — Info banner breaks dry-run command output
_use_virtual_local_rank prints from inside build_launch_command, which run_multigpu_cli calls before the --dry_run branch. With an OVPhysX preset, dry-run stdout becomes the [INFO] banner followed by the command, so callers that shell-parse or eval that output no longer get the exact command the launcher documents. Keep detection side-effect free and emit the banner from run_multigpu_cli after the dry-run return, or send it to stderr.
Greptile SummaryThe PR automatically enables torchrun virtual-local-rank isolation when raw launcher selectors contain OVPhysX, working around OVPhysX 0.5.10 device selection in multi-GPU runs. It also documents the behavior and adds launcher tests, but currently misses the existing
Confidence Score: 4/5The PR should not merge until automatic Literal-token detection covers explicit OVPhysX selections but diverges from worker-side backend resolution for kitless Files Needing Attention: source/isaaclab/isaaclab/cli/multigpu.py and source/isaaclab/test/benchmark/test_multigpu_launcher.py Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Forwarded physics and renderer selectors] --> B{Literal ovphysx token?}
B -- Yes --> C[Append --virtual_local_rank]
B -- No: physics=physx --> D[Launch without workaround]
D --> E[Worker resolves kitless PhysxAutoCfg]
E --> F[OVPhysX plus OVRTX]
F --> G[Device mismatch and distributed hang]
C --> H[One visible GPU per worker as cuda:0]
Reviews (1): Last reviewed commit: "Match the OVPhysX preset only where it c..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| selected.update(name.strip() for name in arg[len(prefix) :].split(",")) | ||
| if VIRTUAL_LOCAL_RANK_PRESET not in selected: |
There was a problem hiding this comment.
When a kitless multi-GPU launch uses physics=physx renderer=rtx, the worker resolves these selectors to OVPhysX and OVRTX, but this literal ovphysx check does not add --virtual_local_rank, causing nonzero ranks to hit the device mismatch and leaving rank 0 hanging during NCCL bootstrap.
1 participant
1. Summary
--num_gpus >= 2when an RTX renderershares the process. The launchers now work around it automatically — no flag to discover.
This repo pins
ovphysx==0.5.10and no published build carries the fix yet.2. Root cause (external)
OVPhysX's direct OVStage attach path does not apply the requested
active_cuda_gpusbefore PhysXcreates its CUDA context. With an RTX renderer in the same process a CUDA context is already
current, so PhysX auto-selects that device. Every rank above local rank 0 then gets a device-0
binding while its tensors live on its own device:
Those ranks die and rank 0 blocks in the NCCL bootstrap. Only
local_rank > 0is affected, whichis why single-GPU runs are fine and why this long presented as an NCCL or topology problem.
3. Fix
When the forwarded presets select OVPhysX, the launcher passes torchrun's
--virtual_local_rank.That sets
LOCAL_RANK=0for every worker and rewritesCUDA_VISIBLE_DEVICESso each sees only itsassigned GPU as device 0, making PhysX's default correct on every rank.
Detection matches the physics backend alone, not an OVPhysX/OVRTX pair. The renderer has no
stable name at launcher time:
renderer=rtxis a placeholder the worker resolves to OVRTX onlyonce it knows no Kit runtime is needed. A pair match reads
{ovphysx, rtx}, misses, and lets thatconfiguration hang — arm 6 below is exactly that case. The cost is engaging the workaround for
OVPhysX with a renderer that does not need it, where it is inert.
No device-selection code changed:
AppLauncheralready derives the device fromLOCAL_RANKbounded by the visible device count, which is the invariant virtual mode preserves. The
--local_ranks_filter 0default still works, because torchrun applies it agent-side against thereal local rank.
There is no flag. A temporary workaround should not add public CLI surface, because removing
that surface later needs a deprecation and migration path — for an option whose only correct value
is the one the launcher already infers. With none, retiring this is a deletion. Both launcher
preamble messages are flushed so they precede the workers' inherited-fd output in a redirected log.
4. Limits
LOCAL_RANK=0; use the global rank to name per-rank files or logs.rsl_rlkeysinit_process_groupand its process identity onRANK;skrldoes not readLOCAL_RANK. The benchmark path only range-checks0 <= local_rank < local_world_size.5. Validation
An 8-GPU node, 8x L40. IsaacLab
0469d0e5cee+ this change. ovphysx 0.5.10,ovrtx 0.4.1.364340, ovstage 0.1.1.355824, torch 2.11.0+cu128.
Isaac-Cartpole-Camera-Direct,64 envs, 3 iterations. Topology: GPU0-3 one PCIe switch (NUMA 0), GPU4-7 the other (NUMA 1).
CUDA_VISIBLE_DEVICESphysics=ovphysx renderer=rtxThe control failed with three
device mismatcherrors on exactly local_rank 1, 2 and 3, and usedthe same renderer as the passing arms — the only variable between them is the workaround. Arms 2-7
pass no virtual-local-rank flag; detection fired on its own each time. Arm 6's banner reports
Renderer rtx (ovrtx)with zero Isaac Sim in the process, confirming the pair-match gap is real.A sampled re-run confirmed physical placement rather than inferring it:
7,6,5,4occupiedGPUs 4-7 and
5,4,1,0occupied GPUs 0,1,4,5.6. Test
test_virtual_local_rank_follows_the_ovphysx_presetis parametrized over the affected andunaffected preset combinations, including
physics=ovphysx renderer=rtx, and asserts the presettokens still reach the worker verbatim. Companion tests cover the bare-flag forwarding to torchrun
and confirm the launcher exposes no opt-in option of its own.
Footnotes
The control was produced with an opt-out flag that this PR's final revision removes, so it is
not reproducible from the shipped CLI. The failure it demonstrates reproduces on any pre-change build. ↩