Skip to content

fix: Order OVRTX mapped-attribute commits after the producing Warp stream - #7166

Merged
kellyguo11 merged 4 commits into
isaac-sim:developfrom
pv-nvidia:pv/ovrtx-mapping-stream-sync
Aug 20, 2026
Merged

fix: Order OVRTX mapped-attribute commits after the producing Warp stream#7166
kellyguo11 merged 4 commits into
isaac-sim:developfrom
pv-nvidia:pv/ovrtx-mapping-stream-sync

Conversation

@pv-nvidia

Copy link
Copy Markdown
Contributor

Description

Filling a CUDA-mapped OVRTX attribute buffer requires the unmap to name the Warp stream that produced the data — an unmap without a CUDA sync performs no synchronization, and the with binding.map(...) form cannot carry one (its __exit__ takes no arguments). Until now the ordering held only by accident: OVRTX's commit kernel launches on the legacy default CUDA stream, which happens to serialize behind Warp's blocking-type streams. That is an implementation detail its API contract does not promise.

This adds isaaclab_ov.renderers.map_attribute_for_warp_writes, a reusable context manager that maps a binding for CUDA writes and unmaps it with the producing Warp stream, and routes the renderer's object and camera transform writes — the only GPU attribute mappings in the codebase — through it. Render-var maps need no changes: CUDA maps already order reads via _map_render_var_to_dlpack, and CPU maps are synchronized by OVRTX before map() returns.

No behavior change on current OVRTX builds; this turns an accidental ordering into a contractual one and gives future call sites one safe pattern.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@pv-nvidia pv-nvidia self-assigned this Aug 19, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Aug 19, 2026
Add map_attribute_for_warp_writes, a reusable context manager that maps
an OVRTX attribute binding for CUDA writes and unmaps it with the
producing Warp stream as the CUDA sync, so OVRTX's commit of the mapped
data waits for the fill on the GPU. The binding's own context manager
cannot carry that sync: its __exit__ takes no arguments and unmaps with
an empty cuda_sync, which the OVRTX API treats as no synchronization.

Route the renderer's object and camera transform writes, the only GPU
attribute mappings in the codebase, through the helper. Previously their
ordering held only through CUDA legacy default-stream serialization
against Warp's blocking-type streams, an implementation detail the OVRTX
contract does not promise. Render-var CUDA maps already order their
reads via _map_render_var_to_dlpack, and render-var CPU maps are
synchronized by OVRTX before map() returns, so neither needs changes.
@pv-nvidia
pv-nvidia force-pushed the pv/ovrtx-mapping-stream-sync branch from 2d65960 to f3dcbe2 Compare August 19, 2026 07:11
@pv-nvidia
pv-nvidia marked this pull request as ready for review August 19, 2026 07:13
@pv-nvidia
pv-nvidia requested a review from a team August 19, 2026 07:13
@pv-nvidia pv-nvidia mentioned this pull request Aug 19, 2026
7 tasks
@greptile-apps

greptile-apps Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR replaces implicit CUDA default-stream ordering with an explicit producer-stream synchronization contract for OVRTX object and camera transform commits.

  • Adds and publicly exports map_attribute_for_warp_writes.
  • Routes both GPU transform mapping paths through the stream-safe helper.
  • Adds API documentation, changelog coverage, and contract tests for normal and exceptional exits.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security defects identified.

The renderer's affected GPU writes now map the intended CUDA device and unmap with the same Warp stream used by their launch or copy, while the new tests cover both successful and exceptional context exits.

Important Files Changed

Filename Overview
source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_mapping.py Adds a balanced map/unmap context manager that commits CUDA-mapped attributes using the producing Warp stream.
source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py Migrates object and camera transform writes from unsynchronized binding context managers to the stream-aware helper.
source/isaaclab_ov/test/test_ovrtx_renderer_contract.py Verifies CUDA device selection, producer-stream propagation, and mapping cleanup when a fill raises.
source/isaaclab_ov/isaaclab_ov/renderers/init.pyi Exposes the new mapping helper as part of the renderers package API.

Sequence Diagram

sequenceDiagram
    participant Renderer
    participant Binding as OVRTX binding
    participant Warp as Warp producer stream
    participant OVRTX as OVRTX commit
    Renderer->>Binding: map(Device.CUDA, device_id)
    Binding-->>Renderer: mapped tensor
    Renderer->>Warp: enqueue transform fill
    Renderer->>Binding: "unmap(stream=Warp.cuda_stream)"
    Binding->>OVRTX: schedule commit after producer stream
    Warp-->>OVRTX: fill completes
    OVRTX-->>Renderer: commit mapped attributes
Loading

Reviews (1): Last reviewed commit: "Order OVRTX mapped-attribute commits aft..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The stream-aware mapping helper correctly centralizes OVRTX attribute commits and migrates both object and camera transform writes, but two concrete issues remain: the new module can initialize OVRTX without the renderer’s required environment/import guard, and its documented bare "cuda" input can map GPU 0 while selecting a stream from another current CUDA device.

  • Design and architecture: Centralizing map/unmap ordering in a reusable context manager matches the existing asynchronous stream-handoff design. However, importing Device at module scope bypasses the established guarded OVRTX initialization path, including OVRTX_SKIP_USD_CHECK, when this module is imported first.
  • API: The new public helper is exported and documented, but its stated support for bare "cuda" is inconsistent: mapping hard-codes device ID 0 while Warp stream lookup may resolve to the current nonzero CUDA device. The device should be resolved once and used consistently for both operations.
  • Implementation: Both changed transform producers now pass their Warp stream to OVRTX during unmap, and try/finally preserves cleanup on errors. Before merge, preserve the guarded OVRTX import behavior and make mapping and stream selection use the same resolved CUDA device.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_mapping.py Outdated
Comment thread source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_mapping.py
wp.get_device resolves the device ident once, so the mapped CUDA device
and the sync stream come from the same device: a bare "cuda" previously
parsed to index 0 while the stream lookup used Warp's current CUDA
device, silently voiding the ordering guarantee on non-zero devices.
The string parser is gone; the resolved device's ordinal and current
stream are used for map and unmap.

The ovrtx import moves inside the helper so importing the module through
the package's lazy exports cannot initialize ovrtx without the guarded
environment ovrtx_renderer establishes (OVRTX_SKIP_USD_CHECK, actionable
install error). Any real binding is created through that path, so ovrtx
is already imported when the helper runs.
The device-index resolution is one testable contract: it delegates to
Warp so a bare "cuda" means Warp's current CUDA device, never a parse
of the string. Pin that in a focused unit test and drop the map-level
test that duplicated it; the map tests keep asserting only what the
context manager owns (device forwarding, producer-stream unmap, and
release on a failed fill).
…ollow-up

Resolving the mapping device through Warp changed behavior for bare
"cuda" strings (current CUDA device instead of index 0). That change is
worth making, but not inside the stream-ordering fix: restore the string
parse so this PR alters no device placement, and mark the bare-cuda /
current-device split with a TODO. The follow-up caches the resolved Warp
device on the renderer instead of re-deriving it from strings.
try:
yield wp.from_dlpack(attr_mapping.tensor, dtype=dtype)
finally:
attr_mapping.unmap(stream=wp.get_stream(device).cuda_stream)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have the stream defined once and reused for all writes? _write_particle_q_slices uses it too!

@pv-nvidia pv-nvidia Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could do that, but my upcoming PR #7169 goes a bit further and caches the warp device itself to work around a bug where passing device id "cuda" would pick up whatever the current device is instead of using the device at the time the ovrtx renderer was created; a potential bug in mgpu scenario

And then we can just do self._warp_device.stream.cuda_stream

Whatever you prefer, I could also do it in this PR but it felt a different task

@kellyguo11
kellyguo11 merged commit 7825dd5 into isaac-sim:develop Aug 20, 2026
74 of 77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants