fix: Order OVRTX mapped-attribute commits after the producing Warp stream - #7166
Conversation
Add map_attribute_for_warp_writes, a reusable context manager that maps an OVRTX attribute binding for CUDA writes and unmaps it with the producing Warp stream as the CUDA sync, so OVRTX's commit of the mapped data waits for the fill on the GPU. The binding's own context manager cannot carry that sync: its __exit__ takes no arguments and unmaps with an empty cuda_sync, which the OVRTX API treats as no synchronization. Route the renderer's object and camera transform writes, the only GPU attribute mappings in the codebase, through the helper. Previously their ordering held only through CUDA legacy default-stream serialization against Warp's blocking-type streams, an implementation detail the OVRTX contract does not promise. Render-var CUDA maps already order their reads via _map_render_var_to_dlpack, and render-var CPU maps are synchronized by OVRTX before map() returns, so neither needs changes.
2d65960 to
f3dcbe2
Compare
Greptile SummaryThe PR replaces implicit CUDA default-stream ordering with an explicit producer-stream synchronization contract for OVRTX object and camera transform commits.
Confidence Score: 5/5The PR appears safe to merge with no actionable correctness or security defects identified. The renderer's affected GPU writes now map the intended CUDA device and unmap with the same Warp stream used by their launch or copy, while the new tests cover both successful and exceptional context exits. Important Files Changed
Sequence DiagramsequenceDiagram
participant Renderer
participant Binding as OVRTX binding
participant Warp as Warp producer stream
participant OVRTX as OVRTX commit
Renderer->>Binding: map(Device.CUDA, device_id)
Binding-->>Renderer: mapped tensor
Renderer->>Warp: enqueue transform fill
Renderer->>Binding: "unmap(stream=Warp.cuda_stream)"
Binding->>OVRTX: schedule commit after producer stream
Warp-->>OVRTX: fill completes
OVRTX-->>Renderer: commit mapped attributes
Reviews (1): Last reviewed commit: "Order OVRTX mapped-attribute commits aft..." | Re-trigger Greptile |
There was a problem hiding this comment.
Isaac Lab Review Bot
The stream-aware mapping helper correctly centralizes OVRTX attribute commits and migrates both object and camera transform writes, but two concrete issues remain: the new module can initialize OVRTX without the renderer’s required environment/import guard, and its documented bare "cuda" input can map GPU 0 while selecting a stream from another current CUDA device.
- Design and architecture: Centralizing map/unmap ordering in a reusable context manager matches the existing asynchronous stream-handoff design. However, importing
Deviceat module scope bypasses the established guarded OVRTX initialization path, includingOVRTX_SKIP_USD_CHECK, when this module is imported first. - API: The new public helper is exported and documented, but its stated support for bare
"cuda"is inconsistent: mapping hard-codes device ID 0 while Warp stream lookup may resolve to the current nonzero CUDA device. The device should be resolved once and used consistently for both operations. - Implementation: Both changed transform producers now pass their Warp stream to OVRTX during unmap, and
try/finallypreserves cleanup on errors. Before merge, preserve the guarded OVRTX import behavior and make mapping and stream selection use the same resolved CUDA device.
Minor fixes needed. Posted 2 actionable findings inline.
Automated review; human maintainers own approval decisions.
wp.get_device resolves the device ident once, so the mapped CUDA device and the sync stream come from the same device: a bare "cuda" previously parsed to index 0 while the stream lookup used Warp's current CUDA device, silently voiding the ordering guarantee on non-zero devices. The string parser is gone; the resolved device's ordinal and current stream are used for map and unmap. The ovrtx import moves inside the helper so importing the module through the package's lazy exports cannot initialize ovrtx without the guarded environment ovrtx_renderer establishes (OVRTX_SKIP_USD_CHECK, actionable install error). Any real binding is created through that path, so ovrtx is already imported when the helper runs.
The device-index resolution is one testable contract: it delegates to Warp so a bare "cuda" means Warp's current CUDA device, never a parse of the string. Pin that in a focused unit test and drop the map-level test that duplicated it; the map tests keep asserting only what the context manager owns (device forwarding, producer-stream unmap, and release on a failed fill).
…ollow-up Resolving the mapping device through Warp changed behavior for bare "cuda" strings (current CUDA device instead of index 0). That change is worth making, but not inside the stream-ordering fix: restore the string parse so this PR alters no device placement, and mark the bare-cuda / current-device split with a TODO. The follow-up caches the resolved Warp device on the renderer instead of re-deriving it from strings.
| try: | ||
| yield wp.from_dlpack(attr_mapping.tensor, dtype=dtype) | ||
| finally: | ||
| attr_mapping.unmap(stream=wp.get_stream(device).cuda_stream) |
There was a problem hiding this comment.
Can we have the stream defined once and reused for all writes? _write_particle_q_slices uses it too!
There was a problem hiding this comment.
Could do that, but my upcoming PR #7169 goes a bit further and caches the warp device itself to work around a bug where passing device id "cuda" would pick up whatever the current device is instead of using the device at the time the ovrtx renderer was created; a potential bug in mgpu scenario
And then we can just do self._warp_device.stream.cuda_stream
Whatever you prefer, I could also do it in this PR but it felt a different task
Description
Filling a CUDA-mapped OVRTX attribute buffer requires the unmap to name the Warp stream that produced the data — an unmap without a CUDA sync performs no synchronization, and the
with binding.map(...)form cannot carry one (its__exit__takes no arguments). Until now the ordering held only by accident: OVRTX's commit kernel launches on the legacy default CUDA stream, which happens to serialize behind Warp's blocking-type streams. That is an implementation detail its API contract does not promise.This adds
isaaclab_ov.renderers.map_attribute_for_warp_writes, a reusable context manager that maps a binding for CUDA writes and unmaps it with the producing Warp stream, and routes the renderer's object and camera transform writes — the only GPU attribute mappings in the codebase — through it. Render-var maps need no changes: CUDA maps already order reads via_map_render_var_to_dlpack, and CPU maps are synchronized by OVRTX beforemap()returns.No behavior change on current OVRTX builds; this turns an accidental ordering into a contractual one and gives future call sites one safe pattern.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there