Skip to content

Fixes Kit camera rendering when running headless#5015

Open
peterd-NV wants to merge 7 commits intoisaac-sim:developfrom
peterd-NV:peterd/fix_headless_camera_rendering
Open

Fixes Kit camera rendering when running headless#5015
peterd-NV wants to merge 7 commits intoisaac-sim:developfrom
peterd-NV:peterd/fix_headless_camera_rendering

Conversation

@peterd-NV
Copy link
Contributor

Description

Previously to get correct camera renderings when running headless the Kit visualizer needed to be specified together with headless. This runs the script without the GUI but still initializes the Kit visualizer.

The Kit visualizer pumps the render update: https://github.com/isaac-sim/IsaacLab/blob/develop/source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py#L173

With PR 4948 the use of headless now overwrites any visualizers which results in incorrect camera observations (particularly after a reset) when running workflows that have cameras headlessly.

Fixes # (issue)

This PR fixes the camera rendering problem in headless mode by pumping the render as Kit would do.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Mar 14, 2026
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 14, 2026

Greptile Summary

This PR fixes two related bugs that caused stale camera observations when running headless with cameras:

  1. AppLauncher headless + visualizer interaction (app_launcher.py): PR Updates Visualizer Resolution and Deprecates Headless Arg #4948 introduced a regression where using --headless unconditionally cleared all visualizers, even when --viz kit was explicitly requested. The Kit visualizer is required to pump omni.kit.app.update() for correct camera frame delivery. The fix removes the erroneous lines that set _cli_visualizer_disable_all = True and _cli_visualizer_types = [] inside the deprecated --headless handling block, so explicitly-requested visualizers are preserved when running headless.

  2. Stale frame after reset() (simulation_context.py + isaac_rtx_renderer_utils.py): The RTX render dedup mechanism uses a (sim_id, step_count) stamp to avoid pumping app.update() more than once per physics step. Because _physics_step_count is never reset in SimulationContext.reset(), the stamp from the last pre-reset frame would match the first post-reset read, returning a stale frame. The fix adds invalidate_render_update() which resets the stamp to the (0, -1) sentinel, guaranteeing a fresh render on the first camera read after any reset.

Key changes:

  • Removed two lines from _resolve_headless_settings that incorrectly cleared the visualizer list when --headless was passed
  • Updated warning message to reflect the new behaviour (visualizers remain active in headless)
  • Added invalidate_render_update() to isaac_rtx_renderer_utils and calls it from SimulationContext.reset() via a guarded dynamic import
  • Version bumps and changelogs updated for both isaaclab (4.5.21) and isaaclab_physx (0.5.12)

Confidence Score: 4/5

  • This PR is safe to merge; the changes are targeted, well-reasoned, and fix a confirmed regression introduced by PR Updates Visualizer Resolution and Deprecates Headless Arg #4948.
  • Both fixes are logically sound and well-scoped. The only issue found is a misleading warning message that fires for --headless --viz none (where visualizers are actually disabled), but this does not affect runtime behavior. No tests were added, which is a minor concern for a rendering-pipeline fix, but the code changes themselves are straightforward and low-risk.
  • Pay close attention to source/isaaclab/isaaclab/app/app_launcher.py — specifically the new warning condition that can incorrectly fire for --headless --viz none.

Important Files Changed

Filename Overview
source/isaaclab/isaaclab/app/app_launcher.py Removes the erroneous clearing of visualizers when --headless is used alongside --viz, fixing camera rendering in headless mode. One minor logic issue: the new "Visualizers are still enabled" warning also fires for --headless --viz none, where visualizers are actually being disabled.
source/isaaclab/isaaclab/sim/simulation_context.py Calls invalidate_render_update() inside reset() to clear the RTX render dedup stamp, ensuring the first camera read post-reset triggers a fresh frame rather than returning the stale pre-reset image.
source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_utils.py Adds invalidate_render_update() which resets _last_render_update_key to the (0, -1) sentinel, guaranteeing the next call to ensure_isaac_rtx_render_update() pumps a fresh RTX frame regardless of the current step count.

Sequence Diagram

sequenceDiagram
    participant User
    participant AppLauncher
    participant SimulationContext
    participant RTXUtils as isaac_rtx_renderer_utils
    participant KitViz as KitVisualizer
    participant OmniApp as omni.kit.app

    User->>AppLauncher: launch(headless=True, viz=kit)
    AppLauncher->>AppLauncher: _resolve_visualizer_settings()<br/>cli_visualizer_types=[kit]
    AppLauncher->>AppLauncher: _resolve_headless_settings()<br/>(no longer clears visualizer list)
    AppLauncher->>SimulationContext: create with KitVisualizer active

    Note over SimulationContext,KitViz: Episode runs...

    User->>SimulationContext: reset()
    SimulationContext->>KitViz: viz.reset(soft)
    SimulationContext->>RTXUtils: invalidate_render_update()<br/>_last_render_update_key = (0,-1)
    SimulationContext->>SimulationContext: physics_manager.play()

    User->>SimulationContext: step() → render()
    SimulationContext->>KitViz: update(dt)
    KitViz->>OmniApp: app.update()  [pumps RTX]
    KitViz-->>RTXUtils: pumps_app_update() = True → dedup stamp updated

    User->>Camera: get_data()
    Camera->>RTXUtils: ensure_isaac_rtx_render_update()
    RTXUtils->>RTXUtils: key == _last_render_update_key?<br/>True → skip (already pumped by Kit)
    RTXUtils-->>Camera: fresh frame returned ✓
Loading

Last reviewed commit: f1eb1eb

Comment on lines 698 to 702
if self._cli_visualizer_explicit:
logger.warning(
"Both '--headless' and '--visualizer/--viz' were provided. "
"Deprecated '--headless' takes precedence and disables all visualizers."
"Visualizers are still enabled so viewport/camera pipeline runs correctly in headless."
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misleading warning fires for --headless --viz none

The second warning says "Visualizers are still enabled so viewport/camera pipeline runs correctly in headless", but this fires whenever self._cli_visualizer_explicit is True — including when the user passes --headless --viz none, which sets self._cli_visualizer_disable_all = True and does disable all visualizers. The guard should exclude the "disable all" case so the message only fires when visualizers are actually being kept active.

Suggested change
if self._cli_visualizer_explicit:
logger.warning(
"Both '--headless' and '--visualizer/--viz' were provided. "
"Deprecated '--headless' takes precedence and disables all visualizers."
"Visualizers are still enabled so viewport/camera pipeline runs correctly in headless."
)
if self._cli_visualizer_explicit and not self._cli_visualizer_disable_all:
logger.warning(
"Both '--headless' and '--visualizer/--viz' were provided. "
"Visualizers are still enabled so viewport/camera pipeline runs correctly in headless."
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant