Skip to content

Implement WebXR Raw Camera Access for the ARKit backend#1769

Open
julapy wants to merge 2 commits into
BabylonJS:masterfrom
julapy:arkit-raw-camera-access
Open

Implement WebXR Raw Camera Access for the ARKit backend#1769
julapy wants to merge 2 commits into
BabylonJS:masterfrom
julapy:arkit-raw-camera-access

Conversation

@julapy

@julapy julapy commented Jul 7, 2026

Copy link
Copy Markdown

What

Implements the WebXR Raw Camera Access module for the ARKit backend, so Babylon.js content can sample the live AR camera image as a texture inside an immersive-ar session:

const cameraAccess = xr.baseExperience.featuresManager.enableFeature(BABYLON.WebXRFeatureName.RAW_CAMERA_ACCESS, "latest");
cameraAccess.onTexturesUpdatedObservable.add((textures) => {
    material.emissiveTexture = textures[0]; // live ARKit camera image
});

Babylon.js's existing WebXRRawCameraAccess feature works against this unmodified — this PR only adds the native hooks it calls (XRView.camera, XRWebGLBinding.getCameraImage).

How

Commit 1 — xr dependency (ARKit backend):

  • Adds CameraTexturePointer / CameraTextureSize to System::Session::Frame::View (null on backends that don't populate them).
  • ARKit already converts the camera image YUV→RGB each frame, but only pre-composites it into the color texture — which is the scene render target and can't double as a sampled camera image. This adds a dedicated BGRA8 texture (recreated on viewport size change, released through the same deleted-texture callback as the color/depth textures) and a second camera pass into it, reusing the shared shader via screenPipelineState with the babylonTexture slot left unbound so the shader takes the camera branch.

Commit 2 — NativeXr plugin (JS surface):

  • XRCamera (new): exposes width/height and wraps the platform camera texture for bgfx using the same createTexture2D + overrideInternal-on-render-thread retry pattern as NativeCamera's CameraDevice.
  • XRView.camera: returns the per-view XRCamera (undefined when the backend provides no camera texture).
  • XRWebGLBinding.getCameraImage(camera): returns the camera image as a Napi::Pointer<Graphics::Texture> — the object type NativeEngine uses as a hardware texture's underlyingResource — so WebXRRawCameraAccess can wrap and sample it directly.

The camera texture is viewport-sized and display-aspect-mapped (same image as the passthrough background), which keeps view.projectionMatrix-based intrinsics consistent with what Babylon's _updateCameraIntrinsics computes.

Testing

Verified on device (iPhone 15 Pro Max, iOS 26.5, Xcode 26, Babylon.js 9.9.1 UMD) via the iOS Playground with ar = true and the feature enabled: the feature attaches, onTexturesUpdatedObservable fires with a 1290×2796 texture, and the live camera image renders on scene geometry inside the AR session at a sustained 60 FPS (camera pass adds well under 1 ms GPU time).

Note: testing immersive-ar on current master first requires a fix for #1767 (NativeXr crashes on session start due to a null active encoder after the threading rework); I applied the guard suggested there while validating this PR. This PR is independent of that fix.

Non-ARKit backends are unaffected (CameraTexturePointer stays null → XRView.camera is undefined, which is the same behavior as before this PR).

🤖 Generated with Claude Code

julapy and others added 2 commits July 7, 2026 15:50
Adds CameraTexturePointer/CameraTextureSize to System::Session::Frame::View so
xr backends can surface the platform camera image alongside the color/depth
render textures. Null on backends that do not populate it.

ARKit implementation: the camera image was already converted YUV->RGB each
frame, but only pre-composited into the color texture — which is the scene
render target and therefore cannot double as a sampled camera image. Allocate
a dedicated BGRA8 texture (recreated on viewport size change, released with
the same deleted-texture callback used by the color/depth textures) and render
a second camera pass into it, reusing the shared shader via screenPipelineState
(no depth/stencil attachments) with the babylonTexture slot left unbound so the
shader takes the camera YUV->RGB branch.

This is the backend half of WebXR Raw Camera Access support
(https://immersive-web.github.io/raw-camera-access/).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aImage)

Implements the JS surface of the WebXR Raw Camera Access module
(https://immersive-web.github.io/raw-camera-access/) on top of the per-view
camera texture provided by the xr backend:

- XRCamera (new): exposes width/height and wraps the platform camera texture
  (MTLTexture on ARKit) for bgfx using the same createTexture2D +
  overrideInternal-on-render-thread retry pattern as NativeCamera's
  CameraDevice. The bgfx handle is owned by a Graphics::Texture whose JS
  lifetime is a Napi::Pointer external.
- XRView.camera: returns the per-view XRCamera (undefined when the backend
  provides no camera texture). XRViewerPose threads the camera texture
  pointer/size from the frame views into XRView::Update.
- XRWebGLBinding.getCameraImage(camera): returns the camera image as a
  Napi::Pointer<Graphics::Texture> — the object type NativeEngine uses as a
  hardware texture's underlyingResource — so Babylon.js's WebXRRawCameraAccess
  feature can wrap and sample it without modification.

Verified on device (iPhone 15 Pro Max, iOS 26.5): Babylon.js 9.9.1's
RAW_CAMERA_ACCESS feature attaches, onTexturesUpdatedObservable fires, and the
live ARKit camera image renders on scene geometry inside an immersive-ar
session at 60 FPS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@julapy

julapy commented Jul 7, 2026

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="EyeJack"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant