Implement WebXR Raw Camera Access for the ARKit backend#1769
Open
julapy wants to merge 2 commits into
Open
Conversation
Adds CameraTexturePointer/CameraTextureSize to System::Session::Frame::View so xr backends can surface the platform camera image alongside the color/depth render textures. Null on backends that do not populate it. ARKit implementation: the camera image was already converted YUV->RGB each frame, but only pre-composited into the color texture — which is the scene render target and therefore cannot double as a sampled camera image. Allocate a dedicated BGRA8 texture (recreated on viewport size change, released with the same deleted-texture callback used by the color/depth textures) and render a second camera pass into it, reusing the shared shader via screenPipelineState (no depth/stencil attachments) with the babylonTexture slot left unbound so the shader takes the camera YUV->RGB branch. This is the backend half of WebXR Raw Camera Access support (https://immersive-web.github.io/raw-camera-access/). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aImage) Implements the JS surface of the WebXR Raw Camera Access module (https://immersive-web.github.io/raw-camera-access/) on top of the per-view camera texture provided by the xr backend: - XRCamera (new): exposes width/height and wraps the platform camera texture (MTLTexture on ARKit) for bgfx using the same createTexture2D + overrideInternal-on-render-thread retry pattern as NativeCamera's CameraDevice. The bgfx handle is owned by a Graphics::Texture whose JS lifetime is a Napi::Pointer external. - XRView.camera: returns the per-view XRCamera (undefined when the backend provides no camera texture). XRViewerPose threads the camera texture pointer/size from the frame views into XRView::Update. - XRWebGLBinding.getCameraImage(camera): returns the camera image as a Napi::Pointer<Graphics::Texture> — the object type NativeEngine uses as a hardware texture's underlyingResource — so Babylon.js's WebXRRawCameraAccess feature can wrap and sample it without modification. Verified on device (iPhone 15 Pro Max, iOS 26.5): Babylon.js 9.9.1's RAW_CAMERA_ACCESS feature attaches, onTexturesUpdatedObservable fires, and the live ARKit camera image renders on scene geometry inside an immersive-ar session at 60 FPS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree company="EyeJack" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements the WebXR Raw Camera Access module for the ARKit backend, so Babylon.js content can sample the live AR camera image as a texture inside an
immersive-arsession:Babylon.js's existing
WebXRRawCameraAccessfeature works against this unmodified — this PR only adds the native hooks it calls (XRView.camera,XRWebGLBinding.getCameraImage).How
Commit 1 — xr dependency (ARKit backend):
CameraTexturePointer/CameraTextureSizetoSystem::Session::Frame::View(null on backends that don't populate them).screenPipelineStatewith thebabylonTextureslot left unbound so the shader takes the camera branch.Commit 2 — NativeXr plugin (JS surface):
XRCamera(new): exposeswidth/heightand wraps the platform camera texture for bgfx using the samecreateTexture2D+overrideInternal-on-render-thread retry pattern as NativeCamera'sCameraDevice.XRView.camera: returns the per-viewXRCamera(undefinedwhen the backend provides no camera texture).XRWebGLBinding.getCameraImage(camera): returns the camera image as aNapi::Pointer<Graphics::Texture>— the object type NativeEngine uses as a hardware texture'sunderlyingResource— soWebXRRawCameraAccesscan wrap and sample it directly.The camera texture is viewport-sized and display-aspect-mapped (same image as the passthrough background), which keeps
view.projectionMatrix-based intrinsics consistent with what Babylon's_updateCameraIntrinsicscomputes.Testing
Verified on device (iPhone 15 Pro Max, iOS 26.5, Xcode 26, Babylon.js 9.9.1 UMD) via the iOS Playground with
ar = trueand the feature enabled: the feature attaches,onTexturesUpdatedObservablefires with a 1290×2796 texture, and the live camera image renders on scene geometry inside the AR session at a sustained 60 FPS (camera pass adds well under 1 ms GPU time).Note: testing
immersive-aron current master first requires a fix for #1767 (NativeXr crashes on session start due to a null active encoder after the threading rework); I applied the guard suggested there while validating this PR. This PR is independent of that fix.Non-ARKit backends are unaffected (
CameraTexturePointerstays null →XRView.cameraisundefined, which is the same behavior as before this PR).🤖 Generated with Claude Code