XR Feature Support Roadmap

[tbirdso]

Motivation

Browser-based virtual and augmented reality support is an evolving technology made feasible by the WebXR API, preceded by now-deprecated APIs such as WebVR and Cardboard. WebXR browser adoption is growing with most major browsers (Chrome, Edge, Firefox) supporting the WebXR API to various extents.

We aim to add key feature support so that developers can deliver browser-based virtual and augmented reality scenes with the vtk.js platform. Features will be accompanied by documentation and example scenes for user and developer accessibility.

This roadmap provides a high-level overview of XR features we would like to pursue in VTK.js along with potential paths for implementation. Roadmap progress is highly dependent on stakeholders and relevant project needs. If you are interested in completing a feature on the roadmap, please consider leading or funding that development contribution to VTK.js.

Target Platforms

We aim to target the intersection of platforms supported by vtk.js and by WebXR; that is, devices and browsers supporting WebGL rendering along with the WebXR API.

Target hardware includes virtual reality and augmented reality headsets, mobile devices with AR capabilities, and holographic displays. Please see VTK.js WebXR documentation for a complete list of target devices.

A complete list of hardware supported by the WebXR standard may be found here.

Roadmap

The features below are planned for eventual support in VTK.js and are listed in rough expected chronological order. A timeline is not currently available for WebXR feature development.

1. Basic XR Rendering Support

We aim to initially support XR rendering for a tailored subset of XR target devices.

• Update from WebVR to WebXR virtual reality rendering support implicit in the vtk.js OpenGL/RenderWindow class: feat(XR): Update to WebXR functionality in OpenGL RenderWindow #2171
• Add mobile AR support to the vtk.js OpenGL/RenderWindow class: feat(XR): Add base WebXR AR support to OpenGL RenderWindow #2202
• Update WebVR examples, add new WebXR examples, and extend existing example applications to demonstrate XR rendering capabilities: https://kitware.github.io/vtk-js/docs/develop_webxr.html

Backlog:

• Clipping range behavior: Odd clipping range behavior with WebXR #2452
• iframe error: OpenGL RenderWindow in sandboxed iframe console error #2537
• permissions error: Examples/Geometry/VR fails to enter VR on first request #2136
• WebXR refactoring: Refactor WebXR Rendering to vtkWebXRRenderWindowHelper #2924

2. Extend Supported Device List

We aim to support advanced rendering to the full list of target XR hardware. Scene interaction is not yet supported.

• Add support for WebGL/OpenXR-compatible augmented reality HMDs with passthrough (Magic Leap, Meta Quest 1/2/3/Pro, Apple Vision Pro): feat(WebXR): Add support for head-mounted augmented reality displays #2824
• Add support for Looking Glass device (https://lookingglassfactory.com/webxr)
• Add support for Microsoft HoloLens 2 headset, which WebXR considers a "VR" device: feat(WebXR): Add support for head-mounted augmented reality displays #2824
• Add examples demonstrating cinematic volume rendering with WebXR: https://kitware.github.io/vtk-js/examples/WebXRHeadFullVolumeCVR.html
• Add support for WebGPU rendering (https://github.com/immersive-web/WebXR-WebGPU-Binding/blob/main/explainer.md)
  • WebGPU / WebXR interoperation is currently in specification phase and is not yet implemented. We should keep an eye on its development.
• Expose parameters for scene initialization (scale, rotation, translation) when launching into an XR scene. Currently resets to center the scene with set scale and rotation when entering XR.
• Optimize rendering to the Looking Glass Factory display via reduced quilt size and on-demand rendering

Backlog:

• Looking Glass refactoring: Draft: Refactor Looking Glass WebXR support #2959
• Looking Glass view count / quilt size: Feature Request: (Re-)Expose user setting for the number of views Looking-Glass/looking-glass-webxr#40
• Looking Glass on-demand rendering: Feature Request: Persist Last Rendered Hologram Looking-Glass/looking-glass-webxr#41

3. Basic XR Controls

We aim to add support for basic XR interactions with VTK.js scenes. We will loosely base interaction infrastructure on the vtkRenderWindowInteractor3D, vtkVRRenderWindowInteractor, and vtkVRInteractorStyle hierarchy defined in the VTK C++ library.

• Evaluate feasibility of porting vtkRenderWindowInteractor3D and vtkOpenXRInteractorStyle.h from VTK C++ OpenXR module to vtk.js. This would help to standardize VR/AR interactions and controls according to the vtk.js interaction system.
• Implement vtk3DRenderWindowInteractor to handle 3D input events
• Implement vtkXRRenderWindowInteractor to extend vtk3DRenderWindowInteractor with XR-specific events
  • The WebXR Gamepads Module extends the Gamepads API for VR and AR controllers, but in practice there are inconsistencies in how VR/AR event data is structured. Mobile AR interactions also typically yield 2D screen touch coordinates while HMD-based VR with controllers yields data with 3D spatial information. Consider whether we need bespoke interactors to handle events on different XR target platforms.
• Implement vtkXRInteractorStyle to define how to handle XR-specific events
  • Consider whether we need more granular interactor styles such as vtkVRHMDInteractorStyle, vtkARMobileInteractorStyle, etc to appropriate handle view behavior
• Implement basic VR controls for user comfort:
  • Locomotion in the direction of the controller
  • Locomotion in the direction of a selected camera view
  • Discrete, "snap" rotation by a fixed angle about the world "up" vector
• Implement a vtkActor to render controller model(s) at controller positions for user comfort

Backlog:

• XRInputSourceEvents: feat(XR): Handle WebXR XRInputSourceEvents #2221

4. Advanced XR Controls

We aim to add support for advanced XR interactions with VTK.js scenes.

• Identify and handle advanced VR controls and scene interactions, including:
  • WebXR hand input module
  • 3D point and cell picking via raycast
  • Locomotion via teleportation
  • World scaling via two-handed grip
  • Model translation, rotation, scaling via grip
  • 3D widget / cutting plane interactions via grip
  • 2D widgets displayed on 3D panels
  • Feature parity with ParaView: https://www.kitware.com/navigation-basics-in-virtual-reality-with-paraview/
• Identify and handle advanced AR controls and scene interactions, including:
  • 2D-to-3D point and cell picking
  • 2D widgets / UI control overlays
  • 2D widgets displayed on 3D panels
  • WebXR anchors for fixing virtual data in a 3D scene
• Identify and handle advanced holographic display controls, including:
  • Optionally disable 3D view interactions while holographic display is running
  • Enforce standard VTK.js 3D view interactions while Looking Glass Factory popup window is running
  • Add VTK.js interactor style support to adjust focal point via the Looking Glass Factory WebXR polyfill
    • May require updates to the LGF WebXR polyfill API

5. Stretch Goals

This goals are not prioritized, but would be nice to have as part of the VTK.js WebXR feature set.

• Add support/example for collaborative XR viewing such as ParaView
• Add support for WebGPU rendering (https://github.com/immersive-web/WebXR-WebGPU-Binding/blob/main/explainer.md)