Skip to content

Client-side stereo detection transfer via ONNX - #1709

Merged
BryonLewis merged 11 commits into
mainfrom
dev/stereo-onnx-transfer
Aug 13, 2026
Merged

Client-side stereo detection transfer via ONNX#1709
BryonLewis merged 11 commits into
mainfrom
dev/stereo-onnx-transfer

Conversation

@mattdawkins

@mattdawkins mattdawkins commented Jun 22, 2026

Copy link
Copy Markdown
Member

Warp a detection from one camera to the other and measure its length entirely
client-side (web + Electron renderer), with no backend, by running VIAME's
epipolar template-matching stereo model (stereo measurement "method 1") exported
to ONNX under onnxruntime-web, plus a client-side port of the triangulation.

This is the client counterpart to the desktop backend stereo service: the
desktop ViewerLoader warps and measures via native IPC (stereoTransferLine /
stereoTransferPoints / stereoMeasureLine); this does the equivalent in the
browser so it also works on the web.

What's here (client/dive-common/use/stereo/)

  • StereoOnnxMatcher — loads the match ONNX model and warps source points to
    target points (epipolar candidate generation + NCC along the curve).
  • calibration.ts / npz.ts — parse .npz/.json calibration in-browser
    (mirrors read_stereo_rig), plus invertRig to swap source/target camera.
  • triangulate.ts — two-view triangulation and stereo measurement (length,
    midpoint, range, RMS) + length aggregation, porting
    viame::core::compute_stereo_measurement and aggregate_lengths. No SVD:
    the DLT null vector comes from the adjugate of AᵀA, as in the ONNX graph.
  • image.ts — RGBA to BT.601 grayscale (matches OpenCV BGR2GRAY used by the NCC).
  • frameSource.ts — read full-res frame pixels from a GeoJS viewer.
  • useStereoOnnxTransfer — warp a box, head/tail line or polygon to the other
    camera, measure once both cameras hold a line, and bulk-warp a camera.
  • Web glue: platform/web-girder/useStereoOnnxWeb.ts, bound to the Viewer's
    stereo-annotation-complete, stereo-track-linked and the Import menu's
    stereo-warp-imported in the web ViewerLoader.vue.

onnxruntime-web is added as a dependency. The exported model is committed at
client/public/models/stereo_match.onnx, which vite build copies into dist/.
The matcher unit test runs against that served model rather than a second copy.

Matching parameters

Held identical to what the desktop interactive stereo service loads from
configs/pipelines/interactive_stereo_template.conf, so both platforms accept
the same matches. Hidden config there and here — no UI on either side.

Config key Value Where it lives here
epipolar_min_disparity / epipolar_max_disparity 2 / 300 px DEFAULT_RANGE in useStereoOnnxWeb
template_matching_threshold 0.5 DEFAULT_THRESHOLD in StereoOnnxMatcher
uniqueness ratio 0 (disabled) desktop applies no uniqueness test
template_size 13 baked into the ONNX graph at export
epipolar_num_samples 5000 baked into the ONNX graph at export

The model was re-exported at --template-size 13 for this: the exporter's own
default is 25, so the previously committed model did not match the service.

Behavior parity with desktop

  • Boxes and polygons warp once; every warped vertex must be a confident match.
  • Head/tail lines re-warp when the source is edited so an auto-generated line
    tracks its source; a human-drawn line is marked stereo_user_line and is
    never overwritten.
  • Measurement writes length (and canonical fishLength), midpoint_x/y/z,
    midpoint_range, stereo_rms and track-level avg_length; a length locked
    with length_method = user_set is preserved.
  • Progress dialog, error dialog and length snackbar mirror the desktop ones.

Enabling on web

isStereoInteractiveModeEnabled() previously hard-returned
isDesktopRuntime() && …, so stereo-annotation-complete was never emitted off
desktop and this feature was unreachable. Since neither the warp nor the
measurement needs a service any more, that gate is removed and the Stereo
Settings panel is no longer desktop-only.

Testing

  • Unit-tested (__tests__/, runs under npm test): calibration parsing,
    grayscale, and point warping validated against the VIAME C++/Python reference
    (~0.25 px) via onnxruntime-web in Node; triangulation and measurement
    validated by projecting known 3D points through the rig, with and without
    distortion, and recovering them (0.01 units at 2.5 m).
  • Full suite: 732 tests pass; lint clean; build:web succeeds.

Note on the reference port

plugins/onnx/triangulate.py applies the Lindstrom correction with
p1ᵀ E p2 (left, right), but for E = [T]ₓR the constraint that holds is
p2ᵀ E p1 — verified numerically against the fixture rig. Using the Python's
ordering perturbs exact correspondences and costs ~0.2% in recovered depth. This
port uses the transpose. The VIAME-side script is probably worth the same fix.

Fixed during live testing

  • The web glue read viewer.multiCamList?.value / aggregateController?.value,
    but Vue unwraps setup refs on the component instance (the desktop ViewerLoader
    reads them bare). Both were undefined, so every stereo op saw zero cameras
    and returned skipped silently.
  • Vite pre-bundled onnxruntime-web into node_modules/.vite/deps, where the
    runtime's sibling .wasm request hit the SPA HTML fallback and failed to
    instantiate — the model never loaded under npm run serve. Excluded from
    optimizeDeps; the production build was already fine.
  • A missing calibration or a failed triangulation now surfaces in the error
    dialog instead of no-oping.

Outstanding

  • Partly live-tested now. Calibration download, the ViewerLoader bindings
    and client-side measurement are confirmed working in a running web viewer
    against a real stereo dataset (length attributes appear on both cameras).
    The warp path (GeoJS frame-pixel read in
    frameSource.geoViewerToImageElement) is still unverified — it was blocked
    behind the two load bugs fixed in this branch and needs a pass now that the
    runtime loads.
  • Segmentation seed transfer is not implemented. Desktop's
    segmentation-stereo-segment warps the seed, runs SAM on the other camera and
    can derive head/tail lines plus the measurement on both. That needs SAM in the
    browser, so type: 'segmentation' returns skipped.
    stereo-annotation-reset and stereo-segmentation-finalize stay unbound on
    web for the same reason — they exist only to undo segmentation side effects.
  • Bulk "Warp to All" is slow and image-sequence only. It decodes each frame
    over HTTP because the GeoJS viewer only holds the on-screen frame; video
    datasets have no imageData, so off-screen frames yield nothing.
  • Measurement includes distortion; the desktop binding does not.
    compute_stereo_measurement_from_calibration builds
    simple_camera_intrinsics(K) with no distortion coefficients, so on a rig
    with significant distortion the web number will differ from desktop. This port
    follows geometry_numpy.py instead. Reduces to identical results when the
    coefficients are zero.
  • The disparity range is scene-dependent and has no UI, matching desktop.
    2–300 px suits the rigs interactive_stereo_template.conf targets, but
    VIAME's batch measurement pipes ship 7–724 for others; a rig outside the
    default silently fails to match until someone passes the range option.
  • The exporter's defaults disagree with the service config (--template-size
    25 vs template_size = 13), and export_stereo_mapping.py's self-check
    reports a sub-pixel right_points mismatch at both sizes (0.037 px at 13,
    0.40 px at 25) from epipolar-candidate ties. Pre-existing, and both sizes
    still reproduce the reference correspondence, but the exporter defaults look
    worth reconciling VIAME-side.
  • DINO-augmented method 2 exists on the VIAME side and is not wired here.
  • No cross-camera undo. Deleting or resetting the source annotation leaves
    the warped feature on the other camera.
  • onChange is unused on web — writes rely on the Track notify path to flag
    unsaved changes, same as desktop. Left as an extension point.

@mattdawkins
mattdawkins force-pushed the dev/stereo-onnx-transfer branch 2 times, most recently from 42ca8d8 to a55690f Compare June 22, 2026 17:11
Base automatically changed from dev/add-interactive-seg-and-stereo to main June 27, 2026 16:07
@mattdawkins
mattdawkins force-pushed the dev/stereo-onnx-transfer branch from a55690f to 4096a90 Compare June 30, 2026 01:05
@mattdawkins mattdawkins changed the title Client-side stereo detection transfer via ONNX (first pass) Client-side stereo detection transfer via ONNX [DRAFT] Jul 8, 2026
Runs VIAME's epipolar template-matching model (stereo method 1) in the browser
with onnxruntime-web to warp a detection (box / head-tail keypoints) from one
camera to the other, with no backend. Includes calibration (.npz/.json) and
grayscale helpers, a platform-agnostic transfer composable, and web ViewerLoader
wiring. Core is validated against the VIAME reference in unit tests.
Length measurement still needs the desktop interactive service, but the
cross-camera warp runs client-side, so stop gating it on the runtime.
Load the rig from the dataset when no session file is stashed, triangulate
lengths client-side, warp polygons, keep human-drawn lines, and surface
progress and failures. Ships the model as a served asset.
Interactive stereo now enables the option off desktop, so bind the event
rather than leaving the checkbox inert.
@mattdawkins
mattdawkins force-pushed the dev/stereo-onnx-transfer branch from 4096a90 to a207367 Compare August 10, 2026 15:06
Desktop keeps these in interactive_stereo_template.conf with no UI; match
that instead of inventing a 512 px ceiling.
Threshold 0.5 with no uniqueness test, and re-export the model at
template_size 13, per interactive_stereo_template.conf. The test now runs
against the served model instead of a second copy.
Vue unwraps setup() refs on the component instance, so multiCamList and
aggregateController never had a .value: every stereo op saw zero cameras and
skipped. Surface a missing calibration instead of no-oping too.
Vite's dep cache put the runtime where its sibling .wasm request hit the SPA
HTML fallback, so the model never loaded under npm run serve.
@mattdawkins mattdawkins changed the title Client-side stereo detection transfer via ONNX [DRAFT] Client-side stereo detection transfer via ONNX Aug 10, 2026
@BryonLewis
BryonLewis force-pushed the dev/stereo-onnx-transfer branch from cb19a98 to 4e4f9ab Compare August 12, 2026 18:14
@BryonLewis

Copy link
Copy Markdown
Collaborator

I think this is about ready, just want to do a bit more local testing before I merge it in.

@BryonLewis
BryonLewis self-requested a review August 13, 2026 13:51

@BryonLewis BryonLewis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it has some built in testing data, and some complications I think this overal ready based on my testing.

@BryonLewis
BryonLewis merged commit 90bb8fb into main Aug 13, 2026
6 of 9 checks passed
@BryonLewis
BryonLewis deleted the dev/stereo-onnx-transfer branch August 13, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants