Add keypoint gui#1744
Open
romleiaj wants to merge 80 commits into
Open
Conversation
Let users pick corresponding points between camera pairs inside DIVE, fit homographies for alignment preview, and persist results for VIAME workflows. This replaces the external keypointgui loop for the common EO/IR registration path. - Add CameraCalibrationStore with blue→red pairing, homography fit, and auto-fit when enabling the overlay or saving calibration - Add CalibrationTools sidebar: camera pair select, correspondence table, points.txt export, overlay preview with direction and opacity - Add CalibrationKeypointLayer for on-image markers and warped overlay - Add homography DLT solver and unit tests - Persist calibration to standalone calibration.json (pairs, points, leftToRight/rightToLeft matrices) and rehydrate on dataset load - Register Camera Calibration panel for multicam datasets in Viewer
Picking was always on native/unwarped viewers, so users couldn't refine
correspondences while visually verifying alignment. DIVE also only ever
fit a full homography, when near-rigid EO/IR rigs often need fewer
points and a more stable low-DOF fit.
- Add client/src/transform.ts: translation/rigid/similarity/affine
estimators alongside the existing homography DLT solver, all sharing
the same Matrix3 primitives so warping/inverse-mapping code doesn't
need to special-case the transform type
- Replace the passive overlay toggle with an alignment mode (Original /
A→B / B→A) that warps the destination camera's pane; add a Picking
Picking correspondences at default zoom is imprecise for cross-modality (EO/IR) rigs since the two panes differ in resolution and scale. When a pair has a fitted transform, panning or zooming either camera now recenters the other on the corresponding point via the homography, and a right-click on either pane snaps both cameras to that location. Also adds a live cursor coordinate readout in the calibration panel.
…bration Closes the remaining keypointgui point-workflow gaps: Clear Last mirrors keypointgui's undo-pending/undo-last-pair semantics, Load points.txt parses the same four-column format with a replace/merge prompt, and the new homography.txt export matches keypointgui's Save Homography output.
…rors Ghost overlay resolution only matched image-sequence quads, silently disabling aligned picking's visual aid for video datasets; now resolves video quads too. Fitting a transform from enough points that are degenerate (e.g. collinear) threw uncaught out of the geojs click handler; failures are now caught and surfaced via a new fitError state instead of crashing the picking flow.
…host Three review findings on the in-app calibration point-picking tool: - Ghost overlay rendered as a parallelogram for projective fits: geojs' canvas quad renderer is affine-only (it builds the draw transform from three corners), so a homography with nonzero perspective terms was drawn as an affine approximation while pickPoint() inverse-mapped clicks through the exact matrix, offsetting ghost-targeted picks from what the user visually aligned. The ghost is now subdivided into an 8x8 grid of sub-quads (single quad when the perspective terms are negligible) whose corners are each mapped through the exact homography and whose textures are cropped to the matching source region, so the rendered warp matches the projective mapping to a couple of pixels even for extreme perspective. Click attribution is unchanged (already exact). - "Save calibration" could never persist a cleared state: the button was gated on the active pair having at least one correspondence, so Clear All / per-row deletes left stale calibration.json / Girder meta saved forever, and pairs other than the active one couldn't be saved either. Save is now always enabled; the >=1-correspondence gating remains only on the points/homography exports that genuinely need it. - Ghost frame went stale on image sequences: ImageAnnotator swaps its quad's <img> asynchronously after the new frame loads, with nothing re-triggering the ghost. The layer now runs a bounded (~1s) requestAnimationFrame loop that re-renders when the source pane's image element changes, and LayerManager additionally watches the ghost source camera's frame ref (video sources are unaffected and skip the polling). Also generalizes solveLinearSystem's degenerate-configuration error message, which is shared by the affine estimator, and adds unit tests for the warp subdivision math. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Linked pan/zoom only recentered the other camera, so zooming left its zoom level untouched; now the target's zoom is set from the source's units-per-pixel scaled by the homography's local scale at the view center (localLinkedScale, extracted to homography.ts and unit-tested). Panel layout: points.txt/homography export-import moves into a collapsed expansion panel and Save calibration moves to the bottom. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
While picking is enabled, a mousedown on an existing marker (or the pending point) within 10 display pixels grabs it instead of panning: the capture-phase listener runs before geojs' interactor, so the drag neither pans the map nor places a new point on release. Coordinates update live through the store (with the usual refit while alignment is active), and hovering a draggable marker shows a grab cursor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BaseLayer's constructor calls initialize() -- which assigns mapNode -- before the subclass field initializers run, so `mapNode = null` erased the assignment and every drag hit-test bailed on a null node. Declare the drag fields without initializers, matching textFeature/quadFeature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Put the Pick points toggle directly under the intro text, above the Camera A / Camera B selectors, so picking is the first thing chosen. Render each calibration marker as a hollow ring (blue pending / red placed) with a small bright center dot (cyan / yellow) at the exact pixel, instead of a solid disc that covered the very point being picked. The dark ring plus bright dot stay legible on both light and dark imagery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the "points.txt / homography files" expansion panel (Export / Load points.txt and Export A->B / B->A homography.txt), the replace/merge load dialog, and their handlers, along with the store's toPointsText, loadPointsFromText, and toHomographyText helpers and their tests. Calibration is picked in-app and persisted with the dataset; the keypointgui-style text IO is no longer needed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y while picking Grabbing or clicking a placed marker (or its panel-table row) selects its correspondence, highlighted in both cameras' panes (a larger white ring with an orange center dot) and the table row. Delete/Backspace removes the selected correspondence -- both cameras' points at once. Clicking empty space clears the selection and places a point as before; selection is authoring state and never persists. Point markers are also now authoring-only: they render just while Pick points is on (the correspondences stay in the store either way). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wrap the correspondences list in a v-expansion-panel (matching the panel's other collapsible sections) so it stays out of the way until expanded, with the pair count in the header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add loadCalibrationText() to the store: parse a { version, pairs[] }
calibration file into correspondences, homographies, and transform-type
choices, deriving a missing matrix direction by inversion and validating
strictly (throws on bad JSON, a missing pairs list, or a singular matrix
without touching current state). A "Load calibration" button in the panel
reads a .json via a file input, confirms before replacing existing
calibration, and warns about cameras not present in the dataset. Works on
both web and desktop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the Camera Calibration panel opens, collapse the left type-filter sidebar and the bottom detections graph to give the picking view more room. Soft default: the normal sidebar/timeline toggles still work while calibrating, and the prior layout is restored when the panel closes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pure helpers (alignedView.ts) resolve a native->reference matrix for every loaded camera from (a) per-camera meta.multiCam transform matrices, read defensively since they are added by a parallel branch, preferred when present and assumed to map camera N -> reference per plan Q3, or (b) calibration-tool pair homographies composed breadth-first through the pair graph. Resolution is all-or-none so a partially aligned display can never render. AlignedViewStore holds the reactive toggle/availability state and composes camera-to-camera mappings through the reference space; it is provided alongside the existing calibration store. Reference camera = first camera in display order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AlignedImageLayer renders a camera's own frame warped into the reference space as an N x N grid of geojs canvas quads (quad-corner warp, decision D1, reusing warpGridSize/subdivideWarpQuads) and hides the annotator's native image quad while active, restoring it byte-identically when cleared. It is created before the annotation layers so it z-orders beneath them, mirrors the native layer's image-enhancement CSS filter, and polls briefly after each trigger to catch asynchronous <img> swaps (same pattern as the calibration ghost). Annotation geometry stays stored in native space (decision D3): BaseLayer gains a draw-time displayTransform applied only in the layers' geojs position accessors, so rectangles, polygons, lines, points, text, and attribute layers land exactly on the warped imagery (and geojs hit-testing/ hover stays consistent since it searches those same positions). LayerManager applies the per-camera matrix to those layers, forces editing off while the aligned view is on (the edit layer draws in native space), hides track tails for warped cameras (they read multi-frame geometry outside FrameDataTrack), and maps locked-camera centering through the same transform. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
useAlignedNavigation attaches pan/zoom handlers to every loaded camera while the aligned view is active: navigation on any camera recenters all others on the same world location via the stored transforms composed through the reference camera (SEAL-TK feature 3). Follows useCalibrationNavigation's re-entrancy-guard pattern, and stands down while either the raw-screen-delta 'sync cameras' toggle or the calibration pair link is on so multiple handlers never fight over the same pan event. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Viewer resolves the per-camera transform set whenever the loaded meta or the calibration store's fitted homographies change: reference = first camera in display order (multiCamList[0]); a per-camera meta.multiCam.cameras[<name>].transform.matrix (row-major 3x3, assumed to map camera N -> reference per plan Q3, read defensively pending the parallel import branch) is preferred over calibration fits. The compact toolbar toggle appears next to the camera selector only when every non-reference camera resolves; while active, a chip flags that displays are warped and editing is disabled. The warp is suspended during calibration point picking so picks are never taken against a warped display (risk K2), the toggle resets on dataset load (no persistence this phase), and all-camera linked pan/zoom is engaged via useAlignedNavigation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The AlignedImageLayer polls its camera's image element briefly after each trigger; during a dataset reload the media controllers are cleared and getController throws. Return null from the lookup closure instead so an in-flight poll tick degrades to 'no image yet' rather than an uncaught error inside requestAnimationFrame. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same gap as the calibration pair link: the handler only recentered other cameras, leaving their zoom untouched. Reuse localLinkedScale to match each target camera's visible extent through its transform. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore Make the calibration store the single source of truth for alignment: the Align button now composes transforms only from the store's pair homographies, so what the panel shows and saves is exactly what Align applies (previously an imported meta.multiCam transform silently overrode in-app calibration). The panel gets Load/Save calibration buttons at the top. Save persists all pairs to the dataset and downloads a portable calibration .json (cameras, transform types, points, homographies per pair); Load replaces the calibration from such a file. Legacy ITK .h5 transforms load into the selected pair as B->A when the platform provides parseItkTransformBuffer (desktop; declared here, implemented with the .h5 reader plumbing). File-loaded homographies carry no points, so the store tracks provenance: a 'loaded' transform survives refit checks that clear under-pointed pairs, drives alignment/ghost/export immediately, and is replaced once enough points are picked and fitted. Clear-pair now also removes a loaded transform. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Load stays at the top; Save moves back to the bottom. Drop the load/save and link-pan/zoom helper captions, and collapse the correspondence list into a default-closed expansion panel (fit errors stay visible outside it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel's save/load format and the desktop's persisted calibration.json were two different JSON shapes; adopt the on-disk pairs shape (left/right, points rows, leftToRight/rightToLeft, transformType) for the panel too, so a panel-saved file, the project's calibration.json, and an import-time seed are all the same interchangeable format. Loading accepts files without the self-identifying "type" key and derives a missing direction by inversion. Also drop the aligned view's unused meta.multiCam transform support (extractMetaCameraTransforms / metaTransforms): external transforms now enter the calibration store itself, so resolveToReferenceTransforms takes the store's homographies directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The calibration JSON is now the single interchange format for the calibration tool: it carries the picked points and both fitted matrix directions, so the keypointgui-era points.txt import/export, the homography.txt export, and the panel-side legacy ITK .h5 hook (parseItkTransformBuffer) have no remaining consumers. Remove them from the store, the panel, and the API spec. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An external model step (e.g. the KAMERA/COLMAP rig solver) can stamp the calibration JSON with a free-form "source" object (model version, swathe, generation time). DIVE never interprets it: the store keeps it through in-app refits, the panel shows it and includes it in saves, the desktop backend persists it in the standalone calibration.json, and the web data model accepts it -- so a refined calibration returning to the producer still says which model version it was refined against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Producer-stamped calibration files carry matrix-only pairs, so a point-backed fit while a source stamp is loaded means a human refined that pair. Derive the flag from existing provenance (rather than storing it) so it survives save/reload naturally, and surface it in the panel as a prompt to save the refinement for the producer's re-solve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fromCalibrationPairs (behind both import-time transform seeding and the project calibration.json loader) silently dropped pairs carrying only one fitted direction -- exactly the matrix-only shape an external producer emits -- so such files never reached the aligned view through import while the panel's Load button (which inverts the missing direction) accepted them. Mirror the panel loader: derive the missing direction by inversion, and keep a pair's points even when its matrix is singular. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Save calibration now only persists to the dataset (the project calibration.json restored on reopen); a separate Export button downloads the portable copy for sharing or producer round trips. The raw screen-delta "sync cameras" toggle is hidden once every camera has a calibration transform: the Align button's transform-aware link is the single control from then on. Raw sync also switches itself off at that point, since the aligned link stands down while it is enabled and its toggle is no longer reachable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Change the user-facing context-panel title (SidebarContext renders each entry's `description`) from "Camera Calibration" to "Manual Alignment" in the context map, the multicam register/unregister calls, and the panel component. The internal component name stays 'CameraCalibration' so the context key and all lookups are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gate the pane-hiding on the Manual Alignment panel being open (calibrationActive) rather than on Pick points being toggled, so on a 3+ camera dataset opening the panel immediately narrows the view to just the Camera A / Camera B pair. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the "Pick points" toggle; picking is enabled when the panel mounts and disabled when it unmounts (closes), mirroring the existing teardown. Everything that keyed off pickingEnabled (markers, cursor readout, aligned- view suspension) now follows the panel's open/closed state automatically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eadiness countdown - Default the transform type to 'similarity' (fits from 2 point pairs) instead of 'homography'; update the tests that assumed the old default. - Replace the transform "needs N points" caption with a live fit-readiness countdown: a grey clock + "N more point pair(s) needed" until enough are picked, then a green check + "Ready to fit". - Rename the "Alignment (in-app aligned picking)" section to "Overlay Warp" and "Ghost opacity" to "Warp Opacity". - Turn the alignment-mode dropdown into a full-width segmented button toggle with concise directional labels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `dirty` computed to the store (a JSON snapshot of the saved-relevant state -- points, homographies, transform types, provenance -- compared against the baseline captured at load and after each save via markSaved()). The Save button now lights up green while there are unsaved changes and greys out (label "Calibration saved") once saved. hydrate() sets the baseline on load; save() calls markSaved() on success. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Locked (committed) points are blue; unlocked points -- the pending point being placed, or a selected one -- are yellow, keeping the hollow-ring + center-dot design. Del/Backspace deletes the unlocked point: the selected correspondence if one is selected, otherwise the pending point mid-placement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a linkedNav flag to the store (default on) that gates the calibration pan/zoom link in useCalibrationNavigation, exposed as a "Fit pan/zoom" checkbox below the transform type. The checkbox is disabled until the active pair has the minimum points to fit its transform, then its indicator colors by robustness: yellow once the transform can be fit (its own minimum, e.g. 2 for Similarity) and green at 12+ point pairs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Instead of showing/hiding the Warp Opacity slider and "Picking for" toggle based on the Overlay Warp mode, keep them always visible and disable (grey out) both -- and their labels -- when the mode is Original, so the panel layout no longer jumps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the "Picking for" (This camera / Ghost overlay) toggle and the pickTarget state and inverse-homography ghost attribution behind it -- it was confusing. New points are now recorded only in the unwarped mode (pickPoint is a no-op while an overlay warp is active), and that mode is renamed from "Original" to "Picking" in the Overlay Warp toggle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Show, at the top of the panel, whether a calibration exists and which cameras satisfy the Align button. A summary line reads "No calibration yet" / "N of M cameras aligned" / "Align ready", and a chip per camera marks the reference (identity), resolved cameras (green, has a fitted path to the reference), and unresolved ones (amber, "needs calibration"). Reuses alignedView.unresolvedCameras so the "all green" state matches exactly when Align becomes available. Shown for any 2+ camera rig. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the static "Align View (requires calibration)" tag with the same per-camera warning the Manual Alignment panel shows: when the aligned view isn't available, the tooltip lists the cameras with no fitted path to the reference, e.g. "Align View — UV · needs calibration". Reads plain "Align View" once every camera resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reword the panel status header and the Align button tooltip to a compact "N/M cameras calibrated" count, with a yellow warning triangle while any camera is unresolved and a green check once all are calibrated. Drops the wordier "N of M cameras aligned" / per-camera-name tooltip phrasing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-camera alignment chips are display-only, but rendered like interactive buttons that highlighted on hover. Add pointer-events: none and disable the ripple so they read as a plain status bar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The unresolved camera chips already read as warnings (amber + alert icon), so the trailing "· needs calibration" is redundant. Chips now show just the camera name (reference still tagged "· reference"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A pair fitted at the default (dropdown untouched) stored no explicit transformType, so the store/UI defaulted it to 'similarity' while the desktop calibration.json layer and the server doc defaulted it to 'homography'. The same picked points then refit to a different model on desktop vs web, and a desktop save/reload silently converted a similarity fit into a homography on the next point edit. Introduce a single DEFAULT_TRANSFORM_TYPE constant and resolve every absent transform type through it (store, panel, and both desktop persistence paths); fix the server comment to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switching between multicam datasets with the Manual Alignment panel open re-runs loadDataset -> calibration.hydrate(), which clears picking and the active pair. The panel establishes those only at mount and is not remounted across the switch, so it was left visibly open but inert: dead markers, selectors still naming the previous dataset's cameras. Watch the camera set and re-default the selectors (or re-establish the pair when the names are unchanged) and re-enable picking whenever it changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
update_metadata serializes with exclude_none, which drops an explicit null, so a previously-stored cameraCalibrationSource could never be removed on web: clearing or hand-refining a calibration left the old producer-provenance stamp attached, mislabeling it. Desktop already handles this. Pop it (and timeFilters) by hand when the client sends null. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The calibration ghost and aligned-warp requestAnimationFrame loops call getCameraImage, whose getController(camera) throws once a dataset reload has cleared the controllers -- an uncaught exception in the rAF callback. Only one of the two callers wrapped it. Swallow it at the source so both loops degrade to "no image" and self-expire. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
useCalibrationNavigation and useAlignedNavigation duplicated the re-entrancy guard, geojs pan/zoom listener bookkeeping, and the zoom-baseline conversion; they differ only in how a source pane's view maps onto a target. Move the common parts into useLinkedViewers so each composable keeps just its mapping and lifecycle. Net -73 lines; behavior unchanged (aligned-nav spec still green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The aligned-view warp and the calibration ghost render a projective warp as a grid of canvas quads. The canvas renderer antialiases each quad's border against the transparent background, so abutting cells meet as two half-transparent edges and show as dark grid lines across the warped image (most visible on low-contrast IR frames). Give subdivideWarpQuads an optional overlap: each cell's crop and warped corners are expanded by that many source pixels (clamped to the image), so neighbors paint over each other's seam with pixel-identical content -- corners still map through the same exact homography. Both renderers pass 2px. Overlap requires opaque drawing, so the ghost's transparency moves from the per-quad style to its (dedicated) feature layer, where it composites once and cannot double-blend in the overlap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
onResize resets every pane to its own native bounds, and resetZoom emits GeoJS pan/zoom events synchronously. The aligned-view (and calibration) pan/zoom links are attached to all panes, so a non-reference pane's native-space reset was broadcast into the shared reference space -- dragging the reference to that pane's native center and stranding the warped panes on an empty corner (a black square on IR/UV). The link's re-entrancy guard only covers events emitted inside a link call, not the externally-triggered resetZoom. Add a `resizing` flag to AggregateMediaController that onResize raises around its resetZoom loop; the linked-viewer navigation ignores pan/zoom events while it is set. The resizeTrigger bump that follows re-snaps every pane from a clean reference view. Also trigger a resize when the context sidebar toggles: the only ResizeObserver watches the controls bar, which is position:absolute in side layout and misses the panes resizing underneath it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Editing was blocked globally while the aligned view was active; gate it per camera on the display transform instead, so the reference camera (which renders unwarped, native == display space) keeps full create and edit behavior while warped cameras stay read-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The calibration pair link only took effect on the first pan/zoom event after toggling. Invoke the link once from camA whenever setup attaches it (toggle-on, picking start, refit, post-resize), matching the aligned-view snap-on-activation behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mapBounds maps an axis-aligned RectBounds through a camera-to-camera homography (AABB of the mapped corners, since a projective transform turns a rectangle into a quad). mapRotatedBounds does the same for a rotated rect, deriving the target rotation from the mapped top-edge direction. mapGeoJSONFeatures deep-copies per-frame track geometry (Point / LineString / Polygon incl. holes) with every coordinate mapped. Groundwork for mirroring tracks drawn under Align View to all cameras. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
While the Align View is active (rig fully calibrated + toggle on), any geometry write on the selected camera now re-projects onto every other camera through the calibrated camera-to-camera homographies, creating the same-id track on cameras that don't have it yet. The camera that received the edit is always the source, so later edits keep the rig in sync (continuous mirror); stored geometry stays native per camera (decision D3) -- each camera receives coordinates mapped into its own image space. Covered write paths: rectangle bounds (incl. rotation, remapped via mapRotatedBounds), recipe GeoJSON (polygons, head/tail lines), direct setTrackFeature, point/annotation removal, and the segmentation predict/confirm/multi-frame/reset flows. Mirrored geometry the source no longer has is dropped (setFeature only upserts by key/type), and a source keyframe deletion removes the mirrored keyframe -- plus the mirrored track when that leaves it empty. Per-camera target frames come from each camera's own controller frame, so the mirror lands on the right local frame under an aligned timeline; cameras with no frame at the current slot are skipped. Off-slot writes (multi-frame segmentation) fall back to the same local frame number. With the aligned view off or suspended everything behaves exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Editing was limited to the unwarped reference camera because the edit layer operates in geojs map coordinates (warped reference space under Align View) while track geometry is stored native per camera. Make LayerManager the transform boundary instead: feed the edit layer display-space copies of the selected track's geometry, and map its draws/edits back to native through the inverted homography before committing, so the edit handles land on warped imagery and storage stays native (decision D3). Segmentation prompt points and the click-outside-edit hit-test are mapped across the same boundary, and the Align View tooltip no longer claims reference-only editing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Based on your feedback that the track drawing wasn't working, and a few bugs I found myself, I made the following changes:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual Alignment & Aligned View (multicam calibration)
A high-level overview of this branch (
dev/keypoint-gui, offdev/sealtk) for reviewers.What it does
Adds an in-app tool for aligning the cameras of a multicam dataset (e.g. an EO/RGB
camera and an IR camera looking at the same scene). Ports the features of https://github.com/Kitware/keypointgui to be within DIVE. The user clicks matching points
across two camera panes; DIVE fits a transform that maps one camera's pixels onto the
other. Once every camera is calibrated to a shared reference, an Aligned View can
warp all cameras into one common frame and link pan/zoom across them during normal
annotation.
Two capabilities:
Why it exists
Cross-modality rigs (EO vs IR) have different resolutions and slight offsets, so the
same animal appears at different pixel locations in each camera. Reviewers need to
confirm a detection in one camera against the other. Previously this required an
external tool (
keypointgui) and file shuffling; this brings the wholepick → fit → align workflow inside DIVE, persisted with the dataset.
GUI elements added
dropdown (Translation → Homography), a rig alignment-status block
("N/M cameras calibrated"), a correspondences table, and Save / Load / Export
buttons. Opening it enters point-picking mode.
yellow = pending/selected), click-to-place, drag-to-refine, click-select + delete,
and an optional ghost overlay of the other camera warped through the current fit.
cameras are calibrated; warps every pane into the reference space and links pan/zoom.
How it's implemented (high level)
homography.ts,transform.ts,alignedView.ts) —self-contained, unit-tested estimators (translation/rigid/similarity/affine/homography
via normalized DLT + Umeyama) and the logic that composes per-pair transforms into a
per-camera "native → reference" mapping. No framework dependencies.
CameraCalibrationStore.ts,AlignedViewStore.ts) — the singlesource of truth for picked points, fitted transforms, and view state. Created in
Viewer.vueand shared with the panel and the render layers via provide/inject.CalibrationKeypointLayer.ts,AlignedImageLayer.ts) — draw themarkers and the warped imagery on GeoJS. Image warps are rendered as a subdivided quad
grid so a projective transform looks correct through GeoJS's affine canvas renderer.
BaseLayer.ts+ one-line change to each annotation layer) —annotation geometry is always stored in native image coordinates and only mapped
into the aligned frame at draw time. The off state is byte-identical to before.
useCalibrationNavigation.ts,useAlignedNavigation.ts, andthe shared
useLinkedViewers.ts) — synchronizes pan/zoom across panes.(
cameraHomographies,cameraCorrespondences,cameraTransformTypes,cameraCalibrationSourceadded toMetadataMutable/apispec.ts/ servermodels.py). Desktop additionally mirrors it to a standalonecalibration.json.Files it touches (~47 files, ~6k lines)
client/src/homography.ts,transform.ts,alignedView.tsclient/src/CameraCalibrationStore.ts,AlignedViewStore.tsclient/src/layers/AnnotationLayers/CalibrationKeypointLayer.ts,layers/AlignedImageLayer.ts,layers/BaseLayer.ts(+ 1-line edits to each annotation layer)client/src/components/annotators/useCalibrationNavigation.ts,useAlignedNavigation.ts,useLinkedViewers.tsclient/dive-common/components/CameraCalibration/CalibrationTools.vue,components/Viewer.vue,LayerManager.vue,controls/Controls.vueclient/dive-common/components/ImportMultiCamDialog/*client/src/provides.ts,dive-common/apispec.ts,dive-common/store/context.tsclient/platform/desktop/backend/native/common.ts,multiCamImport.tsserver/dive_utils/models.py,server/dive_server/crud_dataset.pyPlatform support
Persistence uses the shared metadata API; file load/save use standard browser APIs.
during multicam import): desktop only. On web you import first, then load the
calibration
.jsonfrom the panel — same end state, different entry point.