Add option to infer frame index from filename#1742
Open
romleiaj wants to merge 15 commits into
Open
Conversation
Multicam datasets assume cameras are frame-aligned by position, but cameras in a rig can independently drop frames, so index i isn't necessarily the same real-world moment across cameras. Phase I of SEAL feature 5: derive a best-effort timestamp per frame from its filename and expose it on FrameImage/MediaResource, so later phases can build a real aligned timeline instead of assuming index parity. Parsing is provisional (no real MML sample filenames yet) and fully optional -- undefined/None is the expected common case and must not affect existing datasets. No meta.json schema change: the timestamp is computed at read time from the filename, mirroring how url/ filename are already derived, not persisted.
Multicam playback assumed cameras were frame-aligned by position, but cameras in a rig can independently drop frames, so index i wasn't necessarily the same real-world moment across cameras. Phase II of SEAL feature 5: when every camera has a timestamp on every frame, build a global aligned timeline and resolve each camera's own local frame per slot, blanking a camera's pane when it has none at that instant. Falls back byte-identically to today's positional broadcast whenever any camera lacks full timestamp coverage -- zero risk to existing datasets. Also fixes a latent gap this surfaced: continuous playback bypassed the aggregate controller entirely (each camera free-ran its own frame-advance loop), so aligned mode would have applied to scrubbing but silently reverted to raw per-camera advancement during playback. Centralizes a single tick in the aggregate controller when aligned.
…panes
Phase III of SEAL feature 5. Phase II blanked a camera's image when the
global aligned timeline has no frame for it at a slot, but left two
gaps: LayerManager kept drawing annotations for whatever local frame
the camera was on before blanking, since seek(undefined) deliberately
leaves data.frame untouched and LayerManager's watchers are keyed on
frame number -- so stale boxes floated over an otherwise-blank pane.
The pane itself also gave no visual indication it was blanked rather
than just failing to load.
Expose the existing per-camera hasFrame flag on MediaController, and
have LayerManager watch it: every prior updateLayers() call site now
goes through a new refreshLayers(), which disables all annotation/edit
layers instead whenever hasFrame is false. ImageAnnotator/VideoAnnotator
now render a 'eNo frame at this instant' overlay over a blanked pane.
…ndicator Track create/edit/delete and seek operations were keying off the global aligned slot index instead of the selected camera's local frame, writing detections to wrong frame numbers when alignment is active. Adds selectedCameraFrame() in useModeManager, seekCameraFrame() with an inverse aligned index, per-selected-camera time updates in Viewer, and an aligned-gap indicator bar under the frame slider in Controls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
imageData was initialized as { singleCam: [] } and multicam loading only
added per-camera keys on top, so the leftover empty singleCam entry made
canAlign() disqualify every multicam dataset. Rebuild imageData with
exactly the dataset's cameras before loading (replacing the whole object,
since adding keys via bracket assignment is non-reactive under Vue 2.7)
and restrict the alignedTimeline computed to the multicam camera list so
stale keys can never disqualify alignment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
loadData calls changeCamera(defaultCamera) before progress.loaded, so no annotator (and thus no camera controller) exists yet and the aggregate is the empty stub whose getController() throws. That produced an unhandled promise rejection on every multicam load and skipped selectCamera's edit-mode restore and change-camera emit. selectedCameraFrame() and isCreatingNewDetection() shared the same throw exposure whenever a camera's annotator never mounted. Wrap all three lookups with a graceful fallback (skip the time resync / fall back to the aggregate frame). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Viewer.vue's seekToFrame (track-seek from TrackItem/TrackList/ TrackDetailsPanel/AttributesSubsection keyframe navigation, BottomPanel) and ControlsContainer's Timeline chart click-seek were calling the aggregate seek(), which operates in global slot space under an aligned timeline, with frames produced in the selected camera's local frame space. Translate both through seekCameraFrame(selectedCamera, frame) (a passthrough when alignment is inactive), and drive the Timeline playhead from time.frame -- which tracks the selected camera's local frame -- so charts, playhead, and click-seek all agree on frame space. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gap gradient painted slot s over [s/maxFrame, (s+1)/maxFrame], a full cell to the right of the v-slider thumb (which sits at s/maxFrame), and a trailing gap at s === maxFrame painted [100%, 100+%] -- invisible, since the widening branch only ran for sub-0.25% cells. Extract the computation into a pure computeGapGradient() in alignedTimeline.ts that centers each band on the thumb position ([(s-0.5)/maxFrame, (s+0.5)/maxFrame], clamped to [0, 100]) and unit-test it, including edge slots and min-width widening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setAlignedFrameResolver() reset alignedCurrentFrame to 0 without seeking, so a camera that should be blank at slot 0 kept showing its local frame 0 until the first user seek. Perform an aligned seek to slot 0 on resolver install, and re-apply the current slot (flush: 'post') whenever a camera registers while a resolver is active -- annotators self-seek to their own local frame 0 during init, which covers the common ordering where the resolver is installed before the annotators mount. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…reinstalls Sorting triples camera-name-first swept all of one camera's same-time frames into consecutive single-camera slots, so a calibration dataset where every frame shares one collection timestamp produced a timeline of mostly blank panes instead of positional pairing; tie-break equal timestamps by local index first. Also key the Viewer's resolver install on the slot structure so display-URL swaps (percentile stretch) that leave timestamps unchanged no longer reinstall the resolver and visibly re-seek every camera. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add an "Infer frame index from filename" toggle to the multicam import dialog. When enabled, the equal-frame-count validation is skipped so datasets with dropped frames (e.g. KAMERA, which encodes a capture timestamp per filename) can be imported and aligned downstream by their filename timestamps rather than by exact positional index. Also drop the matching server-side equal-frame-count rejection in create_multicam, keeping the frame-count call only for its validation side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the hand-synced Python + TypeScript filename timestamp parsers with one shared TS module (dive-common/frameTimestamp.ts) used by both platforms. The girder server previously parsed each frame's timestamp in get_media and embedded it in the /media response, but nothing server-side ever read it -- it existed only to reach the client. Move parsing to the client: the web-girder API layer now attaches timestamps from filenames after fetching media (both the single-cam /media response and multicam per-camera imageData from /dive_dataset), mirroring how the desktop backend already parses on media enumeration. - Add dive-common/frameTimestamp.ts (parseFrameTimestamp + attachFrameTimestamps) and move its tests out of sharedUtils.spec.ts. - Point the desktop backend importers at the new shared module. - Delete server dive_utils/timestamp_parser.py and its test; drop the now-dead MediaResource.timestamp field and the get_media parsing call; update the multicam dataset tests that asserted server-side timestamps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Frame Alignment Feature (
dev/frame-alignment)What it does
Multi-camera datasets used to require every camera to have the same number of
frames, aligned by position (frame 5 = frame 5). That breaks for capture systems
like KAMERA that drop frames independently, so frame 5 of one camera isn't the
same instant as frame 5 of another.
This branch aligns cameras by when each frame was captured instead of by
position:
..._20240407_130757.206341_ir.tif).timestamp into the same slot.
shows "No frame at this instant" rather than a stale image.
If timestamps aren't available (or it's a single-camera dataset), it falls back to
the old positional behavior — existing datasets are unaffected.
What a user sees
How it works
client/dive-common/frameTimestamp.ts)used by both platforms: desktop parses local media on load, web parses after
fetching from girder. The server does not parse timestamps.
client/dive-common/alignedTimeline.ts(pure, unit-tested) turnsper-camera timestamped frames into global slots, and computes the gap slots.
AlignedFrameResolverthat maps a global slot to each camera's local frame (or "no frame"), keeping all
panes in sync during seek/scrub.
create_multicam) nolonger reject cameras with differing frame counts.
Note
The KAMERA
YYYYMMDD_HHMMSS.ffffffconvention is confirmed against real sample data(
data/test_data) and locked by a test. Cameras must currently share an exacttimestamp to align (tolerance
0); this can be widened later if real captures showsub-second jitter. The epoch-based fallback patterns are likewise best-effort for
other capture systems.