ui: Show camera viewing ray when hovering a pixel in the image viewer by behnamasadi · Pull Request #4584 · colmap/colmap

behnamasadi · 2026-07-21T20:59:32Z

Summary

Adds a rerun-style interaction to the model viewer: hovering a pixel in a
camera's image draws that pixel's back-projected viewing ray in the 3D view.

Double-click a camera frustum → its photo opens in the image window (existing).
Hover a pixel in that photo → a ray shoots from the camera center through the
pixel into the 3D scene, updating live as the cursor moves and clearing when
it leaves the image.
If the hovered pixel is near an observed keypoint, the ray snaps to and
highlights the corresponding 3D point (yellow); otherwise it extends across
the scene along the pixel's direction.

This makes the 2D image and the 3D reconstruction directly cross-referenceable:
point at anything in a photo and see where it lies in 3D.

Demo

How it works

Capture the hovered pixel — the image window installs an event filter on
its QGraphicsView viewport (mouse tracking on) and maps the cursor to image
pixel coordinates.
Unproject to a world ray — Camera::CamFromImg(pixel) gives the normalized
camera-frame direction (x, y, 1), transformed by the image pose
(Inverse(image.CamFromWorld())) into a world-space ray from
image.ProjectionCenter().
Snap to a 3D point — the nearest observed Point2D with a point3D_id
within ~1% of the image size is found; if present, the ray ends exactly at
that 3D point and highlights it.
Draw in 3D — a dedicated LinePainter (ray) and PointPainter (highlight)
render in ModelViewerWidget::paintGL, using the same
model_scale_ * (world + model_origin_) transform as the rest of the scene.

Notes

Files: model_viewer_widget.{h,cc} (SetHoverRay/ClearHoverRay + painters),
image_viewer_widget.{h,cc} (event filter + unprojection). image_viewer_widget.h
moves graphics_scene_/graphics_view_ from private to protected so the
database image viewer can attach the hover filter.
GUI feature — built with -DGUI_ENABLED=ON and verified interactively; not
covered by the headless unit tests. The ray geometry was checked numerically
(computed pixel direction matches the camera→3D-point direction).

Note on the second commit (cherry-picked)

For convenience this branch also carries a second commit,
"ui: Add reprojection-error image colormap for camera frustums", cherry-picked
from #4579, so both GUI viewer features can be built and demoed from a single
branch. That colormap change is reviewed separately in #4579 — happy to drop
the commit here and keep this PR scoped to the hover-ray change alone if you
prefer.

Hovering the mouse over a pixel in the model viewer's image window now draws the back-projected viewing ray from that camera's center through the pixel into the 3D scene (rerun-style). If the hovered pixel is near an observed keypoint, the ray snaps to and highlights the corresponding 3D point; otherwise it extends across the scene along the pixel's direction. The pixel is unprojected with Camera::CamFromImg and transformed by the image pose; the ray is drawn with a dedicated LinePainter/PointPainter in the model viewer. The image viewer installs an event filter on its graphics view to track hover, and clears the ray when the cursor leaves the image.

Adds a "Reprojection error" option to the image colormap dropdown in the model viewer, coloring each camera frustum by its mean reprojection error (accumulated from the 3D point tracks) through the existing JetColormap. Uses an *absolute* color scale (0 px = blue, "Reproj. error for red [px]" = red) rather than a per-model min/max normalization, so colors stay comparable across models: a well-registered model stays blue/cyan and only genuinely high-error cameras turn red, instead of stretching a tiny error spread over the whole spectrum. The upper bound is user-adjustable via a spinbox (with an explanatory tooltip), and applying the colormap logs the model's actual per-image error range so the scale maps to concrete numbers. Default colormap is unchanged ("Uniform color").

behnamasadi · 2026-07-23T16:29:32Z

@ahojnnes what do you say?

ahojnnes

Thanks, it's a cool visualization to have but I am admittedly not very much convinced this adds a lot of practical value. Could you share a bit how you are using this in your workflow? Happy to be convinced if others find it useful.

Apart from these general concerns, it looks like you accidentally entangled a few other changes into this PR.

ahojnnes · 2026-07-27T07:20:38Z

      image_name_colors_;
 };

+// Color images according to their mean reprojection error, on an *absolute*


This looks like it's an unrelated change to the camera viewing ray functionality, which got accidentally included from one of your other PRs?

ahojnnes · 2026-07-27T07:29:45Z

+  const auto image_it = model_viewer_widget_->images.find(image_id_);
+  if (image_it == model_viewer_widget_->images.end()) {
+    model_viewer_widget_->ClearHoverRay();
+    return;
+  }
+  const Image& image = image_it->second;
+  const auto camera_it = model_viewer_widget_->cameras.find(image.CameraId());
+  if (camera_it == model_viewer_widget_->cameras.end()) {
+    model_viewer_widget_->ClearHoverRay();
+    return;
+  }
+  const Camera& camera = camera_it->second;


This could be extracted once during the ShowImage() call.

ahojnnes · 2026-07-27T07:30:44Z

+  Eigen::Vector3d dir_cam(cam_point->x(), cam_point->y(), 1.0);
+  dir_cam.normalize();


Suggested change

Eigen::Vector3d dir_cam(cam_point->x(), cam_point->y(), 1.0);

dir_cam.normalize();

Eigen::Vector3d cam_ray = cam_point->homogeneous().normalized()

ahojnnes · 2026-07-27T07:31:28Z

+  dir_cam.normalize();
+  const Rigid3d world_from_cam = Inverse(image.CamFromWorld());
+  const Eigen::Vector3d origin = image.ProjectionCenter();
+  const Eigen::Vector3d dir_world = world_from_cam.rotation() * dir_cam;


Suggested change

const Eigen::Vector3d dir_world = world_from_cam.rotation() * dir_cam;

const Eigen::Vector3d world_ray = image.CamFromWorld().rotation.inverse() * dir_cam;

ahojnnes · 2026-07-27T07:32:42Z

+  for (const Point2D& point2D : image.Points2D()) {
+    if (!point2D.HasPoint3D()) {
+      continue;
+    }
+    const double sq_dist = (point2D.xy - pixel).squaredNorm();
+    if (sq_dist >= best_sq_dist) {
+      continue;
+    }
+    const auto p3d_it = model_viewer_widget_->points3D.find(point2D.point3D_id);
+    if (p3d_it == model_viewer_widget_->points3D.end()) {
+      continue;
+    }
+    best_sq_dist = sq_dist;
+    point3D = p3d_it->second.xyz;
+    has_point3D = true;
+  }


Not sure how much value this adds. I would be fine with just showing the ray.

ahojnnes · 2026-07-27T07:34:26Z

+    // Extend across the scene: uniform scale preserves the world direction.
+    const float ray_length = std::max(2.0f * render_origin.norm(), 1.0f);
+    render_end = render_origin + direction.cast<float>() * ray_length;


If the camera happens to be at the origin, then the ray will be very short. How about extracting the max distance to any 3d point in the image and using that? It could be extracted once during ShowImage().

behnamasadi · 2026-08-04T07:50:57Z

@ahojnnes Thanks for your reviews. I have rebased this on top of my other feature: #4579
That feature is about setting the camera frustums based on reprojection error so the user gets an intuitive feeling about the reliability of each camera pose. Regarding your question about how I use this feature, it is basically a visual debugging tool to see if 2D/3D correspondence is correct and they land exactly on the 3D scene, Rerun heavily uses it and I get the idea from there. I mean, if you feel like adding this PR, I can make the changes you mentioned. Regards.

behnamasadi added 2 commits July 23, 2026 18:26

behnamasadi force-pushed the feat/hover-pixel-ray branch from 844f616 to f43845f Compare July 23, 2026 16:27

ahojnnes reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

ui: Show camera viewing ray when hovering a pixel in the image viewer - #4584

ui: Show camera viewing ray when hovering a pixel in the image viewer#4584
behnamasadi wants to merge 2 commits into
colmap:mainfrom
behnamasadi:feat/hover-pixel-ray

behnamasadi commented Jul 21, 2026

Uh oh!

behnamasadi commented Jul 23, 2026

Uh oh!

ahojnnes left a comment

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

ahojnnes Jul 27, 2026

Uh oh!

behnamasadi commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		Eigen::Vector3d dir_cam(cam_point->x(), cam_point->y(), 1.0);
		dir_cam.normalize();

	Eigen::Vector3d dir_cam(cam_point->x(), cam_point->y(), 1.0);
	dir_cam.normalize();
	Eigen::Vector3d cam_ray = cam_point->homogeneous().normalized()

	const Eigen::Vector3d dir_world = world_from_cam.rotation() * dir_cam;
	const Eigen::Vector3d world_ray = image.CamFromWorld().rotation.inverse() * dir_cam;

Uh oh!

Conversation

behnamasadi commented Jul 21, 2026

Summary

Demo

How it works

Notes

Note on the second commit (cherry-picked)

Uh oh!

behnamasadi commented Jul 23, 2026

Uh oh!

ahojnnes left a comment

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

ahojnnes Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

behnamasadi commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants