Skip to content

Add auto-alignment for HDR bracket merging#21420

Open
da-phil wants to merge 3 commits into
darktable-org:masterfrom
da-phil:add_hdr_merge_auto_align
Open

Add auto-alignment for HDR bracket merging#21420
da-phil wants to merge 3 commits into
darktable-org:masterfrom
da-phil:add_hdr_merge_auto_align

Conversation

@da-phil

@da-phil da-phil commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Implements automatic image registration in the lighttable merge HDR path so handheld/tripod brackets can be merged without ghosting, via OpenCV.

Auto-alignment processing steps:

  • Each bracket frame is collapsed from its raw CFA mosaic to a CFA-free luma proxy (~0.625× resolution) by averaging stride-1 2×2 windows, then percentile-normalised and gamma-encoded to an 8-bit image.
  • SIFT keypoints are detected on the 8-bit proxy, filtered to a 6 px scale floor, then spatially balanced to 5000 per frame across a 6×6 grid so neither frame dominates the matcher.
  • Descriptors are matched with FLANN kNN, filtered by Lowe ratio (0.75) and mutual-NN consistency, then spatially subsampled to 1800 matches before RANSAC to avoid bias toward dense bright regions.
  • RANSAC estimates an 8-DOF homography; if it yields fewer than 50 inliers or below 40 % inlier ratio, estimateAffine2D is tried as a fallback. If inliers cluster into ≤ 2 grid cells the model is further degraded to a pure translation to avoid wild extrapolation.
  • The result is sanity-checked (translation < 30 % of the image diagonal, scale within 0.5–2×); frames that fail are accumulated unwarped, so the output is never worse than today's alignment-free merge.
  • The homography is rescaled from proxy to full-sensor coordinates and applied with a CFA-aware resampler that only reads same-color photosites, preserving the Bayer/X-Trans mosaic phase so the warped frame drops directly into the existing weighted accumulator.

A more detailed design overview can be found here:
https://github.com/da-phil/darktable/blob/a5a803c5447894faa208a9507a5c82e5fb38e260/dev-doc/HDR_Alignment_Design.md

Key changes

  • src/common/hdr_alignment.{h,c}: pure-C layer — CFA luma proxy build, percentile normalisation, proxy→full-res homography rescale, CFA-aware same-color warp (Bayer exact, X-Trans approx), reliability/sanity gate, per-frame orchestration
  • src/common/hdr_alignment_cv.cc: OpenCV C++ backend — SIFT detect, FLANN kNN + Lowe ratio + mutual-NN, spatial keypoint balancing (5000-each), findHomography RANSAC, affine fallback, cluster- degradation to translation-only, spatial match subsampling
  • control_jobs.c: wire alignment state into dt_control_merge_hdr_t; set_reference on first frame, align_frame on each subsequent frame
  • Optional USE_OPENCV CMake flag (default ON); without OpenCV the feature compiles to a no-op and the merge is unchanged
  • Seven runtime prefs in darktableconfig.xml.in (auto_align, auto_reference, proxy_scale, feature_gamma, clahe_clip, sift_keypoints, debug_images)
  • DT_DEBUG_HDR_MERGE channel (-d hdr_merge), debug image dump
  • cmocka unit tests: synthetic homography round-trip + probe ranking
  • CI/CD: add OpenCV per-module packages to Dockerfile, Brewfile, ci.yml, nightly.yml; README + Windows packaging docs updated

App images can be found here:
https://github.com/da-phil/darktable/actions/runs/28332610568#artifacts

Disclaimer: this work has been co-created with Claude.

Fixes: #17326

@wpferguson

wpferguson commented Jun 25, 2026

Copy link
Copy Markdown
Member

Are the settings in preferences pretty much set and forget (don't need to be messed with) or useful to be able to adjust without having to go to preferences?

I created an HDR, then adjusted the settings and tried to create another HDR using the same images as the previous. The second HDR overwrote the first HDR. Probably should use the on conflict naming.

EDIT: In my very quick test, the images looked fine. No alignment errors were observed.

@da-phil

da-phil commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Are the settings in preferences pretty much set and forget (don't need to be messed with) or useful to be able to adjust without having to go to preferences?

I believe some of them might be removed after more testing, or we remove them from the GUI and keep them as expert settings in dartablerc only.

The second HDR overwrote the first HDR. Probably should use the on conflict naming.

This behaviour exists since day 1 of the HDR merge feature and is not scope of this PR. I plan to improving the HDR merge feature after this PR, as I discovered a lot of bad behaviours with the actual HDR merge functionality. I want to make it closer to the hdrmerge algorithm, which - by the way - was also recently adopted in vkdt. It prevents ghosting much better as the naive merge algorithm in darktable.
Then we can also talk about other improvements of the actual HDR merge behaviour, such as how do we deal with existing merged DNG files.

EDIT: In my very quick test, the images looked fine. No alignment errors were observed.

Perfect! Can you try some really challenging handheld exposure brackets?

@ralfbrown ralfbrown added feature: new new features to add scope: image processing correcting pixels labels Jun 26, 2026
@wpferguson

Copy link
Copy Markdown
Member

This behaviour exists since day 1 of the HDR merge feature

At that time there was only 1 possible output. This PR introduces settings, making multiple outputs possible. Since settings are available, users will try and tune the HDR output to find the best settings. You could probably steal the conflict code from the exporter or just do a simple test if the file exists and then append _xx to the filename where xx is a 2 digit sequence number.

Can you try some really challenging handheld exposure brackets?

Everything I have can be aligned by simple x y alignments.

Implements automatic image registration in the lighttable merge HDR
path so handheld/tripod brackets can be merged without ghosting, via OpenCV.

Key pieces:
- src/common/hdr_alignment.{h,c}: pure-C layer — CFA luma proxy
  build, percentile normalisation, proxy→full-res homography rescale,
  CFA-aware same-color warp (Bayer exact, X-Trans approx),
  reliability/sanity gate, per-frame orchestration
- src/common/hdr_alignment_cv.cc: OpenCV C++ backend — SIFT detect,
  FLANN kNN + Lowe ratio + mutual-NN, spatial keypoint balancing
  (5000-each), findHomography RANSAC, affine fallback, cluster-
  degradation to translation-only, spatial match subsampling
- control_jobs.c: wire alignment state into dt_control_merge_hdr_t;
  set_reference on first frame, align_frame on each subsequent frame
- Optional USE_OPENCV CMake flag (default ON); without OpenCV the
  feature compiles to a no-op and the merge is unchanged
- Seven runtime prefs in darktableconfig.xml.in (auto_align,
  auto_reference, proxy_scale, feature_gamma, clahe_clip,
  sift_keypoints, debug_images)
- DT_DEBUG_HDR_MERGE channel (-d hdr_merge), debug image dump
- cmocka unit tests: synthetic homography round-trip + probe ranking
- CI/CD: add OpenCV per-module packages to Dockerfile, Brewfile,
  ci.yml, nightly.yml; README + Windows packaging docs updated
@da-phil da-phil force-pushed the add_hdr_merge_auto_align branch from a5a803c to 938a8f3 Compare June 27, 2026 14:07
@da-phil

da-phil commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

At that time there was only 1 possible output. This PR introduces settings, making multiple outputs possible. Since settings are available, users will try and tune the HDR output to find the best settings. You could probably steal the conflict code from the exporter or just do a simple test if the file exists and then append _xx to the filename where xx is a 2 digit sequence number.

I prefer to do this in another PR, I don't want to change too many things at a time.
This PR is only about auto-alginment for the HDR merge functionality and getting feedback from folks who requested the feature.

EDIT: Okay, the more I think about this the more sense it makes, especially for testing the feature with the ability to create multiple new HDR DNG files with different settings will be much easier with that filename numbering schema. If we settle on a specific set of parameter values, we can also remove this again.
Will push another commit soon.

@da-phil

da-phil commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

FYI: I did not test the auto-alignment with X-Trans (Fuji) RAW files. It would be great if anybody could cover this case or provide exposure brackets for testing. Thanks!

@da-phil da-phil marked this pull request as ready for review June 29, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: new new features to add scope: image processing correcting pixels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Better HDR merge with auto-allignment

3 participants