Add opt-in SHA-1 checksum + size tracking for images

[sjmudd]

Context: sidecar files can get lost or separated from their original images and then reconciling those differences can be hard. Adding a size/checksum allows for relocating sidecars against the original image they were generated against. Also deduplication of duplicate images will be easier as the size/checksums can be compared so even files named differently with matching values can most likely be considered to be identical.

Patch

Stores a raw 20-byte SHA-1 digest and file size per image, so a .xmp sidecar can still be matched to its source image after a rename or directory reorganization -- not possible today, since sidecars only record a filename. Revives #2492 (opened 2019, closed by stale-bot, never implemented), discussed on #21864.

• New images.sha1sum BLOB(20) and images.filesize INTEGER columns (schema version 57->58), both nullable, added via plain ALTER TABLE ADD COLUMN (metadata-only in SQLite for nullable, no-default columns -- doesn't rewrite existing rows).
• New plugins/darkroom/compute_checksum preference, default off. Computed lazily on first successful full read via dt_imageio_open(), only once per image (guarded by a new DT_IMAGE_HAS_SHA1SUM flag bit), via a dedicated sequential read pass (GChecksum) rather than hooking into any one loader's own I/O, since only rawspeed exposes a single read buffer.
• Written to the DB immediately; written to the .xmp sidecar (Xmp.darktable.image_checksum as "sha1:<40 uppercase hex chars>", Xmp.darktable.image_size) at the next natural xmp-sync point, gated by the existing write_sidecar_files preference like any other field -- so it's a no-op for users who keep sidecars off.
• Duplicate-with-version carries the identity forward (same physical file); copy-rename does not (new physical file, must recompute lazily).

Transparency: Code generated with help from AI

[ralfbrown]

• Computed lazily on first successful full read

That's good, since otherwise imports would become considerably slower for most users (needing to read the entire file instead of just metadata + embedded preview).

[ralfbrown]

Question after scanning the code: what happens if someone associates a sidecar containing filesize/checksum from another image and then imports (or selects to update DB from sidecar after a check for updated sidecars)? While not the usual use case, people do this for things like checking out PlayRaw submissions or copying around edits.

We've had a number of requests over the years for features that would need the filesize stored in the database, so unconditionally adding that info to the new column on import would enable future PRs implementing such features.

[wpferguson]

What happens if you have XMP files turned off, or worse yet set to after edit? Will computing a checksum and writing it to the XMP count as an edit and populate XMP's for every file opened? That wont sit well with the after edit crowd.

[sjmudd]

Correction of my previous comment.

@ralfbrown:

I had intended to have a cheap "size check" of the image vs what's stored in the side car to catch size differences, but hadn't actually coded this. The check is cheap and should catch most unexpected image file changes. I have now updated the code to check this and if the size changes the checksum will be recomputed and stored in the database. A couple of tests were also added. The side car is not updated unless further edits are made but the value in the database will be correct. This is not perfect but should catch most cases.

[sjmudd]

@wpferguson : if xmp file usage is turned off nothing is written. The code is not trying to write things if the configuration says to not do that. "after edit" behaviour: the checksum calculation does not count as an edit so it will not trigger an xmp write.

[wpferguson]

Suppose I submit an image to pixls.us as a play raw (ask other people to edit it on their systems, then show the result and attach the sidecar file).

I download the sidecar files from the edits I like

I create an original duplicate of the image for each sidecar that I downloaded.

I apply 1 sidecar to 1 duplicate until I've all the sidecars are applied to all the duplicates.

How does darktable handle the various sidecars, each with a different checksum, that all point to the same image?