[AI] Add Lua AI API with model inference, tensor ops, and raw CFA support

[andriiryzhkov]

This adds a Lua scripting API for the AI subsystem, enabling Lua scripts to load ONNX models, run inference, and process images - including raw CFA sensor data. The goal is to let script authors build custom AI workflows (denoise, upscale, tagging, raw processing) without C code and external dependances.

@TurboGit any feedback on the Lua integration would be appreciated, especially the tensor type approach.

Summary

• Add darktable.ai Lua namespace for AI model inference and tensor manipulation
• Provides model discovery, loading with GPU provider selection, and two inference calling conventions (pre-allocated and auto-allocated outputs)
• Tensor type with get/set, crop/paste, dot product, shape introspection, and sRGB/linear conversion
• Image I/O: load from file or darktable library (full pipeline export), raw CFA sensor data access, DNG output with EXIF preservation, 16/32-bit TIFF with ICC profile embedding, PNG/JPEG save
• Includes model_for_task() to get the enabled model for a given task without manual iteration
• Follows darktable Lua conventions: dt_lua_init_singleton for namespace, luaL_checkudata for type safety, dt_lua_image_t for image arguments
• Add Lua AI API to release notes

API overview

-- model discovery and inference
darktable.ai.models()
darktable.ai.model_for_task("denoise")
local ctx = darktable.ai.load_model(model_id)
local output = ctx:run(input)            -- auto-allocate
ctx:run({input}, {output})               -- pre-allocated
ctx:close()

-- tensors
local t = darktable.ai.create_tensor({1, 3, H, W})
t:get({0, 0, y, x})
t:set({0, 0, y, x}, value)
t:shape()  t:ndim()  t:size()
t:crop(y, x, h, w)
t:paste(src, y, x)
t:dot(other)
t:linear_to_srgb()
t:srgb_to_linear()

-- image I/O
darktable.ai.load_image("/path/to/file.png")
darktable.ai.load_image(dt_image_object)
darktable.ai.load_raw(dt_image_object)
darktable.ai.save_dng(tensor, metadata, path)
tensor:save_tiff(path, 16, source_image)
tensor:save(path)

[wpferguson]

@TurboGit, @andriiryzhkov this violates the constraint(understanding?) that Lua isn't supposed to be used to process images in darkroom (though we kind of bend that by manipulating the GUI). The reasoning, AFAIK, was Lua was too slow to provide a decent user experience.

This is going to take some thought...

[andriiryzhkov]

@wpferguson, @TurboGit: I would really appreciate your advice on this.

My motivation comes from the fact that I've seen many attempts to bring AI features to darktable through Lua scripts that call external Python inference. People are already running AI models on their photos - they just do it outside of darktable with manual export/import cycles, and no color management.

This Lua API extension brings model inference inside darktable, which simplifies things significantly: images are loaded through the pipeline with edits applied, color profiles are handled correctly, and there's no external file juggling.

I might not be aware of prior decisions regarding Lua's positioning in darktable, so please let me know if this is the right direction.

[wpferguson]

I might not be aware of prior decisions regarding Lua's positioning in darktable, so please let me know if this is the right direction.

@andriiryzhkov I agree with your arguments about working with external files and hacking on the xmp files.

I don't have a position right now, hence the "This is going to take some thought..".

If the AI takes longer than Lua, then that mitigates the argument about slow processing (i.e. AI is the limiting factor)

I guess I'm going to have to learn AI...

[andriiryzhkov]

@wpferguson:

I don't have a position right now, hence the "This is going to take some thought..".

Sure. No problems.

If the AI takes longer than Lua, then that mitigates the argument about slow processing (i.e. AI is the limiting factor)

To clarify - the proposed Lua AI API does not inject AI processing into the pixel pipeline. There's no Lua callback running per-pixel or per-module in the darkroom pipe. That would indeed be unacceptably slow.

Instead, the AI API works as a batch/offline workflow: the script exports a fully processed image from the library (using the existing export pipeline), passes it as a tensor to an ONNX model for inference, and saves the result as a new file (TIFF/DNG). It's the same workflow people already do with external Python scripts, just without leaving darktable. The AI inference runs on its own thread - it doesn't block the pipeline or the UI.

This would be useful for things like alternative raw denoise, finding and grouping similar images, auto-tagging, and other tasks that work on images as a whole rather than inside the editing pipeline.

I guess I'm going to have to learn AI...

You can count on my help with this.

[wpferguson]

Instead, the AI API works as a batch/offline workflow: the script exports a fully processed image

That's that part that I didn't understand from reading the PR, though in your defense I was a little brain dead yesterday 😄

It's the same workflow people already do with external Python scripts, just without leaving darktable.

That would be an improvement

---

It will take me a little while to get to this, maybe a week or so while I finish up all the other stuff I've got going on.

[TurboGit]

@wpferguson @andriiryzhkov : Clearly Bill is the Lua expert here. So the last word will be from him. On my side I see no issue with that as it is not on darkroom but just scripting using AI feature to create new images (denoised or upsclaled). I don't see what different with a Lua script calling enfuse for example. Well I do see it in the implementation side, one is calling an API the other spawn a process.

My point is that from a user point of view it would be nice to have such an API, so if we can arrange to have a safe approach I'm all for it.

@wpferguson : As you said we already deal with widgets on the Lua side, so maybe not such a big issue :)

[wpferguson]

This is my weekend project 😄

[wpferguson]

I played over the weekend and ...

I roughed out the documentation for the luadocs

I figured the best way to check out the API was to try and use it...

attached is the script I was playing with

ai_example.zip

I was trying to load the default raw denoise model and it couldn't load it. I changed to the default denoise model and it loaded but then I got an inference error when trying to process (probably because the image was raw and the denoise model wasn't).

Here's the log from the runs...

     5.5095 LUA starting AI example
     5.5104 LUA found 8 models
     5.5105 LUA model name - mask sam2.1 hiera small that does Segment Anything 2.1 (Hiera Small) for interactive masking
     5.5105 LUA model name - mask segnext vitb-sax2 hq that does SegNext ViT-B SAx2 HQ fine-tuned for interactive masking
     5.5105 LUA model name - denoise nind that does UNet denoiser trained on NIND dataset
     5.5105 LUA model name - denoise nafnet small that does NAFNet denoiser trained on SIDD dataset
     5.5105 LUA model name - raw denoise nind that does UtNet2 raw denoiser trained on RawNIND dataset
     5.5105 LUA model name - upscale bsrgan that does BSRGAN 2x and 4x blind super-resolution
     5.5105 LUA model name - mask sam2.1 hiera tiny that does Segment Anything 2.1 (Hiera Tiny) for masking
     5.5105 LUA model name - mask depth anything v2 small that does Depth Anything V2 Small for depth-based masking
     5.5105 LUA looking for model for denoise
     5.5105 LUA looking for default model for denoise
     5.5105 LUA found denoise-nind for type denoise
     5.5105 LUA found denoise-nind for model type denoise
     5.5105 LUA looking for model for mask
     5.5105 LUA looking for default model for mask
     5.5105 LUA found mask-object-sam21-small for type mask
     5.5105 LUA found mask-object-sam21-small for model type mask
     5.5105 LUA looking for model for rawdenoise
     5.5105 LUA looking for default model for rawdenoise
     5.5105 LUA found rawdenoise-nind for type rawdenoise
     5.5105 LUA found rawdenoise-nind for model type rawdenoise
     5.5105 LUA looking for model for depth
     5.5105 LUA looking for default model for depth
     5.5105 LUA didn't find any models for type depth
     5.5105 LUA no default model found for depth
     5.5105 LUA looking for model for upscale
     5.5105 LUA looking for default model for upscale
     5.5105 LUA didn't find any models for type upscale
     5.5105 LUA no default model found for upscale
The requested API version [24] is not available, only API versions [1, 20] are supported in this build. Current ORT Version is: 1.20.1
The requested API version [23] is not available, only API versions [1, 20] are supported in this build. Current ORT Version is: 1.20.1
The requested API version [22] is not available, only API versions [1, 20] are supported in this build. Current ORT Version is: 1.20.1
The requested API version [21] is not available, only API versions [1, 20] are supported in this build. Current ORT Version is: 1.20.1
     5.5866 LUA created context
     5.8159 LUA loaded image
     5.8740 LUA ERROR: dtutils.lua: prequire: 223: Error loading ai_example/ai_example 
     5.8741 LUA ERROR: dtutils.lua: prequire: 224: Error returned is ...config/dttest_20716_lua_ai/lua/ai_example/ai_example.lua:227: inference failed (error -3) 
     5.8741 LUA ERROR: script_manager.lua: activate: 577: error loading ai_example/ai_example 
     5.8742 LUA ERROR: script_manager.lua: activate: 578: error message: ...config/dttest_20716_lua_ai/lua/ai_example/ai_example.lua:227: inference failed (error -3) 
   114.9263 LUA INFO: script_manager.lua:  activating ai_example 
   114.9266 LUA starting AI example
   114.9266 LUA found 8 models
   114.9266 LUA model name - mask sam2.1 hiera small that does Segment Anything 2.1 (Hiera Small) for interactive masking
   114.9266 LUA model name - mask segnext vitb-sax2 hq that does SegNext ViT-B SAx2 HQ fine-tuned for interactive masking
   114.9266 LUA model name - denoise nind that does UNet denoiser trained on NIND dataset
   114.9266 LUA model name - denoise nafnet small that does NAFNet denoiser trained on SIDD dataset
   114.9266 LUA model name - raw denoise nind that does UtNet2 raw denoiser trained on RawNIND dataset
   114.9266 LUA model name - upscale bsrgan that does BSRGAN 2x and 4x blind super-resolution
   114.9266 LUA model name - mask sam2.1 hiera tiny that does Segment Anything 2.1 (Hiera Tiny) for masking
   114.9266 LUA model name - mask depth anything v2 small that does Depth Anything V2 Small for depth-based masking
   114.9266 LUA looking for model for denoise
   114.9266 LUA looking for default model for denoise
   114.9266 LUA found denoise-nind for type denoise
   114.9266 LUA found denoise-nind for model type denoise
   114.9266 LUA looking for model for mask
   114.9266 LUA looking for default model for mask
   114.9266 LUA found mask-object-sam21-small for type mask
   114.9266 LUA found mask-object-sam21-small for model type mask
   114.9266 LUA looking for model for rawdenoise
   114.9266 LUA looking for default model for rawdenoise
   114.9267 LUA found rawdenoise-nind for type rawdenoise
   114.9267 LUA found rawdenoise-nind for model type rawdenoise
   114.9267 LUA looking for model for depth
   114.9267 LUA looking for default model for depth
   114.9267 LUA didn't find any models for type depth
   114.9267 LUA no default model found for depth
   114.9267 LUA looking for model for upscale
   114.9267 LUA looking for default model for upscale
   114.9267 LUA didn't find any models for type upscale
   114.9267 LUA no default model found for upscale
   114.9268 LUA ERROR: dtutils.lua: prequire: 223: Error loading ai_example/ai_example 
   114.9268 LUA ERROR: dtutils.lua: prequire: 224: Error returned is ...config/dttest_20716_lua_ai/lua/ai_example/ai_example.lua:223: failed to load model 'rawdenoise-nind' 
   114.9269 LUA ERROR: script_manager.lua: activate: 577: error loading ai_example/ai_example 
   114.9270 LUA ERROR: script_manager.lua: activate: 578: error message: ...config/dttest_20716_lua_ai/lua/ai_example/ai_example.lua:223: failed to load model 'rawdenoise-nind' 
   122.2176 [pwstorage_new] Destroying context 0x63a65bfb5fa0

I have a question. Assuming I get the rawdenoise model to load, how do I adjust the processing parameters?