diff --git a/CHANGELOG.md b/CHANGELOG.md index e516904..2de26cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,12 @@ * **Breaking:** `Image.k_means/2` and `Image.reduce_colors/2` return `{:error, %Image.Error{reason: :invalid_option}}` for an invalid or unknown option instead of raising `NimbleOptions.ValidationError`. `Image.k_means!/2` and `Image.reduce_colors!/2` raise `Image.Error` rather than the NimbleOptions exception. `operation` is set to `k_means` or `reduce_colors`, and `value` is `{key, value}` for an invalid value or the list of keys for unknown options. ([#227](https://github.com/elixir-image/image/pull/227)) +* **Breaking:** `Image.reduce_colors/2` returns `{:u, 8}` instead of `{:f, 32}`. K-means produced a float image, and nothing cast it back. The result is now rounded into the band format it clustered. libvips truncated these float values when casting on save, so output values can change by up to 1 per channel after this change. ([#229](https://github.com/elixir-image/image/pull/229)) + +* **Breaking:** `Image.reduce_colors/2` returns the image in the colorspace it was given. A `:cmyk` image returns four bands and a greyscale image returns one, where previously every image came back as 3-band `:srgb`. ([#229](https://github.com/elixir-image/image/pull/229)) + +* `Image.reduce_colors/2` clamps `:colors` to the number of unique colors in the image. Previously a `:colors` greater than the image's pixel count failed. ([#229](https://github.com/elixir-image/image/pull/229)) + * `Image.affine/3` and `Image.rotate/3` now premultiply alpha explicitly only when the background is non-opaque, since libvips handles the other cases itself. `Image.shear/4` and `Image.translate/4` inherit this. ([#217](https://github.com/elixir-image/image/pull/217)) ### Fixed @@ -48,6 +54,12 @@ * Fix the `:greater_than` and `:less_than` documentation for `Image.chroma_mask/2`, `Image.chroma_key/2` and `Image.replace_color/2`, which described the bounds the wrong way round in all six places they appeared. The mask covers the range between the two, so `:greater_than` is the lower bound and `:less_than` the upper. ([#224](https://github.com/elixir-image/image/pull/224)) +* Fix `Image.reduce_colors/2` silently producing wrong output above 256 colors. A cast to `{:u, 8}` wrapped every color index above 255 back around (256 became 0, 257 became 1, and so on), so those pixels were painted with the color reached by the wraparound. ([#229](https://github.com/elixir-image/image/pull/229)) + +* Fix `Image.reduce_colors/2` raising instead of returning an error tuple, both for the `ArgumentError` that `Scholar.Cluster.KMeans.fit/2` raises for checks it makes outside its option schema, and for the `ArithmeticError` it raises when given a single sample. ([#229](https://github.com/elixir-image/image/pull/229)) + +* Fix `Image.reduce_colors/2` raising when the image could not be converted to a tensor. ([#229](https://github.com/elixir-image/image/pull/229)) + ### Removed * **Breaking:** Removes `Image.Options.WarpPerspective`, replaced by `Image.Options.Mapim`. ([#216](https://github.com/elixir-image/image/pull/216)) @@ -108,6 +120,8 @@ This is the changelog for Image version 0.70.0 released on July 8th, 2026. For * Adds `Image.YUV.valid_encodings/0` and `Image.YUV.valid_colorspaces/0`. +* Adds `Image.Scholar.unique_color_count/1`, which returns the number of distinct colors in an image and accepts either an image or its tensor. ([#229](https://github.com/elixir-image/image/pull/229)) + ### Changed * `Image.average/1` now weights the average by the alpha band so transparent pixels do not contribute; fully transparent images fall back to the unweighted color-band average, and float-format images return unrounded averages. diff --git a/lib/image.ex b/lib/image.ex index abdd5ba..98f09d0 100644 --- a/lib/image.ex +++ b/lib/image.ex @@ -9066,9 +9066,9 @@ defmodule Image do @doc """ Reduces the number of colors in an image. - Takes the `k_means/2` of the image and then - re-colors the image using the returned cluster - colors. + Applies K-means clustering to the pixels of the image + and then re-colors each pixel with the color of the + cluster it was assigned to. ### Arguments @@ -9080,6 +9080,9 @@ defmodule Image do * `:colors` is the number of distinct colors to be used in the returned image. The default is `#{@default_clusters}`. + An image cannot have more distinct colors than it started + with, so `:colors` is clamped to the number of unique colors + in the image. * See also `Scholar.Cluster.KMeans.fit/2` for the available options. @@ -9089,6 +9092,11 @@ defmodule Image do * Note the performance considerations described in `Image.k_means/2` since they also apply to this function. + * Clustering is performed in the `:srgb` colorspace and the + result is converted back to the colorspace of `image`. The + colors of a 16-bit image are therefore drawn from an 8-bit + palette. + * If the intent is to reduce colors in order to reduce the size of an image file it is strongly advised to use the appropriate arguments when calling `Image.write/2`. @@ -9111,6 +9119,9 @@ defmodule Image do """ @doc subject: "Clusters", since: "0.50.0" + @spec reduce_colors(image :: Vimage.t(), options :: Keyword.t()) :: + {:ok, Vimage.t()} | {:error, error()} + def reduce_colors(%Vimage{} = image, options \\ []) do case do_reduce_colors(image, options) do {:error, reason} -> {:error, Image.Error.wrap(reason, operation: :reduce_colors)} @@ -9119,32 +9130,31 @@ defmodule Image do end defp do_reduce_colors(image, options) do - with {:ok, image} <- to_colorspace(image, :srgb) do - kmeans_num_clusters = - Keyword.get(options, :colors, @default_clusters) - - options = - options - |> Keyword.put(:num_clusters, kmeans_num_clusters) - |> Keyword.delete(:colors) - - {width, height, bands} = - Image.shape(image) - - nx_reshaped = - image - |> to_nx!() - |> Nx.reshape({height * width, bands}) - - with {:ok, model} <- Image.Scholar.fit(nx_reshaped, options) do - indicies = - Nx.as_type(model.labels, :u8) - - model.clusters - |> Nx.take(indicies) - |> Nx.reshape({height, width, bands}) - |> Image.from_nx() - end + options = + options + |> Keyword.put(:num_clusters, Keyword.get(options, :colors, @default_clusters)) + |> Keyword.delete(:colors) + + colorspace = Image.colorspace(image) + + with {:ok, srgb} <- to_colorspace(image, :srgb), + {:ok, tensor} <- to_nx(srgb), + {width, height, bands} = Image.shape(srgb), + nx_reshaped = Nx.reshape(tensor, {height * width, bands}), + {:ok, unique_count} <- Image.Scholar.unique_color_count(tensor), + {:ok, model} <- Image.Scholar.fit(nx_reshaped, options, unique_count), + # The clusters are floats, so the recolored image is rounded + # back into the band format of the image being clustered. + {:ok, reduced} <- + model.clusters + |> Nx.take(model.labels) + |> Nx.round() + |> Nx.as_type(Image.band_format(srgb)) + |> Nx.reshape({height, width, bands}) + |> Image.from_nx() do + # Clustering happens in :srgb, so the result is returned to the + # colorspace the caller passed in. + to_colorspace(reduced, colorspace) end end @@ -9152,9 +9162,9 @@ defmodule Image do Reduces the number of colors in an image or raises an exception. - Takes the `k_means/2` of the image and then - re-colors the image using the returned cluster - colors. + Applies K-means clustering to the pixels of the image + and then re-colors each pixel with the color of the + cluster it was assigned to. ### Arguments @@ -9166,6 +9176,9 @@ defmodule Image do * `:colors` is the number of distinct colors to be used in the returned image. The default is `#{@default_clusters}`. + An image cannot have more distinct colors than it started + with, so `:colors` is clamped to the number of unique colors + in the image. * See also `Scholar.Cluster.KMeans.fit/2` for the available options. @@ -9175,6 +9188,11 @@ defmodule Image do * Note the performance considerations described in `Image.k_means/2` since they also apply to this function. + * Clustering is performed in the `:srgb` colorspace and the + result is converted back to the colorspace of `image`. The + colors of a 16-bit image are therefore drawn from an 8-bit + palette. + * If the intent is to reduce colors in order to reduce the size of an image file it is strongly advised to use the appropriate arguments when calling `Image.write/2`. @@ -9197,6 +9215,9 @@ defmodule Image do """ @doc subject: "Clusters", since: "0.51.0" + @spec reduce_colors!(image :: Vimage.t(), options :: Keyword.t()) :: + Vimage.t() | no_return() + def reduce_colors!(%Vimage{} = image, options \\ []) do case reduce_colors(image, options) do {:ok, image} -> image diff --git a/lib/image/scholar.ex b/lib/image/scholar.ex index 72518f6..d2d112a 100644 --- a/lib/image/scholar.ex +++ b/lib/image/scholar.ex @@ -6,8 +6,9 @@ if match?({:module, _module}, Code.ensure_compiled(Scholar.Cluster.KMeans)) and [Scholar](https://hex.pm/packages/scholar) machine-learning primitives. - The primary public API is `unique_colors/1` and `k_means/2` - which underpin `Image.k_means/2` and `Image.reduce_colors/2`. + The public API is `unique_colors/1`, `unique_color_count/1` and + `k_means/2`, which underpin `Image.k_means/2` and + `Image.reduce_colors/2`. """ @@ -102,20 +103,130 @@ if match?({:module, _module}, Code.ensure_compiled(Scholar.Cluster.KMeans)) and end end + @doc """ + Returns the number of unique colors in an image. + + Prefer this over `unique_colors/1` when only the count is + needed. + + ### Arguments + + * `image_or_tensor` is any 3- or 4-band `t:Vix.Vips.Image.t/0` with + `{:u, 8}` band format, or the `{height, width, bands}` tensor of + such an image as returned by `Image.to_nx/2`. Pass the tensor when + one is already to hand, to avoid converting the image twice. + + ### Returns + + * `{:ok, count}` or + + * `{:error, reason}`. + + ### Example + + iex> Image.Scholar.unique_color_count(Image.new!(4, 4, color: :red)) + {:ok, 1} + + """ + @spec unique_color_count(image_or_tensor :: Vimage.t() | Nx.Tensor.t()) :: + {:ok, non_neg_integer()} | {:error, Image.Error.t()} + + def unique_color_count(image_or_tensor) + + def unique_color_count(%Vimage{} = image) do + with {:ok, tensor} <- Image.to_nx(image) do + unique_color_count(tensor) + end + end + + # The rank is checked first so the band lookup below cannot raise. + def unique_color_count(%Nx.Tensor{} = tensor) do + cond do + Nx.rank(tensor) != 3 -> + {:error, + scholar_error( + "unique_color_count requires a {height, width, bands} tensor. " <> + "Found rank #{Nx.rank(tensor)}" + )} + + Nx.axis_size(tensor, 2) not in [3, 4] -> + {:error, + scholar_error( + "unique_color_count requires a 3- or 4-band image. " <> + "Found #{Nx.axis_size(tensor, 2)} bands" + )} + + Nx.type(tensor) != {:u, 8} -> + {:error, + scholar_error( + "unique_color_count requires an 8-bit unsigned image. " <> + "Found #{inspect(Nx.type(tensor))}" + )} + + true -> + {:ok, do_unique_color_count(tensor, Nx.axis_size(tensor, 2))} + end + end + + defp do_unique_color_count(tensor, bands) do + encoded = + tensor + |> encode_colors(bands) + |> Nx.flatten() + |> Nx.sort() + + # Nx.diff/1 needs at least two elements, so a single pixel is counted + # directly. Otherwise the distinct count is one more than the number + # of adjacent unequal pairs. + if Nx.size(encoded) < 2 do + Nx.size(encoded) + else + Nx.to_number(Nx.sum(Nx.not_equal(diff(encoded), 0))) + 1 + end + end + defp scholar_error(message) do %Image.Error{message: message, reason: message} end - # Scholar.Cluster.KMeans.fit/2 validates its options with - # NimbleOptions.validate!/2 which raises on an invalid or unknown - # option, so translate the exception to {:error, %Image.Error{}} here - # at the boundary + # Scholar.Cluster.KMeans.fit/2 raises rather than returning an error: + # NimbleOptions.ValidationError for an invalid or unknown option, and + # ArgumentError for the checks it makes outside its schema. Both are + # translated to {:error, %Image.Error{}} here at the boundary. + # + # `max_clusters` bounds `:num_clusters`, which cannot exceed the + # number of distinct samples to cluster. @doc false - def fit(samples, options) do - {:ok, Scholar.Cluster.KMeans.fit(samples, options)} + def fit(samples, options, max_clusters \\ nil) do + # Scholar raises ArithmeticError for a lone sample rather than + # validating it, so it is checked here to keep the message useful. + if Nx.axis_size(samples, 0) < 2 do + {:error, + scholar_error( + "K-means requires at least 2 samples to cluster. " <> + "Found #{Nx.axis_size(samples, 0)}" + )} + else + {:ok, Scholar.Cluster.KMeans.fit(samples, clamp_clusters(options, max_clusters))} + end rescue exception in NimbleOptions.ValidationError -> {:error, invalid_option(exception)} + + exception in ArgumentError -> + {:error, %Image.Error{reason: :invalid_option, message: Exception.message(exception)}} + end + + defp clamp_clusters(options, nil), do: options + + defp clamp_clusters(options, max_clusters) do + case Keyword.fetch(options, :num_clusters) do + {:ok, num_clusters} when is_integer(num_clusters) -> + Keyword.put(options, :num_clusters, Kernel.min(num_clusters, max_clusters)) + + _other -> + options + end end # An unknown option sets :key to the list of unknown keys and leaves @@ -159,8 +270,6 @@ if match?({:module, _module}, Code.ensure_compiled(Scholar.Cluster.KMeans)) and """ def k_means(%Vimage{} = image, options \\ []) do with {:ok, {_count, colors}} <- unique_colors(image) do - # K-means requires at least as many samples as clusters, so - # the cluster count is clamped to the number of unique colors. # A single unique color (solid image) is duplicated because # the random centroid initialisation needs at least 2 samples. unique_count = Nx.axis_size(colors, 0) @@ -168,16 +277,7 @@ if match?({:module, _module}, Code.ensure_compiled(Scholar.Cluster.KMeans)) and colors = if unique_count == 1, do: Nx.concatenate([colors, colors]), else: colors - options = - case Keyword.fetch(options, :num_clusters) do - {:ok, num_clusters} when is_integer(num_clusters) -> - Keyword.put(options, :num_clusters, Kernel.min(num_clusters, unique_count)) - - _other -> - options - end - - fit(colors, options) + fit(colors, options, unique_count) end end diff --git a/test/image_analysis_coverage_test.exs b/test/image_analysis_coverage_test.exs index 9adc1c7..e44cf44 100644 --- a/test/image_analysis_coverage_test.exs +++ b/test/image_analysis_coverage_test.exs @@ -4,6 +4,8 @@ defmodule Image.AnalysisCoverageTest do import Image.TestSupport alias Vix.Vips.Image, as: Vimage + doctest Image.Scholar + describe "Image.delta_e/3" do test "identical colors have zero difference in all versions" do for version <- [:de00, :de76, :decmc] do @@ -170,6 +172,138 @@ defmodule Image.AnalysisCoverageTest do value: [:unknown_option] }} = Image.reduce_colors(image, unknown_option: true) end + + test "reduce_colors/2 preserves the band format of the image" do + image = Image.new!(2, 2, color: [255, 0, 0]) + + assert {:ok, reduced} = Image.reduce_colors(image, colors: 1, key: Nx.Random.key(1)) + assert Image.band_format(reduced) == Image.band_format(image) + + pixels = Image.to_nx!(reduced) |> Nx.to_flat_list() + assert pixels == List.flatten(List.duplicate([255, 0, 0], 4)) + end + + test "reduce_colors/2 rounds cluster colors rather than truncating them" do + # The two pixels are one apart, so the single centroid lands on + # exactly 100.5, where rounding and truncating disagree. + image = + Image.join!( + [Image.new!(1, 1, color: [100, 100, 100]), Image.new!(1, 1, color: [101, 101, 101])], + across: 2 + ) + + assert {:ok, reduced} = Image.reduce_colors(image, colors: 1, key: Nx.Random.key(1)) + assert Image.to_nx!(reduced) |> Nx.to_flat_list() == List.duplicate(101, 6) + end + + test "reduce_colors/2 returns the image in its original colorspace", %{image: image} do + for colorspace <- [:srgb, :cmyk, :bw, :grey16, :rgb16, :hsv] do + source = Image.to_colorspace!(image, colorspace) + + assert {:ok, reduced} = + Image.reduce_colors(source, colors: 4, key: Nx.Random.key(1), num_runs: 1) + + assert Image.colorspace(reduced) == colorspace + assert Image.bands(reduced) == Image.bands(source) + assert Image.band_format(reduced) == Image.band_format(source) + end + end + + test "reduce_colors/2 returns an error for a single-pixel image" do + assert {:error, %Image.Error{operation: :reduce_colors, message: message}} = + Image.reduce_colors(Image.new!(1, 1, color: :red), colors: 2) + + assert message =~ "at least 2 samples" + end + + test "reduce_colors/2 clamps :colors to the number of unique colors" do + image = + Image.join!([Image.new!(2, 2, color: :red), Image.new!(2, 2, color: :blue)], across: 2) + + assert {:ok, 2} = Image.Scholar.unique_color_count(image) + + assert {:ok, reduced} = Image.reduce_colors(image, colors: 100, key: Nx.Random.key(1)) + assert {:ok, 2} = Image.Scholar.unique_color_count(reduced) + end + + test "reduce_colors/2 uses more than 256 colors when asked" do + image = + image_path("Hong-Kong-2015-07-1998.jpg") + |> Image.open!() + |> Image.thumbnail!(40) + + assert {:ok, {_counts, colors}} = Image.Scholar.unique_colors(image) + assert Nx.axis_size(colors, 0) > 300 + + assert {:ok, reduced} = + Image.reduce_colors(image, + colors: 300, + key: Nx.Random.key(1), + num_runs: 1, + max_iterations: 5 + ) + + assert {:ok, {_counts, reduced_colors}} = Image.Scholar.unique_colors(reduced) + assert Nx.axis_size(reduced_colors, 0) > 256 + end + end + + describe "Image.Scholar.fit/3" do + test "returns an error when Scholar raises outside its option schema" do + samples = Nx.tensor([1, 2, 3, 4]) + + assert {:error, %Image.Error{reason: :invalid_option, message: message}} = + Image.Scholar.fit(samples, num_clusters: 2, key: Nx.Random.key(1)) + + assert message =~ "expected input tensor to have shape" + end + end + + describe "Image.Scholar.unique_color_count/1" do + setup do + image = + image_path("Kip_small.jpg") + |> Image.open!() + |> Image.thumbnail!(32) + + {:ok, %{image: image}} + end + + test "agrees with unique_colors/1", %{image: image} do + assert {:ok, {_counts, colors}} = Image.Scholar.unique_colors(image) + assert {:ok, count} = Image.Scholar.unique_color_count(image) + assert count == Nx.axis_size(colors, 0) + end + + test "an image and its tensor give the same answer", %{image: image} do + assert {:ok, count} = Image.Scholar.unique_color_count(image) + assert {:ok, ^count} = Image.Scholar.unique_color_count(Image.to_nx!(image)) + end + + test "counts a single pixel without calling Nx.diff/1" do + assert {:ok, 1} = Image.Scholar.unique_color_count(Image.new!(1, 1, color: :red)) + end + + test "returns an error for a greyscale image, which it cannot encode", %{image: image} do + assert {:error, %Image.Error{message: message}} = + Image.Scholar.unique_color_count(Image.to_colorspace!(image, :bw)) + + assert message =~ "3- or 4-band" + end + + test "returns an error for a 16-bit image, which it cannot encode", %{image: image} do + assert {:error, %Image.Error{message: message}} = + Image.Scholar.unique_color_count(Image.cast!(image, {:u, 16})) + + assert message =~ "8-bit unsigned" + end + + test "returns an error for a tensor that is not an image", %{image: image} do + assert {:error, %Image.Error{message: message}} = + Image.Scholar.unique_color_count(Nx.flatten(Image.to_nx!(image))) + + assert message =~ "rank 1" + end end describe "Image.preview/1 and Image.p/1" do