feat: add Postgres support to SearchOperations by msmithstubbs · Pull Request #3307 · Logflare/logflare

msmithstubbs · 2026-03-24T04:37:59Z

Adds support for searches against a postgres backend.

Adds a backend_type field to %SearchOperation{}, defaults to `:bigquery', and is set automatically to default backend.
SearchOperations builds the search query for the appropriate backend

Intended for integration tests in CI. Part of #3239

Follows on from #3306 and #3317

use test_connection/1 in test setup
handle custom aggregates
unit tests in SearchOperationTest
unit tests in SearchLV

Ziinc · 2026-03-24T13:18:11Z

+      key = if key == "count", do: "value", else: key
+      Map.put(acc, key, normalize_postgres_value(key, value))


needs to factor in custom aggregates. chart would break for custom agg like p99 i think

Ziinc · 2026-03-24T13:27:18Z

I've added a high level search test, do we also want units tests in SearchOperationTest to mirror the BigQuery coverage?
some unit tests would be nice.

would also be good to have unit test in logs_search_lv_test.exs

Ziinc · 2026-03-26T06:47:13Z

+          |> nested_map_update()
+        end
+
+      {:ok, rows}


We should make the returned map the same as bigquery so that we don't have to normalise in the caller

An Adaptor.QueryResult struct might work to make things more standardised

Handled this in a separate PR #3317

Ziinc · 2026-03-26T06:54:19Z

+      |> TestUtils.wait_for_render("#logs-list-container li")
+
+      assert view |> element("#logs-list-container") |> render() =~ matching_message
+    end


Should refute non matching message

Ziinc · 2026-03-26T06:56:31Z

+      bq_schema = TestUtils.build_bq_schema(%{"event_message" => matching_message})
+
+      assert {:ok, _source_schema} =
+               SourceSchemas.create_or_update_source_schema(source, %{
+                 bigquery_schema: bq_schema,
+                 schema_flat_map: SchemaUtils.bq_schema_to_flat_typemap(bq_schema)
+               })
+


Can be combined into a source_schema factory so that it is a bit more ergonomic.

Ziinc · 2026-03-26T06:56:51Z

+                 schema_flat_map: SchemaUtils.bq_schema_to_flat_typemap(bq_schema)
+               })
+
+      Cachex.clear(Logflare.SourceSchemas.Cache)


Don't need to clear cache if it is inserted before source sup is started

Good point, moved the insert to happen before source sup start.

Ziinc · 2026-03-26T06:58:29Z

+    end
+
+    test "generates expected SQL for all aggregate types", %{base_so: base_so} do
+      for agg <- [:count, :avg, :sum, :max] do


We're only testing a subset of the accepted aggs?

Had thought of this more as an integration test that aggregate types work since each agg is covered by postgres backend transformer test. But it's cheap test to do here, too.

Added full list: :count, :avg, :sum, :max, :countd, :p50, :p95, :p99

Ziinc

@msmithstubbs can you make the test adjustments in a follow up? 🙏

Ziinc · 2026-04-02T00:13:00Z

+      PostgresAdaptor
+      |> expect(:execute_query, fn ^backend, %Ecto.Query{}, [query_type: :search] ->
+        rows = [%{"count" => 2, "timestamp" => unix_timestamp}]
+        {:ok, QueryResult.new(rows, %{total_rows: length(rows)})}
+      end)
+
+      PostgresAdaptor
+      |> expect(:ecto_to_sql, fn %Ecto.Query{}, [] ->
+        {:ok, {"SELECT count(*) FROM test_table", []}}
+      end)


Let's not stub this so thar we have a full integration test

Ziinc · 2026-04-02T00:14:29Z

+      |> expect(:get_default_backend, fn _user -> backend end)
+
+      PostgresAdaptor
+      |> expect(:execute_query, fn ^backend, %Ecto.Query{}, [query_type: :search] ->


Simulating a raise (like bad connection) would be preferred

Ziinc · 2026-04-02T00:15:13Z

+      end)
+
+      PostgresAdaptor
+      |> expect(:execute_query, fn ^backend, %Ecto.Query{} = query, opts ->


Mock here is not referred for comprehensiveness

Ziinc · 2026-04-02T00:17:21Z

+  end
+
+  describe "postgres backend adaptor integration" do
+    setup %{user: user} do


See the SingleTenant test module, there is some test macros that should be used when dealing with single tenant mode that will help with all this provisioning so that we don't need to mock anything

* refactor: backend execute_query/3 returns Adaptor.QueryResult * feat: PostgresAdaptor include total_rows in result * refactor: decouple SearchLV LQL from BigQuery TableSchema * fix: dialyzer warnings * feat: add Postgres support to SearchOperations * style: alias nested modules (credo) * implement PostgresAdaptor.test_connection/1 * fix: postgres adaptor aggregates * fix: execute_query can return error (dialyzer) * test: SearchOperations postgres unit tests * test: SearchLV postgres backend search * test tidy up * refactor: move postgres result normalisation to PostgresAdaptor * test: refute non matching message * test: full list of aggregate types * bump ci * test: put_sql_string matches on backend_type --------- Co-authored-by: Ziinc <Ziinc@users.noreply.github.com>

github-actions Bot assigned msmithstubbs Mar 24, 2026

msmithstubbs force-pushed the feat/postgres-search branch 2 times, most recently from 693cf45 to ee679c4 Compare March 24, 2026 05:24

msmithstubbs mentioned this pull request Mar 24, 2026

refactor: decouple SearchLV LQL from BigQuery TableSchema #3306

Merged

msmithstubbs marked this pull request as ready for review March 24, 2026 06:34

Ziinc requested changes Mar 24, 2026

View reviewed changes

msmithstubbs force-pushed the feat/postgres-search branch 4 times, most recently from b385b5a to 87671ca Compare March 26, 2026 04:27

msmithstubbs requested a review from Ziinc March 26, 2026 06:13

msmithstubbs mentioned this pull request Mar 26, 2026

feat: Monaco editor with LQL support for search field #3239

Merged

Ziinc reviewed Mar 26, 2026

View reviewed changes

msmithstubbs mentioned this pull request Mar 27, 2026

refactor: backend execute_query/3 returns Adaptor.QueryResult #3317

Merged

msmithstubbs force-pushed the feat/postgres-search branch 2 times, most recently from a955253 to cbd6465 Compare March 27, 2026 04:25

refactor: backend execute_query/3 returns Adaptor.QueryResult

104c32a

msmithstubbs force-pushed the feat/postgres-search branch 2 times, most recently from 3bda886 to b9b809f Compare April 1, 2026 03:38

msmithstubbs added 9 commits April 1, 2026 13:41

feat: PostgresAdaptor include total_rows in result

d4bb1da

refactor: decouple SearchLV LQL from BigQuery TableSchema

c4ddf4d

fix: dialyzer warnings

8a3c1d3

feat: add Postgres support to SearchOperations

5591e81

style: alias nested modules (credo)

9e95727

implement PostgresAdaptor.test_connection/1

0bcb70a

fix: postgres adaptor aggregates

13c03b6

fix: execute_query can return error (dialyzer)

b8a15b5

test: SearchOperations postgres unit tests

b027992

msmithstubbs and others added 8 commits April 1, 2026 13:41

test: SearchLV postgres backend search

0fe5fd9

Merge branch 'main' into feat/postgres-search

111f81b

test tidy up

452bda1

refactor: move postgres result normalisation to PostgresAdaptor

de0340e

test: refute non matching message

f6fa992

test: full list of aggregate types

245f8ca

bump ci

c2af619

test: put_sql_string matches on backend_type

ead7a22

msmithstubbs force-pushed the feat/postgres-search branch from b9b809f to ead7a22 Compare April 1, 2026 03:42

Ziinc approved these changes Apr 2, 2026

View reviewed changes

Ziinc reviewed Apr 2, 2026

View reviewed changes

Ziinc merged commit 206bba5 into Logflare:main Apr 2, 2026
12 checks passed

msmithstubbs mentioned this pull request Apr 3, 2026

test: improve SearchOperationsTest postgres coverage #3340

Merged

		key = if key == "count", do: "value", else: key
		Map.put(acc, key, normalize_postgres_value(key, value))

Conversation

msmithstubbs commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Ziinc commented Mar 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ziinc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msmithstubbs commented Mar 24, 2026 •

edited

Loading