DBMON-6748 ClickHouse Async Inserts Monitoring Flush Log Collection by jenny-chung · Pull Request #24549 · DataDog/integrations-core

jenny-chung · 2026-07-14T15:30:15Z

What does this PR do?

Adds flush log collection for ClickHouse async inserts. The check queries system.asynchronous_insert_log and submits one DBM event per flush record. The collection logic runs in the existing query completions job.

Motivation

DBM currently has no visibility into ClickHouse async insert flush health, so customers have to manually query system tables just to see whether flushes are failing, how fast they're running, or how much data they're moving.

Agent payload (in mitm proxy):

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

datadog-datadog-prod-us1-2 · 2026-07-14T15:35:45Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 90.99%
• Overall Coverage: 94.02% (+5.75%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 240253e | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d686cbc0b1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-15T17:33:20Z

+            self._config.query_completions.enabled or self._config.asynchronous_insert_flush_log.enabled
+        ):
+            self.query_completions = ClickhouseQueryCompletions(
+                self, self._config.query_completions, self._config.asynchronous_insert_flush_log


Enable the shared job when only flush logs are configured

When a user enables asynchronous_insert_flush_log but disables query_completions, this still constructs ClickhouseQueryCompletions with self._config.query_completions; ClickhouseQueryLogJob passes that config's enabled value into DBMAsyncJob, whose run_job_loop returns immediately when _enabled is false. In that configuration self.query_completions exists but run_job() is never called, so _collect_flush() is never reached and no async insert flush events are emitted.

Useful? React with 👍 / 👎.

Addressed in commits d96c340 and b8e99f1

chatgpt-codex-connector · 2026-07-15T17:33:20Z

+    {checkpoint_filter}
+    AND event_time_microseconds <= now64(6)


Add the date bound to flush-log collections

On instances that retain more than a few days of system.asynchronous_insert_log, this new query only has the microsecond checkpoint predicate. The existing query-log collectors also bind event_date >= toDate(fromUnixTimestamp64Micro({min_checkpoint_us:UInt64})) so ClickHouse can prune old daily partitions; here _collect_flush_rows discards _min_checkpoint and never adds the date predicate, so every async flush collection can scan historical log partitions that can never match. Please add the same event_date bound and parameter to the flush query.

Useful? React with 👍 / 👎.

Addressed in commit 24d2d2f

…query completions

sangeetashivaji · 2026-07-16T18:52:04Z

+
+    def _flush_error_count(self, error_label):
+        self._check.count(
+            "dd.clickhouse.query_completions.error",


this should be a async insert specific metrics name maybe dd.clickhouse.async_inserts_flush.error

I see, changed the name to be async insert specific in 52c4b33

sangeetashivaji · 2026-07-16T18:54:44Z

+        """
+        Run the async insert flush log collection on its own collection interval
+        """
+        if self._flush_enabled and time.time() - self._last_flush_collection_time >= self._flush_collection_interval:


Let's do one call of time.time(). since between these two calls there will be a few microseconds of elapsed time so the _last_flush_collection_time is later than the value being compared

Makes sense! Addressed in bcce48e

sangeetashivaji · 2026-07-16T19:05:28Z

+            "{checkpoint_filter}", checkpoint_filter
+        )
+        params["min_checkpoint_us"] = min_checkpoint
+        params["max_flush_rows"] = self._max_samples_per_collection


max_samples_per_collection seems to be coming from a different config section, we should have a max_flush_rows option under asynchronous_insert_flush_log in the spec.yaml.
This is confusing for anyone who disables query_completions entirely but runs flush.

Changed to use max_flush_rows option in asynchronous_insert_flush_log config - e35973d

sangeetashivaji · 2026-07-16T19:15:47Z

+            'collection_interval': self._flush_collection_interval,
+            'ddtags': self._tags_no_db,
+            'timestamp': time.time() * 1000,
+            'clickhouse_version': self._check.dbms_version,


let's add service here too 'service': getattr(self._check._config, 'service', None),

Added service in 32a158a

sangeetashivaji · 2026-07-16T19:23:00Z

+
+            event_time_int = self.to_microseconds(event_time_microseconds)
+            flush_time_int = self.to_microseconds(flush_time_microseconds)
+            flush_latency_us = flush_time_int - event_time_int if event_time_int and flush_time_int else 0


this can go negative when the system clocks skew across nodes, might be good to do a flush_latency_us = max(0, flush_time_int - event_time_int) if event_time_int and flush_time_int else 0

Fixed in 2a5ea02

sangeetashivaji · 2026-07-16T19:30:27Z

 SECURE_FIELD_NAMES = frozenset(['tls_ca_cert'])


+class AsynchronousInsertFlushLog(BaseModel):


QQ: I'm guessing this was a generated change but wanted to confirm

yes instance.py is auto generated

eric-weaver · 2026-07-17T18:33:34Z

+            type: number
+            example: 60
+          fleet_configurable: false
+        - name: max_flush_rows


Is limit a clearer? Perhaps max_samples_per_collection? We should strive for following existing naming conventions for similar functionality so that we maintain a cohesive customer experience.

dd-octo-sts · 2026-07-17T18:34:19Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

eric-weaver · 2026-07-17T18:36:37Z

            type: boolean
            example: false
          fleet_configurable: false
+    - name: asynchronous_insert_flush_log


This config name is a bit of a mouthful and I'm not sure customers would easily understand what this feature collects. Let's brainstorm on this a bit 🤔

eric-weaver · 2026-07-17T18:38:42Z

            example: false
          fleet_configurable: false
+    - name: asynchronous_insert_flush_log
+      description: Configure asynchronous insert flush log monitoring


Currently this description is re-stating the config block currently. I think we should move parts of the description for enabled up into here to better describe this whole block and shrink down enabled description. The part below about calling out dbm:true requirement is good to keep there

jenny-chung added 3 commits July 14, 2026 11:09

Add async insert flush log config options

797a518

Add auto generated config files for async insert flush log

bf2bd70

Wire async insert flug log into query completions

517b9a3

temporal-github-worker-1 Bot added agent/review-requested ecosystems/review-requested product/review-requested labels Jul 14, 2026

dd-octo-sts Bot added documentation integration/clickhouse labels Jul 14, 2026

jenny-chung changed the title ~~Jenny chung/clickhouse async inserts flush log~~ DBMON-6748 ClickHouse Async Inserts Monitoring Flush Log Collection Jul 14, 2026

jenny-chung added the qa/required QA is required for this PR and will generate a QA card label Jul 14, 2026

jenny-chung added 5 commits July 15, 2026 08:16

Refactor checkpoint tracking into NodeCheckpoint class

7cc5df7

Sync config models

96c44b6

Add changelog entry for async insert flush records

18b36c4

Collect async insert flush health in query completions job

786fe95

Rename latency_us to flush_latency_us

d686cbc

jenny-chung marked this pull request as ready for review July 15, 2026 17:28

jenny-chung requested review from a team as code owners July 15, 2026 17:28

dd-octo-sts Bot added team/agent-integrations team/database-monitoring-agent labels Jul 15, 2026

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

jenny-chung added 7 commits July 15, 2026 14:25

Add unit tests for async insert flush record collection

91d25a6

Add event date bound to query

24d2d2f

Add event date bound check to tests

ec06fee

Expand the wiring comment to explain why we collapse collection into …

0b07631

…query completions

Add optional enabled override so a subclass can run the shared job

d96c340

Allow flush log collection to run when query completions is disabled

b8e99f1

Add flush only unit test

d0e2ae5

sangeetashivaji reviewed Jul 16, 2026

View reviewed changes

jenny-chung added 6 commits July 17, 2026 11:17

Add service to async insert flush event

32a158a

Prevent negative flush latency from cross node clock skew

2a5ea02

Use a single time.time() call in flush log collection

bcce48e

Use async insert specific metric name for flush errors

52c4b33

Add dedicated max_flush_rows option for flush log collection

e35973d

Regenerate config models and example

240253e

eric-weaver reviewed Jul 17, 2026

View reviewed changes

		SECURE_FIELD_NAMES = frozenset(['tls_ca_cert'])


		class AsynchronousInsertFlushLog(BaseModel):

Uh oh!

Conversation

jenny-chung commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 14, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jenny-chung Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Validation Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jenny-chung commented Jul 14, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 14, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

jenny-chung Jul 17, 2026 •

edited

Loading