cre-5665: durable emitter regression test

[mchain0]

cre-5665

ref. full code fix and test is in PR: #2284

That's a regression test that presents a blocking within the durable emitter and how it breaks 2.54/2.55 vs 2.56/2.57.

Test passes. It clearly demonstrates the regression:

• Disabled (v2.54/v2.55 behavior): 5 emissions = 129µs (instant ErrNotInitialized no-op)
• Enabled (v2.56+ behavior): 5 emissions = 534ms (each blocks ~100ms on insert coalescer flush)

4113x slower

---

Investigation

The DurableEmitterEnabled config default was changed from false to true in core node v2.56.0-rc.0. This causes every workflow metric/telemetry emission to block on a synchronous Postgres INSERT through the durable emitter's insert coalescer, adding ~100ms+ per emission event to workflow execution time.

The following commits between v2.55.0-rc.0 and v2.56.0-rc.0 contributed to the regression:

1. 4e99fa3e76 - Durable Emitter: Load Testing Updates (#23001): Added explicit batch/timeout/retransmit config to application.go. Changed durable emitter from default config to custom config with InsertBatchSize=500, InsertBatchFlushInterval=100ms, BatchSize=1000, MaxPublishTimeout=10s, MessageBufferSize=50000.

2. 302b792a3a - Durable Emitter: TOML Configs (#23096): Made RetransmitBatchSize and EventTTL configurable via TOML. The default change for DurableEmitterEnabled from false to true was introduced in this commit's changes to config_telemetry.go.

3. 51cfafafd9 - Durable emitter- add max queue payload TOML configuration (#23081): Added MaxQueuePayloadBytes config (default 1 GiB).

4. chainlink-common bump (9264d4444ce0 -> 4033d0253977): Includes 3823ee79 (DurableEmitter Optimizations - mark-to-delete, fallback client removal) and 712bad3e (Resource Managers with durable metering for CRE).

Fix Options

Immediate Mitigation

Set DurableEmitterEnabled = false in the node's TOML config under [Telemetry]:

[Telemetry]
DurableEmitterEnabled = false

This restores the v2.55 behavior where GlobalEmit returns ErrNotInitialized immediately.

Proper Fix

Make durableemitter.GlobalEmit() non-blocking. The insert coalescer should queue the event and return immediately without waiting for the INSERT to complete. The current blocking behavior ("block until the multi-row INSERT completes") is inappropriate for the hot path of workflow metric emission.

Alternative Fix

Decouple the durable emitter from the workflow emission path. Instead of calling durableemitter.GlobalEmit() synchronously in emitRawMessage(), emit to the durable emitter asynchronously (e.g., via a goroutine or non-blocking channel send).

[github-actions]

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-common

View full report