OEV-779 Send custom event on atlas error by cll-gg · Pull Request #339 · smartcontractkit/chainlink-evm

cll-gg · 2026-01-22T15:01:15Z

Depends on

feat(svr): add FastLaneAtlasError proto and schema registry workflow chainlink-protos#270
fix: use 'main' branch for prod schema registration credentials chainlink-protos#276
PR review: remove http status code + use uint64 for tx id chainlink-protos#277

github-actions · 2026-01-22T15:03:10Z

⚠️ API Diff Results - Breaking changes detected

📦 Module: `github-com-smartcontractkit-chainlink-evm`

🔴 Breaking Changes (1)

`pkg/txm/clientwrappers/dualbroadcast` (1)

NewMetaMetrics — Type changed:

func(
  - string
  + string, 
  + github.com/smartcontractkit/chainlink-common/pkg/logger.Logger
)
(*MetaMetrics, error)

📄 View full apidiff report

cll-gg · 2026-01-29T14:47:42Z

⚠️ API Diff Results - Breaking changes detected

📦 Module: github-com-smartcontractkit-chainlink-evm

🔴 Breaking Changes (1)

pkg/txm/clientwrappers/dualbroadcast (1)

NewMetaMetrics — Type changed:

func(

~~string~~

string,

github.com/smartcontractkit/chainlink-common/pkg/logger.Logger
)
(*MetaMetrics, error)
📄 View full apidiff report

No problem, NewMetaMetrics is only used in chainlink-evm, see here

Copilot

Pull request overview

This PR adds custom event emission for Atlas errors in the dual broadcast transaction manager. The changes enable tracking of FastLane Atlas errors through OpenTelemetry events sent to Beholder, providing better observability for Atlas-related failures.

Changes:

Added new emitAtlasError and emitAtlasErrorWithHttpStatusCode methods to emit structured error events
Updated MetaMetrics constructor to accept a logger parameter for logging error events
Integrated error event emission at transaction send request and operation failure points

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/txm/clientwrappers/dualbroadcast/meta_metrics.go	Added error emission methods and logger to MetaMetrics struct
pkg/txm/clientwrappers/dualbroadcast/meta_metrics_test.go	Added comprehensive tests for the new error emission functionality
pkg/txm/clientwrappers/dualbroadcast/meta_client.go	Integrated error emission calls at failure points
go.mod	Updated chainlink-protos dependency to include FastLaneAtlasError message
Makefile	Added help text comments to existing targets

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cll-gg · 2026-01-30T13:24:05Z

CI failures are due to smartcontractkit/chainlink-protos#264 not being merged, will fix that when it's merged.

dimriou · 2026-01-30T17:54:05Z

pkg/txm/clientwrappers/dualbroadcast/meta_metrics.go

+}
+
+// emitAtlasErrorWithHttpStatusCode emits an OTel event with an HTTP status code to track FastLane Atlas errors
+func (m *MetaMetrics) emitAtlasErrorWithHttpStatusCode(ctx context.Context, errType string, customURL *url.URL, cause error, httpStatusCode int, tx *types.Transaction) {


If the main code doesn't ever utilize the httpStatusCode param, is there a point for tests to test for it? I feel we can avoid the passthrough method on line 115 and instead have one function.

It's used in the Atlas error struct, see here

It's in the Atlas error struct but besides tests, is there another value passed other than -1?

Ah I see what you mean, good call 👍
I've removed the method and set the status code in the event to -1 for now, will remove the field from the schema in a separate PR.

dimriou · 2026-01-30T17:57:52Z

pkg/txm/clientwrappers/dualbroadcast/meta_metrics.go

+		UseProtoNames:   true,
+		EmitUnpopulated: true,
+	}.Format(msg)
+	m.lggr.Infow("[Beholder.emit]", "message", mStr, "attributes", attrKVs)


I'm on the fence as to whether this needs to be info or debug.

We should only get a lot of errors in case of erroneous circumstances, then I think it'd be good to have these logs

But since this is a node log, we get this information from other sources as well, so this just confirms that it sends the Beholder message on an info level. Anyway, as I mentioned, I don't have a strong preference so we can keep it like that.

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-02T16:54:34Z

pkg/txm/clientwrappers/dualbroadcast/meta_metrics.go

+		ErrorType:      errType,
+		ErrorMessage:   cause.Error(),
+		HttpStatusCode: int32(httpStatusCode),
+		TransactionId:  int64(tx.ID), //nolint:gosec // overflow is acceptable for telemetry


The comment states 'overflow is acceptable for telemetry' but doesn't explain the implications. Consider documenting what happens if overflow occurs and why it's acceptable, or what the expected range of tx.ID values is.

Suggested change

TransactionId: int64(tx.ID), //nolint:gosec // overflow is acceptable for telemetry

// We deliberately cast tx.ID to int64 for the telemetry-only TransactionId field.

// The underlying tx.ID type may be wider than int64, so in theory this cast can

// overflow and wrap according to Go's integer conversion rules. If that happens,

// only the TransactionId reported in telemetry is affected; it does not influence

// on-chain behaviour, database primary keys, or any safety-critical logic.

// In practice, tx.ID values are expected to remain within the int64 range for the

// lifetime of the service, so overflow is not expected in normal operation.

TransactionId: int64(tx.ID), //nolint:gosec // see comment above for justification

cll-gg added 2 commits January 22, 2026 14:57

First pass

32bffc3

Small update

2102173

cll-gg added 3 commits January 22, 2026 15:06

Add nonce as well

145b6fc

Add make help

21050d5

Use fastlane_atlas_error from chainlink-protos

bae4393

cll-gg added a commit to smartcontractkit/chainlink that referenced this pull request Jan 26, 2026

Use chainlink-evm smartcontractkit/chainlink-evm#339

33ab911

cll-gg mentioned this pull request Jan 26, 2026

Send custom event on atlas error and split MetaClient errors from RPC errors smartcontractkit/chainlink#20926

Merged

cll-gg added 6 commits January 26, 2026 14:46

use correct cl-protos version

2fd3d76

Update after seeing it in Staging

81b23b9

Merge branch 'develop' into OEV-779-add-metrics-to-ofa-traffic

4053d0b

Add more logging + unit tests

dcfb1e4

Move emitting to meta_metrics

c5cf2c8

Fix tests

3eeafb8

cll-gg changed the title ~~First pass~~ OEV-779 Send custom event on atlas error Jan 29, 2026

cll-gg added 2 commits January 29, 2026 15:07

Don't send double events

911ce05

Remove duplicate log line

7876bf8

alejoberardino previously approved these changes Jan 30, 2026

View reviewed changes

cll-gg marked this pull request as ready for review January 30, 2026 13:23

cll-gg requested review from a team and dimriou as code owners January 30, 2026 13:23

Copilot AI review requested due to automatic review settings January 30, 2026 13:23

cll-gg requested review from a team as code owners January 30, 2026 13:23

product-security-plaid-production bot requested review from RayXpub, george-dorin and silaslenihan January 30, 2026 13:23

Copilot AI reviewed Jan 30, 2026

View reviewed changes

dimriou reviewed Jan 30, 2026

View reviewed changes

cll-gg requested a review from dimriou February 2, 2026 15:55

Bump to cl-protos merged version

9234588

cll-gg dismissed alejoberardino’s stale review via 9234588 February 2, 2026 15:56

Bump cl-protos once more

97fa34b

cll-gg marked this pull request as draft February 2, 2026 16:41

cll-gg marked this pull request as ready for review February 2, 2026 16:50

cll-gg requested review from alejoberardino and Copilot February 2, 2026 16:50

Copilot AI reviewed Feb 2, 2026

View reviewed changes

cll-gg requested a review from a team February 2, 2026 17:41

No use for http status code

4fcc053

cll-gg mentioned this pull request Feb 3, 2026

PR review: remove http status code + use uint64 for tx id smartcontractkit/chainlink-protos#277

Merged

cll-gg added 2 commits February 3, 2026 12:54

Add comments related to smartcontractkit/chainlink-protos#277

95245ae

Use latest protobuf

06e7e56

dimriou approved these changes Feb 3, 2026

View reviewed changes

george-dorin approved these changes Feb 3, 2026

View reviewed changes

cll-gg merged commit f0190ba into develop Feb 3, 2026
33 checks passed

cll-gg deleted the OEV-779-add-metrics-to-ofa-traffic branch February 3, 2026 13:28

-		TransactionId:  int64(tx.ID), //nolint:gosec // overflow is acceptable for telemetry
+		// We deliberately cast tx.ID to int64 for the telemetry-only TransactionId field.
+		// The underlying tx.ID type may be wider than int64, so in theory this cast can
+		// overflow and wrap according to Go's integer conversion rules. If that happens,
+		// only the TransactionId reported in telemetry is affected; it does not influence
+		// on-chain behaviour, database primary keys, or any safety-critical logic.
+		// In practice, tx.ID values are expected to remain within the int64 range for the
+		// lifetime of the service, so overflow is not expected in normal operation.
+		TransactionId:  int64(tx.ID), //nolint:gosec // see comment above for justification

Conversation

cll-gg commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Depends on

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ API Diff Results - Breaking changes detected

📦 Module: github-com-smartcontractkit-chainlink-evm

🔴 Breaking Changes (1)

pkg/txm/clientwrappers/dualbroadcast (1)

Uh oh!

cll-gg commented Jan 29, 2026

⚠️ API Diff Results - Breaking changes detected

📦 Module: github-com-smartcontractkit-chainlink-evm

🔴 Breaking Changes (1)

pkg/txm/clientwrappers/dualbroadcast (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

cll-gg commented Jan 30, 2026

Uh oh!

dimriou Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

cll-gg Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

cll-gg Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cll-gg Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cll-gg commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

📦 Module: `github-com-smartcontractkit-chainlink-evm`

`pkg/txm/clientwrappers/dualbroadcast` (1)

📦 Module: `github-com-smartcontractkit-chainlink-evm`

`pkg/txm/clientwrappers/dualbroadcast` (1)

dimriou Jan 30, 2026 •

edited

Loading