Add LLMObs.submitFeedback for end-user feedback submission by ddog-thibault-nadin · Pull Request #12129 · DataDog/dd-trace-java

ddog-thibault-nadin · 2026-08-03T14:34:02Z

What does this PR do?

Adds llmobs.submitFeedback(options) for submitting end-user feedback through the Node.js SDK.
Today this is only reachable by calling the Evaluations API directly. (requested by customers)

Testing

Apart from passing new and old tests, I also created a demo app to confirm that things are working.

Here is a span with feedback and evals submitted to it, using the NodeJS SDK.

here is a submitted feedback attached to the key support-ticket-4242 and not a span

Ports the end-user feedback SDK from dd-trace-py (#19347) and dd-trace-js (#9654) to Java. Feedback is submitted through a builder rather than overloads: with 4 target kinds, 5 value types and 3 optional fields, an overload-based API would need ~25 signatures. The builder also makes the runtime type/value coherence checks that Python and Node.js need statically impossible, since each value type has its own setter. Feedback is a v2 eval-metric concept -- event_kind, submitter and the non-score value types only exist there -- while Java evaluations post to v1. Since one batch maps to one HTTP request, the two cannot share a worker, so EvalProcessingWorker is generalised into LLMObsIntakeWorker<T> parameterised by API path, thread and batch serializer. The eval path keeps every parameter it had (queue capacity, flush interval, flush threshold, retry policy, URL and header construction) and its payload is byte-for-byte identical; only the log message wording differs, now that it is templated per payload kind. LLMObsEvalTest pins the v1 payload against regression. submitFeedback goes through its own LLMObsFeedbackProcessor interface rather than a new method on LLMObsEvalProcessor, to avoid breaking binary compatibility for external implementors of a published interface.

datadog-official · 2026-08-03T14:42:01Z

🎯 Code Coverage (details)
• Patch Coverage: 71.79%
• Overall Coverage: 58.01% (+0.11%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 894040e | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-08-03T14:53:54Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.99 s	13.95 s	[-0.5%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.88 s	12.98 s	[-1.9%; +0.3%] (no difference)
startup:petclinic:appsec:Agent	16.96 s	16.76 s	[+0.3%; +2.0%] (maybe worse)
startup:petclinic:iast:Agent	17.01 s	16.86 s	[-0.1%; +1.8%] (no difference)
startup:petclinic:profiling:Agent	16.16 s	16.90 s	[-9.8%; +1.1%] (unstable)
startup:petclinic:sca:Agent	16.74 s	16.56 s	[+0.3%; +1.9%] (maybe worse)
startup:petclinic:tracing:Agent	16.09 s	16.16 s	[-1.2%; +0.3%] (no difference)

Commit: 894040ed · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

datadog-official

The new feedback score builder accepts NaN and infinity, but the Moshi serializer rejects non-finite JSON numbers. A single such feedback item makes the intake worker throw before its network error handling and terminate, preventing subsequent feedback from being submitted; the builder should reject non-finite scores at the call site.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit fcd1138 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-official · 2026-08-04T14:26:15Z

+      public Builder scoreValue(double value) {
+        return value(MetricType.SCORE, value);
+      }


Reject non-finite feedback scores

One malformed numeric feedback can permanently stop the dedicated feedback worker and cause later end-user feedback to be silently stranded until the process restarts.

Assertion details

Input: A caller builds feedback with scoreValue(Double.NaN) or scoreValue(Double.POSITIVE_INFINITY), then the enabled agent flushes the feedback queue.

Expected: The public builder rejects a non-finite score, or the worker safely handles the invalid item without terminating feedback submission.

Actual: scoreValue accepts NaN and infinity. During LLMObsFeedbackEvent serialization, Moshi's JSON writer rejects the non-finite number with IllegalArgumentException. Serialization occurs before the try/catch in LLMObsIntakeWorker.SerializingHandler.flushIfNecessary(), and run() only catches InterruptedException, so the feedback worker thread exits and queued/subsequent feedback is no longer submitted.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

Suggested change

public Builder scoreValue(double value) {

return value(MetricType.SCORE, value);

}

public Builder scoreValue(double value) {

if (!Double.isFinite(value)) {

throw new IllegalArgumentException("score value must be finite");

}

return value(MetricType.SCORE, value);

}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcd1138fe2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-04T14:33:04Z

+      this.targetValue = builder.targetValue;
+      this.label = builder.label;
+      this.metricType = builder.metricType;
+      this.value = builder.value;


Snapshot JSON feedback values before enqueueing

When callers use .jsonValue(details), this stores the caller-owned Map directly in the supposedly immutable Feedback. Since submitFeedback enqueues the Feedback for later serialization on the feedback worker, any mutation to that map after build() or immediately after submission changes what is sent to intake (and concurrent mutation can make serialization fail). Copy and wrap JSON map values the same way tags are copied so the submitted feedback is a stable snapshot.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:33:04Z

      submissionUrl =
          HttpUrl.get(
-              sco.agentUrl.toString()
-                  + DDAgentFeaturesDiscovery.V2_EVP_PROXY_ENDPOINT
-                  + EVAL_METRIC_API_PATH);
-      headers = Headers.of(EVP_SUBDOMAIN_HEADER_NAME, EVAL_METRIC_API_DOMAIN);
+              sco.agentUrl.toString() + DDAgentFeaturesDiscovery.V2_EVP_PROXY_ENDPOINT + apiPath);


Use the shared agent client for feedback proxy submissions

In non-agentless configurations that use a Unix domain socket or named pipe for the Agent, sco.agentUrl is only a placeholder URL while the real socket transport is configured on sco.agentHttpClient. This worker still builds the feedback proxy URL from that placeholder and sends it with its own raw OkHttpClient, so submitFeedback cannot reach the Agent in those supported setups; use the shared agent client (and the shared intake client for agentless) when posting these batches.

Useful? React with 👍 / 👎.

I think was already there, proposed a fix

ddog-thibault-nadin added tag: ai generated Largely based on code generated by an AI or LLM comp: mlobs ML Observability (LLMObs) type: feature Enhancements and improvements labels Aug 3, 2026

ddog-thibault-nadin requested a review from sabrenner August 4, 2026 14:20

ddog-thibault-nadin marked this pull request as ready for review August 4, 2026 14:22

ddog-thibault-nadin requested review from a team as code owners August 4, 2026 14:22

ddog-thibault-nadin requested review from aniszoubiramar and manuel-alvarez-alvarez August 4, 2026 14:22

datadog-official Bot reviewed Aug 4, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Aug 4, 2026

View reviewed changes

ddog-thibault-nadin added 4 commits August 4, 2026 18:01

Align submitFeedback with the dd-trace-py implementation

8f65747

Align the dotted label error message with dd-trace-js

2407617

Keep the intake worker alive when a batch cannot be serialized

4b96f54

Reject non-finite feedback scores and snapshot json values

894040e

ddog-thibault-nadin mentioned this pull request Aug 6, 2026

fix(llmobs): submit eval metrics through the shared HTTP clients #12154

Open

Navigation Menu

Uh oh!

Add LLMObs.submitFeedback for end-user feedback submission - #12129

Add LLMObs.submitFeedback for end-user feedback submission#12129
ddog-thibault-nadin wants to merge 5 commits into
masterfrom
thibault/llmobs-submit-feedback

ddog-thibault-nadin commented Aug 3, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Aug 3, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

dd-octo-sts Bot commented Aug 3, 2026 •

edited

Loading

Uh oh!

datadog-official Bot left a comment

Uh oh!

datadog-official Bot Aug 4, 2026

Uh oh!

ddog-thibault-nadin Aug 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

ddog-thibault-nadin Aug 5, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

ddog-thibault-nadin Aug 5, 2026 •

edited

Loading

Uh oh!

ddog-thibault-nadin Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ddog-thibault-nadin commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Testing

Uh oh!

datadog-official Bot commented Aug 3, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-official Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

ddog-thibault-nadin Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

ddog-thibault-nadin Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

ddog-thibault-nadin Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddog-thibault-nadin Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ddog-thibault-nadin commented Aug 3, 2026 •

edited

Loading

datadog-official Bot commented Aug 3, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

dd-octo-sts Bot commented Aug 3, 2026 •

edited

Loading

ddog-thibault-nadin Aug 5, 2026 •

edited

Loading