Add in LLM Observability Span Processor API by sabrenner · Pull Request #12112 · DataDog/dd-trace-java

sabrenner · 2026-07-30T14:19:45Z

What Does This Do

Adds in a user span processor API to process LLM Observability / Agent Observability spans before they are flushed to Datadog.

Motivation

Being able to redact input / output on certain spans, by analyzing tags on the LLM Observability span.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: MLOB-7902

…ner/llmobs-span-processor

sabrenner · 2026-07-30T14:20:43Z

@codex review

chatgpt-codex-connector · 2026-07-30T14:27:41Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 9f02ef6973

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

dd-octo-sts · 2026-07-30T14:38:20Z

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.87 s	14.72 s	[+0.0%; +1.9%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.63 s	13.73 s	[-1.8%; +0.2%] (no difference)
startup:petclinic:appsec:Agent	17.42 s	17.38 s	[-0.8%; +1.3%] (no difference)
startup:petclinic:iast:Agent	17.41 s	17.49 s	[-1.4%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	17.50 s	17.41 s	[-0.7%; +1.8%] (no difference)
startup:petclinic:sca:Agent	17.62 s	17.05 s	[+2.3%; +4.4%] (significantly worse)
startup:petclinic:tracing:Agent	16.11 s	16.61 s	[-7.2%; +1.1%] (no difference)

Commit: f02ee92d · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

sabrenner

self-review

sabrenner · 2026-08-03T17:46:41Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a6c3dfe35d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sabrenner · 2026-08-04T16:31:03Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: edca6229c6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sabrenner · 2026-08-04T16:54:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

dd-trace-java/dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java

Line 464 in 7d89086

for (LLMObs.Document document : documents) {

Validate retrieval document list elements

For retrieval spans where _ml_obs_tag.output is a List that is not actually List<LLMObs.Document>, this loop performs the unchecked element cast and throws ClassCastException during serialization. Since the branch only checks val instanceof List, a malformed or user-provided list can drop the LLMObs payload instead of being skipped/logged like the non-list case; verify the elements are documents before serializing them as documents.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sabrenner · 2026-08-05T14:17:06Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6492238d7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sabrenner · 2026-08-05T14:52:46Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6998e5192f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sabrenner · 2026-08-05T15:06:34Z

 public interface Mapper<T> {
  void map(T data, Writable packer);

+  default void map(T data, Writable packer, boolean retry) {


added this to help with knowing when mapping is retried for our LLMObsSpanMapper, as we do not want to re-trigger the user span-proessor in that case. I think I've set it up so that it's backwards compatible and CI seems to agree, but let me know if this is an issue!

sabrenner added 2 commits July 29, 2026 13:26

add in llmobs user span processor

1e94d01

Merge branch 'master' of github.com:DataDog/dd-trace-java into sabren…

9f02ef6

…ner/llmobs-span-processor

sabrenner added type: feature Enhancements and improvements comp: mlobs ML Observability (LLMObs) labels Jul 30, 2026

This comment has been minimized.

Sign in to view

sabrenner added 2 commits July 30, 2026 12:36

llmobsspandataadapter test coverage

9266b45

add llmobsinternal to internal-api build.gradle.kts

113074b

sabrenner commented Jul 31, 2026

View reviewed changes

self review fixes

a6c3dfe

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

sabrenner and others added 2 commits August 4, 2026 12:25

handle proper io type for null values of input/output

321245e

Merge branch 'master' into sabrenner/llmobs-span-processor

edca622

chatgpt-codex-connector Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanDataAdapter.java

address retrieval doc serialization

7d89086

chatgpt-codex-connector Bot reviewed Aug 4, 2026

View reviewed changes

further guards

e649223

chatgpt-codex-connector Bot reviewed Aug 5, 2026

View reviewed changes

apply further codex updates - clearing i/o, null error tags

6998e51

chatgpt-codex-connector Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java

Merge branch 'master' into sabrenner/llmobs-span-processor

f07c0f9

sabrenner commented Aug 5, 2026

View reviewed changes

Comment thread internal-api/src/test/groovy/datadog/trace/api/telemetry/LLMObsMetricCollectorTest.groovy

Apply suggestion from @sabrenner

f02ee92

sabrenner mentioned this pull request Aug 5, 2026

Add Support for Annotating Embedding and Retrieval Documents #12149

Open

Uh oh!

Conversation

sabrenner commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

sabrenner commented Jul 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 30, 2026

Uh oh!

dd-octo-sts Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Uh oh!

This comment has been minimized.

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabrenner commented Aug 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabrenner commented Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

sabrenner commented Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabrenner commented Aug 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabrenner commented Aug 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

sabrenner Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

sabrenner commented Jul 30, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 30, 2026 •

edited

Loading