Skip to content

fix(llmobs): submit eval metrics through the shared HTTP clients - #12154

Open
ddog-thibault-nadin wants to merge 1 commit into
masterfrom
thibault/llmobs-shared-agent-client
Open

fix(llmobs): submit eval metrics through the shared HTTP clients#12154
ddog-thibault-nadin wants to merge 1 commit into
masterfrom
thibault/llmobs-shared-agent-client

Conversation

@ddog-thibault-nadin

@ddog-thibault-nadin ddog-thibault-nadin commented Aug 6, 2026

Copy link
Copy Markdown

What Does This Do

Makes EvalProcessingWorker post through the HTTP clients held by SharedCommunicationObjects instead of a raw new OkHttpClient():

  • non-agentless → sco.agentHttpClient
  • agentless → sco.getIntakeHttpClient()

Motivation

When the Agent is reached over a Unix domain socket or a named pipe, SharedCommunicationObjects.parseAgentUrl only stores a placeholder URL:

if (agentUrl.startsWith("unix:")) {
  // provide placeholder agent URL, in practice we'll be tunnelling over UDS
  agentUrl = "http://" + config.getAgentHost() + ":" + config.getAgentPort();
}

The real transport is the socket factory configured on agentHttpClient. The worker built its own client, so it posted to that placeholder host:port where nothing listens: SubmitEvaluation silently reaches nobody in those supported setups.

The agentless branch is not broken in the same way, but its own client ignored getAgentTimeout() and forceClearTextHttpForIntakeClient. getIntakeHttpClient() honours both and is built without UDS/named pipe, which is what a direct intake call needs.

LLMObsSystem.start already calls sco.createRemaining(config) before constructing the worker, so both clients are initialised by then.

@ddog-thibault-nadin ddog-thibault-nadin added type: bug fix Bug fix comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM labels Aug 6, 2026
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 6, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 0.00%
Overall Coverage: 63.25% (+5.32%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aef47d2 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.08 s 13.97 s [-0.0%; +1.6%] (no difference)
startup:insecure-bank:tracing:Agent 12.86 s 13.10 s [-2.7%; -0.9%] (maybe better)
startup:petclinic:appsec:Agent 17.50 s 17.27 s [+0.2%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent 16.83 s 17.51 s [-8.3%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 17.50 s 17.48 s [-0.9%; +1.1%] (no difference)
startup:petclinic:sca:Agent 17.40 s 17.42 s [-1.0%; +0.8%] (no difference)
startup:petclinic:tracing:Agent 16.57 s 16.66 s [-1.6%; +0.5%] (no difference)

Commit: aef47d25 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@ddog-thibault-nadin
ddog-thibault-nadin marked this pull request as ready for review August 6, 2026 10:04
@ddog-thibault-nadin
ddog-thibault-nadin requested a review from a team as a code owner August 6, 2026 10:04

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aef47d25b2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

+ "/"
+ EVAL_METRIC_API_PATH);
headers = Headers.of(DD_API_KEY_HEADER_NAME, config.getApiKey());
httpClient = sco.getIntakeHttpClient();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep HTTPS eval submissions off the cleartext intake client

When DD_FORCE_CLEAR_TEXT_HTTP_FOR_INTAKE_CLIENT is enabled, getIntakeHttpClient() builds the shared intake client as cleartext-only via OkHttpUtils.buildHttpClient, but the agentless eval URL above is still hard-coded to https://api.<site>/.... OkHttp rejects HTTPS requests on a ConnectionSpec.CLEARTEXT-only client, so enabling that supported intake flag for another backend intake now disables LLMObs agentless eval submissions; use an HTTPS-capable client for HTTPS eval URLs or honor the configured LLMObs agentless URL before sharing this client.

Useful? React with 👍 / 👎.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The worker now receives the already-configured shared client in both agent and agentless modes, preserving UDS/named-pipe routing and intake timeout/clear-text settings. Focused compilation and the shared communication tests found no reproducible behavioral regression; no additional tests recommended because the available production lifecycle initializes the shared clients before the worker.

Was this helpful? React 👍 or 👎

📊 Validated against 2 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit aef47d2 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant