feat(llmobs): implement multi-tenant routing context support #7158

PROFeNoM · 2025-12-23T08:33:31Z

What does this PR do?

Adds multi-tenant API key support for LLM Observability, enabling platforms to route LLMObs telemetry to different Datadog organizations from a single process.

New API:

await llmobs.withRoutingContext({ ddApiKey, ddSite }, async () => {
  // All LLMObs spans here route to the specified Datadog org
})

Motivation

Multi-tenant AI platforms need to send LLM Observability data to their customers' Datadog organizations. This enables:

Customer-specific trace visibility
Data isolation between tenants
Compliance with data residency requirements (different DD sites per customer)

Implementation

  withRoutingContext({ ddApiKey, ddSite }, fn)
      ↓ AsyncLocalStorage
  registerLLMObsSpan()
      ↓ captures routing to span tags
  span.finish()
      ↓
  SpanProcessor.process()
      ↓ extracts routing from tags
  writer.append(event, routing)
      ↓ routes to correct buffer
  setInterval flush()
      ↓ flushes each buffer to its endpoint

Key Changes:

withRoutingContext(options, fn) method added to LLMObs SDK (uses AsyncLocalStorage)
Routing context captured at span creation time
BaseLLMObsWriter refactored to multi-buffer architecture (one buffer per API key)
Each buffer flushes independently to its corresponding endpoint
Agentless mode only

Testing

writers/multi-tenant.spec.js - Tests for buffer routing, endpoint construction, API key isolation, context nesting, and concurrency

codecov · 2025-12-23T08:34:01Z

Codecov Report

❌ Patch coverage is 47.88732% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.34%. Comparing base (39fb819) to head (9615841).
⚠️ Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/writers/base.js	52.00%	24 Missing ⚠️
packages/dd-trace/src/llmobs/sdk.js	0.00%	9 Missing ⚠️
packages/dd-trace/src/llmobs/tagger.js	40.00%	3 Missing ⚠️
packages/dd-trace/src/llmobs/noop.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7158      +/-   ##
==========================================
- Coverage   84.49%   84.34%   -0.16%     
==========================================
  Files         524      524              
  Lines       22463    22439      -24     
==========================================
- Hits        18981    18927      -54     
- Misses       3482     3512      +30

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

PROFeNoM · 2025-12-23T08:34:15Z

@codex review

github-actions · 2025-12-23T08:34:18Z

Overall package size

Self size: 4.38 MB
Deduped: 5.2 MB
No deduping: 5.2 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.0 | 68.46 kB | 797.03 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

chatgpt-codex-connector · 2025-12-23T08:34:20Z

To use Codex here, create a Codex account and connect to github.

pr-commenter · 2025-12-23T08:53:02Z

Benchmarks

Benchmark execution time: 2026-01-08 14:35:27

Comparing candidate commit 9615841 in PR branch alex/MLOB-4999_multi-tenant-routing-context-support with baseline commit 39fb819 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 290 metrics, 30 unstable metrics.

…f no routing context is available.

…hance withRoutingContext documentation; add test for empty ddApiKey

BridgeAR · 2026-01-05T20:53:36Z

packages/dd-trace/src/llmobs/writers/base.js

    this._eventType = eventType

-    this._buffer = []
+    this._buffers = new Map()


Please use private properties instead of underscores for all changed properties and new ones. Changing old ones is nice to have as separate commit :)

Please also add a comment to our Claude.md file that highlights this. Otherwise new code changes will likely come up by the AI that use the underscore instead.

Done. Converted to # private syntax where possible (#buffers, #getMaskedRoutingKey, #cleanupEmptyBuffers, #getUrlForRouting)

Kept _getRoutingKey and _getOrCreateBuffer with underscore because LLMObsSpanWriter (subclass) needs to access them.

Also added the convention to AGENTS.md (symlink to claude.md ?) under "Class Properties and Methods".

FYI, I did keep things like get _buffer, set _buffer with the underscore because they are backward compatibility shims. You were not thinking about changing them as well, right?

…S.md

sabrenner

mostly small comments but one larger implementation comment i'm curious on your thoughts on. but overall the logic makes sense to me and the sdk api change looks good pending some type definition updates!

packages/dd-trace/src/llmobs/routing-context.js

packages/dd-trace/src/llmobs/sdk.js

packages/dd-trace/src/llmobs/routing-context.js

packages/dd-trace/src/llmobs/tagger.js

packages/dd-trace/src/llmobs/writers/base.js

packages/dd-trace/test/llmobs/routing-context.spec.js

packages/dd-trace/test/llmobs/writers/multi-tenant.spec.js

…LMObs class, remove legacy routing-context module, and enhance multi-tenant event handling with isolated buffers

sabrenner · 2026-01-07T14:50:45Z

packages/dd-trace/src/llmobs/sdk.js

+    if (!options?.ddApiKey) {
+      throw new Error('ddApiKey is required for routing context')
+    }
+    const currentStore = storage.getStore()


what if someone nests routing contexts

llmobs.withRoutingContext({ ...tenant1Options }, () => { // create span1 llmobs.withRoutingContext({ ...tenant2Options }, () => { // create span2 }) })

i think as it is now, span1 would be routed to tenant1, and span2 would be routed to tenant2, although I'm not sure if a user would expect that, or for span2 to be sent to both tenant1 and tenant2.

maybe we can/should throw if there is already an active routing context? lmk what you think, but i think it could discourage the above if it's not a situation we want to promote. we can always relax that restraint in the future, but maybe we wanna make sure users get a good understanding & signals from our API.

but lmk if you disagree, i'm also fine to keep it as is, maybe with a warning log or smth but still going through with doing the nested routing context

Your understanding is correct. As shown in the nested contexts override outer and restore after test, the inner routing context overrides the outer one, and restores after exit. It can indeed be considered a "silent override".

I personally believe that sending to both is problematic. It's complex, potentially dangerous (more easily to do an accidental data leakage), and harder (from a DEV-X) to reason about imo.

I believe that temporarily switching routing for a subset of operation can be a valid use case, hence why I believe that throwing may not be appropriate.
However, it may indeed be useful to provide visibility and to alert users of a potentially unintended behavior.

Right now, based on your suggestion, I've added a warning message:

Nested routing context detected. Inner context (****-key2) will override outer context (****-key1). Spans created in the inner context will only be sent to the inner context.

WDYT?

I personally believe that sending to both is problematic. It's complex, potentially dangerous (more easily to do an accidental data leakage), and harder (from a DEV-X) to reason about imo.

+1

I believe that temporarily switching routing for a subset of operation can be a valid use case, hence why I believe that throwing may not be appropriate.

ok sounds good! i think a warning log can be fine, and if it become more popular of a use case we can demote it to a debug log or something since it might be nice to have some kind of logged signal still in that case. but im good with a warning log 😄

but i think we can remove the redacted api key bits, idt it's needed for this warning log, but lmk what you think. i think just

[LLM Observability] Nested routing context detected. Inner context will override outer context. Spans created in the inner context will only be sent to the inner context.

is good

packages/dd-trace/src/llmobs/writers/base.js

sabrenner

left a question on the test file added!

sabrenner · 2026-01-07T16:01:07Z

packages/dd-trace/src/llmobs/sdk.js

+      const outerKey = currentStore.routingContext.apiKey
+      const innerKey = options.ddApiKey
+      const maskedOuter = outerKey ? `****${outerKey.slice(-4)}` : ''
+      const maskedInner = innerKey ? `****${innerKey.slice(-4)}` : ''


per my response, if you're ok with it, i think we can forgo logging redacted api keys, and just do the log

sabrenner · 2026-01-07T16:02:49Z

packages/dd-trace/test/llmobs/writers/multi-tenant.spec.js

+    const { storage } = require('../../../src/llmobs/storage')
+    let sdkLogger
+
+    function withRoutingContext (options, fn) {


can these tests be done against the actual sdk? i don't think it's right to re-define the function here and use it in the tests below

… and update tests to reflect changes

AGENTS.md

Co-authored-by: Ruben Bridgewater <ruben@bridgewater.de>

…ions and context handling

PROFeNoM added the semver-minor label Dec 23, 2025

This comment has been minimized.

Sign in to view

PROFeNoM mentioned this pull request Dec 24, 2025

feat(llmobs): implement multi-tenant routing context support DataDog/dd-trace-py#15781

Open

PROFeNoM added 5 commits January 5, 2026 10:15

feat(routing): implement multi-tenant routing context support

d907663

Removed default parameters from getCurrentRouting, now returns null i…

4ff94ff

…f no routing context is available.

restore comment + use size AFTER truncation to make flushing decision

a0361b5

refactor(routing-context): simplify getCurrentRouting function and en…

0c15cfd

…hance withRoutingContext documentation; add test for empty ddApiKey

Backward compat comment + Nested context test

2e4d793

PROFeNoM force-pushed the alex/MLOB-4999_multi-tenant-routing-context-support branch from cd8146f to b34e1f5 Compare January 5, 2026 09:16

fix: API Key Logged in Debug Output

e531090

PROFeNoM force-pushed the alex/MLOB-4999_multi-tenant-routing-context-support branch from b34e1f5 to e531090 Compare January 5, 2026 09:36

chore: remove unnecessary comments

3f33027

PROFeNoM added the llm-observability label Jan 5, 2026

PROFeNoM marked this pull request as ready for review January 5, 2026 09:47

PROFeNoM requested a review from a team as a code owner January 5, 2026 09:47

BridgeAR reviewed Jan 5, 2026

View reviewed changes

Converted to private # syntax, revised tests, and added rule to AGENT…

2496508

…S.md

PROFeNoM requested a review from a team as a code owner January 6, 2026 09:15

PROFeNoM added 4 commits January 6, 2026 10:21

fix

efe918d

#originalEndpoint

185005c

remove some redundancies

edb9db6

add comment for legacy

e60c56f

sabrenner self-requested a review January 6, 2026 14:42

sabrenner reviewed Jan 6, 2026

View reviewed changes

PROFeNoM marked this pull request as draft January 7, 2026 10:42

PROFeNoM added 3 commits January 7, 2026 14:57

refactor(routing-context): implement withRoutingContext directly in L…

5348c0d

…LMObs class, remove legacy routing-context module, and enhance multi-tenant event handling with isolated buffers

refactor

0c444cf

refactor

27759f8

sabrenner reviewed Jan 7, 2026

View reviewed changes

packages/dd-trace/src/llmobs/writers/base.js Show resolved Hide resolved

warn + private methods

99b53b2

PROFeNoM marked this pull request as ready for review January 7, 2026 15:32

sabrenner reviewed Jan 7, 2026

View reviewed changes

refactor(llmobs): simplify warning message for nested routing context…

4495ba2

… and update tests to reflect changes

BridgeAR reviewed Jan 8, 2026

View reviewed changes

AGENTS.md Outdated Show resolved Hide resolved

Update AGENTS.md

4bd005e

Co-authored-by: Ruben Bridgewater <ruben@bridgewater.de>

BridgeAR mentioned this pull request Jan 8, 2026

chore: add vscode settings to enforce eslint to be default editor #7199

Merged

test(llmobs): enhance multi-tenant routing tests with improved assert…

9615841

…ions and context handling

feat(llmobs): implement multi-tenant routing context support #7158

Are you sure you want to change the base?

feat(llmobs): implement multi-tenant routing context support #7158

Uh oh!

Conversation

PROFeNoM commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Implementation

Testing

Uh oh!

codecov bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

PROFeNoM commented Dec 23, 2025

Uh oh!

github-actions bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

chatgpt-codex-connector bot commented Dec 23, 2025

Uh oh!

This comment has been minimized.

pr-commenter bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

PROFeNoM commented Dec 23, 2025 •

edited

Loading

codecov bot commented Dec 23, 2025 •

edited

Loading

github-actions bot commented Dec 23, 2025 •

edited

Loading

pr-commenter bot commented Dec 23, 2025 •

edited

Loading