Skip to content

[python,nodejs] add llm observability parametric initial test#5957

Merged
sabrenner merged 20 commits intomainfrom
sabrenner/llmobs-tests
Jan 13, 2026
Merged

[python,nodejs] add llm observability parametric initial test#5957
sabrenner merged 20 commits intomainfrom
sabrenner/llmobs-tests

Conversation

@sabrenner
Copy link
Copy Markdown
Contributor

@sabrenner sabrenner commented Jan 7, 2026

Motivation

Consistency testing for LLM Observability SDK functions and behaviors.

MLOB-5162

Changes

Adds new parametric test (initial test, to keep scope small, will add more in a subsequent PR) for LLM Observability.

There is one (a bit larger) change which, while it doesn't affect other parametric tests or client implementations for interacting with the tracers, does introduce something non-standard.

  • LLM Observability provides tracing as part of its SDK (manual instrumentation for creating LLM Observability span events)
  • Most SDKs (Java, Node.js, Python) leverage automatic context management and propagation, like normal tracers do
  • Python, Java, and Go LLM Observability SDKs also provide users to start single spans and end them manually with context management. Node.js does not.
  • To close this gap, I've pasted over an implementation from our legacy shared test suite that
    • defines a "trace structure request" (ref), which highlights the desired trace structure for the http clients to recreate with the tracer. this is a dsl for creating traces, instead of manually creating spans and setting parentage in the tests directly, making auto-context-management easier to test.
    • tests would define a simple or complex trace structure to follow, and the clients would translate this into creating a trace with the desired attributes and annotation (LLM Observability concept, similar to tagging) behaviors
    • I moved this whole system over to system-tests, as to not have to rewrite tests

I'm totally down to change the naming of these test request structures, but I felt it important to try and retain them in migrating over to system tests. I know it breaks the pattern a bit, but I'm wondering since it doesn't encroach on existing tests and is fairly well-separated if it's OK to include 😄

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jan 7, 2026

CODEOWNERS have been resolved as:

tests/parametric/test_llm_observability.py                              @DataDog/ml-observability @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/llmobs.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/python/parametric/apm_test_client/llmobs.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/docker_fixtures/spec/llm_observability.py                         @DataDog/ml-observability @DataDog/system-tests-core
.github/CODEOWNERS                                                      @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/server.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/python/parametric/apm_test_client/server.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

@sabrenner sabrenner changed the title [python,nodejs] add llm observability parametric tests [python,nodejs] add llm observability parametric initial test Jan 12, 2026
@sabrenner
Copy link
Copy Markdown
Contributor Author

note for reviewers: only adding one test for now since this PR also introduces a DSL for making LLM Observability traces. I will add more tests in a subsequent PR

@sabrenner sabrenner marked this pull request as ready for review January 12, 2026 15:52
@sabrenner sabrenner requested review from a team as code owners January 12, 2026 15:52
@sabrenner sabrenner requested review from a team as code owners January 12, 2026 15:52
@sabrenner sabrenner requested review from Anilm3, daniel-romano-DD, link04, manuel-alvarez-alvarez, tabgok and taegyunkim and removed request for a team January 12, 2026 15:53
Comment thread .github/CODEOWNERS Outdated
Copy link
Copy Markdown
Collaborator

@cbeauchesne cbeauchesne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a small RC on a comment, then AGTM

Comment thread utils/docker_fixtures/_test_clients/_test_client_parametric.py
@cbeauchesne cbeauchesne dismissed their stale review January 13, 2026 14:51

Change has been applied

Copy link
Copy Markdown
Collaborator

@cbeauchesne cbeauchesne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGTM, but could you get a review from someone familiar with the tested feature ?

Comment thread .github/CODEOWNERS Outdated
@sabrenner sabrenner enabled auto-merge (squash) January 13, 2026 17:13
@sabrenner sabrenner merged commit 8952374 into main Jan 13, 2026
817 of 819 checks passed
@sabrenner sabrenner deleted the sabrenner/llmobs-tests branch January 13, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants