feat(otel): system-tests for HTTP OpenTelemetry semantic conventions (DD_TRACE_OTEL_SEMANTICS_ENABLED) by khanayan123 · Pull Request #7139 · DataDog/system-tests

khanayan123 · 2026-06-16T02:11:19Z

What

system-tests coverage for the opt-in OpenTelemetry HTTP semantic-conventions feature (DD_TRACE_OTEL_SEMANTICS_ENABLED=true), across two export paths.

Spec: https://opentelemetry.io/docs/specs/semconv/http/http-spans/

Principle: the system-tests are the authoritative harness — assertions encode exactly what the spec requires; where a tracer diverges it's recorded as a per-language manifest gate.

Validates three tracer PRs, cross-checked against their actual code and run end-to-end against dd-trace-js#8933:

Two scenarios

Scenario	Path	Validates
`OTEL_SEMANTICS`	DD tracer → agent (v0.4/v0.5)	attribute names, values, mutual-exclusion, `http.route`, resource form, `error.type` — the primary customer path
`OTEL_SEMANTICS_OTLP`	DD tracer → OTLP export	attribute types (`status_code`/`server.port` = int; `method`/`url.*`/`server.address`/`user_agent.original` = string) — only possible on OTLP; the agent protocol flattens everything to meta-strings/metrics-numbers

tests/test_otel_http_semantics.py (DD-agent) and tests/test_otel_http_semantics_otlp.py (OTLP, via interfaces.open_telemetry.get_otel_spans()). A one-line proxy fix (otlp_v1.py) coerces OTLP intValue to a Python int so types are comparable across JSON/binary encodings.

Findings surfaced (for the tracer teams)

dd-trace-js #8933: emits http.response.status_code as a string; per semconv it must be an int. Caught by OTEL_SEMANTICS_OTLP (the DD-agent path masks it). Also: no unknown-method _OTHER normalization (and Node's HTTP parser rejects unknown methods at 400 with no span, so it isn't end-to-end testable on Node — gated incomplete).
dd-trace-dotnet #8791: keeps the URL path in the no-route resource (should be bare {method}); no _OTHER normalization; no error.type mapping.
dd-trace-java #11652: keeps the raw method in the resource for _OTHER (should be HTTP).

E2E verification (dd-trace-js#8933, express4)

OTEL_SEMANTICS: 17 passed, 2 xfailed (the _OTHER cases, gated).
OTEL_SEMANTICS_OTLP: 4 passed, 2 xfailed (the status_code-is-int finding, gated).
Local-only caveat: the weblog's native IAST/appsec layer crashes under x64-on-arm64 qemu, so local runs disable it; the OTel feature is unaffected. (Confirms it's an emulation artifact, not the feature.)

CI fix included

The "Test the test" job was failing because two manifest entries used bug (<free text>) — bug declarations require a JIRA ticket, so _ensure_jira_ticket_as_reason called pytest.exit() during collection and aborted the whole run (surfacing as an unrelated test_library_conf remote-config INTERNALERROR). Changed those two to missing_feature. TEST_THE_TEST now collects/deselects correctly (288 passed, 2311 deselected locally).

Verification

ruff clean; Manifest.validate() passes; test_the_test meta-tests pass; both scenarios collect + run e2e against the JS PR.

🤖 Generated with Claude Code

…tests Add an end-to-end scenario and cross-language tests asserting that HTTP server and client spans honor the OpenTelemetry HTTP semantic conventions when DD_TRACE_OTEL_SEMANTICS_ENABLED=true. - OTEL_SEMANTICS EndToEndScenario sets DD_TRACE_OTEL_SEMANTICS_ENABLED=true - tests/test_otel_http_semantics.py validates the OTel attribute names are emitted and the legacy Datadog names are absent (the flag is mutually exclusive), for both server (url.path/url.scheme) and client (url.full, no url.path/url.query) spans - manifests: dotnet and java left active; other languages marked missing_feature until their tracers ship the flag Validates DataDog/dd-trace-dotnet#8791 and DataDog/dd-trace-java#11652. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-16T02:11:44Z

CODEOWNERS have been resolved as:

tests/test_otel_http_semantics.py                                       @DataDog/system-tests-core
tests/test_otel_http_semantics_otlp.py                                  @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/proxy/traces/otlp_v1.py                                           @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-06-16T02:24:30Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 9 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-8.1 1

🧪 1 Test failed

tests.ffe.test_exposures.Test_FFE_Exposure_Events.test_ffe_multiple_remote_config_files[apache-mod-8.1] from system_tests_suite

AssertionError: Timed out waiting for exposure event for flags [&#39;test-flag-1&#39;, &#39;test-flag-2&#39;] and subject &#39;test-user-multi&#39;
assert False
 &#43;  where False = &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt;(&lt;function wait_for_exposure_event.&lt;locals&gt;.&lt;lambda&gt; at 0x7fb9515c6fc0&gt;, timeout=30)
 &#43;    where &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt; = AgentInterfaceValidator(&#39;agent&#39;).wait_for
 &#43;      where AgentInterfaceValidator(&#39;agent&#39;) = interfaces.agent

self = &lt;tests.ffe.test_exposures.Test_FFE_Exposure_Events object at 0x7fb97c1e4f20&gt;

    def test_ffe_multiple_remote_config_files(self):
        &#34;&#34;&#34;Test that FFE correctly handles multiple remote config files with different flags.&#34;&#34;&#34;
...

Testing the test | System Tests (php, prod) / End-to-end #1 / apache-mod-8.0-zts 1

🧪 1 Test failed

tests.ffe.test_exposures.Test_FFE_Exposure_Events.test_ffe_multiple_remote_config_files[apache-mod-8.0-zts] from system_tests_suite

AssertionError: Timed out waiting for exposure event for flags [&#39;test-flag-1&#39;, &#39;test-flag-2&#39;] and subject &#39;test-user-multi&#39;
assert False
 &#43;  where False = &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt;(&lt;function wait_for_exposure_event.&lt;locals&gt;.&lt;lambda&gt; at 0x7f6c3816ede0&gt;, timeout=30)
 &#43;    where &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt; = AgentInterfaceValidator(&#39;agent&#39;).wait_for
 &#43;      where AgentInterfaceValidator(&#39;agent&#39;) = interfaces.agent

self = &lt;tests.ffe.test_exposures.Test_FFE_Exposure_Events object at 0x7f6c5eedaab0&gt;

    def test_ffe_multiple_remote_config_files(self):
        &#34;&#34;&#34;Test that FFE correctly handles multiple remote config files with different flags.&#34;&#34;&#34;
...

DataDog/system-tests | ruby-app-deployment-mode.amd64.DOA9: [public.ecr.aws/lts/ubuntu:22.04, linux/amd64, 3.1.7]

View all 9 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 9347df6 | Docs | Datadog PR Page | Give us feedback!}

…ked vs tracer PRs Verified assertions against the actual implementations in dd-trace-dotnet#8791 (SpanMetadataOTelRules.cs, HttpOtelHelper.cs) and dd-trace-java#11652 (OtelHttpSemantics.java): - server url.query test (was http.query.string) — present when a query is sent - server error.type test on 5xx — set to the status-code string. java emits this; dd-trace-dotnet does not map error status to error.type, so it is gated missing_feature for dotnet (per-language manifest) - server client.address / network.peer.address validated only when present (Recommended-level), with a clarifying note on the requirement-level approach - left client server.port as an if-present check: dotnet omits default ports and java falls back to 80/443, so always-present would be wrong Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Round out the OTel HTTP semantic-convention coverage: - server http.route: asserts the low-cardinality route template (not the raw URL path), using the established /sample_rate_route/{i} endpoint. http.route is unchanged by the feature but the spec requires it be a template. - client error.type on a 5xx distant call (java emits; gated missing_feature for dd-trace-dotnet, which does not map error status to error.type). - network.peer.address / network.peer.port validated when present (Recommended) on both server and client spans. 15 tests total (9 server, 6 client). ruff + manifest validator + meta-tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s-meta ports Verified the resource/span-name behavior across all three tracer PRs (dd-trace-dotnet#8791, dd-trace-java#11652, dd-trace-js#8933): - test_resource_name: the server entry-span resource is the low-cardinality "{method} {http.route}", never the raw URL path. java rewrites the resource to this form; dotnet and nodejs leave the already-compliant DD resource unchanged. Expected value derived from the same span's http.route so per-tracer template syntax ({i} / {i:int} / :i) cancels out. Gated on the root web span so the java spring.handler child (also type==web) is excluded. - nodejs activated: dd-trace-js#8933 implements the full feature including error.type, so its missing_feature gates are removed (now dotnet+java+nodejs). - numeric attributes (server.port, network.peer.port) read from meta OR metrics: dd-trace-js routes numerics into metrics, java/dotnet keep them in meta strings. 16 tests total. ruff + manifest validator + meta-tests green; collect-only passes for both dotnet and nodejs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ation) Principle: the system-tests are the authoritative harness — assert exactly what the OTel HTTP spec requires and record tracer divergences as manifest gates rather than weakening the assertions. Behavior verified against all three PRs (dd-trace-dotnet#8791, dd-trace-java#11652, dd-trace-js#8933). - client server.port: now asserted present (Required per spec; the distant-call URL uses a non-default port so every compliant tracer emits it). - test_no_route_resource: resource is the bare "{method}", never the URL path. java + js compliant; dd-trace-dotnet keeps the path -> bug gate. - test_request_method_normalization: unknown method -> http.request.method=_OTHER + http.request.method_original. Only java implements it; dotnet + js -> missing_feature. - test_other_method_span_name: per spec the _OTHER resource must be "HTTP". No tracer complies yet (java keeps raw method -> bug; dotnet/js -> missing_feature) — kept as a fully-gated spec anchor that activates as tracers comply. 19 tests total. ruff + manifest validator + meta-tests green; collect-only passes for dotnet, java, and nodejs. Note: the unknown-method tests are the first in system-tests to send a non-standard verb; verified by static analysis, pending a live OTEL_SEMANTICS run to confirm each weblog produces a span for an unknown method (else gate that language incomplete_test_app). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Per the OTel HTTP spec, when http.request.method resolves to _OTHER the span name's method component MUST be the literal "HTTP", not the raw verb. Add OtelHttpSemantics.spanNameMethod() and use it when naming server and client spans, while the http.request.method=_OTHER tag and http.request.method_original keep the raw value. Verified against system-tests OTEL_SEMANTICS (DataDog/system-tests#7139): the previously-gated Test_HttpServerOtelSemantics::test_other_method_span_name now passes (19/19). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…query-drop) Running OTEL_SEMANTICS against dd-trace-js#8933 locally revealed that the express weblog's /make_distant_call drops the target URL's query string, so a distant call to /status?code=500 actually hit /status (no code) and returned 400. The PR was correct (it set error.type="400"); the test was hardcoded to "500". Fix: distant-call an unmatched route (/no_such_route_xyz -> 404, no query needed) and assert error.type equals the span's own status code for any 4xx/5xx client response. Verified end-to-end vs dd-trace-js#8933 (nodejs@6.0.0-pre, express4): 17 passed, 2 xfailed (the _OTHER method-normalization tests, correctly gated missing_feature for nodejs), 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…crash Two things: 1. OTEL_SEMANTICS_OTLP scenario — the OTel-semantics flag plus the APM_TRACING_OTLP exporter env, so HTTP attributes are validated as *typed* OpenTelemetry values (http.response.status_code / server.port as int; method / url.* / server.address / user_agent.original as string) which the Datadog agent protocol cannot represent. - tests/test_otel_http_semantics_otlp.py reads typed attributes via interfaces.open_telemetry.get_otel_spans(). - utils/proxy/traces/otlp_v1.py: coerce OTLP intValue to a Python int so the type is comparable regardless of JSON vs binary OTLP encoding. - Verified e2e vs dd-trace-js#8933 (nodejs@6.0.0-pre): server+client names and types pass; status_code-is-int xfails -> the PR emits http.response.status_code as a STRING instead of an int (finding for the JS PR; the DD-agent path masks this). - Gated the new classes missing_feature for languages where the feature is unimplemented (python/golang/php/ruby/rust) or OTLP is unverified (dotnet/java); nodejs active. 2. Fix the "Test the test" CI crash: two manifest entries used `bug (<text>)` with a non-JIRA reason. add_pytest_marker -> _ensure_jira_ticket_as_reason calls pytest.exit during collection, which aborted the whole TEST_THE_TEST run (surfacing as an unrelated test_library_conf remote-config INTERNALERROR). Changed them to `missing_feature` (dotnet test_no_route_resource, java test_other_method_span_name). TEST_THE_TEST now collects and deselects correctly (288 passed, 2311 deselected locally). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Addresses the repo policy (no new Spock tests): the new OTel semantic-convention logic now lives in the OtelHttpSemantics helper, covered by a JUnit5 OtelHttpSemanticsTest (method _OTHER normalization, "HTTP" span-name, url.full credential redaction, query/fragment stripping, default server.port, error.type). The Spock decorator test files are restored to their upstream state. The decorator flag-gating integration is covered end-to-end by system-tests (DataDog/system-tests#7139) and the serialized status-key rename by DDSpanContextTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ate OTLP int tests The HTTP tracer PR now emits http.response.status_code as a numeric metric (int-typed, so OTLP serializes it as intValue) rather than a meta string — which is what the spec requires. Update the DD-agent server/client test_response_status_code and test_error_type to read the value via the existing _numeric_tag(meta-or-metrics) helper instead of meta only, and compare error.type against the stringified int. Now that the tracer emits an int, the OTLP test_status_code_is_int checks pass, so drop their nodejs missing_feature gates. Verified by replay against the HTTP+DB tracer build: OTEL_SEMANTICS 19 passed, 2 xfailed (method-normalization gates) OTEL_SEMANTICS_OTLP 8 passed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…elopment dd-trace-dotnet branches merging

datadog-system-tests-org · 2026-06-18T16:49:03Z

⚠️ Warnings

🚦 5 Pipeline jobs failed

Testing the test | System Tests (nodejs_lambda, prod) / Build end-to-end (nodejs-alb)

Testing the test | System Tests (nodejs_lambda, prod) / Build end-to-end (nodejs-function-url)

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-8.1 1

View all 5 failed jobs.

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 9347df6 | Docs | Give us feedback!}

khanayan123 and others added 4 commits June 15, 2026 22:26

khanayan123 and others added 2 commits June 16, 2026 14:06

khanayan123 mentioned this pull request Jun 17, 2026

feat(otel): system-tests for DB (postgres) OpenTelemetry semantic conventions #7166

Draft

khanayan123 and others added 2 commits June 17, 2026 13:39

Update the dotnet manifest to run tests. This is dependent on the dev…

9347df6

…elopment dd-trace-dotnet branches merging

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(otel): system-tests for HTTP OpenTelemetry semantic conventions (DD_TRACE_OTEL_SEMANTICS_ENABLED)#7139

feat(otel): system-tests for HTTP OpenTelemetry semantic conventions (DD_TRACE_OTEL_SEMANTICS_ENABLED)#7139
khanayan123 wants to merge 9 commits into
mainfrom
ayan.khan/otel-http-semantics-system-tests

khanayan123 commented Jun 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jun 16, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

datadog-system-tests-org Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

khanayan123 commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Two scenarios

Findings surfaced (for the tracer teams)

E2E verification (dd-trace-js#8933, express4)

CI fix included

Verification

Uh oh!

github-actions Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jun 16, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

datadog-system-tests-org Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khanayan123 commented Jun 16, 2026 •

edited

Loading

github-actions Bot commented Jun 16, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jun 16, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

datadog-system-tests-org Bot commented Jun 18, 2026 •

edited

Loading