Improving smoke test capabilities

[PerfectSlayer]

This PR was created to preview the end goal of the task. It is still early work and will be split into reviewable chunks once ready. Check additional notes for the PRs ready to review.

What Does This Do

This PR introduces smoke tests framework based on JUnit.

It provides the following capabilities:

• Register smoke applications
  • Either web server application or CLI application
  • Can be extended to new kind of applications too
  • Lifecycle handled as part of the JUnit engine
  • Registered for parameter resolution
• Provide full trace content and structure verification
  • With assert rules and matchers (similar to instrumentation testing)
• Receive traces from test agent
  • No longer test against our own instrumentation
  • Open to snapshot testing
  • Introduce test agent session support to tracer headers
• Rely on the existing CI test agent
  • Will run test agent locally if docker engine is found
  • Can run the old mock server as alternative when no docker env is available

Additionally, this PR ports two smoke tests as example:

• opentelemetry - a basic CLI app
• spring-boot-rabbit - a complex multiple apps / queues setup
• dynamic-config - a use case that relies on remote config push and poll

Motivation

Additional Notes

This will make some tests run with the test agent side car and enable trace invariant gate too (check_test_agent_results.sh).

Related pull-requests ready for review:

• Add test agent session token support #12050
• Avoid null collection on decoded spans #12087
• Add APM test agent trace decoding #12089
• Add decoded span assert rules #12088
• Add smoke test JUnit framework #12111
• Migrate OpenTelemetry smoke test to JUnit #12127
• Migrate Spring Boot RabbitMQ smoke test to JUnit #12128
• Migrate dynamic config smoke tests to JUnit #12148

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: APMLP-1247

[datadog-prod-us1-5]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 57.90% (-0.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0408389 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.63 s	[+0.1%; +2.1%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.65 s	13.71 s	[-1.4%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	16.28 s	16.79 s	[-7.1%; +1.0%] (no difference)
startup:petclinic:iast:Agent	17.00 s	17.01 s	[-0.9%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.77 s	16.89 s	[-1.9%; +0.4%] (no difference)
startup:petclinic:sca:Agent	16.92 s	16.62 s	[+0.7%; +2.9%] (maybe worse)
startup:petclinic:tracing:Agent	16.11 s	16.20 s	[-1.4%; +0.2%] (no difference)

Commit: 04083896 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[datadog-prod-us1-5]

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

Added EnabledIfDockerAvailable, the shared JUnit condition referenced by the migrated Docker-backed smoke tests. The condition enables those tests only when Testcontainers can connect to Docker.

Commit fix to this PR

---

View in Datadog | Reviewed commit 54102e9 · Any feedback? Reach out in #deveng-pr-agent