Skip to content

Add EVP flagevaluation system tests#7146

Merged
leoromanovsky merged 12 commits into
mainfrom
leo.romanovsky/ffe-evp-flagevaluation-system-tests
Jun 17, 2026
Merged

Add EVP flagevaluation system tests#7146
leoromanovsky merged 12 commits into
mainfrom
leo.romanovsky/ffe-evp-flagevaluation-system-tests

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Motivation

Server-side FFE EVP flagevaluation needs system-test coverage for the worker-visible event contract and SDK-side aggregation behavior before language-by-language rollout.

Changes

  • Adds tests/ffe/test_flag_eval_evp.py for basic EVP emission, repeated-count aggregation, bounded context, runtime default, load aggregation, bounded burst aggregation, high-cardinality aggregation, and production-cap degradation.
  • Adds batched /ffe request support to weblog fixtures so degradation is exercised with 10,050 evaluations without making the test an HTTP throughput benchmark.
  • Adds feature_flags_evp_flagevaluation as a distinct feature marker from dynamic evaluation, exposures, and OTel evaluation metrics.
  • Wires manifests so SDK languages keep this new test disabled for first review with missing_feature (FFL-2446); non-applicable proxy/lambda/OTel manifests are marked irrelevant.

Decisions

  • Tests validate visible EVP fields from /api/v2/flagevaluations without vendoring dd-source schemas.
  • OpenFeature reason is neither accepted in payloads nor used as an aggregation dimension.
  • targeting_rule is expected only when real targeting-rule metadata exists; current no-rule fixtures assert it is omitted.
  • Degradation is reached naturally by one request with 10,050 distinct evaluations for one flag; degraded events must omit context and targeting key while preserving visible flag, variant, allocation, and count fields.
  • System-tests do not force private SDK queue caps. The bounded burst and degradation tests exercise async handoff, aggregation, and degradation end-to-end; SDK-internal queue-overflow accounting remains SDK unit-test territory.
  • Language enablement happens in separate draft sibling PRs after each SDK has local passing evidence.

Evidence

Base PR checks on current head 2e7ad616bc525ee76698f34a686da549701b1914:

./run.sh TEST_THE_TEST tests/test_the_test/test_conventions.py::test_utils

Result: 1 passed in 0.74s.

Current manifest state:

  • golang, nodejs, python, ruby, java, .NET, php: tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446).
  • Proxy, lambda, and OTel-only manifests are marked irrelevant where the server-side EVP weblog scenario does not apply.

Validated sibling evidence includes PHP #7160: ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library php --weblog apache-mod-8.2 -k "test_flag_eval_evp" - PASS, 8 passed, 2577 deselected in 63.53s. Degradation payload from that run: events=10001, total=10050, full_total=10000, degraded_count=1, degraded_total=50.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/__init__.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/fixtures.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_kong.yml                                                  @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/envoy.yml                                                     @DataDog/system-tests-core
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/haproxy.yml                                                   @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/java_lambda.yml                                               @DataDog/system-tests-core
manifests/java_otel.yml                                                 @DataDog/system-tests-core
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/nodejs_lambda.yml                                             @DataDog/system-tests-core
manifests/nodejs_otel.yml                                               @DataDog/system-tests-core
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/python_lambda.yml                                             @DataDog/apm-python @DataDog/asm-python
manifests/python_otel.yml                                               @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/ruby_lambda.yml                                               @DataDog/system-tests-core
manifests/rust.yml                                                      @DataDog/apm-rust
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/test_exposures.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/test_flag_eval_metrics.py                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/dotnet/weblog/Controllers/FeatureFlagEvaluatorController.cs  @DataDog/apm-dotnet @DataDog/asm-dotnet @DataDog/system-tests-core
utils/build/docker/golang/app/_shared/common/ffe.go                     @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/java/spring-boot/src/main/java/com/datadoghq/system_tests/springboot/featureflag/FeatureFlagEvaluatorController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/express/app.js                                @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/php/common/ffe.php                                   @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/open_feature_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jun 16, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 39ce169 | Docs | Datadog PR Page | Give us feedback!

@leoromanovsky leoromanovsky marked this pull request as ready for review June 16, 2026 19:28
@leoromanovsky leoromanovsky requested review from a team as code owners June 16, 2026 19:28
@leoromanovsky leoromanovsky requested review from avara1986, cataphract, claponcet, jandro996, mhlidd and quinna-h and removed request for a team June 16, 2026 19:28
@leoromanovsky leoromanovsky requested review from dd-oleksii and sameerank and removed request for a team June 16, 2026 19:28

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29743d2271

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/ffe/test_flag_eval_evp.py Outdated
Comment thread manifests/python_lambda.yml
Comment thread manifests/ruby.yml
Comment thread utils/_features.py Outdated
Comment thread tests/ffe/test_flag_eval_evp.py Outdated
Comment thread tests/ffe/test_flag_eval_evp.py Outdated
@leoromanovsky leoromanovsky merged commit 14f65e4 into main Jun 17, 2026
701 checks passed
@leoromanovsky leoromanovsky deleted the leo.romanovsky/ffe-evp-flagevaluation-system-tests branch June 17, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants