feat(reduce transform): add max_groups option to bound memory usage - #25972
Open
karan-vk wants to merge 1 commit into
Open
feat(reduce transform): add max_groups option to bound memory usage#25972karan-vk wants to merge 1 commit into
karan-vk wants to merge 1 commit into
Conversation
Adds a max_groups option to the reduce transform that limits the number of groups kept in memory at any one time. When a new group would exceed the limit, the group that has gone the longest without receiving an event is flushed downstream to make room. Each early flush increments the new reduce_max_groups_exceeded_total internal metric.
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
|
🔗 Commit SHA: b3f84a0 | Docs | Datadog PR Page | Give us feedback! |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
max_groupsoption to thereducetransform that limits the number of groups kept in memory at any one time. This is a safeguard against unbounded memory growth (and eventual OOM) whengroup_bycontains high-cardinality fields.When a new group would exceed the limit, the group that has gone the longest without receiving an event is flushed downstream to make room, so no data is dropped. Each early flush increments the new
reduce_max_groups_exceeded_totalinternal metric and logs a rate-limited warning.When
max_groupsis not set, behavior is unchanged.Vector configuration
How did you test this PR?
max_groups: 0is rejected at config parse time, and when the limit is exceeded the least recently updated group is flushed while events for existing groups do not trigger a flush.cargo test --lib transforms::reduce(17 passed)cargo clippy --lib --workspace -- -D warningscargo fmt --all -- --checkcargo vdev check eventscargo vdev build component-docsIs this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
reducetransform memory use #3027reducetransform #9498