Skip to content

feat(reduce transform): add max_groups option to bound memory usage - #25972

Open
karan-vk wants to merge 1 commit into
vectordotdev:masterfrom
karan-vk:reduce-max-groups
Open

feat(reduce transform): add max_groups option to bound memory usage#25972
karan-vk wants to merge 1 commit into
vectordotdev:masterfrom
karan-vk:reduce-max-groups

Conversation

@karan-vk

Copy link
Copy Markdown

Summary

Adds a max_groups option to the reduce transform that limits the number of groups kept in memory at any one time. This is a safeguard against unbounded memory growth (and eventual OOM) when group_by contains high-cardinality fields.

When a new group would exceed the limit, the group that has gone the longest without receiving an event is flushed downstream to make room, so no data is dropped. Each early flush increments the new reduce_max_groups_exceeded_total internal metric and logs a rate-limited warning.

When max_groups is not set, behavior is unchanged.

Vector configuration

transforms:
  my_reduce:
    type: reduce
    inputs: ["my_source"]
    group_by: ["request_id"]
    max_groups: 10000

How did you test this PR?

  • Added unit tests: max_groups: 0 is rejected at config parse time, and when the limit is exceeded the least recently updated group is flushed while events for existing groups do not trigger a flush.
  • cargo test --lib transforms::reduce (17 passed)
  • cargo clippy --lib --workspace -- -D warnings
  • cargo fmt --all -- --check
  • cargo vdev check events
  • Regenerated component docs with cargo vdev build component-docs

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Adds a max_groups option to the reduce transform that limits the number
of groups kept in memory at any one time. When a new group would exceed
the limit, the group that has gone the longest without receiving an
event is flushed downstream to make room. Each early flush increments
the new reduce_max_groups_exceeded_total internal metric.
@karan-vk
karan-vk requested review from a team as code owners July 30, 2026 03:20
@github-actions github-actions Bot added docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: transforms Anything related to Vector's transform components domain: external docs Anything related to Vector's external, public documentation and removed docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. labels Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 30, 2026

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b3f84a0 | Docs | Datadog PR Page | Give us feedback!

@karan-vk

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@pront pront added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation domain: transforms Anything related to Vector's transform components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Safeguard reduce transform memory use

2 participants