compute: reject a negative record count in accumulable reduce - #38010
Open
def- wants to merge 1 commit into
Open
compute: reject a negative record count in accumulable reduce#38010def- wants to merge 1 commit into
def- wants to merge 1 commit into
Conversation
`AccumulableErrorCheck` accumulates a record count per key, and `finalize_accum` returns that count directly for `count(*)`. The check only ever compared the count against zero: one arm caught a net-zero count paired with a non-zero accumulation, and another caught negative accumulation for unsigned sums. A NEGATIVE count passed both, so it reached `finalize_accum` unexamined and surfaced as a user-visible negative number with no error and no log line. A materialized view over such an aggregate then persisted the nonsense value with a positive multiplicity, making it durable in a derived shard and indistinguishable from a real result. A well formed collection has non-negative multiplicities, so a negative count means the input is already corrupt. Report it as an error instead. The check is per key rather than per aggregate, because the record count is a property of the input records, not of any one aggregate. This is defense in depth, not a root cause fix: it only fires once something upstream has already driven a collection negative. Its value is turning silently wrong answers into a diagnosable error, which is exactly what has been missing when invalid retractions show up in production. Note the check is inherently per key. A global aggregate accumulates every record under one empty key, so an unmatched retraction there is absorbed into a positive total and stays invisible. Only grouped aggregates are covered. Extends test/testdrive/avro-cdcv2.td with a regression case that injects an unmatched retraction through `ENVELOPE MATERIALIZE`, the only source envelope that passes upstream diffs through verbatim, and asserts both that a grouped count errors and that the error propagates into a materialized view over it. A positive control covers real retractions, so the new arm cannot fire on well formed input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
def-
force-pushed
the
compute-negative-record-count
branch
from
August 3, 2026 08:37
c2450cd to
69dba7f
Compare
Contributor
|
Fwiw, worth taking some care here as this is also the mechanism that we use to diagnose flawed data. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AccumulableErrorCheckaccumulates a record count per key, andfinalize_accumreturns that count directly forcount(*). The check only ever compared the count against zero: one arm caught a net-zero count paired with a non-zero accumulation, and another caught negative accumulation for unsigned sums. A NEGATIVE count passed both, so it reachedfinalize_accumunexamined and surfaced as a user-visible negative number with no error and no log line. A materialized view over such an aggregate then persisted the nonsense value with a positive multiplicity, making it durable in a derived shard and indistinguishable from a real result.A well formed collection has non-negative multiplicities, so a negative count means the input is already corrupt. Report it as an error instead. The check is per key rather than per aggregate, because the record count is a property of the input records, not of any one aggregate.
This is defense in depth, not a root cause fix: it only fires once something upstream has already driven a collection negative. Its value is turning silently wrong answers into a diagnosable error, which is exactly what has been missing when invalid retractions show up in production.
Note the check is inherently per key. A global aggregate accumulates every record under one empty key, so an unmatched retraction there is absorbed into a positive total and stays invisible. Only grouped aggregates are covered.