fix: guard against invalid span data in performance detectors#119467
fix: guard against invalid span data in performance detectors#119467billyvg wants to merge 1 commit into
Conversation
Fix KeyError in MNPlusOneDB detector when db spans lack a 'description' key (data scrubbing can drop this field), and fix ValueError in LargeHTTPPayload detector when encoded_body_size is a non-numeric string like '[NaN]' (produced by Relay data scrubbing). Fixes SENTRY-5QVD, Fixes SENTRY-5R9R Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014nHHCFa15GuZSXmrie1JtV
|
🚨 Warning: This pull request contains Frontend and Backend changes! It's discouraged to make changes to Sentry's Frontend and Backend in a single pull request. The Frontend and Backend are not atomically deployed. If the changes are interdependent of each other, they must be separated into two pull requests and be made forward or backwards compatible, such that the Backend or Frontend can be safely deployed independently. Have questions? Please ask in the |
c39be13 to
7c89a16
Compare
| @@ -255,7 +255,7 @@ def _maybe_performance_problem(self) -> PerformanceProblem | None: | |||
| return PerformanceProblem( | |||
| fingerprint=self._fingerprint(db_span["hash"], common_parent_span), | |||
There was a problem hiding this comment.
db_span["hash"] can raise KeyError on scrubbed spans, same pattern being fixed one line below
This line accesses db_span["hash"] directly while line 258 is being changed to .get("description", "") to avoid a KeyError on scrubbed spans; the same span can be missing hash, so the KeyError just moves to line 256. Use db_span.get("hash", "").
Evidence
_first_relevant_db_span()(line 291) returns a span filtered only onspan["op"].startswith("db"); it never verifies ahashkey exists.- The PR fixes
db_span["description"]->.get("description", "")on line 258 because Relay scrubbing can strip keys from the same span dict. _is_span_eligibleinlarge_http_payload_detector.pytreats hash as optional (span.get("hash", None)), and_fingerprintusesparent_span.get("hash") or "", showing the codebase already treatshashas possibly absent._fingerprintconcatenatesdb_hashinto a string, so a missing key raises KeyError at line 256 before the fixed line 258 is reached;.get("hash", "")avoids the crash.
Identified by Warden sentry-backend-bugs · 2VV-BT2
Backend Test FailuresFailures on
|
Fix two high-actionability production errors in the spans performance-problem detection pipeline, both triggered by missing or scrubbed span data.
Problem 1:
KeyError: 'description'— SENTRY-5QVD (285k events, high actionability)File:
src/sentry/issue_detection/detectors/mn_plus_one_db_span_detector.py:258The MN+1 DB detector accessed
db_span["description"]directly. Relay data scrubbing can strip this key entirely, causing aKeyErrorthat is caught by the outerdetect_performance_problemsexception handler and logged as "Failed to detect performance problems". With 285k events, this was the most impactful of the two bugs.Fix: Use
.get("description", "")instead.Problem 2:
ValueError: invalid literal for int() with base 10: '[NaN]'— SENTRY-5R9R (1.3k events, high actionability)File:
src/sentry/issue_detection/detectors/large_http_payload_detector.py:59The large HTTP payload detector converts
http.response_content_lengthfrom a string to an int when it arrives as a string. However, Relay data scrubbing can replace numeric values with the sentinel string'[NaN]', which is not a valid integer literal.Fix: Wrap the
int()conversion in atry/except ValueErrorand skip the span if conversion fails.Evidence
Both errors share the same culprit (
spans.consumers.process_segments.process_segment) and the same outer error message ("Failed to detect performance problems"), confirming they originate from the same exception-handling wrapper inperformance_detection.py.Session: https://claude.ai/code/session_014nHHCFa15GuZSXmrie1JtV
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
Generated by Claude Code