Skip to content

fix: guard against invalid span data in performance detectors#119467

Draft
billyvg wants to merge 1 commit into
masterfrom
claude/serene-cannon-j60bqn
Draft

fix: guard against invalid span data in performance detectors#119467
billyvg wants to merge 1 commit into
masterfrom
claude/serene-cannon-j60bqn

Conversation

@billyvg

@billyvg billyvg commented Jul 12, 2026

Copy link
Copy Markdown
Member

Fix two high-actionability production errors in the spans performance-problem detection pipeline, both triggered by missing or scrubbed span data.

Problem 1: KeyError: 'description'SENTRY-5QVD (285k events, high actionability)

File: src/sentry/issue_detection/detectors/mn_plus_one_db_span_detector.py:258

The MN+1 DB detector accessed db_span["description"] directly. Relay data scrubbing can strip this key entirely, causing a KeyError that is caught by the outer detect_performance_problems exception handler and logged as "Failed to detect performance problems". With 285k events, this was the most impactful of the two bugs.

Fix: Use .get("description", "") instead.

Problem 2: ValueError: invalid literal for int() with base 10: '[NaN]'SENTRY-5R9R (1.3k events, high actionability)

File: src/sentry/issue_detection/detectors/large_http_payload_detector.py:59

The large HTTP payload detector converts http.response_content_length from a string to an int when it arrives as a string. However, Relay data scrubbing can replace numeric values with the sentinel string '[NaN]', which is not a valid integer literal.

Fix: Wrap the int() conversion in a try/except ValueError and skip the span if conversion fails.

Evidence

Both errors share the same culprit (spans.consumers.process_segments.process_segment) and the same outer error message ("Failed to detect performance problems"), confirming they originate from the same exception-handling wrapper in performance_detection.py.

Session: https://claude.ai/code/session_014nHHCFa15GuZSXmrie1JtV

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.


Generated by Claude Code

Fix KeyError in MNPlusOneDB detector when db spans lack a 'description'
key (data scrubbing can drop this field), and fix ValueError in
LargeHTTPPayload detector when encoded_body_size is a non-numeric string
like '[NaN]' (produced by Relay data scrubbing).

Fixes SENTRY-5QVD, Fixes SENTRY-5R9R

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014nHHCFa15GuZSXmrie1JtV
@github-actions github-actions Bot added Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components labels Jul 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🚨 Warning: This pull request contains Frontend and Backend changes!

It's discouraged to make changes to Sentry's Frontend and Backend in a single pull request. The Frontend and Backend are not atomically deployed. If the changes are interdependent of each other, they must be separated into two pull requests and be made forward or backwards compatible, such that the Backend or Frontend can be safely deployed independently.

Have questions? Please ask in the #discuss-dev-infra channel.

@billyvg billyvg force-pushed the claude/serene-cannon-j60bqn branch from c39be13 to 7c89a16 Compare July 12, 2026 02:58
@@ -255,7 +255,7 @@ def _maybe_performance_problem(self) -> PerformanceProblem | None:
return PerformanceProblem(
fingerprint=self._fingerprint(db_span["hash"], common_parent_span),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

db_span["hash"] can raise KeyError on scrubbed spans, same pattern being fixed one line below

This line accesses db_span["hash"] directly while line 258 is being changed to .get("description", "") to avoid a KeyError on scrubbed spans; the same span can be missing hash, so the KeyError just moves to line 256. Use db_span.get("hash", "").

Evidence
  • _first_relevant_db_span() (line 291) returns a span filtered only on span["op"].startswith("db"); it never verifies a hash key exists.
  • The PR fixes db_span["description"] -> .get("description", "") on line 258 because Relay scrubbing can strip keys from the same span dict.
  • _is_span_eligible in large_http_payload_detector.py treats hash as optional (span.get("hash", None)), and _fingerprint uses parent_span.get("hash") or "", showing the codebase already treats hash as possibly absent.
  • _fingerprint concatenates db_hash into a string, so a missing key raises KeyError at line 256 before the fixed line 258 is reached; .get("hash", "") avoids the crash.

Identified by Warden sentry-backend-bugs · 2VV-BT2

@github-actions

Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on bac4f12 in this run:

tests/sentry/issue_detection/test_m_n_plus_one_db_detector.py::MNPlusOneDBDetectorTest::test_handles_db_spans_missing_descriptionlog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/issue_detection/test_m_n_plus_one_db_detector.py:190: in test_handles_db_spans_missing_description
    assert len(problems) == 1
E   assert 0 == 1
E    +  where 0 = len([])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants