fix(spans): use decompressed bytes size for oversized segment check

[lvthanh03]

In add-buffer, we compared the compressed segment sizes with max_segment_bytes, but when flushing, we use the decompressed segment size to determine whether we should drop the segment. So, we don't see spop called when the compressed size is under 20MiB but the uncompressed size is oversized.

This PR changes the EVALSHA logic to use the uncompressed sizes to determine if a segment is oversized if we should skip merging.

We also remove the spop step since we cannot determine which spans to evict from the compressed set members alone , since redis memory usage of the set doesn't reflects its uncompressed ingested size.

[github-actions]

Backend Test Failures

Failures on 8f070c7 in this run:

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[single-nochunk] — log

tests/sentry/spans/test_buffer.py:997: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='140276986122592'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[cluster-nochunk] — log

tests/sentry/spans/test_buffer.py:894: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[cluster-nochunk] — log

tests/sentry/spans/test_buffer.py:997: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='140367602863472'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[single-chunk1] — log

tests/sentry/spans/test_buffer.py:894: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[cluster-chunk1] — log

tests/sentry/spans/test_buffer.py:997: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='139859978048176'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[cluster-chunk1] — log

tests/sentry/spans/test_buffer.py:894: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[single-chunk1] — log

tests/sentry/spans/test_buffer.py:997: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='139881317701616'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[single-nochunk] — log

tests/sentry/spans/test_buffer.py:894: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

[evanh]

I don't think this is correct, if you remove spop_end_time_ms then you need to check arg_cleanup_end_time_ms and use that if it exists.

[github-actions]

Backend Test Failures

Failures on 2ab9f67 in this run:

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[single-nochunk] — log

tests/sentry/spans/test_buffer.py:996: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='140269687285008'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[cluster-nochunk] — log

tests/sentry/spans/test_buffer.py:893: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[single-chunk1] — log

tests/sentry/spans/test_buffer.py:996: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='140260886920704'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[single-nochunk] — log

tests/sentry/spans/test_buffer.py:893: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[single-chunk1] — log

tests/sentry/spans/test_buffer.py:893: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})

tests/sentry/spans/test_buffer.py::test_dropped_spans_emit_outcomes[cluster-chunk1] — log

tests/sentry/spans/test_buffer.py:996: in test_dropped_spans_emit_outcomes
    assert mock_track_outcome.called, "track_outcome should be called when spans are dropped"
E   AssertionError: track_outcome should be called when spans are dropped
E   assert False
E    +  where False = <MagicMock name='track_outcome' id='139732280236016'>.called

tests/sentry/spans/test_buffer.py::test_max_segment_spans_limit[cluster-chunk1] — log

tests/sentry/spans/test_buffer.py:893: in test_max_segment_spans_limit
    assert len(retained_span_ids) < len(all_span_ids), "Some spans should have been evicted"
E   AssertionError: Some spans should have been evicted
E   assert 5 < 5
E    +  where 5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})
E    +  and   5 = len({'aaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbb', 'cccccccccccccccc', 'dddddddddddddddd', 'eeeeeeeeeeeeeeee'})