Skip to content

[fix](regression) Stabilize delete bitmap cache metrics test#65535

Draft
shuke987 wants to merge 1 commit into
apache:masterfrom
shuke987:codex/fix-delete-bitmap-metrics-cache-order
Draft

[fix](regression) Stabilize delete bitmap cache metrics test#65535
shuke987 wants to merge 1 commit into
apache:masterfrom
shuke987:codex/fix-delete-bitmap-metrics-cache-order

Conversation

@shuke987

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A

Related PR: #64515

Problem Summary:

test_delete_bitmap_metrics verifies each replica's durable delete bitmap and lazy aggregate-cache snapshot. In multi-replica P0 environments, the case warmed cache entries before iteration and fetched the large global /brpc_metrics payload inside the per-replica loop. On build 202230, the third replica was inspected about 32 seconds after warm-up; its durable bitmap was still correct (7/7) but the capacity-bounded LRU snapshot was empty.

This change sorts tablet rows by ReplicaId to match use_fix_replica ordering, pins and queries each replica immediately before checking that same replica's cache, restores the session variable in finally, and fetches the global metrics payload once after all replica checks.

The patched single case passed 3 consecutive lock-serialized runs on the authorized agent5 test cluster.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#64515

Problem Summary:

The case warmed aggregate delete-bitmap cache entries before iterating over
replicas, then fetched the large global brpc metrics payload inside that loop.
On a loaded multi-replica P0 cluster, later replicas could be checked tens of
seconds after warm-up even though the cache is a capacity-bounded LRU.

Pin and query each replica immediately before inspecting its cache. Sort the
tablet rows by ReplicaId to match use_fix_replica ordering, restore the session
variable in finally, and fetch the global metrics payload once after all
per-replica checks.

### Release note

None

### Check List (For Author)

- Test
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
        - [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
    - [x] No.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@shuke987

Copy link
Copy Markdown
Collaborator Author

/review

@shuke987

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed after two rounds. I found one blocking issue: cloud multi-replica mode can still inspect a different BE's local aggregate-cache than the BE that served the warmup scan. I left the inline comment at the warmup query.

Checkpoint conclusions:

  • Goal/test coverage: this one-file regression change stabilizes the local and normal-cloud paths, but it does not prove the intended per-BE aggregate-cache behavior when cloud multi-replica placement is enabled.
  • Scope/focus: the change is small and focused; no additional user focus was provided.
  • Concurrency/lifecycle/config/compatibility/persistence/data writes/FE-BE variables: no product-code or protocol changes; the relevant config-sensitive path is cloud multi-replica selection.
  • Parallel paths: local/shared-nothing, normal cloud, and cloud multi-replica paths were checked; cloud multi-replica is the remaining gap.
  • Regression-test standards/results: the deterministic result check remains ordered; the added warmup query is not an output assertion; no .out file was changed. No tests/builds were run in this review environment per the prompt.
  • Observability/performance: moving /brpc_metrics polling out of the replica loop preserves the same fixed-backend metric coverage and avoids repeated large metrics scrapes.

Round 2 convergence reviewers returned NO_NEW_VALUABLE_FINDINGS for the current ledger/comment set. Requesting changes for the inline issue.

// use_fix_replica selects replicas in ReplicaId order. Populate and
// inspect each replica immediately because the agg cache is an LRU.
sql "set use_fix_replica=${ri};"
sql "select * from ${testTable};"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still looks flaky when cloud multi-replica placement is enabled. The test records tablet.BackendId from SHOW TABLETS, then warms with use_fix_replica=${ri} and probes /count_agg_cache on that stored backend. In cloud mode, though, CloudTablet has one logical CloudReplica, and CloudReplica.getBackendIdImpl() chooses a physical BE with rand.nextInt(Config.cloud_replica_num) on each backend-resolution call when enable_cloud_multi_replica is true. The SHOW TABLETS call and the later scan can therefore pick different physical BEs, while /count_agg_cache only snapshots the target BE's local agg cache. Please either pin/disable cloud multi-replica for this per-BE cache assertion, or query the actual BE selected by the warmup scan before checking its cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants