[fix](regression) Stabilize delete bitmap cache metrics test#65535
[fix](regression) Stabilize delete bitmap cache metrics test#65535shuke987 wants to merge 1 commit into
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64515 Problem Summary: The case warmed aggregate delete-bitmap cache entries before iterating over replicas, then fetched the large global brpc metrics payload inside that loop. On a loaded multi-replica P0 cluster, later replicas could be checked tens of seconds after warm-up even though the cache is a capacity-bounded LRU. Pin and query each replica immediately before inspecting its cache. Sort the tablet rows by ReplicaId to match use_fix_replica ordering, restore the session variable in finally, and fetch the global metrics payload once after all per-replica checks. ### Release note None ### Check List (For Author) - Test - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
|
/review |
|
run buildall |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Automated review completed after two rounds. I found one blocking issue: cloud multi-replica mode can still inspect a different BE's local aggregate-cache than the BE that served the warmup scan. I left the inline comment at the warmup query.
Checkpoint conclusions:
- Goal/test coverage: this one-file regression change stabilizes the local and normal-cloud paths, but it does not prove the intended per-BE aggregate-cache behavior when cloud multi-replica placement is enabled.
- Scope/focus: the change is small and focused; no additional user focus was provided.
- Concurrency/lifecycle/config/compatibility/persistence/data writes/FE-BE variables: no product-code or protocol changes; the relevant config-sensitive path is cloud multi-replica selection.
- Parallel paths: local/shared-nothing, normal cloud, and cloud multi-replica paths were checked; cloud multi-replica is the remaining gap.
- Regression-test standards/results: the deterministic result check remains ordered; the added warmup query is not an output assertion; no .out file was changed. No tests/builds were run in this review environment per the prompt.
- Observability/performance: moving /brpc_metrics polling out of the replica loop preserves the same fixed-backend metric coverage and avoids repeated large metrics scrapes.
Round 2 convergence reviewers returned NO_NEW_VALUABLE_FINDINGS for the current ledger/comment set. Requesting changes for the inline issue.
| // use_fix_replica selects replicas in ReplicaId order. Populate and | ||
| // inspect each replica immediately because the agg cache is an LRU. | ||
| sql "set use_fix_replica=${ri};" | ||
| sql "select * from ${testTable};" |
There was a problem hiding this comment.
This still looks flaky when cloud multi-replica placement is enabled. The test records tablet.BackendId from SHOW TABLETS, then warms with use_fix_replica=${ri} and probes /count_agg_cache on that stored backend. In cloud mode, though, CloudTablet has one logical CloudReplica, and CloudReplica.getBackendIdImpl() chooses a physical BE with rand.nextInt(Config.cloud_replica_num) on each backend-resolution call when enable_cloud_multi_replica is true. The SHOW TABLETS call and the later scan can therefore pick different physical BEs, while /count_agg_cache only snapshots the target BE's local agg cache. Please either pin/disable cloud multi-replica for this per-BE cache assertion, or query the actual BE selected by the warmup scan before checking its cache.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64515
Problem Summary:
test_delete_bitmap_metricsverifies each replica's durable delete bitmap and lazy aggregate-cache snapshot. In multi-replica P0 environments, the case warmed cache entries before iteration and fetched the large global/brpc_metricspayload inside the per-replica loop. On build 202230, the third replica was inspected about 32 seconds after warm-up; its durable bitmap was still correct (7/7) but the capacity-bounded LRU snapshot was empty.This change sorts tablet rows by
ReplicaIdto matchuse_fix_replicaordering, pins and queries each replica immediately before checking that same replica's cache, restores the session variable infinally, and fetches the global metrics payload once after all replica checks.The patched single case passed 3 consecutive lock-serialized runs on the authorized agent5 test cluster.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)