perf(metrics): cache request series handles

[moonming]

Summary

• cache registered Prometheus counter/histogram handles for the paired proxy + LLM request series
• hash and look up the borrowed request label tuple once instead of rebuilding and hashing four full metric keys per request
• bound the additional cache to 1,024 label sets; eviction only falls back to recorder registration and does not change metric correctness
• preserve every existing metric name, label set, value, and absent-vs-zero behavior
• keep the standalone proxy-only and LLM-only public recorders uncached so they continue to register only the metric families they actually record

Root cause

The supplied AISIX flame graph attributes material request-path CPU to metrics::Key::eq, generate_key_hash, label construction, sorting, and drops. The request completion path emitted the proxy and LLM counter/histogram pairs separately, rebuilding the same large dynamic label tuple for each series.

The production chat and messages paths always emit the proxy and LLM request series together with the same duration, so the new paired recorder performs one collision-safe cache lookup and reuses the registered handles.

GetBusbar A/B

Measured with the upstream GetBusbar/benchmarking runner at commit 3715e6c952b3a25f99dfb6a7bf9ac7436a79cdf8, OpenAI-only, on the same ARM64 8-vCPU VM. Gateway cores were pinned to 0-3, load generator to 4-5, and mock upstream to 6-7. The adaptive sweep used 6-second rungs. Base was AISIX 6202977; optimized production code was 44fb078 (the later commits only add tests and restore standalone recorder semantics; production handlers still use the paired cached path).

Metric	Base	Cached handles	Observed delta
Added latency p99	344 us	170 us	-50.6%
1 ms p99 frontier	35,636 RPS	36,328 RPS	+1.94%
5 ms p99 frontier	36,165 RPS	36,328 RPS	+0.45%
10 ms p99 frontier	36,165 RPS	36,855 RPS	+1.91%
Unbounded frontier	36,165 RPS @ c=64	36,855 RPS @ c=128	+1.91%
Failed requests	0	0	unchanged
Steady RSS	100.26 MiB	100.65 MiB	+0.39 MiB
Peak RSS	101.11 MiB	102.10 MiB	+0.99 MiB

Five additional interleaved c=64, 10-second loadgen trials:

• base: 34,912 / 35,706 / 33,236 / 35,528 / 29,820 RPS (median 34,912)
• optimized: 36,464 / 33,139 / 34,272 / 35,986 / 35,822 RPS (median 35,822)
• every trial reported zero failures

Interpretation and limitations

These short samples overlap materially and the base set contains a low outlier. The table is therefore reported as a directional local observation, not as a statistically significant percentage improvement or a merge criterion. The broader repeatable performance investigation remains tracked by api7/AISIX-Cloud#1199.

The optimized GetBusbar load tail was classified plateaued=false at +1.63 MiB/min. Its 60-second post-load recovery was flat at 100.895 MiB, but that window is not sufficient to claim a long-term steady-state RSS plateau. The only bounded-memory claim made by this PR is structural: the additional handle cache cannot retain more than 1,024 label sets, which is covered by a regression test.

Correctness coverage

• repeated label sets reuse one registered handle set
• concurrent first misses converge on one cache entry and preserve exact counter/histogram counts
• distinct full label sets remain isolated under a forced 64-bit hash collision
• eviction and re-registration continue accumulating the original Prometheus series while the cache stays at exactly 1,024 entries
• failure counters increment exactly once
• standalone proxy-only and LLM-only recorders do not expose unrelated zero-valued metric families

Validation

• cargo fmt --all -- --check
• cargo test -p aisix-obs (152 passed, 5 real-cloud tests ignored)
• cargo test -p aisix-proxy (763 passed)
• cargo test -p aisix-server (87 passed)
• cargo clippy -p aisix-obs -p aisix-proxy -p aisix-server --all-targets -- -D warnings
• GetBusbar real network E2E against its mock upstream/load generator

Part of api7/AISIX-Cloud#1199.

[coderabbitai]

Warning

Review limit reached

@moonming, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 51 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3dd653b2-fda4-4a5e-85d3-9025c32c7790

📥 Commits

Reviewing files that changed from the base of the PR and between 0866202 and 975f981.

📒 Files selected for processing (3)

• crates/aisix-obs/src/metrics.rs
• crates/aisix-proxy/src/chat.rs
• crates/aisix-proxy/src/messages.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}