feat(metrics): add block transaction count and SR set change monitoring

[warku123]

What does this PR do?

Add two new metrics enhancements for better network monitoring:

1. Block Transaction Count Histogram

   • Replace tron:block_empty_total counter with tron:block_transaction_count histogram
   • Track transaction count distribution per block with buckets: [0, 10, 50, 100, 200, 500, 1000, 2000, 5000, 10000]
   • Empty blocks can be monitored via le=0.0 bucket

2. SR Set Change Monitoring

   • Add tron:sr_set_change counter to track SR membership changes
   • Detect SR additions and removals at each maintenance time interval
   • Labels: action (add/remove), witness (SR address)

Changes:

• Add overloaded init() method in MetricsHistogram for custom buckets
• Add BLOCK_TRANSACTION_COUNT histogram metric
• Add SR_SET_CHANGE counter with SR_ADD/SR_REMOVE labels
• Implement SR set change detection logic in BlockChainMetricManager
• Update PrometheusApiServiceTest with comprehensive test coverage

Why are these changes required?

• Transaction histogram provides richer insights than simple counter for network analysis while maintaining empty block detection capability
• SR set change monitoring enables tracking of network consensus participant changes for governance analysis

Compared to #6602, this PR provides a cleaner implementation focused solely on metrics enhancement without additional complexity.

This PR has been tested by:

• Unit Tests (updated PrometheusApiServiceTest)
• Manual Testing

Follow up

N/A

Extra details

• Closes [Feature] Add Prometheus metrics for empty blocks and SR set changes #6590
• Related to feat(metrics): add block transaction count histogram for empty block monitoring #6602 (alternative cleaner implementation)

[halibobo1205]

The witness label uses addresses. Although the Active SR pool is relatively limited (27 slots), as SRs rotate in and out over time, label values will accumulate monotonically — they only grow, never shrink. This is a known Prometheus anti-pattern.

Practical impact assessment:

• TRON's SR candidate pool is not infinite, and rotation frequency is low, so it won't explode in the short term

• However, as a long-running node metric, months or years of operation could still accumulate a significant number of time series

Possible approaches:

• If the address dimension is truly needed, keep the current design, but document this characteristic

• Or remove the witness label entirely, keeping only action (add/remove), and output address details via logs instead. Follow the principle that Prometheus should be used for alerting and awareness, while logs are for precise investigation and root-cause analysis.

[warku123]

Thanks for raising the high-cardinality concern — it's a valid Prometheus best practice to watch for.

However, I believe the witness label is acceptable here given TRON's specific constraints:

1. Bounded candidate pool: The SR candidate pool is finite. Even over years of operation, the total number of distinct addresses that have ever held an SR slot has a practical ceiling — far from the
   unbounded growth typical of high-cardinality anti-patterns.
2. Low rotation frequency: SR set changes only occur at maintenance intervals (every 6 hours), so new label values are introduced very infrequently.
3. Diagnostic value: The primary purpose of this metric is to identify which SR was added or removed. Dropping the witness label would reduce the metric to just "something changed", requiring operators to
   cross-reference logs — losing the at-a-glance visibility in Grafana dashboards that makes this metric useful in the first place.

In summary, the worst-case time series count is roughly candidate_pool_size × 2 (add/remove), which stays well within Prometheus's comfortable operating range.

That said, I agree it's worth documenting this characteristic. Could you suggest what form you'd prefer — a code comment at the metric registration site, a note in the PR description, or something else?

[halibobo1205]

prefer to a code comment at the metric registration site.

[halibobo1205]

L174, L175, and L179 can use txCount.

[halibobo1205]

Two questions regarding observability adoption of the new metrics:

1. PromQL query guide: Could you provide some common query examples for the new tron:block_transaction_count histogram and tron:sr_set_change counter? For instance:

   • How to query empty block rate via the le="0.0" bucket
   • How to calculate P50/P99 transaction counts over a given time window
   • How to query SR membership changes from the most recent maintenance cycle

2. Grafana dashboard examples: Are there any accompanying Grafana dashboard JSONs or screenshots? If so, it would be helpful to include them in the PR description or documentation, so node operators can import and use them directly, reducing the onboarding effort.

[lxcmyf]

metricsService.applyBlock(block) runs before dbManager.pushBlock(block) in TronNetDelegate.processBlock(), so this histogram is updated for blocks that are later rejected or only live on a fork. For an empty-block monitoring metric, that produces false positives per witness and lets invalid peer traffic skew the distribution. This should be recorded only after the block has been applied successfully.

[warku123]

Good catch. I actually noticed this ordering issue during my investigation — metricsService.applyBlock() is called before dbManager.pushBlock(), meaning all metrics in BlockChainMetricManager.applyBlock() are recorded for blocks that may later be rejected.

However, this is not specific to the histogram I added — the existing metrics in the same method (e.g., MINER_LATENCY, TXS, NET_LATENCY, duplicate witness detection) all share the same problem. I chose to keep the same pattern to maintain architectural consistency and avoid expanding the scope of this PR.

If you agree, I'd like to open a separate issue to move metricsService.applyBlock() after dbManager.pushBlock() succeeds in TronNetDelegate.processBlock(), which would fix the timing problem for all metrics at once.

[lxcmyf]

The SR-set change detection is reading nextMaintenanceTime and getActiveWitnesses() before the current block is applied. In the real pipeline those values are updated during dbManager.pushBlock() / consensus.applyBlock(), so on the maintenance block this branch still sees the old schedule and old maintenance timestamp. That makes tron:sr_set_change show up one block late, and the added test masks the problem by mutating both stores manually before generating the next block.

[warku123]

Same root cause as above — since metricsService.applyBlock() runs before dbManager.pushBlock(), the maintenance time and active witness list haven't been updated yet at the point of reading.

In fact, this is precisely why the SR-set change detection relies on nextMaintenanceTime change as the trigger — because getActiveWitnesses() still returns the old schedule at this point, comparing against the previous nextMaintenanceTime is the only way to detect that a maintenance cycle has occurred and capture the witness list transition.

This would be addressed together in that follow-up issue: once metricsService.applyBlock() is moved after a successful pushBlock(), the SR-set change detection will naturally read the up-to-date state without needing this workaround.