PHOENIX-7986 : Replace IndexCDCConsumer's exponential idle backoff with a fixed poll interval

[palashc]

What changes were proposed in this pull request?

When an eventually consistent index consumer polls and finds no new CDC events, it now sleeps a fixed interval instead of walking HBase's RETRY_BACKOFF array.

• New config phoenix.index.cdc.consumer.idle.poll.interval.ms, default 4000, with per-consumer jitter so consumers don't poll in lock-step after a RegionServer restart.
• retryCount is reset on any non-throwing iteration, so it now counts only consecutive exceptions. Error paths and the data-visibility retry loop keep their exponential backoff.

Why are the changes needed?

Idle backoff caps at a 200x multiplier — 40s at the default 200ms retry pause, reached after ~96s of inactivity. A write to a quiet region therefore waits up to timestampBufferMs + 40s (~45s) before the consumer looks for it, versus ~5.6s measured on a region under steady write load.

That backoff is borrowed from RPC retry logic meant to relieve a failing server. An empty poll indicates nothing is wrong, so the escalation buys nothing and costs a 7x worse tail on exactly the low-traffic tables where a single write is most likely to be noticed.

It also makes cdcIndexUpdateLag unusable as an SLI: idle consumers emit a 5s → 45s sawtooth, so percentiles describe the backoff schedule rather than index freshness.

Does this PR introduce any user-facing change?

Yes. Worst-case index visibility on an idle region drops from timestampBufferMs + 40s to timestampBufferMs + ~4.8s. Steady-state latency under load is unchanged. Idle CDC query load rises from one poll per region per 40s to one per ~4s. The new config is optional and defaults to the behavior above.

How was this patch tested?

IndexCDCConsumerLagIT passes unchanged.

No new test was added. sleepWithLagSampling emits a sample every lagSampleIntervalMs regardless of total sleep length, so the existing count-based assertion is invariant to sleep duration and cannot observe this change either way. A meaningful assertion needs the lag value rather than the count — MetricHistogram exposes only add() and getCount(), and cold-start samples floored at now - consumerStartTime would have to be excluded first.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor

[palashc]

Build: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2597/