Refresh threshold cache manager changes

[amurphy-cl]

I get tied in knots trying to describe these cache things, so I had Claude explain it pretty:

Where we are today

Each of the 4 instances keeps its own in-process cache. Nothing is shared. When someone creates an experiment, the instance that handled the write clears its own cache — the other three don't know anything happened. TTL expiry is our only cross-instance sync mechanism.

That's why the TTL is short. But a short TTL is a blunt instrument: it's simultaneously (a) how fresh the data is and (b) how often every key goes completely cold. And when a key goes cold, the refill happens inside a user's request — that user waits on the database.

What refreshThreshold changes

It splits those two jobs apart:

• Max age (CACHING_REFRESH_THRESHOLD) — how stale data is allowed to get. Once an entry is older than this, the next read returns the cached copy immediately and kicks off a refill in the background.
• TTL — now only a hard expiry for keys that stop being read. It stops governing freshness.

The numbers, using your example

Per cache key, across 4 instances:

┌─────────────────────────────────────┬─────────────────────┬─────────────────────────┐
│                                     │ 60s TTL, no refresh │ 3600s TTL + 60s max age │
├─────────────────────────────────────┼─────────────────────┼─────────────────────────┤
│ DB reads per minute                 │ 4                   │ 4                       │
├─────────────────────────────────────┼─────────────────────┼─────────────────────────┤
│ Requests that wait on those reads   │ 4                   │ 0                       │
├─────────────────────────────────────┼─────────────────────┼─────────────────────────┤
│ Worst-case cross-instance staleness │ 60s                 │ ~60s                    │
└─────────────────────────────────────┴─────────────────────┴─────────────────────────┘

Same database load. Same freshness. Nobody waits. That's the whole pitch.

And the ceiling is now adjustable in the direction we actually want. Today, going from 60s → 15min TTL to save DB load means a new experiment is invisible on 3 of 4 instances for up to 15 minutes. With background refresh, the TTL can be an hour and the cluster still converges in ~60 seconds.

Two secondary wins

• No stampede at expiry. Today, a burst of requests arriving right when a key expires all queue behind one DB query. With refresh, they're served the cached value while the refill happens off to the side.
• DB blips stop reaching users. A failed background refresh is logged and the previous value keeps serving. Today a failed cold-miss refill becomes a user-facing error.

Honest limits

• Still traffic-driven. A key nobody reads for an hour expires and the next reader pays a normal cold miss. That's the intended, acceptable case.
• Not instant propagation. Worst case is max-age plus one refresh duration, and requests during that window get the old value. If we ever need true immediacy, that's the bigger overhaul (shared Redis cache or polling for updates) — this change is deliberately not that.
• This buys latency and sync, not DB savings. An actively-read key refreshes every 60s whether or not anything changed. Same as today's short TTL, so it's not a regression — but "long TTL alone" would be cheaper on the DB, at the cost of the sync problem we're trying to fix.
• Slightly more memory held. Longer TTLs mean entries linger; CACHING_MAX_KEYS (default 500) still caps it.

Recommended settings

Long TTL as a cleanup bound, max-age set to whatever freshness we actually want:

CACHING_TTL_EXPERIMENTS=3600
CACHING_TTL_FEATURE_FLAGS=3600
CACHING_TTL_SEGMENTS=3600
CACHING_REFRESH_THRESHOLD=60

Leaving CACHING_REFRESH_THRESHOLD unset (or 0) disables background refresh entirely and reverts to today's plain TTL behavior — so this ships behind a config switch with a zero-risk off state.