Skip to content

Add cache proxy origin backpressure#792

Open
bill-ph wants to merge 1 commit into
mainfrom
codex/proxy-origin-backpressure
Open

Add cache proxy origin backpressure#792
bill-ph wants to merge 1 commit into
mainfrom
codex/proxy-origin-backpressure

Conversation

@bill-ph

@bill-ph bill-ph commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add per-pod origin-fill backpressure for cacheable cache misses via ORIGIN_MAX_IN_FLIGHT (default 0, disabled until configured from production metrics)
  • queue distinct origin-fill leaders after local cache lookup, peer lookup, and singleflight dedupe; same-key waiters share one slot
  • return 503 Service Unavailable with Retry-After: 1 if a queued request is canceled before origin fetch starts
  • expose queue metrics: cache_proxy_origin_fetches_queued and cache_proxy_origin_fetch_queue_wait_seconds{outcome}
  • add a dedicated just test-cache-proxy recipe and CI job

Notes

  • local cache hits, peer hits, non-GET passthrough, and CONNECT tunnels do not consume origin slots
  • cache_proxy_origin_fetches_in_flight remains active origin fills only
  • configure a positive ORIGIN_MAX_IN_FLIGHT value only after observing production origin-fill, retry, and queue metrics
  • dashboard panels for the new queue metrics should be a separate grafana-dashboards follow-up

Validation

  • red step: TestNewCacheProxyOriginBackpressureDefaultsDisabled failed while the default was still 64
  • red step: focused backpressure tests failed before limiter implementation
  • go test -v ./cmd/cache-proxy -run 'TestNewCacheProxyOriginBackpressureDefaultsDisabled|TestHandleProxyOriginBackpressure|TestNonNegativeIntEnvOrDefault'
  • just test-cache-proxy
  • go test -race ./cmd/cache-proxy -run 'TestHandleProxyOriginBackpressure|TestNonNegativeIntEnvOrDefault' -count=20
  • just lint
  • just test-unit

@bill-ph bill-ph force-pushed the codex/proxy-origin-backpressure branch from b2f3a59 to 35cb55d Compare June 17, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant