Skip to content

fix(data-api): remove chain chips from By Provider pivot #1700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 2 commits into from
Jul 29, 2026
Merged

fix(data-api): remove chain chips from By Provider pivot #1700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f3cf5e5
feat(pm-venues): add Metaculus + Betfair to api-latency and rate-limi…
Flotapponnier Jul 29, 2026
86e43f2
fix(data-api): remove chain chips from pivot, keep only region sub-sc…
Flotapponnier Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 39 additions & 5 deletions benchmarks/pm-api-latency.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,14 +105,20 @@ per_chain_explainer:
- slug: smarkets
h2: Smarkets API latency
body: "Smarkets is a UK-regulated peer-to-peer betting exchange with a public CLOB order book endpoint. The harness probes /v3/markets/{id}/ for price (5s interval) and /v3/markets/{id}/quotes/ for the book (5s interval). No authentication is required for either. No visible CDN layer sits between the probe and origin, so latency reflects the API server response time directly. Smarkets is the only non-US venue in the cohort and its primary user base and server infrastructure are UK-centric, which affects the relative latency from US East vs EU West probe nodes."
- slug: metaculus
h2: Metaculus API latency
body: "Metaculus is a public forecasting aggregator with no real-money markets, probed via its /api2/questions/{id}/ endpoint every 5 seconds from three regions. No authentication is required and no CDN cache layer was detected on the single-question endpoint. Metaculus has no order book and no WebSocket: all data access is REST. The uptime panel shows the live health gauge for the question API; the latency reflects the Django origin server without a CDN buffer. For builders querying aggregate community forecasts rather than market prices, Metaculus is the only venue in this cohort with a purely public read API at sub-second latency."
- slug: betfair
h2: Betfair API latency
body: "Betfair is the world's largest regulated betting exchange, probed via its Exchange API listMarketBook endpoint (POST with JSON body). Unlike every other venue in this cohort, all Betfair API calls are POST and require X-Application (developer AppKey) plus X-Authentication (session token). Betfair's infrastructure is UK-centric, so latency from us-east and sgp is higher than from eu-west. The latency shown is the full round trip of a listMarketBook call on a pinned binary market. For how Betfair behaves under rising request rates, see the pm-rate-limits bench."

source: https://github.com/ChainBench/OpenChainBench/tree/main/harnesses/pm-rate-limits

prometheus:
window: 24h
expected_freshness_seconds: 300

rank_matrix_query: 1000 * label_replace(histogram_quantile(0.50, sum by (venue, region, le) (rate(pmapi_request_duration_seconds_bucket{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets",conn="warm",class="price",cache!="hit"}[24h]))), "provider", "$1", "venue", "(.+)")
rank_matrix_query: 1000 * label_replace(histogram_quantile(0.50, sum by (venue, region, le) (rate(pmapi_request_duration_seconds_bucket{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets|metaculus|betfair",conn="warm",class="price",cache!="hit"}[24h]))), "provider", "$1", "venue", "(.+)")

dimensions:
venue:
Expand All @@ -124,6 +130,8 @@ dimensions:
- { value: myriad, label: Myriad }
- { value: predictit, label: PredictIt }
- { value: smarkets, label: Smarkets }
- { value: metaculus, label: Metaculus }
- { value: betfair, label: Betfair }
region:
- { value: all, label: All regions }
- { value: us-east, label: US East }
Expand All @@ -134,29 +142,29 @@ metric_panels:
- id: uptime_24h
label: Uptime 24h
description: "Share of probe cycles that succeeded over the last 24 hours, averaged across the three probe regions. 100 percent means every 5 second probe of the venue's API came back healthy."
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets"}[24h]))
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets|metaculus|betfair"}[24h]))
label_key: venue
unit: pct
higher_is_better: true
- id: uptime_7d
label: Uptime 7d
description: "Same health gauge averaged over 7 days. Short outages that vanish from the 24h figure stay visible here for a week."
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets"}[7d]))
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets|metaculus|betfair"}[7d]))
label_key: venue
unit: pct
higher_is_better: true
- id: uptime_30d
label: Uptime 30d
description: "Same health gauge averaged over 30 days. Feeds the ledger's 30d window toggle so a one-hour outage that's already aged out of the 24h panel still shows up in the table over a month."
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets"}[30d]))
metric: 100 * avg(avg_over_time(pmapi_health{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets|metaculus|betfair"}[30d]))
label_key: venue
unit: pct
higher_is_better: true
tab: false
- id: cold_connect_p50
label: Cold connect
description: "TCP plus TLS handshake time on the once a minute cold probe with keep alives disabled. The startup cost a brand new client pays before its first request."
metric: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_connect_seconds_bucket{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets"}[24h])) by (le))
metric: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_connect_seconds_bucket{source="direct",venue=~"polymarket|kalshi|limitless|manifold|myriad|predictit|smarkets|metaculus|betfair"}[24h])) by (le))
label_key: venue
unit: ms
higher_is_better: false
Expand Down Expand Up @@ -259,3 +267,29 @@ providers:
success: clamp_max(sum(rate(pmapi_requests_total{venue="smarkets",source="direct",class="price",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="smarkets",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="smarkets",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{venue="smarkets",source="direct",class="price",conn="warm",cache!="hit"}[1h])) by (le))

- slug: metaculus
name: Metaculus
tag: "Public forecasting platform, no auth, no order book, binary questions probed via /api2/questions/{id}/"
formula: "p50 of warm round trips against the single question endpoint /api2/questions/{id}/, successful requests only, 24h window. No CDN cache layer detected."
queries:
p50: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
p90: 1000 * histogram_quantile(0.90, sum(rate(pmapi_request_duration_seconds_bucket{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
p99: 1000 * histogram_quantile(0.99, sum(rate(pmapi_request_duration_seconds_bucket{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
mean: 1000 * sum(rate(pmapi_request_duration_seconds_sum{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[24h])) / sum(rate(pmapi_request_duration_seconds_count{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[24h]))
success: clamp_max(sum(rate(pmapi_requests_total{venue="metaculus",source="direct",class="price",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="metaculus",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="metaculus",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{venue="metaculus",source="direct",class="price",conn="warm",cache!="hit"}[1h])) by (le))

- slug: betfair
name: Betfair
tag: "UK FCA-regulated exchange, auth required, price endpoint is listMarketBook POST"
formula: "p50 of warm round trips against the Exchange API listMarketBook endpoint (POST, pinned binary market), successful requests only, 24h window. Requires BETFAIR_APP_KEY + credentials to run."
queries:
p50: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
p90: 1000 * histogram_quantile(0.90, sum(rate(pmapi_request_duration_seconds_bucket{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
p99: 1000 * histogram_quantile(0.99, sum(rate(pmapi_request_duration_seconds_bucket{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[24h])) by (le))
mean: 1000 * sum(rate(pmapi_request_duration_seconds_sum{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[24h])) / sum(rate(pmapi_request_duration_seconds_count{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[24h]))
success: clamp_max(sum(rate(pmapi_requests_total{venue="betfair",source="direct",class="price",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="betfair",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="betfair",source="direct",class="price",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{venue="betfair",source="direct",class="price",conn="warm",cache!="hit"}[1h])) by (le))
38 changes: 36 additions & 2 deletions benchmarks/pm-rate-limits.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,14 +111,20 @@ per_chain_explainer:
- slug: smarkets
h2: Smarkets rate limits
body: "Smarkets showed no visible rate limiting during reconnaissance probes at up to 6 rapid requests per second. The daily ramp uses conservative tiers (10, 25, 50 requests per 10 seconds) with the standard abort guard. The CLOB quotes endpoint returns full bid-offer depth for the pinned market on every request. Unlike Polymarket which queues bursts under Cloudflare, Smarkets returns direct origin responses, so the ramp records actual latency scaling rather than queue inflation."
- slug: metaculus
h2: Metaculus rate limits
body: "Metaculus does not document API rate limits. The harness uses the most conservative ramp in the cohort (5/10/20 requests per 10 seconds) with the standard abort guard. There is no order book endpoint and no WebSocket. Because Metaculus is a forecasting aggregator rather than a betting exchange, its API is designed for low-frequency access; polling faster than once per 5 seconds on a single question likely hits undocumented limits. Warm price endpoint p50: {{p50:metaculus}}."
- slug: betfair
h2: Betfair rate limits
body: "Betfair's Exchange API documents a rate limit of 5 requests per second per AppKey for the listMarketBook endpoint, enforced with 429 responses. The daily ramp uses tiers of 10/25/50 requests per 10 seconds and stops at the first 429, consistent with the documented contract. Unlike Polymarket, Betfair does not queue under Cloudflare — it applies a hard token bucket. All API calls are POST with JSON body and require X-Application (AppKey) plus X-Authentication (session token). Betfair is the only venue in this cohort where the price probe is a POST request. Warm book p50: {{p50:betfair}}."

source: https://github.com/ChainBench/OpenChainBench/tree/main/harnesses/pm-rate-limits

prometheus:
window: 24h
expected_freshness_seconds: 300

rank_matrix_query: 1000 * label_replace(histogram_quantile(0.50, sum by (venue, region, le) (rate(pmapi_request_duration_seconds_bucket{cache!="hit",conn="warm",class="book"}[24h]) or rate(pmapi_request_duration_seconds_bucket{cache!="hit",conn="warm",class="price",venue=~"myriad|predictit"}[24h]))), "provider", "$1", "venue", "(.+)")
rank_matrix_query: 1000 * label_replace(histogram_quantile(0.50, sum by (venue, region, le) (rate(pmapi_request_duration_seconds_bucket{cache!="hit",conn="warm",class="book"}[24h]) or rate(pmapi_request_duration_seconds_bucket{cache!="hit",conn="warm",class="price",venue=~"myriad|predictit|metaculus"}[24h]))), "provider", "$1", "venue", "(.+)")

dimensions:
venue:
Expand All @@ -130,6 +136,8 @@ dimensions:
- { value: myriad, label: Myriad }
- { value: predictit, label: PredictIt }
- { value: smarkets, label: Smarkets }
- { value: metaculus, label: Metaculus }
- { value: betfair, label: Betfair }
region:
- { value: all, label: All regions }
- { value: us-east, label: US East }
Expand All @@ -140,7 +148,7 @@ metric_panels:
- id: ramp_added_latency
label: Ramp added latency
description: "Worst tier of the daily ramp: p50 latency during the burst minus the warm baseline p50 of the previous hour, same region. Venues that queue under load instead of returning 429 show up here."
metric: 1000 * max(max_over_time(pmapi_ramp_added_latency_seconds{}[24h]))
metric: 1000 * max(max_over_time(pmapi_ramp_added_latency_seconds{venue!="predictit",venue!="myriad",venue!="metaculus"}[24h]))
label_key: venue
unit: ms
higher_is_better: false
Expand Down Expand Up @@ -267,3 +275,29 @@ providers:
success: clamp_max(sum(rate(pmapi_requests_total{venue="smarkets",class="book",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="smarkets",class="book",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="smarkets",class="book",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="smarkets",class="book",conn="warm"}[1h])) by (le))

- slug: metaculus
name: Metaculus
tag: "Public forecasting platform, no auth, no book endpoint, conservative ramp 5/10/20 per 10s"
formula: "p50 of warm price requests against /api2/questions/{id}/, successful requests only, 24h window. No book endpoint. Ramp at 5/10/20 per 10s (limits undocumented)."
queries:
p50: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="metaculus",class="price",conn="warm"}[24h])) by (le))
p90: 1000 * histogram_quantile(0.90, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="metaculus",class="price",conn="warm"}[24h])) by (le))
p99: 1000 * histogram_quantile(0.99, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="metaculus",class="price",conn="warm"}[24h])) by (le))
mean: 1000 * sum(rate(pmapi_request_duration_seconds_sum{cache!="hit",venue="metaculus",class="price",conn="warm"}[24h])) / sum(rate(pmapi_request_duration_seconds_count{cache!="hit",venue="metaculus",class="price",conn="warm"}[24h]))
success: clamp_max(sum(rate(pmapi_requests_total{venue="metaculus",class="price",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="metaculus",class="price",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="metaculus",class="price",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="metaculus",class="price",conn="warm"}[1h])) by (le))

- slug: betfair
name: Betfair
tag: "UK FCA-regulated exchange, auth required, listMarketBook POST, token bucket rate limiting"
formula: "p50 of warm listMarketBook POST requests against the Exchange API for the pinned binary market, successful requests only, 24h window. Requires BETFAIR_APP_KEY + credentials. Ramp at 10/25/50 per 10s, stops on first 429."
queries:
p50: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="betfair",class="book",conn="warm"}[24h])) by (le))
p90: 1000 * histogram_quantile(0.90, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="betfair",class="book",conn="warm"}[24h])) by (le))
p99: 1000 * histogram_quantile(0.99, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="betfair",class="book",conn="warm"}[24h])) by (le))
mean: 1000 * sum(rate(pmapi_request_duration_seconds_sum{cache!="hit",venue="betfair",class="book",conn="warm"}[24h])) / sum(rate(pmapi_request_duration_seconds_count{cache!="hit",venue="betfair",class="book",conn="warm"}[24h]))
success: clamp_max(sum(rate(pmapi_requests_total{venue="betfair",class="book",conn="warm",outcome="ok"}[24h])) / sum(rate(pmapi_requests_total{venue="betfair",class="book",conn="warm",outcome!="probe_invalid"}[24h])), 1)
sample_size: sum(increase(pmapi_requests_total{venue="betfair",class="book",conn="warm",outcome!="probe_invalid"}[24h]))
series: 1000 * histogram_quantile(0.50, sum(rate(pmapi_request_duration_seconds_bucket{cache!="hit",venue="betfair",class="book",conn="warm"}[1h])) by (le))
Loading
Loading