feat(benchmark): expose demographics and standings breakdown in SDK

[RapidPoseidon]

What

Exposes the two new leaderboard demographic endpoints through the Python SDK so programmatic consumers can pull voter composition and per-segment standings for a benchmark.

Adds two methods to RapidataBenchmarkManager (client.mri):

• get_demographics(benchmark_id, run_id=None, filters=None) → BenchmarkDemographics
  Voter composition per dimension (ageBucket, gender, occupation, country, language) — raw votes and share, including the "unknown" bucket; shares sum to 1.
• get_standings_breakdown(benchmark_id, dimension, run_id=None, filters=None, use_weighted_scoring=False) → BenchmarkStandingsBreakdown
  Global standings plus one segment per bucket of dimension (incl. "unknown"), each with that segment's votes and standings.

Both accept the same demographic/tag/run filters as standings/query and a use_weighted_scoring flag (default False = raw counts).

How it respects the three-tier structure

generated client → service wrapper → manager. The backend endpoints aren't in the checked-in generated api_client yet, so this adds a thin typed wrapper (BenchmarkDemographicsApi) that talks to the same low-level RapidataApiClient the generated APIs use (param_serialize / call_api / response_deserialize), mirroring their filter-serialization and error-wrapping. It's exposed on LeaderboardService as benchmark_demographics_api. Once the backend ships and the OpenAPI client is regenerated, the generated methods can supersede the wrapper.

Honesty

Field names stay faithful to the API contract: votes, share, the "unknown" buckets are all preserved (not dropped into a flattened DataFrame). Docstrings note that ageBucket/gender/occupation are estimated (inferred), not self-declared, and that segments are raw vote counts by default.

Backend contract built against:

• GET /benchmark/{benchmarkId}/demographics
• GET /benchmark/{benchmarkId}/standings/breakdown?dimension=AgeBucket

Validation

• python -c "from rapidata import RapidataClient" → OK; new public classes re-exported from both __init__.py files.
• pyright src/rapidata/rapidata_client → 0 errors.
• End-to-end unit exercise with a mocked API client: query-param explosion (country[in]=US, dimension, useWeightedScoring), model parsing (incl. unknown buckets + confidenceInterval), and filter validation all verified.
• mkdocs build → succeeds; docs added under mri_advanced.md (Voter Demographics section).

Scope

This is the SDK leg of the "Expose Demographic Information for Benchmarks" RFC (phase 4). Sibling PRs build in parallel:

• rapidata-backend — the AllowAnonymous endpoints
• app-frontend — the UI

This PR only touches rapidata-python-sdk.

🔗 Session: https://node-bfba29d9.poseidon.rapidata.internal/

[RapidPoseidon]

Folded in contract update (from parent — backend leg merged as RapidataAI/rapidata-backend#4821)

Pushed a follow-up commit reconciling with the locked endpoint shapes:

• Removed use_weighted_scoring from get_standings_breakdown(...) and the thin wrapper. Correction from the parent: neither new endpoint exposes weighting — both are raw-count only. (The pre-existing standings/query still has it; these two new endpoints do not.) get_demographics(...) never had a weighting param.
• Completed the standings item shape to match the merged contract: added proprietaryName (optional), logo (optional), status (StandingStatus), and isDisabled (bool) alongside id/name/family/score/wins/totalMatches/confidenceInterval. Vote counts, shares, and the always-present "unknown" buckets are preserved.

No conflict with anything already shipped — the wrapper stays (per repo convention) until #4821 is deployed and the OpenAPI client is regenerated. Re-validated: pyright 0 errors, mocked end-to-end parse of the full item shape, mkdocs build clean.

[RapidPoseidon]

⚠️ CI not scheduling on the follow-up commits — needs a human nudge

The type-check (pyright) workflow ran green on the first commit, but GitHub has not scheduled a run for the follow-up commits (9e1bdcc, 17734b5). No github-actions check suite is being created for the branch's synchronize events — verified over a 30-minute automated watch plus an empty-commit re-trigger. Other PRs on the repo get Actions runs, so this is a GitHub-side scheduling hiccup specific to this branch, not a code failure. My session token lacks the scope to close/reopen the PR or poke Actions.

To kick CI: close + reopen this PR (fires a reopened event, which the workflow listens for), or push any commit from an account with full scopes.

Local verification of the follow-up (same checks CI runs):

• pyright src/rapidata/rapidata_client → 0 errors, 0 warnings
• Mocked end-to-end parse of the full locked standings-item shape (status, isDisabled, proprietaryName, logo, confidenceInterval) + confirmation that use_weighted_scoring is gone from both the manager method and the wrapper
• mkdocs build → clean

The last commit (17734b5) is an empty ci: re-trigger checks commit; feel free to drop it on squash-merge.

[LinoGiger]

i really hate how you expose all of this. please fix it in a way that follows the sdk standards. so if i want the standings of a specific benchmark i call .get_standings on the benchmark etc. please also keep in mind what actually makes sense to be exposed and expose it in an intuitive way

[RapidPoseidon]

Thanks @LinoGiger — reworked to follow the SDK standards. Pushed in 02d3bfd.

Now exposed on the benchmark object, like the other standings methods (no more manager methods, no benchmark_id argument, no bespoke result classes):

benchmark = client.mri.get_benchmark_by_id("...")

# voter composition — DataFrame: dimension, value, votes, share
benchmark.get_demographics(tags=["landscape"], run_id="...")

# standings split by a demographic dimension — DataFrame:
# segment, segment_votes, name, wins, total_matches, score
benchmark.get_standings_breakdown(dimension="AgeBucket", tags=["landscape"])

What changed vs the first version:

• Moved onto RapidataBenchmark, sitting right next to get_overall_standings / get_win_loss_matrix, and taking the same explicit tags / leaderboard_ids / run_id filters (no generic filters dict).
• Returns pandas.DataFrame like every other standings method, instead of the five custom pydantic classes — those are gone, along with their top-level re-exports.
• Only what makes sense is exposed: get_standings_breakdown returns just the per-segment standings; the overall standings already live on get_overall_standings, so I don't duplicate them. Vote counts, shares and the "unknown" buckets are preserved as columns/rows.
• The backend deployed in the meantime, so I merged main, dropped the temporary hand-written wrapper entirely, and now call the generated benchmark_api demographics/breakdown methods directly.

Net diff is now just two files: rapidata_benchmark.py and the docs. Re-requesting your review.

[LinoGiger]

what i still don't like is that everything is just strings - we already have some of them typed no?

[RapidPoseidon]

Good catch @LinoGiger — switched to the typed enum we already have (7f1668b, a05547e):

• get_standings_breakdown(dimension=...) now takes BenchmarkDemographicDimension (the generated enum) instead of a hand-rolled string Literal. A raw string is still coerced for convenience, and an invalid value raises a clear ValueError.
• get_demographics reports the same vocabulary: the DataFrame dimension column now holds the BenchmarkDemographicDimension values (AgeBucket, Gender, Occupation, Country, Language) instead of raw camelCase strings, so both methods speak the same typed language.
• Re-exported BenchmarkDemographicDimension from the top level: from rapidata import BenchmarkDemographicDimension.

from rapidata import BenchmarkDemographicDimension

benchmark.get_standings_breakdown(dimension=BenchmarkDemographicDimension.AGEBUCKET)

demographics = benchmark.get_demographics()
demographics[demographics["dimension"] == BenchmarkDemographicDimension.COUNTRY]

The return stays a pandas.DataFrame to match get_overall_standings / get_win_loss_matrix. If by "some of them" you also meant the row values (model name/score/etc.) should come back as the typed standings objects rather than DataFrame columns, say the word and I'll switch the return — I kept it a DataFrame for consistency with the existing standings methods. Re-requesting your review.

[LinoGiger]

like one big thing that you're completely missing is to filter the standings / matrix based on those filters. please add that

[RapidPoseidon]

Added it (1fa4573). The demographic dimensions are now filters on every read that hits the vote-query endpoints, so you can compute standings/matrices from just the votes of a given voter group.

Added country, language, gender, age_bucket, occupation, run_id (alongside the existing tags / leaderboard_ids) to:

• RapidataBenchmark.get_overall_standings, get_win_loss_matrix, get_demographics, get_standings_breakdown
• RapidataLeaderboard.get_standings, get_win_loss_matrix

from rapidata import Gender, AgeGroup, BenchmarkDemographicDimension

# standings among young women in the US/UK
benchmark.get_overall_standings(
    gender=[Gender.FEMALE],
    age_bucket=[AgeGroup.BETWEEN_18_29],
    country=["US", "GB"],
)

# same filters work on the matrix and the breakdown
benchmark.get_win_loss_matrix(gender=[Gender.FEMALE])
benchmark.get_standings_breakdown(
    dimension=BenchmarkDemographicDimension.COUNTRY, gender=[Gender.FEMALE]
)

Typing: gender / age_bucket take the SDK's existing typed Gender / AgeGroup enums (converted to backend values internally); country / language / occupation are open value sets, so plain list[str] like tags. A shared _vote_filters helper builds the backend filter shape so all six methods stay consistent.

pyright 0 errors, black clean, mkdocs build clean, and an end-to-end test exercising all six methods (enum→backend-string conversion, unset filters as no-ops). Re-requesting your review.