Experiment with ES|QL lookup joins for stack rollups

[ejsmith]

Summary

• keeps this experiment stacked on Upgrade Elasticsearch to 9.5.0 #2416, whose current head already contains the latest main
• makes the canonical stack index an Elasticsearch 9.5 lookup index with one primary shard
• replaces the event endpoint's four stack modes with dedicated /stack-rollups and /stack-rollups/stats resources
• executes event filtering, stack filtering, aggregation, ranking, totals, and deterministic cursor selection in ES|QL with LOOKUP JOIN
• uses before/after keyset cursors for both ranked rollups and stored-stack management; no stack list uses page-number pagination
• removes the mode/fallback/legacy switch, the hidden stack_new filter behavior, and the 20,000-stack-id materialization path
• updates both Svelte and Angular stack consumers and isolates local Elasticsearch/Kibana on ports 9215/5615
• replaces the incompatible Elasticsearch 8-based AppHost health-check package with a small HTTP cluster-health check

Why pursue this

The old stack page was not really paging stacks. It was:

1. splitting a mixed event/stack filter;
2. searching the stack index and materializing up to 20,000 matching stack ids;
3. sending those ids back to Elasticsearch in an event query;
4. asking a terms aggregation for skip + limit + 1 ranked buckets;
5. discarding the first skip buckets in the API; and
6. loading the final stack documents in another request.

That has four structural problems:

• broad filters fail at 20,000 matching stacks;
• deep pages get progressively more expensive and eventually hit the maximum skip;
• ranking by sum, minimum, or cardinality depends on shard-local terms candidates outside Elasticsearch's documented safe ordering cases; and
• the four mode values implicitly changed columns, aggregation shape, sort, and even filtering (stack_new) inside a generic events endpoint.

The join pipeline is a much better fit for the resource we are returning:

filtered events
  -> LOOKUP JOIN matching, non-deleted stack metadata
  -> STATS total/users/first/last BY stack_id
  -> keyset predicate on (sort metric, stack_id)
  -> SORT + LIMIT page_size + 1
  -> hydrate only the returned stack page

All rollup rows now have the same explicit columns. “Most frequent,” “most users,” and “new” are ordinary filters/sorts or saved views, not server-side modes. The chart uses a typed stats endpoint built from the same event/stack semantics.

How much better

These are isolated Elasticsearch 9.5 runs on the same development machine. Each scenario used two warmups and seven measured iterations with alternating execution order. They are synthetic benchmarks, not production capacity numbers, but they directly compare the former query with the new query.

5,000 stacks / 15,000 events / page size 25

Scenario	Former median / p95	Join median / p95	Median change
First page	44.4 / 50.9 ms	16.2 / 18.5 ms	63.5% faster
Page 100	97.1 / 122.4 ms	16.6 / 20.0 ms	82.9% faster
Stack filter	45.5 / 46.8 ms	16.1 / 18.5 ms	64.7% faster
Full API list	n/a	22.2 / 24.7 ms	n/a
Full API stats	n/a	23.4 / 23.9 ms	n/a

25,000 stacks / 25,000 events / page size 25

Scenario	Former median / p95	Join median / p95	Result
First page	failed at 20,000-stack limit	21.7 / 37.3 ms	join succeeds
Page 100	failed at 20,000-stack limit	23.8 / 25.2 ms	join succeeds
Stack filter	27.2 / 37.3 ms	23.4 / 26.0 ms	14.2% faster
Full API list	n/a	26.8 / 28.2 ms	n/a
Full API stats	n/a	27.9 / 30.2 ms	n/a

The important result is not only lower latency. The join removes the cardinality cliff and keeps returned work page-sized as the cursor moves deeper. Page 100 is effectively the same latency as page 1 instead of more than doubling.

API and implementation model

• /stack-rollups is the time-bound ranked stack summary resource. It has fixed rollup columns and supports total, users, first_occurrence, and last_occurrence sorts in both directions.
• /stack-rollups/stats is typed and replaces the stack page's free-form aggregation string.
• /stacks remains the stored stack-document resource used for project stack management. It uses Foundatio repository search-after cursors; it does not need a join.
• cursor tokens include the sort, metric, stack id, resolved UTC time range, and a query fingerprint. Forward and backward traversal use the metric plus stack-id tie-breaker.
• cursors are deterministic live keyset cursors, not point-in-time snapshots.

Intentional breaking changes

• stack_recent, stack_frequent, stack_new, and stack_users event modes are removed and return 400.
• stack list contracts no longer expose or consume page; callers use before/after plus limit.
• “new stacks” is now the explicit first_occurrence:[start TO end] filter instead of a hidden mode mutation.
• generic event count is no longer used for stack-page totals/charts.
• there is no compatibility flag, fallback query, or legacy runtime switch.

Tradeoffs and prerequisites

• Elasticsearch 9.5 is required.
• lookup indices require exactly one primary shard. That simplifies and bounds reads but needs write-throughput and shard-size monitoring before production adoption.
• lookup joins are not snapshots; inserts or updates between cursor requests can move rows, like other live keyset pagination.
• ES|QL COUNT_DISTINCT remains approximate. The overall stack total uses a 40,000 precision threshold and the benchmark accepts a 1% cardinality tolerance.
• partial ES|QL results are disabled so shard failures fail the request instead of returning silently incomplete rankings.

Verification

• full backend suite against isolated Elasticsearch 9.5 on port 9215: 2,891 passed, 4 opt-in skipped, 0 failed
• event endpoint suite: 139 passed
• stack endpoint suite: 52 passed
• exact controlled-data parity for all eight metric/direction sorts, including ties and forward/back traversal
• malformed, conflicting, and fingerprint-mismatched cursor coverage
• opt-in 5k and 25k repeatable benchmark committed in StackRollupBenchmarkTests
• Svelte unit tests: 611 passed across 73 files
• npm run validate: formatting/lint passed; Svelte diagnostics 0 errors, 0 warnings
• npm run build: production build passed
• Playwright against the isolated local app: ranked cursor paging, project-management cursor paging, cache reuse, stack status mutations/failures, detail navigation, chart interaction, WebSocket reconciliation, and notification stress passed
• fresh CI-mode AppHost proof against a uniquely scoped Elasticsearch 9.5 instance on port 9218: stack alias targeted stacks-v2, index.mode=lookup, one primary shard, and the cursor E2E passed
• local /next/ and /api/v2/about smoke checks returned 200
• OpenAPI and endpoint-manifest snapshots regenerated and verified
• hosted CI run 32626696061: version, frontend, API coverage, Docker, and full Aspire/Playwright E2E jobs passed

Official references: LOOKUP JOIN, lookup prerequisites, QSTR, BUCKET, COUNT_DISTINCT, and terms aggregation ordering.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	37%	35%	286	❌
Exceptionless.Core	76%	68%	10491	✔
Exceptionless.Web	85%	69%	7901	✔
Exceptionless.AppHost	70%	65%	181	➖
Summary	79% (25816 / 32710)	68% (12120 / 17868)	18859	✔