feat(mcp): add hyperdx_event_deltas tool

[brandon-pereira]

Summary

Add the hyperdx_event_deltas MCP tool that compares two row groups (target vs baseline) and ranks properties by how much their value distributions differ. Uses the same algorithm as the in-app Event Deltas view (DBDeltaChart).

What changed

New files

• packages/common-utils/src/core/eventDeltas.ts — Pure algorithm functions extracted from the UI: flattenData, getPropertyStatistics, computeComparisonScore, semanticBoost, field classification helpers, sampling config, and rankProperties. Shared by both the UI and MCP server.
• packages/api/src/mcp/tools/query/eventDeltas.ts — The hyperdx_event_deltas MCP tool. Validates input, queries ClickHouse for target/baseline row samples, runs rankProperties, and returns ranked property deltas.
• packages/common-utils/src/__tests__/eventDeltas.test.ts — Unit tests for the shared algorithm.
• packages/api/src/mcp/__tests__/deltasTool.test.ts — Integration tests for the MCP tool (schema validation, time range rejection, algorithm ranking against ClickHouse data).

Modified files

• packages/api/src/mcp/tools/query/index.ts — Register eventDeltas tool.
• packages/app/src/components/deltaChartUtils.ts — Replaced inline algorithm definitions with re-exports from @hyperdx/common-utils/dist/core/eventDeltas. UI-only helpers (mergeValueStatisticsMaps, applyTopNAggregation, computeEntropyScore, filter key converters, constants) remain in place.

How the tool works

Given a source ID, a target group (time window + optional filter), and a baseline group, the tool:

1. Samples rows from each group via renderChartConfig
2. Runs the shared rankProperties algorithm to score every property by distribution delta
3. Returns a ranked list of properties with per-value target%/baseline%/diff% breakdowns

Typical uses: slow vs fast spans, before vs after a deploy, failing vs succeeding requests.

[changeset-bot]

🦋 Changeset detected

Latest commit: 71ad6aa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/api	Patch
@hyperdx/common-utils	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	May 22, 2026 10:41pm

[github-actions]

E2E Test Results

✅ All tests passed • 176 passed • 3 skipped • 1236s

Status	Count
✅ Passed	176
❌ Failed	0
⚠️ Flaky	6
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Large diff: 1317 production lines changed (threshold: 1000)
• Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 4
• Production lines changed: 1317 (+ 447 in test files, excluded from tier calculation)
• Branch: brandon/event-deltas-mcp-tool
• Author: brandon-pereira

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

Deep Review

🟡 P2 -- recommended

• packages/api/src/mcp/tools/query/eventDeltas.ts:321 -- ClickhouseClient is constructed without requestTimeout, so it inherits the 1-hour default (packages/common-utils/src/clickhouse/index.ts:455); server-side max_execution_time: 30 does not bound a stuck TCP read or a pre-execution hang, so the MCP tool can block for up to an hour holding the session.
  • Fix: Pass an explicit requestTimeout (≈35s) to new ClickhouseClient(...) and/or attach a wall-clock setTimeout that calls abortController.abort() and is cleared on success.
  • _{reliability, adversarial}
• packages/api/src/mcp/tools/query/eventDeltas.ts:539 -- Response is returned via JSON.stringify(output) with no size cap; every sibling MCP tool (runEventPatterns, describeSource, helpers) wraps user-shaped output in trimToolResponse (~50KB cap), so wide trace schemas at topN=100, topValuesPerProperty=20 can ship 200-400KB into the MCP conversation.
  • Fix: Run the final output through the existing trimToolResponse helper used by the other query tools.
  • _adversarial
• packages/api/src/mcp/tools/query/eventDeltas.ts:386 -- metadata.getColumns(...).catch(() => null) swallows every column-meta failure; with columnMeta=[], isDenylisted and isIdField always return false, so ID-like columns (TraceId, SpanId, request IDs) rank near the top and the agent gets a polluted result with only a soft columnMetaUnavailable: true flag buried in summary.
  • Fix: Log the underlying error (so operators can detect chronic getColumns failures) and surface a columnMetaError field in the summary describing the cause.
  • _{adversarial, reliability}
• packages/api/src/mcp/__tests__/deltasTool.test.ts:91-107 -- New behavior covers only one of six validation/error branches (target endTime <= startTime); the baseline-window error, the identical-groups guard (eventDeltas.ts:257-275), the source-not-found, wrong-source-kind, connection-not-found, and zero-rows warning branches are all new and untested.
  • Fix: Add integration tests for each isError: true branch and the summary.warning zero-rows path.
  • _{testing, kieran-typescript, correctness}
• packages/api/src/mcp/tools/query/eventDeltas.ts:373 -- let targetSql, baselineSql; is declared without a type annotation, so under the repo's noImplicitAny: false they widen to any; downstream targetSql.sql and targetSql.params (lines 419-420, 427-428) are also any, removing type checks from the most safety-critical interface in the file.
  • Fix: Annotate the declarations (let targetSql: Awaited<ReturnType<typeof renderChartConfig>> | undefined) or restructure so the destructured const [targetSql, baselineSql, colsResult] = await Promise.all(...) is reused at the query sites.
  • _{kieran-typescript}

🔵 P3 nitpicks (22)

• packages/api/src/mcp/tools/query/eventDeltas.ts:435-440 -- targetRes as { json: () => Promise<{ data: any[] }> } casts away the typed return of ClickhouseClient.query<'JSON'>(...); the elsewhere-in-repo pattern is await res.json() directly.
  • Fix: Drop the cast and use await targetRes.json<{ data: Record<string, unknown> }>().
• packages/api/src/mcp/tools/query/eventDeltas.ts:247-250 -- input.target.where ?? '' and input.target.whereLanguage ?? 'lucene' are dead fallbacks because the zod schema already applies .default('') / .default('lucene').
  • Fix: Remove the ?? fallbacks and rely on the zod-inferred non-undefined types.
• packages/api/src/mcp/tools/query/eventDeltas.ts:348 -- whereLanguage: group.where ? group.whereLanguage : 'sql' silently rewrites a user-supplied lucene choice to sql when where is empty; harmless today but a latent compatibility bug.
  • Fix: Pass group.whereLanguage unconditionally and let the renderer handle the empty-where short-circuit.
• packages/api/src/mcp/tools/query/eventDeltas.ts:415-454 -- The shared AbortController is only signalled inside catch; ClickHouse will not abort the in-flight server query unless cancel_http_readonly_queries_on_client_close is enabled, so a sibling query can keep running for up to the 30s max_execution_time after the tool returns.
  • Fix: Add a finally block that always calls abortController.abort() and consider rejecting fast via Promise.allSettled + explicit abort on first rejection.
  • _{correctness, adversarial, reliability}
• packages/api/src/mcp/tools/query/eventDeltas.ts:257-275 -- The sameWhere && sameWindow guard uses exact string and timestamp equality; an agent trivially bypasses with AND 1=1 or a 1ms endTime shift and gets back zero-score rankings labelled as success.
  • Fix: Either soften the guard to a warning, or detect the bypassed-but-empty-result case in the success path and surface it explicitly.
• packages/api/src/mcp/tools/query/eventDeltas.ts:519-526 -- Empty target/baseline returns success with properties: [] plus a summary.warning string; the identical-groups branch at line 257 returns isError: true for a semantically similar "won't yield useful output" condition, so the contract is asymmetric.
  • Fix: Choose one direction -- either make empty-rows isError, or document explicitly in the description that warning means "do not trust properties".
• packages/api/src/mcp/tools/query/eventDeltas.ts:24-41 -- Time-range validity is enforced at runtime via parseGroupTimeRange but is invisible to the LLM agent reading the zod schema; the agent learns the constraint only by erroring once.
  • Fix: Express endTime > startTime via .refine on groupSchema so it surfaces in the introspected tool schema.
• packages/api/src/mcp/tools/query/eventDeltas.ts:376-411 -- Generic catch at line 401 emits Failed to build sample queries: ... with no indication of whether the failing input is target.where, baseline.where, or column metadata.
  • Fix: Wrap each promise with a per-side .catch that re-throws with a prefix (target: / baseline: / columnMeta:).
• packages/common-utils/src/core/eventDeltas.ts:25 -- The eslint-disable-next-line security/detect-object-injection sits above the function recurse(...) declaration, which has no injection sink; the three real sinks each carry their own inline disable, so the line-25 annotation is dead and misleads readers.
  • Fix: Move the justification to a block comment above the function (covering all three sinks) or delete the bare disable.
• packages/common-utils/src/core/eventDeltas.ts:23-44 -- Public flattenData and RankPropertiesOptions use Record<string, any>, defeating downstream type-checking; the algorithm never accesses typed paths and only String()-coerces values.
  • Fix: Switch the public signatures to Record<string, unknown>.
• packages/common-utils/src/core/eventDeltas.ts:189-221 -- isHighCardinality's parameters still use outlier*/inlier* while the surrounding module standardised on target/baseline; rankProperties (lines 385-391) maps target→outlier inline at every call.
  • Fix: Rename the parameters to targetValueOccurences/baselineValueOccurences; brand-new exported function, no external callers.
• packages/common-utils/src/core/eventDeltas.ts:27 -- Object(cur) !== cur is a clever primitive check that allocates a boxed wrapper per recursion step; the modern cur === null || typeof cur !== 'object' is more obvious and avoids the allocation.
  • Fix: Replace the idiom with the explicit primitive check.
• packages/app/src/components/deltaChartUtils.ts:13-33 -- stripTypeWrappers is named in the export { ... } from re-export AND imported again at line 33 for local use; the comment explaining why is itself a smell.
  • Fix: Use the import-then-re-export pattern (import { ... } from '...'; export { ... };) so the symbol list is stated once.
• packages/app/src/components/deltaChartUtils.ts:1 -- Re-exporting the new flattenData and getPropertyStatistics silently fixes two old UI bugs (empty-array sentinel was dead code; divisor ?? 0 could yield Infinity; raw value Map keys collided across types) -- the behaviour change is intentional but undocumented.
  • Fix: Note the behaviour change in the changeset or commit body so reviewers of DBDeltaChart/PropertyComparisonChart know the silent fixes happened.
  • _{correctness, testing}
• packages/api/src/mcp/tools/query/eventDeltas.ts:58-67 -- MCP tool defaults sampleSize to 500 with min=100, while common-utils exports SAMPLE_SIZE=1000, MIN_SAMPLE_SIZE=500, MAX_SAMPLE_SIZE=5000; the two callers have meaningfully different definitions of "minimum useful sample" with no explanation.
  • Fix: Either reference the shared constants in the zod schema or comment why the MCP tool intentionally diverges.
• packages/api/src/mcp/tools/query/eventDeltas.ts:217-541 -- The inline tool handler is ~325 lines and mixes validation, query construction, query execution, and output shaping; the same isError: true as const, content: [...] early-return pattern is repeated seven times.
  • Fix: Extract validateInputs, fetchAndValidateSource, sampleBothGroups, and shapeResponse as module-private helpers and add a small errText(msg) helper.
• packages/api/src/mcp/tools/query/eventDeltas.ts:117-150 -- topDeltasForProperty re-implements per-value share normalisation that computeComparisonScore (common-utils/src/core/eventDeltas.ts:232-265) already computes; a future change to share semantics has to land in two places.
  • Fix: Extract a shared normalizeValueShares (or perValueDeltas) helper in common-utils and call it from both sites.
• packages/api/src/mcp/__tests__/deltasTool.test.ts:20-23 -- Test setup uses explicit let team: any; let user: any; let logSource: any; let connection: any;; types are inferable from getLoggedInAgent, Source.create, and Connection.create.
  • Fix: Drop the : any annotations or type via Awaited<ReturnType<typeof getLoggedInAgent>>.
• packages/common-utils/src/__tests__/eventDeltas.test.ts:61, 68, 72 -- expect(computeComparisonScore(...)).toBe(100) and .toBe(0) use exact float equality; current inputs happen to terminate in binary, but a refactor changing operation order could flake.
  • Fix: Use toBeCloseTo for non-identity score comparisons.
• packages/common-utils/src/core/eventDeltas.ts:35 -- for (const p in cur) walks inherited enumerable properties; safe today because the input is JSON.parse-derived, but the line-25 eslint-disable justification ("known object keys via recursion") is misleading about why.
  • Fix: Switch to for (const p of Object.keys(cur)) to make safety explicit.
• packages/api/src/mcp/tools/query/eventDeltas.ts:161-212 -- The description string is ~50 lines and mixes well-justified WHY/WHEN guidance with a WHAT-restatement of topDeltas / targetCount / baselineCount field shapes that will drift if the response evolves.
  • Fix: Trim the OUTPUT SHAPE paragraph to the one fact agents can't derive ("topDeltas covers per-value comparison; no nested target/baseline distributions") and drop the field-by-field recitation.
• packages/common-utils/src/core/eventDeltas.ts:62, 304, 311 -- getPropertyStatistics, getStableSampleExpression, and computeEffectiveSampleSize lack JSDoc while their siblings have it; same for parseGroupTimeRange and topDeltasForProperty in the MCP file.
  • Fix: Add a one-line JSDoc to each bare exported helper.

---

Reviewers (7): correctness, adversarial, testing, security, kieran-typescript, maintainability, reliability, performance.

Testing gaps:

• flattenData arrays-of-objects (arr[0].x, arr[1].x) and top-level empty input (flattenData({})) are not asserted.
• computeComparisonScore one-sided-empty branch (outlierSum === 0 || inlierSum === 0, core/eventDeltas.ts:245-255) is not exercised.
• getBaseColumnName, nested stripTypeWrappers, isIdField Array(String) branch, computeEffectiveSampleSize, and getStableSampleExpression have no direct unit tests.
• The behavioural changes from the UI extraction (?? 1 instead of ?? 0; String(value) coercion of null/undefined/non-string values at core/eventDeltas.ts:88-105) are not pinned by assertions.
• The whereLanguage: 'sql' code path through the MCP integration test is never exercised end-to-end.
• topDeltasForProperty's targetTotal === 0 short-circuit and tie-breaking sort are not directly tested.
• No assertion that abortController.abort() actually cancels the in-flight HTTP request when one of the two parallel queries rejects.