Skip to content

feat(aicontext): enrich /context with type enums, profile detail, and service identity#30456

Open
pmbrull wants to merge 4 commits into
mainfrom
pmbrull/la-paz
Open

feat(aicontext): enrich /context with type enums, profile detail, and service identity#30456
pmbrull wants to merge 4 commits into
mainfrom
pmbrull/la-paz

Conversation

@pmbrull

@pmbrull pmbrull commented Jul 24, 2026

Copy link
Copy Markdown
Member

Describe your changes

Enriches the AI /context endpoint so the ai-platform Analytics Agent can stop making its separate schema (getEntityByFqn) and tableProfile/latest calls — closing the gaps identified in open-metadata/ai-platform#930.

Schema (aiContext.json) — every addition is directly consumed by the agent, nothing speculative:

  • fieldContext.dataTypeEnum — raw ColumnDataType name (e.g. BIGINT) for programmatic type branching; dataType keeps carrying the display form (e.g. bigint(20)). This was the only schema-call gap.
  • columnProfileSummary.uniqueProportion / mean / median / cardinalityDistribution — the exact profile signals the agent's per-column stats builder consumes (PK detection, numeric ranges, exact/sampled value lists). cardinalityDistribution reuses the existing generated class: the inline object in table.json is hoisted to a named definition with an explicit javaType matching the class it already generated, so Java/Kotlin consumers see no change.
  • observability.profileSample / profileSampleType — sample coverage metadata, needed to gate "Exact values" labels on full-dataset profiles.
  • Top-level name, service (entity reference) and populated serviceType — query execution needs the service id/name for provenance (workbench links, chart follow-ups) without re-fetching the entity.

Deliberately not added: sample data, valuesCount, or opt-in "give me all" flags — the agent consumes none of them, and the added fields are small enough that the default payload stays lean. The markdown rendering is unchanged; only the JSON projection carries the new fields.

Note on permissions: /context requires VIEW_ALL, which subsumes VIEW_DATA_PROFILE, so embedding the profile does not widen access.

Companion PR (consumer side): open-metadata/ai-platform — linked in comments once opened.

Type of change

  • Improvement

Checklist

  • I have read the CONTRIBUTING document.
  • I have added tests that prove my fix is effective or that my feature works. (AIContextBuilderTest)

🤖 Generated with Claude Code

Greptile Summary

Enriches AI asset context responses with service identity, raw column types, and detailed profile metadata.

  • Adds table service references and service types to generated AI context.
  • Includes profile sampling, uniqueness, numeric statistics, and cardinality distributions.
  • Hoists the cardinality-distribution schema into a reusable definition and regenerates TypeScript models.
  • Extends AI context builder tests for the new fields.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure eligible for this follow-up review remains.

No blocking failure remains.

Important Files Changed

Filename Overview
openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java Populates asset name, table service identity, raw type enums, sample metadata, and detailed column-profile statistics.
openmetadata-spec/src/main/resources/json/schema/type/aiContext.json Extends the canonical AI context contract with identity, type, sampling, and profile fields.
openmetadata-spec/src/main/resources/json/schema/entity/data/table.json Hoists cardinality distribution into a reusable schema definition with an explicit Java type.
openmetadata-service/src/test/java/org/openmetadata/service/aicontext/AIContextBuilderTest.java Adds coverage for raw type enums, profile details, sampling metadata, and table service identity.
openmetadata-ui/src/main/resources/ui/src/generated/type/aiContext.ts Regenerates the TypeScript AI context contract from the expanded schema.

Sequence Diagram

sequenceDiagram
    participant Consumer as AI/MCP Consumer
    participant Context as /context Endpoint
    participant Builder as AIContextBuilder
    participant Profile as Table Profile Repository
    Consumer->>Context: Request asset context
    Context->>Builder: Build context for entity
    Builder->>Profile: Load latest table profile
    Profile-->>Builder: Sampling and column statistics
    Builder-->>Context: Enriched AIContext
    Context-->>Consumer: Identity, schema, and profile details
Loading

Reviews (4): Last reviewed commit: "Merge branch 'main' into pmbrull/la-paz" | Re-trigger Greptile

Context used:

Enrich the /context payload so the ai-platform Analytics Agent can drop
its separate schema and tableProfile/latest calls
(open-metadata/ai-platform#930):

- fieldContext.dataTypeEnum: raw ColumnDataType name for programmatic
  type branching (dataType keeps the display form)
- columnProfileSummary: uniqueProportion, mean, median, and the
  cardinalityDistribution (reusing the generated class, now a named
  definition in table.json)
- observability: profileSample + profileSampleType so callers can
  sample-gate the column stats
- top-level name, service ref, and populated serviceType so query
  execution has the service identity without re-fetching the entity

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Jul 24, 2026
@pmbrull

pmbrull commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

Consumer-side PR: open-metadata/ai-platform#931 (drops the redundant schema + profile calls in the Analytics Agent once this lands).

@github-actions

Copy link
Copy Markdown
Contributor

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

Copilot AI review requested due to automatic review settings July 24, 2026 09:55
@github-actions
github-actions Bot requested a review from a team as a code owner July 24, 2026 09:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Review follow-up: populateProfile mixed profile-level metadata with
per-column summary construction; hoist the latter into
toColumnProfileSummary and map columns via a stream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@pmbrull

pmbrull commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

Review follow-up in 06702e6: extracted the per-column summary construction out of populateProfile into toColumnProfileSummary and mapped columns via a stream — populateProfile now only handles the profile-level metadata.

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 65%
65.67% (76782/116913) 49.52% (46062/93011) 50.76% (13891/27364)

Copilot AI review requested due to automatic review settings July 24, 2026 13:18
@pmbrull
pmbrull requested a review from a team as a code owner July 24, 2026 13:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@gitar-bot

gitar-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 1 resolved / 1 findings

Refactors the AI context builder to extract per-column summary logic and enriches /context responses with service identity, raw data type enums, and detailed profiling metrics. No issues found.

✅ 1 resolved
Quality: populateProfile exceeds method-size / column-mapping could be extracted

📄 openmetadata-service/src/main/java/org/openmetadata/service/aicontext/AIContextBuilder.java:408-422
With the new fields, populateProfile now spans ~25 lines and mixes two concerns (profile-level metadata and per-column summary building), exceeding the ≤15-line/one-responsibility guideline. Extract the ColumnProfileSummary construction into a helper (e.g. toColumnProfileSummary(Column)) and map columns via listOrEmpty(...).stream() to keep each method focused. Minor and consistent with the pre-existing structure, so optional.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 574068096a99a29fb1b571b5ca7ef07f6a819b8d in Playwright run 30096396615, attempt 1.

✅ 738 passed · ❌ 0 failed · 🟡 2 flaky · ⏭️ 5 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 54m 29s

⏱️ Max setup 3m 18s · max shard execution 19m 43s · max shard-job elapsed before upload 23m 21s · reporting 5s

🌐 196.50 requests/attempt · 2.54 app boots/UI scenario · 12.60% common-shard skew

Optimization targets still in progress:

  • Application boot ratio was 2.54 per UI scenario (1942 boots / 764 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 156 0 0 3 0 0
✅ Shard chromium-02 166 0 0 0 0 0
🟡 Shard chromium-03 144 0 1 0 0 0
🟡 Shard chromium-04 134 0 1 0 0 0
✅ Shard data-asset-rules-01 61 0 0 0 0 0
✅ Shard domain-isolation-01 14 0 0 0 0 0
✅ Shard global-state-01 23 0 0 0 0 0
✅ Shard ingestion-01 1 0 0 0 0 0
✅ Shard reindex-01 2 0 0 0 0 0
✅ Shard search-01 10 0 0 0 0 0
✅ Shard search-rbac-01 27 0 0 2 0 0
🟡 2 flaky test(s) (passed on retry)
  • Pages/Entity.spec.tsDomain Propagation (shard chromium-03, 1 retry)
  • Pages/CustomProperties.spec.tsShould clear search and show all properties for apiEndpoint in right panel (shard chromium-04, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants