fix(ingestion): resolve Airbyte source/destination config from public API for lineage#30426
Conversation
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
🔴 Playwright Results — 3 test failure(s)✅ 4533 passed · ❌ 3 failed · 🟡 23 flaky · ⏭️ 102 skipped
Genuine Failures (failed on all attempts)❌
|
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
dbf8529 to
d945339
Compare
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
| # Internal API (`/sources/get`) returns `sourceName` + `connectionConfiguration`; | ||
| # the public API (`/api/public/v1/sources/{id}`) returns `sourceType` + `configuration`. | ||
| sourceName: Optional[str] = None # noqa: N815, UP045 | ||
| sourceType: Optional[str] = None # noqa: N815, UP045 |
There was a problem hiding this comment.
I imagine this is returned by the sources/{sourceId} endpoint from Airbyte?
There was a problem hiding this comment.
Yes — these fields come from the Airbyte public API: GET /api/public/v1/sources/{sourceId} and GET /api/public/v1/destinations/{destinationId}, which return sourceType/destinationType (slug, e.g. postgres) and configuration. The internal API (/sources/get) instead returns sourceName (display name) and connectionConfiguration. Confirmed against a live self-hosted instance — the model now accepts both shapes via resolved_type/resolved_configuration.
harshach
added
the
safe to test
| # while the public API (`api/public/v1`) reports them as slugs (e.g. "postgres"). | ||
| # These maps let lineage resolution accept either form. | ||
| SOURCE_TYPE_LOOKUP = { | ||
| AirbyteSource.MYSQL.value: AirbyteSource.MYSQL, |
There was a problem hiding this comment.
do we need more of these types? to cover all the connectors that Airbyte supports?
There was a problem hiding this comment.
Good question. This PR intentionally keeps the scope narrow — it only remaps the already-supported connector types (Postgres/MySQL/MSSQL/MongoDB) to their public-API slugs, so it stays a focused bug fix for the public-API lineage path rather than changing coverage.
Happy to extend it, though. The highest-value additions are warehouse destinations — Snowflake/BigQuery/Redshift — whose configuration exposes database + schema directly, so they'd need only a lookup entry plus the existing default extraction. That would also close the Snowflake-destination case in #26993. Additional DB sources (Oracle, Redshift, ClickHouse, etc.) are doable too but each needs its own config-key handling.
Would you prefer I expand the lookups (starting with warehouse destinations) in this PR, or land this focused fix first and follow up with a coverage-expansion PR?
… API for lineage
Airbyte pipeline lineage read the connector type and config only from the
internal-API fields (sourceName/destinationName, connectionConfiguration). On the
public API (api/public/v1) those live under sourceType/destinationType and
configuration, so schema/database resolved to None and no lineage was produced.
The connector-type guard also only matched internal display names ('Postgres'),
not public API slugs ('postgres').
- Response models expose resolved_type/resolved_configuration that fall back to
the public-API fields.
- Add SOURCE/DESTINATION_TYPE_LOOKUP mapping both display names and public slugs
to the supported connector enums.
- Lineage helpers use the resolved values.
Adds a public-API lineage regression test; internal-API behavior is unchanged.
Addresses the public-API portion of open-metadata#26993 (warehouse destinations such as
Snowflake remain a separate enhancement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Cale <cale.anderson93@gmail.com>
d945339 to
60b0e08
Compare
Code Review ✅ Approved 1 resolved / 1 findingsFixes Airbyte pipeline lineage generation for the public API by adding support for sourceType/destinationType and configuration fields alongside normalization lookup maps. No issues found. ✅ 1 resolved✅ Edge Case: MongoDB source config can raise AttributeError
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
3 participants
Fixes #30428
Addresses the public-API portion of #26993.
Problem
Airbyte pipeline lineage is not produced when the service uses the public API (
api/public/v1), even for connector types that are otherwise supported (Postgres/MySQL/MSSQL).Two causes, both in the source/destination resolution used by
get_source_table_details/get_destination_table_details:Config field name. The models only read
sourceName/destinationNameandconnectionConfiguration. The public API returns the connector type undersourceType/destinationTypeand the config underconfiguration:So
connectionConfigurationisNone→schema/databaseresolve toNone→ lineage silently skipped.Connector-type identifier. The support guard compares against internal-API display names (
AirbyteSource.POSTGRES = "Postgres",MSSQL = "Microsoft SQL Server (MSSQL)"), but the public API returns slugs (postgres,mssql,mysql). Even after (1), the guard fails and returnsNone.Fix
AirbyteSourceResponse/AirbyteDestinationResponsenow also acceptsourceType/destinationTypeandconfiguration, and exposeresolved_type/resolved_configurationthat fall back across both API shapes.SOURCE_TYPE_LOOKUP/DESTINATION_TYPE_LOOKUPmaps inconstants.pyrecognize both display names and public-API slugs.Internal-API behavior is unchanged (display names still map).
Testing
test_yield_pipeline_lineage_details_public_apiasserting lineage is produced from a public-API-shaped source/destination (mirrors the existing internal-API test).Scope
This makes lineage work on the public API for the already-supported DB connector types. Adding warehouse destinations (Snowflake/BigQuery/Redshift) — the destination used in #26993 — is a larger enhancement (which destination configs to support) and is intentionally left out of this PR; noted in #26993.
Greptile Summary
Updates Airbyte lineage resolution to support public-API response fields and connector slugs.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failures remain.
Important Files Changed
Reviews (3): Last reviewed commit: "fix(ingestion): resolve Airbyte source/d..." | Re-trigger Greptile