Skip to content

Support shared catalog config across tables in iceberg-source#6727

Open
lawofcycles wants to merge 1 commit intoopensearch-project:mainfrom
lawofcycles:feature/iceberg-source-shared-catalog
Open

Support shared catalog config across tables in iceberg-source#6727
lawofcycles wants to merge 1 commit intoopensearch-project:mainfrom
lawofcycles:feature/iceberg-source-shared-catalog

Conversation

@lawofcycles
Copy link
Copy Markdown
Contributor

Description

Adds support for a top level catalog configuration in iceberg-source that applies to all tables by default. When a table specifies its own catalog, it fully replaces the top level definition.

This reduces configuration duplication when multiple tables share the same catalog.

iceberg:
 catalog:
   type: rest
   uri: "http://iceberg-rest-catalog:8181"
   io-impl: "org.apache.iceberg.aws.s3.S3FileIO"
 tables:
    - table_name: "db.table_a"
     identifier_columns: ["id"]
    - table_name: "db.table_b"
     identifier_columns: ["id"]
     catalog:
       type: glue
       warehouse: "s3://other-bucket/warehouse"
       io-impl: "org.apache.iceberg.aws.s3.S3FileIO"

Issues Resolved

Partially addresses #6726 (catalog config sharing). Per-table shuffle overrides will be addressed after #6682 is merged.

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant