Skip to content

[AMORO-4268][AMS] Support HDFS owner impersonation for optimizing commits - #4333

Open
johntomcat7408-cmyk wants to merge 1 commit into
apache:masterfrom
johntomcat7408-cmyk:feature-4268-hdfs-owner-impersonation
Open

[AMORO-4268][AMS] Support HDFS owner impersonation for optimizing commits#4333
johntomcat7408-cmyk wants to merge 1 commit into
apache:masterfrom
johntomcat7408-cmyk:feature-4268-hdfs-owner-impersonation

Conversation

@johntomcat7408-cmyk

Copy link
Copy Markdown
Contributor

Why are the changes needed?

AMS currently constructs optimizing commits with tables whose Hadoop FileIO runs as the catalog service identity. In secured, table-owner-controlled warehouses, optimizer tasks can succeed while the AMS commit fails to write metadata or clean up files.

This is the first scoped part of #4268. It covers AMS-side optimizing commits only; optimizer task writes and other maintenance operations are unchanged.

Brief change log

  • Add opt-in hdfs.impersonation.enabled configuration, disabled by default. An explicit table value overrides the direct catalog value and the catalog table. default.
  • Scope owner impersonation to table loading for AMS optimizing commits so the resulting FileIO retains the owner identity for commit-time reads, writes, deletes, and executor-thread cleanup.
  • Resolve the current owner from Hive Metastore for Hive-backed Iceberg, Mixed-Iceberg, and Mixed-Hive tables, and from refreshed Iceberg metadata for other Iceberg, Mixed-Iceberg, and internal tables.
  • Require SIMPLE or KERBEROS authentication, HadoopFileIO, and a non-empty owner. Proxy and configuration failures are propagated without falling back to the catalog service user.
  • Keep ordinary table loads unchanged and document the Hadoop proxy-user and authorization requirements.

How was this patch tested?

  • Added positive and negative tests for configuration precedence, owner resolution, caller and executor identities, concrete FileIO entry points, unsupported configurations, missing owners, and no-fallback behavior.
  • Screenshots are not applicable because this change has no UI surface.
  • Ran focused tests and validation locally.

Commands and results:

  • .\mvnw.cmd test -pl amoro-format-iceberg -am -Dtest=TestAuthenticatedHadoopFileIO,TestTableOwnerResolver -Dsurefire.failIfNoSpecifiedTests=false — 17 tests passed.
  • .\mvnw.cmd test -pl amoro-format-mixed/amoro-mixed-hive -am -Dtest=TestMixedHiveTablesImpersonation -Dsurefire.failIfNoSpecifiedTests=false — 2 tests passed.
  • TestInternalIcebergHandlerImpersonation — 1 test passed in the AMS-focused run.
  • .\mvnw.cmd validate -pl amoro-format-iceberg,amoro-format-mixed/amoro-mixed-hive,amoro-ams -am — all 15 reactor modules passed Spotless and Checkstyle validation.
  • git diff HEAD^ HEAD --check — passed.

The AMS-focused run could not initialize TestOptimizingQueue's Hive metastore on Windows because HADOOP_HOME/winutils.exe is unavailable. A real Kerberos/MiniDFS/HMS environment and the full test suite were not run locally.

Compatibility and risks

  • The feature is opt-in and disabled by default.
  • There are no Thrift, REST, database schema, or dependency changes.
  • When enabled, an invalid owner, unsupported authentication/FileIO, or Hadoop proxy authorization failure intentionally fails the optimizing commit instead of retrying as the service user.
  • The conventional hdfs owner is not special-cased; cluster proxy-user policy remains authoritative.

Provenance and contribution boundaries

The implementation is original to this PR. It builds on the existing TableMetaStore#doAsImpersonating and authenticated FileIO patterns already present in upstream commit 96a4d781aac60c4080b3f4a41322848e2308649a. No code was copied from open or unmerged pull requests or third-party sources.

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs/admin-guides/managing-catalogs.md

…mits

Add opt-in table-owner proxy impersonation to AMS-side optimizing commits. Resolve the effective owner from Hive Metastore or refreshed Iceberg metadata, keep ordinary table loads unchanged, and fail without service-user fallback.

Document Hadoop proxy-user requirements and cover configuration precedence, owner resolution, FileIO identity, and failure paths. Optimizer task writes and other maintenance operations remain out of scope.
@github-actions github-actions Bot added type:docs Improvements or additions to documentation module:ams-server Ams server module module:mixed-hive Hive moduel for Mixed Format module:common labels Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module module:common module:mixed-hive Hive moduel for Mixed Format type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant