Skip to content

Core, Hive: Fix NPE in commit status check when a create commit fails - #17464

Open
superdiaodiao wants to merge 2 commits into
apache:mainfrom
superdiaodiao:fix-create-commit-status-npe
Open

Core, Hive: Fix NPE in commit status check when a create commit fails#17464
superdiaodiao wants to merge 2 commits into
apache:mainfrom
superdiaodiao:fix-create-commit-status-npe

Conversation

@superdiaodiao

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

When a CREATE TABLE commit fails with a non-specific exception (e.g. a Thrift socket timeout from HMS), HiveTableOperations#doCommit calls checkCommitStatus(...) to determine whether the commit landed. For a table that was never persisted, doRefresh() swallows NoSuchObjectException (expected for creates) and refreshes from a null metadata location, leaving current() as null. checkCurrentMetadataLocation then dereferences the null metadata and throws an NPE on every status-check attempt. The NPEs are suppressed (suppressFailureWhenFinished), the status stays UNKNOWN, and the user gets a CommitStateUnknownException recommending manual intervention - even though the outcome was knowable.

This PR null-guards the commit status supplier in two places:

  • BaseMetastoreTableOperations#checkCurrentMetadataLocation: if refresh() yields no metadata, the table does not exist in the catalog, so the new metadata location cannot be current or in history - return false.
  • HiveViewOperations#checkCurrentMetadataLocation: same pattern and null-guard for the view counterpart.

With the guard in place:

This revives the core part of #6499, which diagnosed and fixed the same problem in Dec 2022 but was closed by the stale bot without a decision. The code has since moved (the status check was refactored into BaseMetastoreOperations by #12637), so the guard now lives in checkCurrentMetadataLocation.

Are these changes tested?

Yes - a new core unit test, TestBaseMetastoreTableOperations, with a minimal BaseMetastoreTableOperations subclass whose doRefresh() mimics HiveTableOperations for a never-persisted table (missing table is not an error when no metadata location is known -> refresh from null -> current metadata stays null):

  • strictStatusCheckIsFailureWhenTableWasNeverPersisted - fails without the fix (the NPE is suppressed and the status stays UNKNOWN) and passes with it.
  • statusCheckIsUnknownWhenTableWasNeverPersisted - pins the relaxed-check semantics for the same scenario.

Verified locally: the strict test fails on main without the fix and passes with it; :iceberg-core:test and :iceberg-hive-metastore:test both pass with the change.

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Hive integration test drives the actual create-commit-failure path end-to-end. The predecessor PR #6499 included a TestHiveCommits integration test that mocked persistTable to throw when base==null and asserted no NPE plus correct CommitStatus. The new TestBaseMetastoreTableOperations uses a minimal stub that bypasses the real HiveTableOperations.doRefresh() entirely, so the NoSuchObjectException-swallow-then-refreshFromMetadataLocation(null) wiring is not directly verified. A Hive integration test would cover the full end-to-end path and guard against a future doRefresh refactor silently breaking the fix.

@uros-b

uros-b commented Aug 2, 2026

Copy link
Copy Markdown
Member

Also, no corresponding test in TestHiveViewCommits for the create-view-commit-fails path. The HiveViewOperations null guard is only transitively covered by the base-class stub; a parallel view test would pin the view-specific doRefresh wiring against future refactors.

Change-Id: Ida46b3caa2ae3c40c99a86df89c36b1442952f2f
@superdiaodiao

Copy link
Copy Markdown
Author

Added a TestHiveViewCommits case for the create-view-commit-fails path in 7105ad0. It drives a failed create through the real HiveViewOperations (spied persistTable throwing TException), asserts the relaxed status check maps to UNKNOWN, and pins the view-specific doRefresh wiring: refresh() yields null metadata for a never-persisted view and checkCurrentMetadataLocation resolves to false instead of NPE-ing. The reflection-based supplier assertion fails on main without the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hive: NullPointerException in checkCommitStatus when CREATE TABLE commit fails, masking FAILURE as CommitStateUnknownException

2 participants