HIVE-29595: HiveClientCache does not respect metastore.client.impl#6462
Open
jlalwani-amazon wants to merge 1 commit intoapache:masterfrom
Open
HIVE-29595: HiveClientCache does not respect metastore.client.impl#6462jlalwani-amazon wants to merge 1 commit intoapache:masterfrom
jlalwani-amazon wants to merge 1 commit intoapache:masterfrom
Conversation
jlalwani-amazon
changed the title
jlalwani-amazon
changed the title
jlalwani-amazon
changed the title
HiveClientCache (used by HCatalog's HCatInputFormat/HCatOutputFormat) hardcodes HiveMetaStoreClient, ignoring the metastore.client.impl config. This means any custom IMetaStoreClient implementation configured via hive.metastore.client.impl is not used when accessing Hive tables through HCatalog. This patch makes HiveClientCache respect the configured metastore client implementation by: 1. getNonCachedHiveMetastoreClient() now reads the class name from MetastoreConf.ConfVars.METASTORE_CLIENT_IMPL and passes it to RetryingMetaStoreClient.getProxy(). Previously hardcoded to HiveMetaStoreClient. 2. CacheableHiveMetaStoreClient refactored from 'extends HiveMetaStoreClient' to 'implements InvocationHandler'. It now wraps any IMetaStoreClient via Proxy.newProxyInstance, delegating metastore calls to the underlying client while managing cache lifecycle (acquire/release/teardown). 3. ICacheableMetaStoreClient changed from 'extends IMetaStoreClient' to 'extends Closeable' since the proxy handles IMetaStoreClient. 4. getOrCreate() creates the client via getNonCachedHiveMetastoreClient and wraps it in the proxy. 5. invoke() unwraps InvocationTargetException so callers see the real exception rather than a reflection wrapper.
jlalwani-amazon
force-pushed
the
HIVE-12679-hcatalog-pluggable-metastore-client
branch
from
6cb7754 to
a8a2067
Compare
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
HIVE-12679 (fixed in 4.2.0) added support for pluggable
IMetaStoreClientviametastore.client.implin the main Hive path (HiveMetaStoreClientBuilder). However,HiveClientCache— used by HCatalog'sHCatInputFormat/HCatOutputFormat— was not updated and still hardcodesHiveMetaStoreClient:getNonCachedHiveMetastoreClient()callsRetryingMetaStoreClient.getProxy(hiveConf, true)which hardcodesHiveMetaStoreClient.class.getName()CacheableHiveMetaStoreClient extends HiveMetaStoreClient— only works with the Thrift clientThis means custom
IMetaStoreClientimplementations (e.g., AWS Glue Data Catalog) are ignored when accessing Hive tables through HCatalog, even though they work correctly through the main Hive path.Fix
getNonCachedHiveMetastoreClient()now reads the class name fromMetastoreConf.ConfVars.METASTORE_CLIENT_IMPLand passes it toRetryingMetaStoreClient.getProxy(). Retry behavior is preserved.CacheableHiveMetaStoreClientrefactored fromextends HiveMetaStoreClienttoimplements InvocationHandler. It wraps anyIMetaStoreClientviaProxy.newProxyInstance, delegating metastore calls to the underlying client while managing cache lifecycle (acquire/release/teardown).ICacheableMetaStoreClientchanged fromextends IMetaStoreClienttoextends Closeablesince the proxy handlesIMetaStoreClient.invoke()unwrapsInvocationTargetExceptionso callers see the real exception.Testing
mvn test -pl metastore -Dtest=TestHiveClientCache -Drat.skip=true -am— 5/5 pass, 0 failures.Note to committers
I don't have Apache JIRA access to file a new ticket. Could a committer please create one (component: Metastore, HCatalog) and I'll update the PR title? This is a follow-up to HIVE-12679 which missed the HCatalog code path.