HADOOP-19872: Exclude Engine API when building with OpenSSL 3.x#8550
Open
rohit-kb wants to merge 1 commit into
Open
HADOOP-19872: Exclude Engine API when building with OpenSSL 3.x#8550rohit-kb wants to merge 1 commit into
rohit-kb wants to merge 1 commit into
Conversation
|
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Pull request overview
This PR addresses native hadoop-common build failures against OpenSSL 3.x builds where the legacy Engine API is removed (e.g., RHEL 10 with OPENSSL_NO_ENGINE) by compiling out ENGINE_* usage in OpensslSecureRandom.c.
Changes:
- Guarded Engine API function pointers and dynamic symbol loading on UNIX with
#if OPENSSL_VERSION_NUMBER < 0x30000000L. - Updated UNIX thread-locking callback compilation to only apply to OpenSSL
< 1.1.0. - Disabled
rdrandengine initialization/cleanup logic when compiling against OpenSSL 3.x.
Comments suppressed due to low confidence (1)
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:319
- The OpenSSL 3.x guard added here disables all ENGINE usage at compile time, but the Windows initSR path still unconditionally LOAD_DYNAMIC_SYMBOL()s ENGINE_* symbols. On an OpenSSL 3 build where the Engine API is removed (OPENSSL_NO_ENGINE), Windows will throw UnsatisfiedLinkError during initSR before RAND_bytes/ERR_get_error are usable. Please apply the same OPENSSL_VERSION_NUMBER < 0x30000000L (or equivalent) conditional around the Windows ENGINE_* symbol loads (and related typedef/static declarations if needed), leaving RAND_bytes/ERR_get_error loading unconditional.
#if OPENSSL_VERSION_NUMBER < 0x30000000L
#if OPENSSL_VERSION_NUMBER < 0x10100000L
locks_setup();
dlsym_ENGINE_load_rdrand();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
pan3793
changed the title
Member
|
@rohit-kb, this part has been broken again and again. Actually, the SSL ENGINE API is used solely to opt into Intel's RDRAND hardware RNG as OpenSSL's default random source. And the claim of
is dated and should be qualified. "Much faster" was a reasonable statement circa 2014 (HADOOP-10693) but is no longer a safe blanket claim on modern JDKs. I'd like to request a re-evaluation of the performance on the modern JDK, and if it does not have a performance advantage over |
Contributor
Author
|
Thanks @pan3793 for the review. I will try to benchmark this. We can try to address the JDK implementation for the same (if legit) in a separate jira as that one is going to be a bigger change I think. |
|
Thanks for creating this PR. I can confirm that building on UBI 10 works. My understanding of this PR is that it turns the functions using the engine api into no-ops without replacement. Is this correct? If this is true, how does this change impact functionality of existing deployments? Is an upgrade of Hadoop clusters without some form of (meta)data migration even possible? I don't know enough about how this API is/was used, but I imagine that any keys/certs/secrets or encrypted (meta)data generated previously will not be usable anymore after the upgrade. Am I correct ? |
Contributor
Author
|
Thanks for confirming UBI 10 build! Regarding your concern, The ENGINE API change only affects how the random bytes are produced internally (RDRAND via ENGINE vs RDRAND via OpenSSL's built-in provider in version 3.x). The output is on par random bytes in both cases. Existing keys, certificates, and encrypted data should not be affected. Here is the overall flow I believe: Encryption:
Decryption:
OpensslSecureRandom is never used in decryption. Our change only affects how the random bytes are generated inside OpensslSecureRandom. Whether those random bytes came from Intel RDRAND via the ENGINE path or via OpenSSL's built-in provider. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Native build of hadoop-common fails on RHEL 10 with:
Root Cause: RHEL 10 ships OpenSSL 3.2.x, which was compiled with OPENSSL_NO_ENGINE — completely removing the OpenSSL Engine API (ENGINE_* functions and constants). The native file OpensslSecureRandom.c used ENGINE_METHOD_RAND and other Engine API symbols without any OpenSSL 3.x compatibility guard, causing a compile-time failure.
Fix: Added #if OPENSSL_VERSION_NUMBER < 0x30000000L guards in OpensslSecureRandom.c to exclude Engine API usage when building against OpenSSL 3.x
Contains content generated by Claude
How was this patch tested?
RHEL 10 build passed. No breakage reported in RHEL 9 arm64 too.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html