Fenrir fixes for AES-GCM/GMAC and RSA-PSS JNI wrappers

[cconlon]

This PR includes five Fenrir fixes:

• F-3767: Add (inLen != 0 && in == NULL) input validation to the AES-GCM encrypt wrapper, matching the existing check in the decrypt wrapper.

• F-3768: Make AesGmac.verify() throw WolfCryptException for operational errors while still returning false for an authentication tag mismatch, matching its documented contract. Adds tests for the invalid argument cases.

• F-3995: Apply the same policy to the RSA-PSS VerifyCheck wrapper. A BAD_PADDING_E signature mismatch returns false in both salt length branches, all other errors throw, including those detected before the native verify call. Updates the rsaPssVerifyWithDigest() javadoc and adds tests pinning both sides of the boundary.

• F-4435 / F-4436: Initialize the GMAC size locals read by debug logging in wc_Gmac and wc_GmacVerify, so WOLFCRYPT_JNI_DEBUG_ON builds no longer read uninitialized values on the argument validation path.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Improves JNI wrapper behavior and docs so operational/native errors throw exceptions while cryptographic mismatches return false, and tightens native input/initialization safety.

Changes:

• Align AES-GCM encrypt JNI validation and initialize GMAC debug-size locals to prevent uninitialized reads in debug builds.
• Update AES-GMAC and RSA-PSS verify semantics to throw on operational errors but return false for authentication/signature mismatches; add tests to pin behavior.
• Refine Javadoc contracts for updated verify behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/test/java/com/wolfssl/wolfcrypt/test/RsaTest.java	Adds tests to ensure RSA-PSS operational errors throw and mismatches return false.
src/test/java/com/wolfssl/wolfcrypt/test/AesGmacTest.java	Adds tests asserting AesGmac.verify() throws on invalid args.
src/main/java/com/wolfssl/wolfcrypt/Rsa.java	Updates verify Javadoc contract to distinguish mismatch vs operational error.
src/main/java/com/wolfssl/wolfcrypt/AesGmac.java	Implements “throw on operational error, false on tag mismatch” policy.
jni/jni_rsa.c	Adjusts RSA-PSS verify wrapper error handling to throw except BAD_PADDING_E.
jni/jni_aesgmac.c	Initializes size locals to avoid uninitialized reads on validation/debug paths.
jni/jni_aesgcm.c	Adds missing (inLen != 0 && in == NULL) validation in encrypt wrapper.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.