Skip to content

feat: add support for aes_decrypt expression#3497

Open
rafafrdz wants to merge 9 commits intoapache:mainfrom
rafafrdz:feat/add-support-aes_decrypt
Open

feat: add support for aes_decrypt expression#3497
rafafrdz wants to merge 9 commits intoapache:mainfrom
rafafrdz:feat/add-support-aes_decrypt

Conversation

@rafafrdz
Copy link
Contributor

Summary

  • Added support for Spark aes_decrypt by mapping AesDecrypt to native scalar function aes_decrypt.
  • Added tests for aes_decrypt in CometStringExpressionSuite covering both:
    • default args: aes_decrypt(input, key)
    • explicit args: aes_decrypt(input, key, mode, padding, aad)
  • Updated spark_expressions_support.md to mark aes_decrypt as supported.
    Closes [Feature] Support Spark expression: aes_decrypt #3188

@rafafrdz rafafrdz force-pushed the feat/add-support-aes_decrypt branch from 3b7cc7f to eb318d1 Compare February 13, 2026 13:11
@rafafrdz rafafrdz marked this pull request as draft February 13, 2026 13:26
@rafafrdz rafafrdz marked this pull request as ready for review February 17, 2026 22:47
@andygrove
Copy link
Member

Thanks for the contribution @rafafrdz. I took a quick look and this looks nice. I kicked off CI.

On the testing side, the current tests only cover GCM mode with a 256-bit key. Since the Rust implementation also supports CBC and ECB modes with PKCS padding, and handles 128-bit, 192-bit, and 256-bit keys, it would be great to add test coverage for those combinations.

It might also be worth adding a null input test, maybe inserting a row with null encrypted data and verifying the result is null.

@rafafrdz
Copy link
Contributor Author

rafafrdz commented Feb 26, 2026

Thanks for the contribution @rafafrdz. I took a quick look and this looks nice. I kicked off CI.

On the testing side, the current tests only cover GCM mode with a 256-bit key. Since the Rust implementation also supports CBC and ECB modes with PKCS padding, and handles 128-bit, 192-bit, and 256-bit keys, it would be great to add test coverage for those combinations.

It might also be worth adding a null input test, maybe inserting a row with null encrypted data and verifying the result is null.

Done 😄 @andygrove

@rafafrdz rafafrdz requested a review from andygrove March 4, 2026 11:41
Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Rust test CI failure is unrelated to this PR (a pre-existing as_slice() ambiguity in page_util.rs). Merging latest from main should fix it. All Spark test suites pass across 3.4, 3.5, and 4.0.

One thing I noticed is that AesDecrypt extends RuntimeReplaceable in Spark, so by the time Comet's serde runs it has already been replaced with a StaticInvoke. That means the CometAesDecrypt registration in miscExpressions for classOf[AesDecrypt] will never match. Only the CometAesDecryptStaticInvoke path through statics.scala is used in practice. You could simplify things by removing CometAesDecrypt and its registration in QueryPlanSerde.scala.

AesDecrypt extends RuntimeReplaceable in Spark, so it is always
rewritten to StaticInvoke before Comet's serde runs. Remove the
direct CometAesDecrypt handler and CometAesDecryptHelper, keeping
only the CometAesDecryptStaticInvoke path.
@rafafrdz rafafrdz force-pushed the feat/add-support-aes_decrypt branch from 3a32725 to 4f9dcaa Compare March 19, 2026 12:29
@rafafrdz rafafrdz requested a review from andygrove March 19, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support Spark expression: aes_decrypt

2 participants