feat: add support for aes_decrypt expression#3497
feat: add support for aes_decrypt expression#3497rafafrdz wants to merge 9 commits intoapache:mainfrom
aes_decrypt expression#3497Conversation
3b7cc7f to
eb318d1
Compare
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Outdated
Show resolved
Hide resolved
|
Thanks for the contribution @rafafrdz. I took a quick look and this looks nice. I kicked off CI. On the testing side, the current tests only cover GCM mode with a 256-bit key. Since the Rust implementation also supports CBC and ECB modes with PKCS padding, and handles 128-bit, 192-bit, and 256-bit keys, it would be great to add test coverage for those combinations. It might also be worth adding a null input test, maybe inserting a row with null encrypted data and verifying the result is null. |
Done 😄 @andygrove |
andygrove
left a comment
There was a problem hiding this comment.
The Rust test CI failure is unrelated to this PR (a pre-existing as_slice() ambiguity in page_util.rs). Merging latest from main should fix it. All Spark test suites pass across 3.4, 3.5, and 4.0.
One thing I noticed is that AesDecrypt extends RuntimeReplaceable in Spark, so by the time Comet's serde runs it has already been replaced with a StaticInvoke. That means the CometAesDecrypt registration in miscExpressions for classOf[AesDecrypt] will never match. Only the CometAesDecryptStaticInvoke path through statics.scala is used in practice. You could simplify things by removing CometAesDecrypt and its registration in QueryPlanSerde.scala.
AesDecrypt extends RuntimeReplaceable in Spark, so it is always rewritten to StaticInvoke before Comet's serde runs. Remove the direct CometAesDecrypt handler and CometAesDecryptHelper, keeping only the CometAesDecryptStaticInvoke path.
3a32725 to
4f9dcaa
Compare
Summary
aes_decryptby mapping AesDecrypt to native scalar functionaes_decrypt.Closes [Feature] Support Spark expression: aes_decrypt #3188