-
Notifications
You must be signed in to change notification settings - Fork 270
feat: implement cast from whole numbers to binary format #3083
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@parthchandra , @andygrove these are the changes to support int to binary cast and bool to decimal cast operation . I have also had to change the base test to make sure that we use DSL (since Int -> Binary casts are only supported in the DSL side of Spark and would raise an encoding exception if we use Spark SQL) . Also, Int -> Binary casts do not support Try or ANSI eval modes in Spark and I have followed the same convention here as well |
|
Fixed format issue with |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3083 +/- ##
============================================
+ Coverage 56.12% 58.97% +2.84%
- Complexity 976 1371 +395
============================================
Files 119 167 +48
Lines 11743 15562 +3819
Branches 2251 2586 +335
============================================
+ Hits 6591 9177 +2586
- Misses 4012 5086 +1074
- Partials 1140 1299 +159 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
andygrove
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding int→binary and boolean→decimal cast support!
Issue: Dead code
The function canCastToBinary (lines 354-358 in CometCast.scala) is defined but never called anywhere. This appears to be unused code that should be removed.
private def canCastToBinary(fromType: DataType): SupportLevel = fromType match {
case DataTypes.ByteType | DataTypes.ShortType | DataTypes.IntegerType | DataTypes.LongType =>
Compatible()
case _ => Unsupported(Some(s"Cast from BinaryType to $fromType is not supported"))
}The actual logic for int→binary casts is correctly handled in the canCastFromByte, canCastFromShort, canCastFromInt, and canCastFromLong functions.
Otherwise, the implementation looks good:
- Correctly limits int→binary to LEGACY mode only (matching Spark)
- Boolean→Decimal implementation handles precision/scale correctly
- Tests use column data from parquet (not just literals)
- Compatibility docs updated
- ANSI and Try modes correctly disabled for int→binary tests
This review was generated with AI assistance.
|
There is a test failure: |
|
Thank you @andygrove ,I am working on the test failures (I believe there is a bug on the DSL side vs using Spark SQL) |
910fe4f to
69d8290
Compare
Which issue does this PR close?
try_castis only supported starting Spark 4x)Relavent spark code :
https://github.com/apache/spark/blob/f89fbd49c45045b44259341c6c73e30ee9c46dd0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L682
(Note that only legacy mode routes the code here while we Try and ANSI are not supported)
Closes #.
Rationale for this change
What changes are included in this PR?
How are these changes tested?