[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast by raminqaf · Pull Request #28758 · apache/flink

raminqaf · 2026-07-16T09:55:59Z

What is the purpose of the change

Follow-up to the initial VARIANT-to-primitive cast support (FLINK-37925). It tightens the cast semantics, turns the string cast into a real value cast, adds a decodability check, and documents the resulting behavior.

Numeric casts from a VARIANT now reject values that do not fit the target instead of silently wrapping. An out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. This follows the behavior of Spark's variant casts.
CAST(VARIANT AS CHAR/VARCHAR) now extracts the scalar value, so a stored string is returned unquoted (foo), while objects and arrays return their JSON representation. JSON_STRING remains the way to get the JSON text, where a string stays quoted ("foo"). Previously the cast reused the JSON serialization and was indistinguishable from JSON_STRING.

Brief change log

Add VariantCastUtils (flink-table-runtime): range-checked numeric narrowing (truncate toward zero, throw on overflow) and scalar-to-string extraction.
VariantToPrimitiveCastRule: route integer and DECIMAL targets through the checked helpers; keep FLOAT/DOUBLE lenient.
VariantToStringCastRule: value extraction instead of JSON serialization; JSON_STRING (JsonStringCallGen) is left unchanged.
LogicalTypeCasts: express VARIANT cast validation in the per-target rules and drop the JSON_STRING cast hint.
Fix the Variant.getInstant javadoc to reference Type.TIMESTAMP_LTZ and document microsecond timestamp precision.
Docs: VARIANT value encoding, the PARSE_JSON type mapping, the string-cast vs JSON_STRING distinction, and that NaN/infinity are not valid JSON (witha store-as-string workaround).

Verifying this change

This change added and updated tests:

CastFunctionITCase: overflow cases (CAST fails, TRY_CAST returns NULL), lenient FLOAT/DOUBLE, and string-cast value extraction (a string returns unquoted, objects return JSON).
LogicalTypeCastsTest: the VARIANT cast-support matrix.
BinaryVariantInternalBuilderTest: PARSE_JSON rejects NaN, Infinity, and -Infinity.
BinaryVariantTest: checkFullySupported accepts supported types and rejects every primitive type-code above 16.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes, javadoc-only change to the @PublicEvolving Variant interface (no signature change)
The serializers: no
The runtime per-record code paths (performance sensitive): yes, only on the VARIANT-to-numeric cast path
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no (it refines existing FLINK-37925 behavior)
If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Opus 4.8

flinkbot · 2026-07-16T10:01:19Z

CI report:

f9bd3a0 UNKNOWN
23e41c6 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin · 2026-07-17T17:24:45Z

+        // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+        // error and TRY_PARSE_JSON returns NULL.


why do we assume that
IIRC Jackson has a property allowing to read them

NaN and Infinity are not part of the JSON specification: https://www.rfc-editor.org/info/rfc8259/#section-6

Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.

Jackson does not allow it by default and you need to switch a flag to allow it. To represent them, the user has to define them as string and cast them as float. The workaround is documented in the variant type

then the question: why do we support these values in some parts of the code and do not support in others?
Do we have a plan to be consistent for all parts?

In Variant we don't and I don't think we should because it follows the JSON spec. Vendors like Snowflake and Databricks also don't allow it.
For cast we do CAST('Nan' AS FLOAT) and it is a valid case

snuyanzin · 2026-07-20T09:27:32Z

+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
+```
+{{< /hint >}}


what is difference here from non VARIANT case?

snuyanzin · 2026-07-20T09:34:38Z

+                        // Spark-style overflow: an integer value outside the target range fails
+                        // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+                        .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
                        .testResult(
-                                call("PARSE_JSON", "40000").cast(SMALLINT()),
-                                "CAST(PARSE_JSON('40000') AS SMALLINT)",
-                                (short) -25536,
-                                SMALLINT().notNull())
-                        .testResult(
-                                call("PARSE_JSON", "128").cast(TINYINT()),
-                                "CAST(PARSE_JSON('128') AS TINYINT)",
-                                (byte) -128,
-                                TINYINT().notNull())
+                                call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+                                "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+                                null,
+                                SMALLINT())
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")


why do we do this this way?

Better to have same way as regular cast

Then there is SqlConformance#checkedArithmetic which we can make use to switch between different behavior

The idea here is that the user first casts (converts) the value from variant to the actual type it has and then performs an explicit cast if they want too. In this case the 128 is an INT so the user first needs to cast it INT and then if they want to explicitly cast to TINYINT.

yes, however it does not answer why we expect NULL after that

to conclude on this: whereas for regular casts people opt-in on the overflow, for variant casts they should first extract the correct value before opt-in to overflow.

davidradl · 2026-07-20T15:41:13Z

+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.


why microsecond not nanoseconds?

@davidradl This is how the implementation of the BinaryVariant defines it.

@Override public Instant getInstant() throws VariantTypeException { checkType(Type.TIMESTAMP_LTZ, getType()); return microsToInstant(BinaryVariantUtil.getLong(value, pos)); }

The implementation is done based on the Encoding types defined in Parquet. Flink implements the types from 0 (NULL) to 16 (STRING). The rest of the types: 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.

davidradl · 2026-07-20T15:43:17Z

+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+


I am curious why Date does not go to Flink date which would seem more natural

@davidradl I think the implementation of DATE is also based on Parquet: https://parquet.apache.org/docs/file-format/types/logicaltypes/

DATE is used for a logical date type, without a time of day. It must annotate an int32 that stores the number of days from the Unix epoch, 1 January 1970.

@Override public LocalDate getDate() throws VariantTypeException { checkType(Type.DATE, getType()); return LocalDate.ofEpochDay(BinaryVariantUtil.getLong(value, pos)); }

twalthr

Thank you @raminqaf. I left some feedback.

twalthr · 2026-07-21T13:11:46Z

+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON


did you double check this behavior with other vendors? sounds incorrect to me.

while objects and arrays return their JSON

this is what JSON_STRING should be good for.

For objects, arrays, bytes we rely on the Variant#toJson() implementation. All other types (i.e., String) we convert to literal String. This is because for a case like CAST(PARSE_JSON('foo') AS STRING)) we want to return a string literal foo and not a JSON String "foo".

twalthr · 2026-07-21T13:28:24Z

            LogicalType inputLogicalType,
            LogicalType targetLogicalType) {
-        return methodCall(inputTerm, "toString");
+        return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);


according to LogicalTypeCasts, we support VARCHAR and CHAR both with various length. this cast rule should consider the target length as well. I guess it should throw if length exceeds?

when the target is bounded CHAR(n)/VARCHAR(n), CharVarCharTrimPadCastRule kicks in, calls the variant→string rule, and trims/pads — the identical path every other to-string cast uses.

I added tests to prove it, and they pass:

CAST(PARSE_JSON('"foobar"') AS VARCHAR(3)) → foo (trimmed)

CAST(PARSE_JSON('"ab"') AS CHAR(5)) → ab (padded)

twalthr · 2026-07-21T13:30:19Z

+                        // Spark-style overflow: an integer value outside the target range fails
+                        // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+                        .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
                        .testResult(
-                                call("PARSE_JSON", "40000").cast(SMALLINT()),
-                                "CAST(PARSE_JSON('40000') AS SMALLINT)",
-                                (short) -25536,
-                                SMALLINT().notNull())
-                        .testResult(
-                                call("PARSE_JSON", "128").cast(TINYINT()),
-                                "CAST(PARSE_JSON('128') AS TINYINT)",
-                                (byte) -128,
-                                TINYINT().notNull())
+                                call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+                                "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+                                null,
+                                SMALLINT())
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")


to conclude on this: whereas for regular casts people opt-in on the overflow, for variant casts they should first extract the correct value before opt-in to overflow.

twalthr · 2026-07-21T13:33:14Z

+        final DecimalData decimal =
+                DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+        if (decimal == null) {
+            throw new ArithmeticException(


is it guaranteed that this becomes a TableRuntimeException eventually? or how do we treat other cast cases currently?

Changed it to TableRuntimeException

twalthr · 2026-07-22T14:27:16Z

-     * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+     * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+     * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+     * {@code JSON_STRING}); objects and arrays use their JSON representation.


objects and arrays use their JSON representation

Let's not do this. People can use JSON_STRING for this. Let stay conservative and throw an error. Maybe pointing towards JSON_STRING.

… fit the target type and allow string cast Follow-up to the initial VARIANT-to-primitive cast support. Numeric casts from a VARIANT now reject values that do not fit the target type instead of silently wrapping: an out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST, matching Spark's variant cast behavior. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. The new VariantCastUtils performs the checked narrowing by truncating toward zero and range-checking the value. CAST(VARIANT AS CHAR/VARCHAR) is now allowed and returns the JSON string representation, so the previous JSON_STRING-only restriction and its cast hint are removed. VARIANT cast validation is expressed directly in the per-target rules of LogicalTypeCasts. The VARIANT section of the data types reference documents the overflow behavior and the double-cast pattern for wrap-around narrowing.

…pe mapping Describe which value kinds a VARIANT can hold, including that TIMESTAMP and TIMESTAMP_LTZ use microsecond precision and DATE a day count, and that there is no TIME kind. Add a table showing how PARSE_JSON maps JSON values to variant kinds, and explain that PARSE_JSON cannot produce FLOAT, DATE, TIMESTAMP, TIMESTAMP_LTZ, or BYTES because JSON has no literal for them; those kinds come from other producers such as VariantBuilder or format conversions. Point the PARSE_JSON function reference to the VARIANT data type section for the mapping details.

…nt interface Note that getDateTime and getInstant return values with microsecond precision, which the LocalDateTime and Instant return types do not convey on their own. Fix the getInstant javadoc to reference Type.TIMESTAMP_LTZ, the type it actually accepts, instead of Type.TIMESTAMP.

twalthr

Another round of comments.

twalthr · 2026-07-23T08:01:05Z

+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.


Suggested change

are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.

are stored with microsecond precision.

twalthr · 2026-07-23T08:03:27Z

+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.


also include the 1e400 example here.

twalthr · 2026-07-23T08:04:54Z

-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A


Suggested change

targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A

targets accept any wider numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A

twalthr · 2026-07-23T08:05:41Z

+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a


Let's not go into the details of CHAR/VARCHAR at this line

Suggested change

handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a

handled the same way. Casting a `VARIANT` to `STRING` extracts the scalar value, so a

twalthr · 2026-07-23T08:07:03Z

+padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
+otherwise `CAST` fails and `TRY_CAST` returns `NULL`.


people should know how they work already

Suggested change

padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,

otherwise `CAST` fails and `TRY_CAST` returns `NULL`.

padding or truncation.

twalthr · 2026-07-23T08:14:27Z

+                return staticCall(VariantCastUtils.class, "toLongExact", number);
            case FLOAT:
-                return "floatValue";
+                return methodCall(number, "floatValue");


Are there any data loss checks in this method? Asking AI leads to exactness loss.

can I store a full long in a float?

The short answer is yes, in terms of size (range), but no, in terms of exactness (precision).

If you attempt to store a large long inside a float, the application will not crash, but you will likely lose data through rounding errors.

twalthr · 2026-07-23T08:14:42Z

+                return methodCall(number, "floatValue");
            case DOUBLE:
-                return "doubleValue";
+                return methodCall(number, "doubleValue");


Are there any data loss checks in this method? Asking AI leads to exactness loss.

twalthr · 2026-07-23T08:19:58Z

+     * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+     * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+     * {@code JSON_STRING}); a variant holding an object, array, or binary value is not castable and
+     * raises {@link TableRuntimeException}.
+     *
+     * <p>The target length is enforced strictly, with no padding or truncation: a {@code VARCHAR}
+     * value must not be longer than {@code targetLength}, and a {@code CHAR} value must match it
+     * exactly. A value that does not fit raises {@link TableRuntimeException}.


nit: Too much comment. Code is shorter than this JavaDoc.

twalthr · 2026-07-23T08:22:17Z

+    }
+
+    private static BigDecimal convertToBigDecimal(Number number) {
+        if (number == null) {


Why this null check here?

twalthr · 2026-07-23T08:22:58Z

+
+    public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+        final DecimalData decimal =
+                DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);


Where is max precision of 38 checked?

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from beec0e0 to e03672b Compare July 16, 2026 10:07

raminqaf commented Jul 16, 2026

View reviewed changes

Comment thread docs/content/docs/sql/reference/data-types.md Outdated

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from f02dbdc to 49d4bca Compare July 16, 2026 10:45

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 16, 2026

raminqaf force-pushed the FLINK-37925-followup branch 6 times, most recently from 3750b25 to f9bd3a0 Compare July 17, 2026 11:38

raminqaf changed the title ~~[FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string~~ [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast Jul 17, 2026

snuyanzin reviewed Jul 17, 2026

View reviewed changes

Comment thread ...k-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java Outdated

snuyanzin reviewed Jul 17, 2026

View reviewed changes

snuyanzin reviewed Jul 20, 2026

View reviewed changes

davidradl reviewed Jul 20, 2026

View reviewed changes

raminqaf force-pushed the FLINK-37925-followup branch from 02f5ffc to e9c07c4 Compare July 21, 2026 09:41

twalthr reviewed Jul 21, 2026

View reviewed changes

raminqaf force-pushed the FLINK-37925-followup branch from 0a0d8d0 to 8f09c44 Compare July 22, 2026 11:00

twalthr reviewed Jul 22, 2026

View reviewed changes

raminqaf and others added 6 commits July 22, 2026 18:31

[FLINK-37925] Address feedbacks

fd2676f

[FLINK-37925] Add strikt cast validation for VARCHAR and CHAR

db85cdf

[FLINK-37925] Fail on casting Variant array, objects, bytes to String

23e41c6

raminqaf force-pushed the FLINK-37925-followup branch from 4dc9135 to 23e41c6 Compare July 22, 2026 16:31

twalthr reviewed Jul 23, 2026

View reviewed changes

		// NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
		// error and TRY_PARSE_JSON returns NULL.

	are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
	are stored with microsecond precision.

	targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
	targets accept any wider numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A

	handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
	handled the same way. Casting a `VARIANT` to `STRING` extracts the scalar value, so a

		padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
		otherwise `CAST` fails and `TRY_CAST` returns `NULL`.

	padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
	otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
	padding or truncation.

Uh oh!

Conversation

raminqaf commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raminqaf Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raminqaf Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

raminqaf commented Jul 16, 2026 •

edited

Loading

flinkbot commented Jul 16, 2026 •

edited

Loading

raminqaf Jul 19, 2026 •

edited

Loading

snuyanzin Jul 20, 2026 •

edited

Loading

raminqaf Jul 21, 2026 •

edited

Loading