Skip to content

Wide-decimal overflow reports a different value than Spark #5211

Description

@peterxcli

Describe the bug

In ANSI mode, Comet's wide-decimal arithmetic overflow reports a different value parameter from Spark. PR #5169 fixes the exception type and propagates the error class, SQLSTATE, and query context, but Comet still formats the rescaled i256 result while Spark formats its pre-toPrecision Decimal.

This is a follow-up to #5072.

Steps to reproduce

Using Spark 4.1.2, run the query once with Comet disabled and once with Comet enabled:

SET spark.sql.ansi.enabled = true;

CREATE TABLE tbl (_1 DECIMAL(20, 0)) USING PARQUET;
INSERT INTO tbl VALUES (11000000000000000000);

SELECT _1 * _1 FROM tbl;

Spark reports:

[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003

Comet reports:

[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000.000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003

Expected behavior

Comet should match Spark's value parameter and report 121000000000000000000000000000000000000 without the output-scale suffix.

Additional context

The regression in spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala added by #5169 documents the current difference: Spark formats its pre-toPrecision Decimal, while Comet formats the rescaled i256 value. The test intentionally compares the structured error fields and query context without comparing this differing message parameter.

Matching Spark is not only a scale-formatting change. Spark's Decimal.$times multiplies with MATH_CONTEXT, which is MathContext(38, HALF_UP), so the value in Spark's error has already been rounded to 38 significant digits before toPrecision is attempted. The reproduction above rounds exactly because 1.21e38 has trailing zeros; products with non-zero low digits can differ from Comet's exact i256 even after natural-scale formatting. A fix must reproduce this significant-digit rounding as well as the scale formatting.

Metadata

Metadata

Assignees

Labels

good first issueGood for newcomerspriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions