From 68a6f2f22015931fcadee13c14d90ed1e89081f3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 17 Jul 2026 13:29:11 -0600 Subject: [PATCH 01/19] Add Comet 1.0.0 release announcement blog post --- .../blog/2026-07-31-datafusion-comet-1.0.0.md | 190 ++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 content/blog/2026-07-31-datafusion-comet-1.0.0.md diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-07-31-datafusion-comet-1.0.0.md new file mode 100644 index 00000000..c7ff36c2 --- /dev/null +++ b/content/blog/2026-07-31-datafusion-comet-1.0.0.md @@ -0,0 +1,190 @@ +--- +layout: post +title: Apache DataFusion Comet 1.0.0 Release +date: 2026-07-31 +author: pmc +categories: [subprojects] +--- + + + +[TOC] + +The Apache DataFusion PMC is pleased to announce version 1.0.0 of the [Comet](https://datafusion.apache.org/comet/) subproject. + +Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for +improved performance and efficiency without requiring any code changes. + +This is a major milestone. Comet began as a code donation in early 2024 and has shipped steadily ever since, +one release at a time. Version 1.0.0 marks the point where the project is mature enough to commit to a stable +release line: broad Apache Spark coverage, ANSI SQL semantics, native Parquet and Iceberg scans, and a native +shuffle, all validated continuously against Spark's own test suites. This release covers roughly six weeks of +development since 0.17.0 and is the result of merging over 140 PRs from 19 contributors. See the +[change log] for more information. + +[change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md + +## What 1.0 Means + +Reaching 1.0 is less about any single new feature than about the accumulated maturity of the project: + +- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.1 out of the same + codebase, with dedicated Maven profiles, shim sources, and CI matrices for each. +- **ANSI SQL by default.** Spark 4 enables ANSI semantics by default. Comet implements ANSI behavior for the + expressions it supports natively, including arithmetic overflow checks, ANSI cast behavior, and `try_*` + variants, so ANSI workloads keep accelerating rather than falling back. +- **A correctness harness, not just unit tests.** Comet runs the full Apache Spark SQL test suite through its + native execution path against every supported Spark version. Running Spark's own correctness tests is what + surfaces semantic shifts before they reach user workloads, and it is the foundation of the confidence behind + a 1.0 release. +- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. The very + first deprecations under that policy are announced in this release (see [Deprecation Notice](#deprecation-notice) + below). + +The rest of this post covers what is new since 0.17.0. + +## Native Expression Performance + +A large share of this release is dedicated to making Comet's native scalar expressions faster. These kernels +run per-row or per-batch, so improvements here compound across every query that uses them: + +- **`get_json_object`** ([#4907](https://github.com/apache/datafusion-comet/pull/4907)): roughly 4x faster. +- **`regexp_extract`** ([#4894](https://github.com/apache/datafusion-comet/pull/4894)) and **`parse_url`** + ([#4893](https://github.com/apache/datafusion-comet/pull/4893)): optimized regex and URL parsing paths. +- **Casts**: a faster floating-point-to-decimal cast + ([#4940](https://github.com/apache/datafusion-comet/pull/4940)), an optimized integer-to-integer cast + ([#4920](https://github.com/apache/datafusion-comet/pull/4920)), and shared no-overflow fast paths in + `CheckOverflow` ([#4937](https://github.com/apache/datafusion-comet/pull/4937)) and + `DecimalRescaleCheckOverflow` ([#4938](https://github.com/apache/datafusion-comet/pull/4938)). +- **String and array kernels**: `lpad` ([#4919](https://github.com/apache/datafusion-comet/pull/4919)), + `unhex` ([#4876](https://github.com/apache/datafusion-comet/pull/4876)), + `size` ([#4877](https://github.com/apache/datafusion-comet/pull/4877)), + `arrays_overlap` ([#4906](https://github.com/apache/datafusion-comet/pull/4906)), + `escape_string` ([#4902](https://github.com/apache/datafusion-comet/pull/4902)), and the + `try_*` arithmetic kernel ([#4910](https://github.com/apache/datafusion-comet/pull/4910)). + +To make this kind of work repeatable, the release also adds a +[scalar expression optimization guide](https://github.com/apache/datafusion-comet/pull/4933) documenting how +to benchmark a kernel, keep its output bit-identical to Spark, and gate changes on a no-regression check. + +On the scan side, Parquet reads pick up several improvements: full Parquet metadata (including the page index) +is now cached via DataFusion's `CachedParquetFileReaderFactory` +([#4707](https://github.com/apache/datafusion-comet/pull/4707)), identity casts are unwrapped in the schema +adapter so Parquet statistics pruning can engage +([#4730](https://github.com/apache/datafusion-comet/pull/4730)), and filter pushdown configuration has been +revised ([#4722](https://github.com/apache/datafusion-comet/pull/4722)). + +## New Expression and Aggregate Support + +This release expands the set of Spark expressions and aggregates that run natively: + +- **Aggregates**: `approx_percentile` / `percentile_approx` + ([#4801](https://github.com/apache/datafusion-comet/pull/4801)) and exact `percentile` / `median` + ([#4542](https://github.com/apache/datafusion-comet/pull/4542)). +- **Grouping**: `grouping()` and `grouping_id()` indicator functions + ([#4815](https://github.com/apache/datafusion-comet/pull/4815)). +- **Intervals**: interval types with `make_ym_interval` and `make_dt_interval` + ([#4541](https://github.com/apache/datafusion-comet/pull/4541)). +- **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)) and `split_part` via + `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)). +- **Array / map**: `array_prepend` ([#4716](https://github.com/apache/datafusion-comet/pull/4716)), + `size()` for `MapType` ([#4580](https://github.com/apache/datafusion-comet/pull/4580)), `ElementAt` over + `MapType` ([#4697](https://github.com/apache/datafusion-comet/pull/4697)), and removal of the constraint on + arrays of nested elements ([#4714](https://github.com/apache/datafusion-comet/pull/4714)). +- **Date/time**: native `TimestampNTZ` inputs for `hour` / `minute` / `second` + ([#4753](https://github.com/apache/datafusion-comet/pull/4753)) and + `PreciseTimestampConversion` for native time-window grouping + ([#4784](https://github.com/apache/datafusion-comet/pull/4784)). +- **Windows**: extended native window function support + ([#4209](https://github.com/apache/datafusion-comet/pull/4209)) and Spark 4 decimal window average + ([#4749](https://github.com/apache/datafusion-comet/pull/4749)). + +## Experimental PyArrow UDF Support + +This release adds experimental support for accelerated PyArrow UDFs +([#4234](https://github.com/apache/datafusion-comet/pull/4234)), allowing PyArrow-based user-defined functions +to participate in native execution instead of forcing a fallback to Spark. When the feature is disabled, Comet +now hints at the native PyArrow UDF path in its fallback reasons +([#4892](https://github.com/apache/datafusion-comet/pull/4892)) so users know the option exists. This is an +early-stage feature and we welcome feedback from users experimenting with it. + +## Iceberg + +- **Iceberg 1.11 support** ([#4840](https://github.com/apache/datafusion-comet/pull/4840)): adds support for + Iceberg 1.11, audits the existing Iceberg diffs, bumps the iceberg-rust dependency, and adds a + `run-iceberg-tests` CI trigger. +- **Delete-file correctness** ([#4760](https://github.com/apache/datafusion-comet/pull/4760)): the native scan + now sizes Iceberg delete files correctly, avoiding dropped deletes. +- **Exchange-reuse correctness** ([#4812](https://github.com/apache/datafusion-comet/pull/4812)): fixed a case + where Iceberg native scan exchange reuse with different pushed filters could produce wrong results. + +## Native Parquet I/O and Cloud + +- **gzip Parquet writes** ([#4930](https://github.com/apache/datafusion-comet/pull/4930)): native Parquet + writes now support gzip compression. +- **Azure authentication** ([#4783](https://github.com/apache/datafusion-comet/pull/4783)): the native + Parquet scan now supports Azure authentication, complementing the existing S3 credential support. + +## Execution and Fallback Improvements + +- **Mixed partial/final aggregation** ([#4861](https://github.com/apache/datafusion-comet/pull/4861)): + `sum` and non-decimal `avg` can now run with the partial and final aggregation stages split across Spark and + Comet, keeping more aggregation plans partially native. +- **Stage-based fallback** ([#4519](https://github.com/apache/datafusion-comet/pull/4519)): a more granular + fallback mechanism that operates at the stage level. +- **Codegen dispatch for opt-in serdes** ([#4728](https://github.com/apache/datafusion-comet/pull/4728)): + expressions reported as `Unsupported` can now route through JVM codegen dispatch for opt-in serdes, and + native opt-in expressions surface as compatible-by-default with a `COMET-INFO` plan hint + ([#4721](https://github.com/apache/datafusion-comet/pull/4721)). +- **Runtime cleanup** ([#4734](https://github.com/apache/datafusion-comet/pull/4734)): the Tokio runtime is + now released on driver and executor exit. +- **Contrib scan SPI** ([#4700](https://github.com/apache/datafusion-comet/pull/4700)): a core SPI for + contrib leaf scans (`CometScanWithPlanData`), the first part of splitting the Delta integration into a + contrib module. + +## Deprecation Notice + +With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic versioning +([#4857](https://github.com/apache/datafusion-comet/pull/4857)): + +- **JDK 11** is deprecated and scheduled for removal in Comet 1.1.0. +- **Apache Spark 3.4** is deprecated and scheduled for removal in Comet 1.1.0. + +Users on these platforms should plan to move to JDK 17+ and Spark 3.5 or later before upgrading to 1.1.0. + +## Compatibility + +Supported platforms include: + +- **Spark 3.4.3** with Java 11/17 and Scala 2.12/2.13 (deprecated, removal in 1.1.0) +- **Spark 3.5.8** with Java 11/17 and Scala 2.12/2.13 +- **Spark 4.0.2** with Java 17 and Scala 2.13 +- **Spark 4.1.1** with Java 17 and Scala 2.13 + +See the [Spark Version Compatibility] page for known limitations specific to each version. + +[Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html + +This release upgrades to **DataFusion 54** and **Arrow 58.3**. + +## Get Started with Comet 1.0.0 + +Ready to try it out? Follow the [Comet 1.0.0 Installation Guide](https://datafusion.apache.org/comet/user-guide/1.0/installation.html) +to get up and running, then point Comet at your existing Spark workloads and see the speedup for yourself. From 0ce09ac53fd4e77c1a49bea8e2aa6f1b7730af76 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 27 Jul 2026 14:39:06 -0600 Subject: [PATCH 02/19] Update Comet 1.0.0 blog post with commits merged since draft Incorporates PRs merged to datafusion-comet main between 2026-07-17 and 2026-07-27, updates PR/contributor counts, and bumps DataFusion version to 54.1. Adds a new Shuffle Improvements section. --- .../blog/2026-07-31-datafusion-comet-1.0.0.md | 60 ++++++++++++++++--- 1 file changed, 51 insertions(+), 9 deletions(-) diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-07-31-datafusion-comet-1.0.0.md index c7ff36c2..81a9f5e6 100644 --- a/content/blog/2026-07-31-datafusion-comet-1.0.0.md +++ b/content/blog/2026-07-31-datafusion-comet-1.0.0.md @@ -36,7 +36,7 @@ This is a major milestone. Comet began as a code donation in early 2024 and has one release at a time. Version 1.0.0 marks the point where the project is mature enough to commit to a stable release line: broad Apache Spark coverage, ANSI SQL semantics, native Parquet and Iceberg scans, and a native shuffle, all validated continuously against Spark's own test suites. This release covers roughly six weeks of -development since 0.17.0 and is the result of merging over 140 PRs from 19 contributors. See the +development since 0.17.0 and is the result of merging over 190 PRs from 22 contributors. See the [change log] for more information. [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md @@ -70,9 +70,19 @@ run per-row or per-batch, so improvements here compound across every query that ([#4893](https://github.com/apache/datafusion-comet/pull/4893)): optimized regex and URL parsing paths. - **Casts**: a faster floating-point-to-decimal cast ([#4940](https://github.com/apache/datafusion-comet/pull/4940)), an optimized integer-to-integer cast - ([#4920](https://github.com/apache/datafusion-comet/pull/4920)), and shared no-overflow fast paths in + ([#4920](https://github.com/apache/datafusion-comet/pull/4920)), shared no-overflow fast paths in `CheckOverflow` ([#4937](https://github.com/apache/datafusion-comet/pull/4937)) and - `DecimalRescaleCheckOverflow` ([#4938](https://github.com/apache/datafusion-comet/pull/4938)). + `DecimalRescaleCheckOverflow` ([#4938](https://github.com/apache/datafusion-comet/pull/4938)), a ~40% faster + `float64`-to-`utf8` cast ([#4918](https://github.com/apache/datafusion-comet/pull/4918)), an optimized + `decimal128`-to-`utf8` cast ([#4924](https://github.com/apache/datafusion-comet/pull/4924)), string-to-date + parsing up to 2x faster ([#4917](https://github.com/apache/datafusion-comet/pull/4917)), + `parse_string_to_decimal` 30-40% faster ([#4916](https://github.com/apache/datafusion-comet/pull/4916)), and + `cast_binary_to_string` up to 27x faster on binary-format styles + ([#4912](https://github.com/apache/datafusion-comet/pull/4912)). +- **Decimal and date/time kernels**: `date_trunc` more than 2x faster + ([#4915](https://github.com/apache/datafusion-comet/pull/4915)), + `spark_ceil` 3x faster ([#4926](https://github.com/apache/datafusion-comet/pull/4926)), and a vectorized + `spark_unscaled_value` 9x faster ([#4972](https://github.com/apache/datafusion-comet/pull/4972)). - **String and array kernels**: `lpad` ([#4919](https://github.com/apache/datafusion-comet/pull/4919)), `unhex` ([#4876](https://github.com/apache/datafusion-comet/pull/4876)), `size` ([#4877](https://github.com/apache/datafusion-comet/pull/4877)), @@ -96,14 +106,21 @@ revised ([#4722](https://github.com/apache/datafusion-comet/pull/4722)). This release expands the set of Spark expressions and aggregates that run natively: - **Aggregates**: `approx_percentile` / `percentile_approx` - ([#4801](https://github.com/apache/datafusion-comet/pull/4801)) and exact `percentile` / `median` - ([#4542](https://github.com/apache/datafusion-comet/pull/4542)). + ([#4801](https://github.com/apache/datafusion-comet/pull/4801)), exact `percentile` / `median` + ([#4542](https://github.com/apache/datafusion-comet/pull/4542)), + `approx_count_distinct` ([#4819](https://github.com/apache/datafusion-comet/pull/4819)), and native + `collect_list` / `array_agg` ([#4720](https://github.com/apache/datafusion-comet/pull/4720)). - **Grouping**: `grouping()` and `grouping_id()` indicator functions ([#4815](https://github.com/apache/datafusion-comet/pull/4815)). - **Intervals**: interval types with `make_ym_interval` and `make_dt_interval` - ([#4541](https://github.com/apache/datafusion-comet/pull/4541)). -- **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)) and `split_part` via - `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)). + ([#4541](https://github.com/apache/datafusion-comet/pull/4541)), + `CalendarIntervalType` support ([#4898](https://github.com/apache/datafusion-comet/pull/4898)), and + `multiply_dt_interval` via codegen dispatch + ([#4900](https://github.com/apache/datafusion-comet/pull/4900)). +- **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)), + `split_part` via `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)), + native `levenshtein` ([#4105](https://github.com/apache/datafusion-comet/pull/4105)), and native + `randstr` compatible with Spark ([#5035](https://github.com/apache/datafusion-comet/pull/5035)). - **Array / map**: `array_prepend` ([#4716](https://github.com/apache/datafusion-comet/pull/4716)), `size()` for `MapType` ([#4580](https://github.com/apache/datafusion-comet/pull/4580)), `ElementAt` over `MapType` ([#4697](https://github.com/apache/datafusion-comet/pull/4697)), and removal of the constraint on @@ -130,10 +147,16 @@ early-stage feature and we welcome feedback from users experimenting with it. - **Iceberg 1.11 support** ([#4840](https://github.com/apache/datafusion-comet/pull/4840)): adds support for Iceberg 1.11, audits the existing Iceberg diffs, bumps the iceberg-rust dependency, and adds a `run-iceberg-tests` CI trigger. +- **Iceberg table format V3** ([#4991](https://github.com/apache/datafusion-comet/pull/4991)): native table + decryption for V3 tables, with fallback to Spark for other V3 features. Follow-up + [#5020](https://github.com/apache/datafusion-comet/pull/5020) applies the same diff changes across other + Iceberg versions. - **Delete-file correctness** ([#4760](https://github.com/apache/datafusion-comet/pull/4760)): the native scan now sizes Iceberg delete files correctly, avoiding dropped deletes. - **Exchange-reuse correctness** ([#4812](https://github.com/apache/datafusion-comet/pull/4812)): fixed a case where Iceberg native scan exchange reuse with different pushed filters could produce wrong results. +- **Native serde dedup** ([#4982](https://github.com/apache/datafusion-comet/pull/4982)): dedupes Iceberg + residuals and delete files in the native scan serde, reducing planning overhead. ## Native Parquet I/O and Cloud @@ -153,12 +176,31 @@ early-stage feature and we welcome feedback from users experimenting with it. expressions reported as `Unsupported` can now route through JVM codegen dispatch for opt-in serdes, and native opt-in expressions surface as compatible-by-default with a `COMET-INFO` plan hint ([#4721](https://github.com/apache/datafusion-comet/pull/4721)). +- **Collated predicate fallback** ([#4948](https://github.com/apache/datafusion-comet/pull/4948)): Comet + now falls back to Spark for predicates whose operands use non-default collations, avoiding subtle + correctness issues. +- **ANSI cast to date** ([#5014](https://github.com/apache/datafusion-comet/pull/5014)): invalid calendar + dates now raise `CAST_INVALID_INPUT` under ANSI mode, matching Spark's behavior. +- **Local scan nullability** ([#4843](https://github.com/apache/datafusion-comet/pull/4843)): local table scan + child nullability is now widened to match the native kernels, fixing a class of nullability mismatches. +- **Config aliases** ([#4979](https://github.com/apache/datafusion-comet/pull/4979)): a `withAlternative` + alias mechanism lets `CometConf` entries carry old names during renames without breaking existing + configurations. - **Runtime cleanup** ([#4734](https://github.com/apache/datafusion-comet/pull/4734)): the Tokio runtime is now released on driver and executor exit. - **Contrib scan SPI** ([#4700](https://github.com/apache/datafusion-comet/pull/4700)): a core SPI for contrib leaf scans (`CometScanWithPlanData`), the first part of splitting the Delta integration into a contrib module. +## Shuffle Improvements + +- **Native shuffle memory cap** ([#4989](https://github.com/apache/datafusion-comet/pull/4989)): a new + `spark.comet.shuffle.maxBufferBytes` config caps native shuffle writer memory to bound worst-case usage. +- **Shuffle IPC schema encoding** ([#5006](https://github.com/apache/datafusion-comet/pull/5006)): the IPC + schema is now encoded once per writer instead of per block, cutting per-batch shuffle overhead. +- **BatchCoalescer bypass** ([#5003](https://github.com/apache/datafusion-comet/pull/5003)): shuffle bypasses + the `BatchCoalescer` for batches that are already appropriately sized. + ## Deprecation Notice With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic versioning @@ -182,7 +224,7 @@ See the [Spark Version Compatibility] page for known limitations specific to eac [Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html -This release upgrades to **DataFusion 54** and **Arrow 58.3**. +This release upgrades to **DataFusion 54.1** and **Arrow 58.3**. ## Get Started with Comet 1.0.0 From c26ea16843583f11d4f22227db71f5abee3a808d Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 31 Jul 2026 15:11:49 -0600 Subject: [PATCH 03/19] Update Comet 1.0.0 blog post with commits merged since last update --- .../blog/2026-07-31-datafusion-comet-1.0.0.md | 183 ++++++++++++++++-- 1 file changed, 165 insertions(+), 18 deletions(-) diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-07-31-datafusion-comet-1.0.0.md index 81a9f5e6..1578f085 100644 --- a/content/blog/2026-07-31-datafusion-comet-1.0.0.md +++ b/content/blog/2026-07-31-datafusion-comet-1.0.0.md @@ -36,7 +36,7 @@ This is a major milestone. Comet began as a code donation in early 2024 and has one release at a time. Version 1.0.0 marks the point where the project is mature enough to commit to a stable release line: broad Apache Spark coverage, ANSI SQL semantics, native Parquet and Iceberg scans, and a native shuffle, all validated continuously against Spark's own test suites. This release covers roughly six weeks of -development since 0.17.0 and is the result of merging over 190 PRs from 22 contributors. See the +development since 0.17.0 and is the result of merging over 220 PRs from 21 contributors. See the [change log] for more information. [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md @@ -45,8 +45,11 @@ development since 0.17.0 and is the result of merging over 190 PRs from 22 contr Reaching 1.0 is less about any single new feature than about the accumulated maturity of the project: -- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.1 out of the same - codebase, with dedicated Maven profiles, shim sources, and CI matrices for each. +- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2 out of the same + codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental + Spark 4.2 profile for early evaluation. The + [Spark version adoption and support-lifetime policy](https://github.com/apache/datafusion-comet/pull/4977) + now states how long each Spark minor stays supported. - **ANSI SQL by default.** Spark 4 enables ANSI semantics by default. Comet implements ANSI behavior for the expressions it supports natively, including arithmetic overflow checks, ANSI cast behavior, and `try_*` variants, so ANSI workloads keep accelerating rather than falling back. @@ -54,9 +57,18 @@ Reaching 1.0 is less about any single new feature than about the accumulated mat native execution path against every supported Spark version. Running Spark's own correctness tests is what surfaces semantic shifts before they reach user workloads, and it is the foundation of the confidence behind a 1.0 release. -- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. The very - first deprecations under that policy are announced in this release (see [Deprecation Notice](#deprecation-notice) - below). +- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. The + [versioning policy](https://github.com/apache/datafusion-comet/pull/5056) spells out what that covers: + because Comet is a plugin rather than a library, its configuration is its primary API surface, so + `spark.comet.*` keys, an explicitly enumerated public Java and Scala API, and query results documented as + Compatible are all part of the compatibility surface. Behavior changes in a minor release now require a + `spark.comet.legacy.*` key that restores the previous behavior, and each release records its behavior changes + in a user-facing upgrade guide. Correctness fixes are exempt from being treated as breaking changes. The very + first deprecations under that policy are announced in this release (see + [Deprecation Notice](#deprecation-notice) below). +- **Documented limitations.** Every open correctness issue is now surfaced in the generated compatibility guide + ([#5085](https://github.com/apache/datafusion-comet/pull/5085)), down to the expression level, so you can see + where Comet is known to diverge from Spark before you hit it in production rather than after. The rest of this post covers what is new since 0.17.0. @@ -98,8 +110,71 @@ On the scan side, Parquet reads pick up several improvements: full Parquet metad is now cached via DataFusion's `CachedParquetFileReaderFactory` ([#4707](https://github.com/apache/datafusion-comet/pull/4707)), identity casts are unwrapped in the schema adapter so Parquet statistics pruning can engage -([#4730](https://github.com/apache/datafusion-comet/pull/4730)), and filter pushdown configuration has been -revised ([#4722](https://github.com/apache/datafusion-comet/pull/4722)). +([#4730](https://github.com/apache/datafusion-comet/pull/4730)), filter pushdown configuration has been +revised ([#4722](https://github.com/apache/datafusion-comet/pull/4722)), the native scan now passes a metadata +size hint so a single read usually captures the footer, matching the Iceberg path +([#4717](https://github.com/apache/datafusion-comet/pull/4717)), and the native Parquet scan seeds its reader +options from the session config so Parquet settings you already set take effect +([#5107](https://github.com/apache/datafusion-comet/pull/5107)). + +Two lower-level costs were also removed: Comet no longer makes excessive timer calls in the native execution +loop ([#4739](https://github.com/apache/datafusion-comet/pull/4739)), and plan-data injection is now an O(1) +lookup by operator kind rather than a scan ([#4535](https://github.com/apache/datafusion-comet/pull/4535)). + +## Correctness + +A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most of +them found by running Spark's own SQL test suites through Comet's native path: + +- **ANSI error semantics**: `round` on a `Long` with a large negative scale now overflows instead of silently + returning zero ([#5082](https://github.com/apache/datafusion-comet/pull/5082)), `Long.MinValue / -1` raises + `ARITHMETIC_OVERFLOW` ([#5084](https://github.com/apache/datafusion-comet/pull/5084)), floating-point + `x % 0.0` raises `REMAINDER_BY_ZERO` instead of returning `NaN` + ([#5081](https://github.com/apache/datafusion-comet/pull/5081)), `make_decimal` honors its + fail-on-error flag and throws `NUMERIC_VALUE_OUT_OF_RANGE` rather than returning null + ([#5080](https://github.com/apache/datafusion-comet/pull/5080)), invalid calendar dates raise + `CAST_INVALID_INPUT` ([#5014](https://github.com/apache/datafusion-comet/pull/5014)), and errors from nested + casts now propagate ([#4675](https://github.com/apache/datafusion-comet/pull/4675)). +- **Wrong results**: `count` no longer returns zero when the native scan is disabled + ([#4795](https://github.com/apache/datafusion-comet/pull/4795)), Spark's legacy `null IN ()` behavior is + honored ([#5127](https://github.com/apache/datafusion-comet/pull/5127)), `pow` matches Java's `Math.pow` on + the edge cases where C and Java disagree ([#5033](https://github.com/apache/datafusion-comet/pull/5033)), + `percentile` matches Spark's interpolation precision + ([#4792](https://github.com/apache/datafusion-comet/pull/4792)), `date_trunc` handles DST boundaries in + non-UTC sessions and no longer produces schema mismatches through shuffle and sort + ([#4761](https://github.com/apache/datafusion-comet/pull/4761)), `flatten` handles null sub-arrays + ([#4822](https://github.com/apache/datafusion-comet/pull/4822)), `ArrayInsert` handles null source arrays + ([#4726](https://github.com/apache/datafusion-comet/pull/4726)), and the `array_filter` / `array_compact` + fast path is restricted to the lambda variable + ([#4848](https://github.com/apache/datafusion-comet/pull/4848)). +- **Strings and encodings**: `CAST(binary AS string)` now decodes exactly as the JVM's + `new String(bytes, UTF_8)` does, including the surrogate cases where Rust's lossy decoder diverges, instead + of reinterpreting bytes unchecked ([#4763](https://github.com/apache/datafusion-comet/pull/4763)), and + shuffle tolerates non-UTF-8 bytes rather than failing + ([#4524](https://github.com/apache/datafusion-comet/pull/4524)). +- **Timestamps**: reading a Parquet timestamp-with-timezone column as `TimestampNTZ` is now rejected on Spark + 3.x, matching Spark's own refusal, instead of silently returning the UTC instant + ([#4357](https://github.com/apache/datafusion-comet/pull/4357)). +- **Deep expression trees**: long chains of associative bitwise, `Add`, and `Multiply` operators are rebalanced + so plans no longer hit protobuf's recursion limit + ([#4588](https://github.com/apache/datafusion-comet/pull/4588)). +- **Native shuffle**: scalar subqueries used in partitioning expressions are now registered, fixing + "Subquery N not found" failures ([#4869](https://github.com/apache/datafusion-comet/pull/4869)), and constant + column vectors are materialized on Comet's serialize and export paths + ([#4532](https://github.com/apache/datafusion-comet/pull/4532)). +- **Collations**: Comet falls back for predicates whose operands use non-default collations + ([#4948](https://github.com/apache/datafusion-comet/pull/4948)), for Spark 4 datetime expressions under + non-default collations ([#4693](https://github.com/apache/datafusion-comet/pull/4693)), and for + `str_to_map` ([#4701](https://github.com/apache/datafusion-comet/pull/4701)). +- **Conservative fallbacks** where native behavior could not be made to match: decimal `SUM` / `AVG` over + sliding window frames ([#4732](https://github.com/apache/datafusion-comet/pull/4732)), + `FromUnixTime` with a non-default format ([#4847](https://github.com/apache/datafusion-comet/pull/4847)), + `CreateArray` with struct-nullability-divergent children + ([#4533](https://github.com/apache/datafusion-comet/pull/4533)), and native V1 scans on filesystem schemes + that `object_store` does not support ([#4525](https://github.com/apache/datafusion-comet/pull/4525)). +- **Error reporting**: native Parquet read failures now surface as Spark's `FAILED_READ_FILE` + ([#4536](https://github.com/apache/datafusion-comet/pull/4536)), and a DataFusion 54.1.0 Parquet page-index + regression is worked around ([#5132](https://github.com/apache/datafusion-comet/pull/5132)). ## New Expression and Aggregate Support @@ -114,14 +189,16 @@ This release expands the set of Spark expressions and aggregates that run native ([#4815](https://github.com/apache/datafusion-comet/pull/4815)). - **Intervals**: interval types with `make_ym_interval` and `make_dt_interval` ([#4541](https://github.com/apache/datafusion-comet/pull/4541)), - `CalendarIntervalType` support ([#4898](https://github.com/apache/datafusion-comet/pull/4898)), and + `CalendarIntervalType` support ([#4898](https://github.com/apache/datafusion-comet/pull/4898)), `multiply_dt_interval` via codegen dispatch - ([#4900](https://github.com/apache/datafusion-comet/pull/4900)). + ([#4900](https://github.com/apache/datafusion-comet/pull/4900)), and interval codegen dispatch for nested + values and native shuffle ([#4976](https://github.com/apache/datafusion-comet/pull/4976)). - **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)), `split_part` via `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)), native `levenshtein` ([#4105](https://github.com/apache/datafusion-comet/pull/4105)), and native `randstr` compatible with Spark ([#5035](https://github.com/apache/datafusion-comet/pull/5035)). - **Array / map**: `array_prepend` ([#4716](https://github.com/apache/datafusion-comet/pull/4716)), + the `shuffle()` array function ([#4797](https://github.com/apache/datafusion-comet/pull/4797)), `size()` for `MapType` ([#4580](https://github.com/apache/datafusion-comet/pull/4580)), `ElementAt` over `MapType` ([#4697](https://github.com/apache/datafusion-comet/pull/4697)), and removal of the constraint on arrays of nested elements ([#4714](https://github.com/apache/datafusion-comet/pull/4714)). @@ -132,6 +209,19 @@ This release expands the set of Spark expressions and aggregates that run native - **Windows**: extended native window function support ([#4209](https://github.com/apache/datafusion-comet/pull/4209)) and Spark 4 decimal window average ([#4749](https://github.com/apache/datafusion-comet/pull/4749)). +- **Internal expressions**: native `empty2null` + ([#4683](https://github.com/apache/datafusion-comet/pull/4683)), which appears in write paths, and the + shuffle-side infrastructure for Spark's `TimeType` + ([#4398](https://github.com/apache/datafusion-comet/pull/4398)). + +## Native Sampling + +Spark's `SampleExec` now runs natively when sampling without replacement +([#5110](https://github.com/apache/datafusion-comet/pull/5110)), covering `DataFrame.sample`, SQL +`TABLESAMPLE`, and `DataFrame.randomSplit`. The native operator ports Spark's `BernoulliCellSampler` on top of +`XorShiftRandom` and seeds per partition exactly as Spark does, so it selects the same rows for a given seed — +sampling stays reproducible whether or not Comet is enabled. Sampling with replacement still falls back to +Spark. ## Experimental PyArrow UDF Support @@ -175,12 +265,24 @@ early-stage feature and we welcome feedback from users experimenting with it. - **Codegen dispatch for opt-in serdes** ([#4728](https://github.com/apache/datafusion-comet/pull/4728)): expressions reported as `Unsupported` can now route through JVM codegen dispatch for opt-in serdes, and native opt-in expressions surface as compatible-by-default with a `COMET-INFO` plan hint - ([#4721](https://github.com/apache/datafusion-comet/pull/4721)). -- **Collated predicate fallback** ([#4948](https://github.com/apache/datafusion-comet/pull/4948)): Comet - now falls back to Spark for predicates whose operands use non-default collations, avoiding subtle - correctness issues. -- **ANSI cast to date** ([#5014](https://github.com/apache/datafusion-comet/pull/5014)): invalid calendar - dates now raise `CAST_INVALID_INPUT` under ANSI mode, matching Spark's behavior. + ([#4721](https://github.com/apache/datafusion-comet/pull/4721)). Casts join this path + ([#5079](https://github.com/apache/datafusion-comet/pull/5079)): an incompatible or unsupported cast now runs + Spark's own generated code inside the Comet pipeline instead of pulling the whole operator back to Spark, so + one awkward cast no longer de-accelerates a query. `sort_array` under strict floating-point mode + ([#4637](https://github.com/apache/datafusion-comet/pull/4637)) and `concat` under non-`UTF8_BINARY` + collations ([#4640](https://github.com/apache/datafusion-comet/pull/4640)) opt in the same way. +- **JVM columnar-to-row by default** ([#5114](https://github.com/apache/datafusion-comet/pull/5114)): isolated + benchmarking showed the native columnar-to-row converter is roughly 3.7x slower per row than the JVM + implementation at the default batch size — and up to 15.7x slower for small batches — because of a fixed + per-batch JNI and FFI cost, with no end-to-end benefit in TPC-DS runs. It is now disabled by default. +- **Clearer fallback reasons**: unsupported scalar serde cases are reported in the support level + ([#4745](https://github.com/apache/datafusion-comet/pull/4745)), mixed-execution fallback messages name the + incompatible aggregate functions ([#4750](https://github.com/apache/datafusion-comet/pull/4750)), unsupported + metadata column names appear in fallback reasons + ([#4758](https://github.com/apache/datafusion-comet/pull/4758)), the compatibility guide URLs in fallback + messages are corrected ([#4854](https://github.com/apache/datafusion-comet/pull/4854)), and the spurious + "WriteFilesExec is not supported" message is suppressed + ([#4928](https://github.com/apache/datafusion-comet/pull/4928)). - **Local scan nullability** ([#4843](https://github.com/apache/datafusion-comet/pull/4843)): local table scan child nullability is now widened to match the native kernels, fixing a class of nullability mismatches. - **Config aliases** ([#4979](https://github.com/apache/datafusion-comet/pull/4979)): a `withAlternative` @@ -188,6 +290,8 @@ early-stage feature and we welcome feedback from users experimenting with it. configurations. - **Runtime cleanup** ([#4734](https://github.com/apache/datafusion-comet/pull/4734)): the Tokio runtime is now released on driver and executor exit. +- **Quieter logs** ([#5155](https://github.com/apache/datafusion-comet/pull/5155)): logging levels on frequent + call sites have been lowered so Comet no longer floods executor logs at default settings. - **Contrib scan SPI** ([#4700](https://github.com/apache/datafusion-comet/pull/4700)): a core SPI for contrib leaf scans (`CometScanWithPlanData`), the first part of splitting the Delta integration into a contrib module. @@ -200,6 +304,48 @@ early-stage feature and we welcome feedback from users experimenting with it. schema is now encoded once per writer instead of per block, cutting per-batch shuffle overhead. - **BatchCoalescer bypass** ([#5003](https://github.com/apache/datafusion-comet/pull/5003)): shuffle bypasses the `BatchCoalescer` for batches that are already appropriately sized. +- **Single-partition shuffle** ([#5004](https://github.com/apache/datafusion-comet/pull/5004)): a redundant + concatenation layer has been removed from the single-partition path. + +## Configuration Changes + +Because configuration is Comet's primary API surface, 1.0 is the release where the naming gets cleaned up. +Every rename below registers the old key as an alias, so existing configurations keep working and log a +deprecation warning when an old key is read: + +- **Unified shuffle prefix** ([#4986](https://github.com/apache/datafusion-comet/pull/4986)): shuffle configs + were spread across four disjoint prefixes (`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`, + `spark.comet.native.shuffle.*`, `spark.comet.shuffle.*`). They now all live under `spark.comet.shuffle.*`, + with `.jvm.` and `.native.` sub-namespaces matching the `spark.comet.shuffle.mode` value you already set. +- **Grouped explain configs** ([#5026](https://github.com/apache/datafusion-comet/pull/5026)): the orphan + explain-related configs are collected under a single `spark.comet.explain.*` prefix. +- **`spark.comet.version`** ([#5049](https://github.com/apache/datafusion-comet/pull/5049)): the loaded Comet + build version is now exposed as a runtime config, so you can confirm which Comet a cluster is actually + running with `spark.conf.get` or `SET`. +- **Removed dead configs**: the Parquet parallel-IO knobs + ([#4981](https://github.com/apache/datafusion-comet/pull/4981)) and + `spark.comet.use.lazyMaterialization` ([#4998](https://github.com/apache/datafusion-comet/pull/4998)) had no + remaining effect and were misleading anyone tuning against them. +- **Removed async columnar shuffle** ([#4985](https://github.com/apache/datafusion-comet/pull/4985)): the + `spark.comet.columnar.shuffle.async.*` path was off by default and untested, and has been removed rather than + carried into a stable release line. + +## Documentation + +The documentation received a substantial overhaul for 1.0: + +- A [design refresh](https://github.com/apache/datafusion-comet/pull/4353) of the docs site, with versioned + user guides gaining captioned sidebar sections + ([#4699](https://github.com/apache/datafusion-comet/pull/4699)) and a set of accessibility and navigation + fixes ([#4858](https://github.com/apache/datafusion-comet/pull/4858)). +- The supported-expressions page now shows how each expression is implemented — native, codegen dispatch, or + hybrid ([#5028](https://github.com/apache/datafusion-comet/pull/5028)), generated from the planner itself + rather than maintained by hand. +- An expanded [tuning guide](https://github.com/apache/datafusion-comet/pull/4908) covering the performance and + memory configs that matter in practice, plus guidance on `spark.sql.files.maxPartitionBytes` + ([#4931](https://github.com/apache/datafusion-comet/pull/4931)). +- An [updated roadmap](https://github.com/apache/datafusion-comet/pull/5064) for the post-1.0 work, and a new + [blog posts and talks page](https://github.com/apache/datafusion-comet/pull/5043). ## Deprecation Notice @@ -218,13 +364,14 @@ Supported platforms include: - **Spark 3.4.3** with Java 11/17 and Scala 2.12/2.13 (deprecated, removal in 1.1.0) - **Spark 3.5.8** with Java 11/17 and Scala 2.12/2.13 - **Spark 4.0.2** with Java 17 and Scala 2.13 -- **Spark 4.1.1** with Java 17 and Scala 2.13 +- **Spark 4.1.2** with Java 17/21 and Scala 2.13 +- **Spark 4.2** with Java 17 and Scala 2.13 (experimental, for early evaluation only) See the [Spark Version Compatibility] page for known limitations specific to each version. [Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html -This release upgrades to **DataFusion 54.1** and **Arrow 58.3**. +This release upgrades to **DataFusion 54.1** and **Arrow 58.4**. ## Get Started with Comet 1.0.0 From 0a28ee3565383b1ab3b5a630dc766e1b51f9c22a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 09:23:27 -0600 Subject: [PATCH 04/19] Update Comet 1.0.0 blog post for final branch-1.0 state --- .../blog/2026-07-31-datafusion-comet-1.0.0.md | 66 ++++++++++++++----- 1 file changed, 51 insertions(+), 15 deletions(-) diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-07-31-datafusion-comet-1.0.0.md index 1578f085..15e577ea 100644 --- a/content/blog/2026-07-31-datafusion-comet-1.0.0.md +++ b/content/blog/2026-07-31-datafusion-comet-1.0.0.md @@ -36,7 +36,7 @@ This is a major milestone. Comet began as a code donation in early 2024 and has one release at a time. Version 1.0.0 marks the point where the project is mature enough to commit to a stable release line: broad Apache Spark coverage, ANSI SQL semantics, native Parquet and Iceberg scans, and a native shuffle, all validated continuously against Spark's own test suites. This release covers roughly six weeks of -development since 0.17.0 and is the result of merging over 220 PRs from 21 contributors. See the +development since 0.17.0 and consists of 244 commits from 23 contributors. See the [change log] for more information. [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md @@ -45,7 +45,7 @@ development since 0.17.0 and is the result of merging over 220 PRs from 21 contr Reaching 1.0 is less about any single new feature than about the accumulated maturity of the project: -- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2 out of the same +- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.9, 4.0.4, and 4.1.3 out of the same codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental Spark 4.2 profile for early evaluation. The [Spark version adoption and support-lifetime policy](https://github.com/apache/datafusion-comet/pull/4977) @@ -68,7 +68,9 @@ Reaching 1.0 is less about any single new feature than about the accumulated mat [Deprecation Notice](#deprecation-notice) below). - **Documented limitations.** Every open correctness issue is now surfaced in the generated compatibility guide ([#5085](https://github.com/apache/datafusion-comet/pull/5085)), down to the expression level, so you can see - where Comet is known to diverge from Spark before you hit it in production rather than after. + where Comet is known to diverge from Spark before you hit it in production rather than after. Notes for bugs + that have since been fixed were removed in the same pass + ([#5154](https://github.com/apache/datafusion-comet/pull/5154)). The rest of this post covers what is new since 0.17.0. @@ -117,17 +119,36 @@ size hint so a single read usually captures the footer, matching the Iceberg pat options from the session config so Parquet settings you already set take effect ([#5107](https://github.com/apache/datafusion-comet/pull/5107)). -Two lower-level costs were also removed: Comet no longer makes excessive timer calls in the native execution -loop ([#4739](https://github.com/apache/datafusion-comet/pull/4739)), and plan-data injection is now an O(1) -lookup by operator kind rather than a scan ([#4535](https://github.com/apache/datafusion-comet/pull/4535)). +Query planning and plan serialization also got cheaper. `QueryContext` SQL text is now interned into a per-plan +pool ([#5204](https://github.com/apache/datafusion-comet/pull/5204)), which makes serialized plans up to 20x +smaller on TPC-DS — every plan crosses the JNI boundary, so this shrinks per-query overhead across the board. +Plan-data injection is now an O(1) lookup by operator kind +([#4535](https://github.com/apache/datafusion-comet/pull/4535)) and no longer rebuilds operators it does not +touch ([#5220](https://github.com/apache/datafusion-comet/pull/5220)). Comet also makes far fewer timer calls +in the native execution loop ([#4739](https://github.com/apache/datafusion-comet/pull/4739)), and nested array +equality now uses Arrow's comparator ([#5176](https://github.com/apache/datafusion-comet/pull/5176)). ## Correctness A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most of them found by running Spark's own SQL test suites through Comet's native path: -- **ANSI error semantics**: `round` on a `Long` with a large negative scale now overflows instead of silently - returning zero ([#5082](https://github.com/apache/datafusion-comet/pull/5082)), `Long.MinValue / -1` raises +- **Casts**: casting a string to `boolean`, an integral type, `float`/`double`, or `decimal` now uses Spark's + exact whitespace-trimming rules ([#5150](https://github.com/apache/datafusion-comet/pull/5150)). Comet's + kernels had used four different trim sets, three of them wrong, so results diverged in both directions — + returning null where Spark parses a value, and returning a value where Spark returns null. Casting + `float`/`double` to `decimal` now rounds the shortest decimal string form as Spark does, rather than the + binary value ([#5136](https://github.com/apache/datafusion-comet/pull/5136)), which matters for values such + as `0.5153125` whose binary form sits just below the rounding tie the string form lands on; `NaN` and infinity + now return null even in ANSI mode, matching Spark. Decimal promotion also uses the per-expression eval mode + ([#5171](https://github.com/apache/datafusion-comet/pull/5171)). +- **ANSI error semantics**: the codegen dispatcher's null short-circuit no longer swallows errors Spark raises + ([#5219](https://github.com/apache/datafusion-comet/pull/5219)) — Spark evaluates null-intolerant expressions + per node and left to right, so short-circuiting on the union of input ordinals skipped subtrees Spark would + have evaluated, losing their errors. Roughly 70 built-in expressions route through this dispatcher and ANSI is + on by default in Spark 4, so this affected a wide surface. Also: `round` on a `Long` with a large negative + scale now overflows instead of silently returning zero + ([#5082](https://github.com/apache/datafusion-comet/pull/5082)), `Long.MinValue / -1` raises `ARITHMETIC_OVERFLOW` ([#5084](https://github.com/apache/datafusion-comet/pull/5084)), floating-point `x % 0.0` raises `REMAINDER_BY_ZERO` instead of returning `NaN` ([#5081](https://github.com/apache/datafusion-comet/pull/5081)), `make_decimal` honors its @@ -196,7 +217,9 @@ This release expands the set of Spark expressions and aggregates that run native - **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)), `split_part` via `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)), native `levenshtein` ([#4105](https://github.com/apache/datafusion-comet/pull/4105)), and native - `randstr` compatible with Spark ([#5035](https://github.com/apache/datafusion-comet/pull/5035)). + `randstr` ([#5035](https://github.com/apache/datafusion-comet/pull/5035)) and `uuid` + ([#5034](https://github.com/apache/datafusion-comet/pull/5034)), both bit-for-bit compatible with Spark for a + given seed. - **Array / map**: `array_prepend` ([#4716](https://github.com/apache/datafusion-comet/pull/4716)), the `shuffle()` array function ([#4797](https://github.com/apache/datafusion-comet/pull/4797)), `size()` for `MapType` ([#4580](https://github.com/apache/datafusion-comet/pull/4580)), `ElementAt` over @@ -241,10 +264,14 @@ early-stage feature and we welcome feedback from users experimenting with it. decryption for V3 tables, with fallback to Spark for other V3 features. Follow-up [#5020](https://github.com/apache/datafusion-comet/pull/5020) applies the same diff changes across other Iceberg versions. +- **Metadata columns** ([#4752](https://github.com/apache/datafusion-comet/pull/4752)): the native Iceberg scan + now supports the `_pos`, `_spec`, `_file`, and `_partition` metadata columns. - **Delete-file correctness** ([#4760](https://github.com/apache/datafusion-comet/pull/4760)): the native scan now sizes Iceberg delete files correctly, avoiding dropped deletes. - **Exchange-reuse correctness** ([#4812](https://github.com/apache/datafusion-comet/pull/4812)): fixed a case where Iceberg native scan exchange reuse with different pushed filters could produce wrong results. +- **Scan disambiguation** ([#5180](https://github.com/apache/datafusion-comet/pull/5180)): Iceberg scans that + share a `metadata_location` are now told apart, rather than being conflated. - **Native serde dedup** ([#4982](https://github.com/apache/datafusion-comet/pull/4982)): dedupes Iceberg residuals and delete files in the native scan serde, reducing planning overhead. @@ -283,6 +310,12 @@ early-stage feature and we welcome feedback from users experimenting with it. messages are corrected ([#4854](https://github.com/apache/datafusion-comet/pull/4854)), and the spurious "WriteFilesExec is not supported" message is suppressed ([#4928](https://github.com/apache/datafusion-comet/pull/4928)). +- **Expression coverage in extended explain** ([#5201](https://github.com/apache/datafusion-comet/pull/5201)): + the extended explain summary previously reported operator coverage but said nothing about expressions. It now + ends with a line such as `Comet accelerated 14 expressions (14 native, 1 codegen dispatch)`, so you can see + how much of a plan's expression evaluation runs in native DataFusion kernels versus Spark's generated code + inside the dispatcher. Operator counts also handle `ReusedSubquery` and `CometSubqueryBroadcast` correctly + ([#5206](https://github.com/apache/datafusion-comet/pull/5206)). - **Local scan nullability** ([#4843](https://github.com/apache/datafusion-comet/pull/4843)): local table scan child nullability is now widened to match the native kernels, fixing a class of nullability mismatches. - **Config aliases** ([#4979](https://github.com/apache/datafusion-comet/pull/4979)): a `withAlternative` @@ -318,13 +351,16 @@ deprecation warning when an old key is read: `spark.comet.native.shuffle.*`, `spark.comet.shuffle.*`). They now all live under `spark.comet.shuffle.*`, with `.jvm.` and `.native.` sub-namespaces matching the `spark.comet.shuffle.mode` value you already set. - **Grouped explain configs** ([#5026](https://github.com/apache/datafusion-comet/pull/5026)): the orphan - explain-related configs are collected under a single `spark.comet.explain.*` prefix. + explain-related configs are collected under a single `spark.comet.explain.*` prefix, and the PyArrow UDF + config is renamed to `pyarrowUDF` for consistency + ([#5197](https://github.com/apache/datafusion-comet/pull/5197)). - **`spark.comet.version`** ([#5049](https://github.com/apache/datafusion-comet/pull/5049)): the loaded Comet build version is now exposed as a runtime config, so you can confirm which Comet a cluster is actually running with `spark.conf.get` or `SET`. - **Removed dead configs**: the Parquet parallel-IO knobs - ([#4981](https://github.com/apache/datafusion-comet/pull/4981)) and - `spark.comet.use.lazyMaterialization` ([#4998](https://github.com/apache/datafusion-comet/pull/4998)) had no + ([#4981](https://github.com/apache/datafusion-comet/pull/4981)), + `spark.comet.use.lazyMaterialization` ([#4998](https://github.com/apache/datafusion-comet/pull/4998)), and + `spark.comet.exceptionOnDatetimeRebase` ([#5221](https://github.com/apache/datafusion-comet/pull/5221)) had no remaining effect and were misleading anyone tuning against them. - **Removed async columnar shuffle** ([#4985](https://github.com/apache/datafusion-comet/pull/4985)): the `spark.comet.columnar.shuffle.async.*` path was off by default and untested, and has been removed rather than @@ -362,9 +398,9 @@ Users on these platforms should plan to move to JDK 17+ and Spark 3.5 or later b Supported platforms include: - **Spark 3.4.3** with Java 11/17 and Scala 2.12/2.13 (deprecated, removal in 1.1.0) -- **Spark 3.5.8** with Java 11/17 and Scala 2.12/2.13 -- **Spark 4.0.2** with Java 17 and Scala 2.13 -- **Spark 4.1.2** with Java 17/21 and Scala 2.13 +- **Spark 3.5.9** with Java 11/17 and Scala 2.12/2.13 +- **Spark 4.0.4** with Java 17 and Scala 2.13 +- **Spark 4.1.3** with Java 17/21 and Scala 2.13 - **Spark 4.2** with Java 17 and Scala 2.13 (experimental, for early evaluation only) See the [Spark Version Compatibility] page for known limitations specific to each version. From d8a539a9ed50d1b33956a65f87e85876b88d2b2c Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 09:39:08 -0600 Subject: [PATCH 05/19] fixes --- content/blog/2026-07-31-datafusion-comet-1.0.0.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-07-31-datafusion-comet-1.0.0.md index 15e577ea..2f3ba677 100644 --- a/content/blog/2026-07-31-datafusion-comet-1.0.0.md +++ b/content/blog/2026-07-31-datafusion-comet-1.0.0.md @@ -32,18 +32,17 @@ The Apache DataFusion PMC is pleased to announce version 1.0.0 of the [Comet](ht Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for improved performance and efficiency without requiring any code changes. -This is a major milestone. Comet began as a code donation in early 2024 and has shipped steadily ever since, -one release at a time. Version 1.0.0 marks the point where the project is mature enough to commit to a stable -release line: broad Apache Spark coverage, ANSI SQL semantics, native Parquet and Iceberg scans, and a native -shuffle, all validated continuously against Spark's own test suites. This release covers roughly six weeks of -development since 0.17.0 and consists of 244 commits from 23 contributors. See the -[change log] for more information. +This release covers roughly six weeks of +development since 0.17.0 and consists of 244 commits from 23 contributors. See the [change log] for more information. + [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md ## What 1.0 Means -Reaching 1.0 is less about any single new feature than about the accumulated maturity of the project: +The 1.0.0 release is the culmination of more than two year's work since the project was [donated] as an Apache DataFusion subproject in March, 2024, and is less about any single new feature than about the accumulated maturity of the project: + +[donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ - **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.9, 4.0.4, and 4.1.3 out of the same codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental @@ -131,7 +130,7 @@ equality now uses Arrow's comparator ([#5176](https://github.com/apache/datafusi ## Correctness A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most of -them found by running Spark's own SQL test suites through Comet's native path: +them found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's expression implementations with all supported versions of Spark. - **Casts**: casting a string to `boolean`, an integral type, `float`/`double`, or `decimal` now uses Spark's exact whitespace-trimming rules ([#5150](https://github.com/apache/datafusion-comet/pull/5150)). Comet's From 769cee76e1d5bfd9d5d074fc84da3973e57978f7 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 09:41:03 -0600 Subject: [PATCH 06/19] change date --- ...sion-comet-1.0.0.md => 2026-08-07-datafusion-comet-1.0.0.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename content/blog/{2026-07-31-datafusion-comet-1.0.0.md => 2026-08-07-datafusion-comet-1.0.0.md} (99%) diff --git a/content/blog/2026-07-31-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md similarity index 99% rename from content/blog/2026-07-31-datafusion-comet-1.0.0.md rename to content/blog/2026-08-07-datafusion-comet-1.0.0.md index 2f3ba677..dc8475da 100644 --- a/content/blog/2026-07-31-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -1,7 +1,7 @@ --- layout: post title: Apache DataFusion Comet 1.0.0 Release -date: 2026-07-31 +date: 2026-08-07 author: pmc categories: [subprojects] --- From 6d81c8b9225c12eee7ef1dd9d290a1b9b1e05c62 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 10:08:51 -0600 Subject: [PATCH 07/19] Trim changelog feel from Comet 1.0.0 blog post Remove inline PR links throughout, compress long enumerations into grouped lists, and consolidate the Iceberg / correctness / config sections into shorter prose. The change log link at the top is now the pointer for per-PR detail. --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 437 ++++++------------ 1 file changed, 151 insertions(+), 286 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index dc8475da..2c13b2fe 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -32,22 +32,22 @@ The Apache DataFusion PMC is pleased to announce version 1.0.0 of the [Comet](ht Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for improved performance and efficiency without requiring any code changes. -This release covers roughly six weeks of -development since 0.17.0 and consists of 244 commits from 23 contributors. See the [change log] for more information. - +This release covers roughly six weeks of development since 0.17.0 and consists of 244 commits from 23 +contributors. See the [change log] for the full list of changes. [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md ## What 1.0 Means -The 1.0.0 release is the culmination of more than two year's work since the project was [donated] as an Apache DataFusion subproject in March, 2024, and is less about any single new feature than about the accumulated maturity of the project: +The 1.0.0 release is the culmination of more than two years of work since the project was [donated] as an +Apache DataFusion subproject in March 2024, and is less about any single new feature than about the +accumulated maturity of the project: [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ - **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.9, 4.0.4, and 4.1.3 out of the same codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental - Spark 4.2 profile for early evaluation. The - [Spark version adoption and support-lifetime policy](https://github.com/apache/datafusion-comet/pull/4977) + Spark 4.2 profile for early evaluation. A published Spark version adoption and support-lifetime policy now states how long each Spark minor stays supported. - **ANSI SQL by default.** Spark 4 enables ANSI semantics by default. Comet implements ANSI behavior for the expressions it supports natively, including arithmetic overflow checks, ANSI cast behavior, and `try_*` @@ -56,190 +56,106 @@ The 1.0.0 release is the culmination of more than two year's work since the proj native execution path against every supported Spark version. Running Spark's own correctness tests is what surfaces semantic shifts before they reach user workloads, and it is the foundation of the confidence behind a 1.0 release. -- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. The - [versioning policy](https://github.com/apache/datafusion-comet/pull/5056) spells out what that covers: - because Comet is a plugin rather than a library, its configuration is its primary API surface, so - `spark.comet.*` keys, an explicitly enumerated public Java and Scala API, and query results documented as - Compatible are all part of the compatibility surface. Behavior changes in a minor release now require a - `spark.comet.legacy.*` key that restores the previous behavior, and each release records its behavior changes - in a user-facing upgrade guide. Correctness fixes are exempt from being treated as breaking changes. The very - first deprecations under that policy are announced in this release (see - [Deprecation Notice](#deprecation-notice) below). -- **Documented limitations.** Every open correctness issue is now surfaced in the generated compatibility guide - ([#5085](https://github.com/apache/datafusion-comet/pull/5085)), down to the expression level, so you can see - where Comet is known to diverge from Spark before you hit it in production rather than after. Notes for bugs - that have since been fixed were removed in the same pass - ([#5154](https://github.com/apache/datafusion-comet/pull/5154)). +- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. Because Comet + is a plugin rather than a library, its configuration is its primary API surface, so `spark.comet.*` keys, an + explicitly enumerated public Java and Scala API, and query results documented as Compatible are all part of + the compatibility surface. Behavior changes in a minor release now require a `spark.comet.legacy.*` key that + restores the previous behavior, and each release records its behavior changes in a user-facing upgrade + guide. Correctness fixes are exempt from being treated as breaking changes. The first deprecations under + that policy are announced in this release (see [Deprecation Notice](#deprecation-notice) below). +- **Documented limitations.** Every open correctness issue is now surfaced in the generated compatibility + guide, down to the expression level, so you can see where Comet is known to diverge from Spark before you + hit it in production rather than after. The rest of this post covers what is new since 0.17.0. ## Native Expression Performance A large share of this release is dedicated to making Comet's native scalar expressions faster. These kernels -run per-row or per-batch, so improvements here compound across every query that uses them: - -- **`get_json_object`** ([#4907](https://github.com/apache/datafusion-comet/pull/4907)): roughly 4x faster. -- **`regexp_extract`** ([#4894](https://github.com/apache/datafusion-comet/pull/4894)) and **`parse_url`** - ([#4893](https://github.com/apache/datafusion-comet/pull/4893)): optimized regex and URL parsing paths. -- **Casts**: a faster floating-point-to-decimal cast - ([#4940](https://github.com/apache/datafusion-comet/pull/4940)), an optimized integer-to-integer cast - ([#4920](https://github.com/apache/datafusion-comet/pull/4920)), shared no-overflow fast paths in - `CheckOverflow` ([#4937](https://github.com/apache/datafusion-comet/pull/4937)) and - `DecimalRescaleCheckOverflow` ([#4938](https://github.com/apache/datafusion-comet/pull/4938)), a ~40% faster - `float64`-to-`utf8` cast ([#4918](https://github.com/apache/datafusion-comet/pull/4918)), an optimized - `decimal128`-to-`utf8` cast ([#4924](https://github.com/apache/datafusion-comet/pull/4924)), string-to-date - parsing up to 2x faster ([#4917](https://github.com/apache/datafusion-comet/pull/4917)), - `parse_string_to_decimal` 30-40% faster ([#4916](https://github.com/apache/datafusion-comet/pull/4916)), and - `cast_binary_to_string` up to 27x faster on binary-format styles - ([#4912](https://github.com/apache/datafusion-comet/pull/4912)). -- **Decimal and date/time kernels**: `date_trunc` more than 2x faster - ([#4915](https://github.com/apache/datafusion-comet/pull/4915)), - `spark_ceil` 3x faster ([#4926](https://github.com/apache/datafusion-comet/pull/4926)), and a vectorized - `spark_unscaled_value` 9x faster ([#4972](https://github.com/apache/datafusion-comet/pull/4972)). -- **String and array kernels**: `lpad` ([#4919](https://github.com/apache/datafusion-comet/pull/4919)), - `unhex` ([#4876](https://github.com/apache/datafusion-comet/pull/4876)), - `size` ([#4877](https://github.com/apache/datafusion-comet/pull/4877)), - `arrays_overlap` ([#4906](https://github.com/apache/datafusion-comet/pull/4906)), - `escape_string` ([#4902](https://github.com/apache/datafusion-comet/pull/4902)), and the - `try_*` arithmetic kernel ([#4910](https://github.com/apache/datafusion-comet/pull/4910)). - -To make this kind of work repeatable, the release also adds a -[scalar expression optimization guide](https://github.com/apache/datafusion-comet/pull/4933) documenting how -to benchmark a kernel, keep its output bit-identical to Spark, and gate changes on a no-regression check. - -On the scan side, Parquet reads pick up several improvements: full Parquet metadata (including the page index) -is now cached via DataFusion's `CachedParquetFileReaderFactory` -([#4707](https://github.com/apache/datafusion-comet/pull/4707)), identity casts are unwrapped in the schema -adapter so Parquet statistics pruning can engage -([#4730](https://github.com/apache/datafusion-comet/pull/4730)), filter pushdown configuration has been -revised ([#4722](https://github.com/apache/datafusion-comet/pull/4722)), the native scan now passes a metadata -size hint so a single read usually captures the footer, matching the Iceberg path -([#4717](https://github.com/apache/datafusion-comet/pull/4717)), and the native Parquet scan seeds its reader -options from the session config so Parquet settings you already set take effect -([#5107](https://github.com/apache/datafusion-comet/pull/5107)). - -Query planning and plan serialization also got cheaper. `QueryContext` SQL text is now interned into a per-plan -pool ([#5204](https://github.com/apache/datafusion-comet/pull/5204)), which makes serialized plans up to 20x -smaller on TPC-DS — every plan crosses the JNI boundary, so this shrinks per-query overhead across the board. -Plan-data injection is now an O(1) lookup by operator kind -([#4535](https://github.com/apache/datafusion-comet/pull/4535)) and no longer rebuilds operators it does not -touch ([#5220](https://github.com/apache/datafusion-comet/pull/5220)). Comet also makes far fewer timer calls -in the native execution loop ([#4739](https://github.com/apache/datafusion-comet/pull/4739)), and nested array -equality now uses Arrow's comparator ([#5176](https://github.com/apache/datafusion-comet/pull/5176)). +run per-row or per-batch, so improvements here compound across every query that uses them. Expressions with +optimized implementations in this release include: + +- **Casts** between numeric, string, decimal, and date types, including a faster float-to-decimal cast, an + optimized integer-to-integer cast, shared no-overflow fast paths in `CheckOverflow` and + `DecimalRescaleCheckOverflow`, and a `cast_binary_to_string` that is up to 27x faster on binary-format + styles. +- **JSON, regex, and URL parsing**: `get_json_object`, `regexp_extract`, and `parse_url`. +- **Date/time and decimal kernels**: `date_trunc`, `spark_ceil`, and a vectorized `spark_unscaled_value`. +- **String and array kernels**: `lpad`, `unhex`, `size`, `arrays_overlap`, `escape_string`, and the `try_*` + arithmetic kernel. + +To make this kind of work repeatable, the release also adds a scalar expression optimization guide +documenting how to benchmark a kernel, keep its output bit-identical to Spark, and gate changes on a +no-regression check. + +Parquet reads pick up several improvements as well. Full Parquet metadata, including the page index, is now +cached via DataFusion's `CachedParquetFileReaderFactory`; identity casts are unwrapped in the schema adapter +so Parquet statistics pruning can engage; filter pushdown configuration has been revised; the native scan +passes a metadata size hint so a single read usually captures the footer; and the native Parquet scan seeds +its reader options from the session config so Parquet settings you already set take effect. + +Query planning and plan serialization also got cheaper. `QueryContext` SQL text is now interned into a +per-plan pool, which makes serialized plans up to 20x smaller on TPC-DS — every plan crosses the JNI +boundary, so this shrinks per-query overhead across the board. Plan-data injection is now an O(1) lookup by +operator kind and no longer rebuilds operators it does not touch. Comet also makes far fewer timer calls in +the native execution loop. ## Correctness -A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most of -them found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's expression implementations with all supported versions of Spark. - -- **Casts**: casting a string to `boolean`, an integral type, `float`/`double`, or `decimal` now uses Spark's - exact whitespace-trimming rules ([#5150](https://github.com/apache/datafusion-comet/pull/5150)). Comet's - kernels had used four different trim sets, three of them wrong, so results diverged in both directions — - returning null where Spark parses a value, and returning a value where Spark returns null. Casting - `float`/`double` to `decimal` now rounds the shortest decimal string form as Spark does, rather than the - binary value ([#5136](https://github.com/apache/datafusion-comet/pull/5136)), which matters for values such - as `0.5153125` whose binary form sits just below the rounding tie the string form lands on; `NaN` and infinity - now return null even in ANSI mode, matching Spark. Decimal promotion also uses the per-expression eval mode - ([#5171](https://github.com/apache/datafusion-comet/pull/5171)). -- **ANSI error semantics**: the codegen dispatcher's null short-circuit no longer swallows errors Spark raises - ([#5219](https://github.com/apache/datafusion-comet/pull/5219)) — Spark evaluates null-intolerant expressions - per node and left to right, so short-circuiting on the union of input ordinals skipped subtrees Spark would - have evaluated, losing their errors. Roughly 70 built-in expressions route through this dispatcher and ANSI is - on by default in Spark 4, so this affected a wide surface. Also: `round` on a `Long` with a large negative - scale now overflows instead of silently returning zero - ([#5082](https://github.com/apache/datafusion-comet/pull/5082)), `Long.MinValue / -1` raises - `ARITHMETIC_OVERFLOW` ([#5084](https://github.com/apache/datafusion-comet/pull/5084)), floating-point - `x % 0.0` raises `REMAINDER_BY_ZERO` instead of returning `NaN` - ([#5081](https://github.com/apache/datafusion-comet/pull/5081)), `make_decimal` honors its - fail-on-error flag and throws `NUMERIC_VALUE_OUT_OF_RANGE` rather than returning null - ([#5080](https://github.com/apache/datafusion-comet/pull/5080)), invalid calendar dates raise - `CAST_INVALID_INPUT` ([#5014](https://github.com/apache/datafusion-comet/pull/5014)), and errors from nested - casts now propagate ([#4675](https://github.com/apache/datafusion-comet/pull/4675)). -- **Wrong results**: `count` no longer returns zero when the native scan is disabled - ([#4795](https://github.com/apache/datafusion-comet/pull/4795)), Spark's legacy `null IN ()` behavior is - honored ([#5127](https://github.com/apache/datafusion-comet/pull/5127)), `pow` matches Java's `Math.pow` on - the edge cases where C and Java disagree ([#5033](https://github.com/apache/datafusion-comet/pull/5033)), - `percentile` matches Spark's interpolation precision - ([#4792](https://github.com/apache/datafusion-comet/pull/4792)), `date_trunc` handles DST boundaries in - non-UTC sessions and no longer produces schema mismatches through shuffle and sort - ([#4761](https://github.com/apache/datafusion-comet/pull/4761)), `flatten` handles null sub-arrays - ([#4822](https://github.com/apache/datafusion-comet/pull/4822)), `ArrayInsert` handles null source arrays - ([#4726](https://github.com/apache/datafusion-comet/pull/4726)), and the `array_filter` / `array_compact` - fast path is restricted to the lambda variable - ([#4848](https://github.com/apache/datafusion-comet/pull/4848)). -- **Strings and encodings**: `CAST(binary AS string)` now decodes exactly as the JVM's - `new String(bytes, UTF_8)` does, including the surrogate cases where Rust's lossy decoder diverges, instead - of reinterpreting bytes unchecked ([#4763](https://github.com/apache/datafusion-comet/pull/4763)), and - shuffle tolerates non-UTF-8 bytes rather than failing - ([#4524](https://github.com/apache/datafusion-comet/pull/4524)). -- **Timestamps**: reading a Parquet timestamp-with-timezone column as `TimestampNTZ` is now rejected on Spark - 3.x, matching Spark's own refusal, instead of silently returning the UTC instant - ([#4357](https://github.com/apache/datafusion-comet/pull/4357)). -- **Deep expression trees**: long chains of associative bitwise, `Add`, and `Multiply` operators are rebalanced - so plans no longer hit protobuf's recursion limit - ([#4588](https://github.com/apache/datafusion-comet/pull/4588)). -- **Native shuffle**: scalar subqueries used in partitioning expressions are now registered, fixing - "Subquery N not found" failures ([#4869](https://github.com/apache/datafusion-comet/pull/4869)), and constant - column vectors are materialized on Comet's serialize and export paths - ([#4532](https://github.com/apache/datafusion-comet/pull/4532)). -- **Collations**: Comet falls back for predicates whose operands use non-default collations - ([#4948](https://github.com/apache/datafusion-comet/pull/4948)), for Spark 4 datetime expressions under - non-default collations ([#4693](https://github.com/apache/datafusion-comet/pull/4693)), and for - `str_to_map` ([#4701](https://github.com/apache/datafusion-comet/pull/4701)). +A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most +of them found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's expression +implementations with all supported versions of Spark. Highlights: + +- **Casts** to `boolean`, integral types, `float`/`double`, and `decimal` now use Spark's exact + whitespace-trimming rules. Comet's kernels had used four different trim sets, three of them wrong, so + results diverged in both directions — returning null where Spark parses a value, and returning a value + where Spark returns null. Casts from `float`/`double` to `decimal` now round the shortest decimal string + form as Spark does, and return null for `NaN` and infinity even in ANSI mode. +- **ANSI error semantics.** The codegen dispatcher's null short-circuit no longer swallows errors Spark + raises. Spark evaluates null-intolerant expressions per node and left to right, so short-circuiting on the + union of input ordinals skipped subtrees Spark would have evaluated, losing their errors. Roughly 70 + built-in expressions route through this dispatcher and ANSI is on by default in Spark 4, so this affected a + wide surface. Other ANSI fixes cover overflow on `round` with a large negative scale, `Long.MinValue / -1`, + floating-point remainder by zero, `make_decimal` fail-on-error, invalid calendar dates, and errors from + nested casts. +- **Wrong results** in `count` when the native scan is disabled, Spark's legacy `null IN ()` behavior, `pow` + edge cases where C and Java disagree, `percentile` interpolation precision, `date_trunc` across DST + boundaries, `flatten` and `ArrayInsert` with null inputs, and the `array_filter` / `array_compact` fast + path. +- **Strings and encodings.** `CAST(binary AS string)` now decodes exactly as the JVM's + `new String(bytes, UTF_8)` does, including the surrogate cases where Rust's lossy decoder diverges, and + shuffle tolerates non-UTF-8 bytes rather than failing. +- **Deep expression trees.** Long chains of associative bitwise, `Add`, and `Multiply` operators are + rebalanced so plans no longer hit protobuf's recursion limit. +- **Collations.** Comet now falls back for predicates whose operands use non-default collations, for Spark 4 + datetime expressions under non-default collations, and for `str_to_map`. - **Conservative fallbacks** where native behavior could not be made to match: decimal `SUM` / `AVG` over - sliding window frames ([#4732](https://github.com/apache/datafusion-comet/pull/4732)), - `FromUnixTime` with a non-default format ([#4847](https://github.com/apache/datafusion-comet/pull/4847)), - `CreateArray` with struct-nullability-divergent children - ([#4533](https://github.com/apache/datafusion-comet/pull/4533)), and native V1 scans on filesystem schemes - that `object_store` does not support ([#4525](https://github.com/apache/datafusion-comet/pull/4525)). -- **Error reporting**: native Parquet read failures now surface as Spark's `FAILED_READ_FILE` - ([#4536](https://github.com/apache/datafusion-comet/pull/4536)), and a DataFusion 54.1.0 Parquet page-index - regression is worked around ([#5132](https://github.com/apache/datafusion-comet/pull/5132)). + sliding window frames, `FromUnixTime` with a non-default format, `CreateArray` with + struct-nullability-divergent children, and native V1 scans on filesystem schemes that `object_store` does + not support. +- **Error reporting.** Native Parquet read failures now surface as Spark's `FAILED_READ_FILE`, and a + DataFusion 54.1.0 Parquet page-index regression is worked around. ## New Expression and Aggregate Support This release expands the set of Spark expressions and aggregates that run natively: -- **Aggregates**: `approx_percentile` / `percentile_approx` - ([#4801](https://github.com/apache/datafusion-comet/pull/4801)), exact `percentile` / `median` - ([#4542](https://github.com/apache/datafusion-comet/pull/4542)), - `approx_count_distinct` ([#4819](https://github.com/apache/datafusion-comet/pull/4819)), and native - `collect_list` / `array_agg` ([#4720](https://github.com/apache/datafusion-comet/pull/4720)). -- **Grouping**: `grouping()` and `grouping_id()` indicator functions - ([#4815](https://github.com/apache/datafusion-comet/pull/4815)). -- **Intervals**: interval types with `make_ym_interval` and `make_dt_interval` - ([#4541](https://github.com/apache/datafusion-comet/pull/4541)), - `CalendarIntervalType` support ([#4898](https://github.com/apache/datafusion-comet/pull/4898)), - `multiply_dt_interval` via codegen dispatch - ([#4900](https://github.com/apache/datafusion-comet/pull/4900)), and interval codegen dispatch for nested - values and native shuffle ([#4976](https://github.com/apache/datafusion-comet/pull/4976)). -- **String**: `base64` ([#4778](https://github.com/apache/datafusion-comet/pull/4778)), - `split_part` via `StringSplitSQL` ([#4592](https://github.com/apache/datafusion-comet/pull/4592)), - native `levenshtein` ([#4105](https://github.com/apache/datafusion-comet/pull/4105)), and native - `randstr` ([#5035](https://github.com/apache/datafusion-comet/pull/5035)) and `uuid` - ([#5034](https://github.com/apache/datafusion-comet/pull/5034)), both bit-for-bit compatible with Spark for a - given seed. -- **Array / map**: `array_prepend` ([#4716](https://github.com/apache/datafusion-comet/pull/4716)), - the `shuffle()` array function ([#4797](https://github.com/apache/datafusion-comet/pull/4797)), - `size()` for `MapType` ([#4580](https://github.com/apache/datafusion-comet/pull/4580)), `ElementAt` over - `MapType` ([#4697](https://github.com/apache/datafusion-comet/pull/4697)), and removal of the constraint on - arrays of nested elements ([#4714](https://github.com/apache/datafusion-comet/pull/4714)). -- **Date/time**: native `TimestampNTZ` inputs for `hour` / `minute` / `second` - ([#4753](https://github.com/apache/datafusion-comet/pull/4753)) and - `PreciseTimestampConversion` for native time-window grouping - ([#4784](https://github.com/apache/datafusion-comet/pull/4784)). -- **Windows**: extended native window function support - ([#4209](https://github.com/apache/datafusion-comet/pull/4209)) and Spark 4 decimal window average - ([#4749](https://github.com/apache/datafusion-comet/pull/4749)). -- **Internal expressions**: native `empty2null` - ([#4683](https://github.com/apache/datafusion-comet/pull/4683)), which appears in write paths, and the - shuffle-side infrastructure for Spark's `TimeType` - ([#4398](https://github.com/apache/datafusion-comet/pull/4398)). +- **Aggregates**: `approx_percentile` / `percentile_approx`, exact `percentile` / `median`, + `approx_count_distinct`, and native `collect_list` / `array_agg`. +- **Grouping**: `grouping()` and `grouping_id()`. +- **Intervals**: interval types via `make_ym_interval` and `make_dt_interval`, `CalendarIntervalType`, + `multiply_dt_interval`, and interval codegen dispatch for nested values and native shuffle. +- **String**: `base64`, `split_part` via `StringSplitSQL`, native `levenshtein`, and native `randstr` and + `uuid` — both bit-for-bit compatible with Spark for a given seed. +- **Array / map**: `array_prepend`, the `shuffle()` array function, `size()` for `MapType`, and `ElementAt` + over `MapType`. +- **Date/time**: native `TimestampNTZ` inputs for `hour` / `minute` / `second` and + `PreciseTimestampConversion` for native time-window grouping. +- **Windows**: extended native window function support and Spark 4 decimal window average. ## Native Sampling -Spark's `SampleExec` now runs natively when sampling without replacement -([#5110](https://github.com/apache/datafusion-comet/pull/5110)), covering `DataFrame.sample`, SQL +Spark's `SampleExec` now runs natively when sampling without replacement, covering `DataFrame.sample`, SQL `TABLESAMPLE`, and `DataFrame.randomSplit`. The native operator ports Spark's `BernoulliCellSampler` on top of `XorShiftRandom` and seeds per partition exactly as Spark does, so it selects the same rows for a given seed — sampling stays reproducible whether or not Comet is enabled. Sampling with replacement still falls back to @@ -247,97 +163,60 @@ Spark. ## Experimental PyArrow UDF Support -This release adds experimental support for accelerated PyArrow UDFs -([#4234](https://github.com/apache/datafusion-comet/pull/4234)), allowing PyArrow-based user-defined functions -to participate in native execution instead of forcing a fallback to Spark. When the feature is disabled, Comet -now hints at the native PyArrow UDF path in its fallback reasons -([#4892](https://github.com/apache/datafusion-comet/pull/4892)) so users know the option exists. This is an -early-stage feature and we welcome feedback from users experimenting with it. +This release adds experimental support for accelerated PyArrow UDFs, allowing PyArrow-based user-defined +functions to participate in native execution instead of forcing a fallback to Spark. When the feature is +disabled, Comet now hints at the native PyArrow UDF path in its fallback reasons so users know the option +exists. This is an early-stage feature and we welcome feedback from users experimenting with it. ## Iceberg -- **Iceberg 1.11 support** ([#4840](https://github.com/apache/datafusion-comet/pull/4840)): adds support for - Iceberg 1.11, audits the existing Iceberg diffs, bumps the iceberg-rust dependency, and adds a - `run-iceberg-tests` CI trigger. -- **Iceberg table format V3** ([#4991](https://github.com/apache/datafusion-comet/pull/4991)): native table - decryption for V3 tables, with fallback to Spark for other V3 features. Follow-up - [#5020](https://github.com/apache/datafusion-comet/pull/5020) applies the same diff changes across other - Iceberg versions. -- **Metadata columns** ([#4752](https://github.com/apache/datafusion-comet/pull/4752)): the native Iceberg scan - now supports the `_pos`, `_spec`, `_file`, and `_partition` metadata columns. -- **Delete-file correctness** ([#4760](https://github.com/apache/datafusion-comet/pull/4760)): the native scan - now sizes Iceberg delete files correctly, avoiding dropped deletes. -- **Exchange-reuse correctness** ([#4812](https://github.com/apache/datafusion-comet/pull/4812)): fixed a case - where Iceberg native scan exchange reuse with different pushed filters could produce wrong results. -- **Scan disambiguation** ([#5180](https://github.com/apache/datafusion-comet/pull/5180)): Iceberg scans that - share a `metadata_location` are now told apart, rather than being conflated. -- **Native serde dedup** ([#4982](https://github.com/apache/datafusion-comet/pull/4982)): dedupes Iceberg - residuals and delete files in the native scan serde, reducing planning overhead. +Comet now supports Iceberg 1.11 and Iceberg table format V3 (with native table decryption for V3 tables and +fallback to Spark for other V3 features). The native Iceberg scan supports the `_pos`, `_spec`, `_file`, and +`_partition` metadata columns, sizes delete files correctly to avoid dropped deletes, disambiguates scans that +share a `metadata_location`, and dedupes residuals and delete files in the native scan serde. A prior case +where Iceberg native scan exchange reuse with different pushed filters could produce wrong results is also +fixed. ## Native Parquet I/O and Cloud -- **gzip Parquet writes** ([#4930](https://github.com/apache/datafusion-comet/pull/4930)): native Parquet - writes now support gzip compression. -- **Azure authentication** ([#4783](https://github.com/apache/datafusion-comet/pull/4783)): the native - Parquet scan now supports Azure authentication, complementing the existing S3 credential support. +Native Parquet writes now support gzip compression, and the native Parquet scan supports Azure authentication, +complementing the existing S3 credential support. ## Execution and Fallback Improvements -- **Mixed partial/final aggregation** ([#4861](https://github.com/apache/datafusion-comet/pull/4861)): - `sum` and non-decimal `avg` can now run with the partial and final aggregation stages split across Spark and - Comet, keeping more aggregation plans partially native. -- **Stage-based fallback** ([#4519](https://github.com/apache/datafusion-comet/pull/4519)): a more granular - fallback mechanism that operates at the stage level. -- **Codegen dispatch for opt-in serdes** ([#4728](https://github.com/apache/datafusion-comet/pull/4728)): - expressions reported as `Unsupported` can now route through JVM codegen dispatch for opt-in serdes, and - native opt-in expressions surface as compatible-by-default with a `COMET-INFO` plan hint - ([#4721](https://github.com/apache/datafusion-comet/pull/4721)). Casts join this path - ([#5079](https://github.com/apache/datafusion-comet/pull/5079)): an incompatible or unsupported cast now runs - Spark's own generated code inside the Comet pipeline instead of pulling the whole operator back to Spark, so - one awkward cast no longer de-accelerates a query. `sort_array` under strict floating-point mode - ([#4637](https://github.com/apache/datafusion-comet/pull/4637)) and `concat` under non-`UTF8_BINARY` - collations ([#4640](https://github.com/apache/datafusion-comet/pull/4640)) opt in the same way. -- **JVM columnar-to-row by default** ([#5114](https://github.com/apache/datafusion-comet/pull/5114)): isolated - benchmarking showed the native columnar-to-row converter is roughly 3.7x slower per row than the JVM - implementation at the default batch size — and up to 15.7x slower for small batches — because of a fixed - per-batch JNI and FFI cost, with no end-to-end benefit in TPC-DS runs. It is now disabled by default. -- **Clearer fallback reasons**: unsupported scalar serde cases are reported in the support level - ([#4745](https://github.com/apache/datafusion-comet/pull/4745)), mixed-execution fallback messages name the - incompatible aggregate functions ([#4750](https://github.com/apache/datafusion-comet/pull/4750)), unsupported - metadata column names appear in fallback reasons - ([#4758](https://github.com/apache/datafusion-comet/pull/4758)), the compatibility guide URLs in fallback - messages are corrected ([#4854](https://github.com/apache/datafusion-comet/pull/4854)), and the spurious - "WriteFilesExec is not supported" message is suppressed - ([#4928](https://github.com/apache/datafusion-comet/pull/4928)). -- **Expression coverage in extended explain** ([#5201](https://github.com/apache/datafusion-comet/pull/5201)): - the extended explain summary previously reported operator coverage but said nothing about expressions. It now - ends with a line such as `Comet accelerated 14 expressions (14 native, 1 codegen dispatch)`, so you can see - how much of a plan's expression evaluation runs in native DataFusion kernels versus Spark's generated code - inside the dispatcher. Operator counts also handle `ReusedSubquery` and `CometSubqueryBroadcast` correctly - ([#5206](https://github.com/apache/datafusion-comet/pull/5206)). -- **Local scan nullability** ([#4843](https://github.com/apache/datafusion-comet/pull/4843)): local table scan - child nullability is now widened to match the native kernels, fixing a class of nullability mismatches. -- **Config aliases** ([#4979](https://github.com/apache/datafusion-comet/pull/4979)): a `withAlternative` - alias mechanism lets `CometConf` entries carry old names during renames without breaking existing - configurations. -- **Runtime cleanup** ([#4734](https://github.com/apache/datafusion-comet/pull/4734)): the Tokio runtime is - now released on driver and executor exit. -- **Quieter logs** ([#5155](https://github.com/apache/datafusion-comet/pull/5155)): logging levels on frequent - call sites have been lowered so Comet no longer floods executor logs at default settings. -- **Contrib scan SPI** ([#4700](https://github.com/apache/datafusion-comet/pull/4700)): a core SPI for - contrib leaf scans (`CometScanWithPlanData`), the first part of splitting the Delta integration into a - contrib module. +- **Mixed partial/final aggregation.** `sum` and non-decimal `avg` can now run with the partial and final + aggregation stages split across Spark and Comet, keeping more aggregation plans partially native. +- **Codegen dispatch for opt-in serdes.** Expressions reported as `Unsupported` can now route through JVM + codegen dispatch for opt-in serdes, and native opt-in expressions surface as compatible-by-default with a + `COMET-INFO` plan hint. Casts join this path: an incompatible or unsupported cast now runs Spark's own + generated code inside the Comet pipeline instead of pulling the whole operator back to Spark, so one + awkward cast no longer de-accelerates a query. `sort_array` under strict floating-point mode and `concat` + under non-`UTF8_BINARY` collations opt in the same way. +- **JVM columnar-to-row by default.** Isolated benchmarking showed the native columnar-to-row converter is + roughly 3.7x slower per row than the JVM implementation at the default batch size — and up to 15.7x slower + for small batches — because of a fixed per-batch JNI and FFI cost, with no end-to-end benefit in TPC-DS + runs. It is now disabled by default. +- **Clearer fallback reasons.** Unsupported scalar serde cases are reported in the support level, + mixed-execution fallback messages name the incompatible aggregate functions, unsupported metadata column + names appear in fallback reasons, and the spurious "WriteFilesExec is not supported" message is suppressed. +- **Expression coverage in extended explain.** The extended explain summary previously reported operator + coverage but said nothing about expressions. It now ends with a line such as `Comet accelerated 14 + expressions (14 native, 1 codegen dispatch)`, so you can see how much of a plan's expression evaluation + runs in native DataFusion kernels versus Spark's generated code inside the dispatcher. +- **Quieter logs and runtime cleanup.** Logging levels on frequent call sites have been lowered so Comet no + longer floods executor logs at default settings, and the Tokio runtime is now released on driver and + executor exit. ## Shuffle Improvements -- **Native shuffle memory cap** ([#4989](https://github.com/apache/datafusion-comet/pull/4989)): a new - `spark.comet.shuffle.maxBufferBytes` config caps native shuffle writer memory to bound worst-case usage. -- **Shuffle IPC schema encoding** ([#5006](https://github.com/apache/datafusion-comet/pull/5006)): the IPC - schema is now encoded once per writer instead of per block, cutting per-batch shuffle overhead. -- **BatchCoalescer bypass** ([#5003](https://github.com/apache/datafusion-comet/pull/5003)): shuffle bypasses - the `BatchCoalescer` for batches that are already appropriately sized. -- **Single-partition shuffle** ([#5004](https://github.com/apache/datafusion-comet/pull/5004)): a redundant - concatenation layer has been removed from the single-partition path. +- **Native shuffle memory cap.** A new `spark.comet.shuffle.maxBufferBytes` config caps native shuffle writer + memory to bound worst-case usage. +- **Shuffle IPC schema encoding.** The IPC schema is now encoded once per writer instead of per block, + cutting per-batch shuffle overhead. +- **BatchCoalescer bypass.** Shuffle bypasses the `BatchCoalescer` for batches that are already appropriately + sized. +- **Single-partition shuffle.** A redundant concatenation layer has been removed from the single-partition + path. ## Configuration Changes @@ -345,47 +224,33 @@ Because configuration is Comet's primary API surface, 1.0 is the release where t Every rename below registers the old key as an alias, so existing configurations keep working and log a deprecation warning when an old key is read: -- **Unified shuffle prefix** ([#4986](https://github.com/apache/datafusion-comet/pull/4986)): shuffle configs - were spread across four disjoint prefixes (`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`, - `spark.comet.native.shuffle.*`, `spark.comet.shuffle.*`). They now all live under `spark.comet.shuffle.*`, - with `.jvm.` and `.native.` sub-namespaces matching the `spark.comet.shuffle.mode` value you already set. -- **Grouped explain configs** ([#5026](https://github.com/apache/datafusion-comet/pull/5026)): the orphan - explain-related configs are collected under a single `spark.comet.explain.*` prefix, and the PyArrow UDF - config is renamed to `pyarrowUDF` for consistency - ([#5197](https://github.com/apache/datafusion-comet/pull/5197)). -- **`spark.comet.version`** ([#5049](https://github.com/apache/datafusion-comet/pull/5049)): the loaded Comet - build version is now exposed as a runtime config, so you can confirm which Comet a cluster is actually - running with `spark.conf.get` or `SET`. -- **Removed dead configs**: the Parquet parallel-IO knobs - ([#4981](https://github.com/apache/datafusion-comet/pull/4981)), - `spark.comet.use.lazyMaterialization` ([#4998](https://github.com/apache/datafusion-comet/pull/4998)), and - `spark.comet.exceptionOnDatetimeRebase` ([#5221](https://github.com/apache/datafusion-comet/pull/5221)) had no - remaining effect and were misleading anyone tuning against them. -- **Removed async columnar shuffle** ([#4985](https://github.com/apache/datafusion-comet/pull/4985)): the - `spark.comet.columnar.shuffle.async.*` path was off by default and untested, and has been removed rather than - carried into a stable release line. +- **Unified shuffle prefix.** Shuffle configs were spread across four disjoint prefixes + (`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`, `spark.comet.native.shuffle.*`, + `spark.comet.shuffle.*`). They now all live under `spark.comet.shuffle.*`, with `.jvm.` and `.native.` + sub-namespaces matching the `spark.comet.shuffle.mode` value you already set. +- **Grouped explain configs.** The orphan explain-related configs are collected under a single + `spark.comet.explain.*` prefix, and the PyArrow UDF config is renamed to `pyarrowUDF` for consistency. +- **`spark.comet.version`.** The loaded Comet build version is now exposed as a runtime config, so you can + confirm which Comet a cluster is actually running with `spark.conf.get` or `SET`. +- **Removed dead configs.** The Parquet parallel-IO knobs, `spark.comet.use.lazyMaterialization`, and + `spark.comet.exceptionOnDatetimeRebase` had no remaining effect and were misleading anyone tuning against + them. +- **Removed async columnar shuffle.** The `spark.comet.columnar.shuffle.async.*` path was off by default and + untested, and has been removed rather than carried into a stable release line. ## Documentation -The documentation received a substantial overhaul for 1.0: - -- A [design refresh](https://github.com/apache/datafusion-comet/pull/4353) of the docs site, with versioned - user guides gaining captioned sidebar sections - ([#4699](https://github.com/apache/datafusion-comet/pull/4699)) and a set of accessibility and navigation - fixes ([#4858](https://github.com/apache/datafusion-comet/pull/4858)). -- The supported-expressions page now shows how each expression is implemented — native, codegen dispatch, or - hybrid ([#5028](https://github.com/apache/datafusion-comet/pull/5028)), generated from the planner itself - rather than maintained by hand. -- An expanded [tuning guide](https://github.com/apache/datafusion-comet/pull/4908) covering the performance and - memory configs that matter in practice, plus guidance on `spark.sql.files.maxPartitionBytes` - ([#4931](https://github.com/apache/datafusion-comet/pull/4931)). -- An [updated roadmap](https://github.com/apache/datafusion-comet/pull/5064) for the post-1.0 work, and a new - [blog posts and talks page](https://github.com/apache/datafusion-comet/pull/5043). +The documentation received a substantial overhaul for 1.0: a design refresh of the docs site, versioned user +guides with captioned sidebar sections, and a set of accessibility and navigation fixes. The +supported-expressions page now shows how each expression is implemented — native, codegen dispatch, or hybrid +— generated from the planner itself rather than maintained by hand. An expanded tuning guide covers the +performance and memory configs that matter in practice, and there is an updated post-1.0 roadmap and a new +blog posts and talks page. ## Deprecation Notice -With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic versioning -([#4857](https://github.com/apache/datafusion-comet/pull/4857)): +With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic +versioning: - **JDK 11** is deprecated and scheduled for removal in Comet 1.1.0. - **Apache Spark 3.4** is deprecated and scheduled for removal in Comet 1.1.0. From 80b2dba03345cbf3454554a1853e5e7392dd10a2 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 10:26:37 -0600 Subject: [PATCH 08/19] Condense Correctness section to stats plus illustrative examples Replace the full enumeration of ~30 correctness fixes with a stat, two representative examples (whitespace trimming, ANSI short-circuit), and a short note on conservative fallbacks. Full list stays in the change log. --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 53 +++++++------------ 1 file changed, 20 insertions(+), 33 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 2c13b2fe..6cae57b1 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -102,39 +102,26 @@ the native execution loop. ## Correctness -A 1.0 release is only as good as its results. This release fixes a broad set of divergences from Spark, most -of them found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's expression -implementations with all supported versions of Spark. Highlights: - -- **Casts** to `boolean`, integral types, `float`/`double`, and `decimal` now use Spark's exact - whitespace-trimming rules. Comet's kernels had used four different trim sets, three of them wrong, so - results diverged in both directions — returning null where Spark parses a value, and returning a value - where Spark returns null. Casts from `float`/`double` to `decimal` now round the shortest decimal string - form as Spark does, and return null for `NaN` and infinity even in ANSI mode. -- **ANSI error semantics.** The codegen dispatcher's null short-circuit no longer swallows errors Spark - raises. Spark evaluates null-intolerant expressions per node and left to right, so short-circuiting on the - union of input ordinals skipped subtrees Spark would have evaluated, losing their errors. Roughly 70 - built-in expressions route through this dispatcher and ANSI is on by default in Spark 4, so this affected a - wide surface. Other ANSI fixes cover overflow on `round` with a large negative scale, `Long.MinValue / -1`, - floating-point remainder by zero, `make_decimal` fail-on-error, invalid calendar dates, and errors from - nested casts. -- **Wrong results** in `count` when the native scan is disabled, Spark's legacy `null IN ()` behavior, `pow` - edge cases where C and Java disagree, `percentile` interpolation precision, `date_trunc` across DST - boundaries, `flatten` and `ArrayInsert` with null inputs, and the `array_filter` / `array_compact` fast - path. -- **Strings and encodings.** `CAST(binary AS string)` now decodes exactly as the JVM's - `new String(bytes, UTF_8)` does, including the surrogate cases where Rust's lossy decoder diverges, and - shuffle tolerates non-UTF-8 bytes rather than failing. -- **Deep expression trees.** Long chains of associative bitwise, `Add`, and `Multiply` operators are - rebalanced so plans no longer hit protobuf's recursion limit. -- **Collations.** Comet now falls back for predicates whose operands use non-default collations, for Spark 4 - datetime expressions under non-default collations, and for `str_to_map`. -- **Conservative fallbacks** where native behavior could not be made to match: decimal `SUM` / `AVG` over - sliding window frames, `FromUnixTime` with a non-default format, `CreateArray` with - struct-nullability-divergent children, and native V1 scans on filesystem schemes that `object_store` does - not support. -- **Error reporting.** Native Parquet read failures now surface as Spark's `FAILED_READ_FILE`, and a - DataFusion 54.1.0 Parquet page-index regression is worked around. +A 1.0 release is only as good as its results. This release fixes roughly 30 divergences from Spark, spanning +casts, ANSI error semantics, wrong-result bugs, string and encoding handling, collations, and error +reporting. Most were found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's +expression implementations with all supported versions of Spark. See the [change log] for the full list; two +representative fixes give a sense of the shape of the work: + +- **Whitespace trimming in string-to-numeric casts.** Casts to `boolean`, integral types, `float`/`double`, + and `decimal` now use Spark's exact trim rules. Comet's kernels had used four different trim sets, three of + them wrong, so results diverged in both directions — returning null where Spark parses a value, and + returning a value where Spark returns null. +- **ANSI errors swallowed by null short-circuit.** The codegen dispatcher's null short-circuit no longer + swallows errors Spark raises. Spark evaluates null-intolerant expressions per node and left to right, so + short-circuiting on the union of input ordinals skipped subtrees Spark would have evaluated, losing their + errors. Roughly 70 built-in expressions route through this dispatcher and ANSI is on by default in Spark 4, + so this affected a wide surface. + +Alongside the bug fixes, Comet also adds conservative fallbacks where native behavior could not be made to +match — decimal `SUM` / `AVG` over sliding window frames, `FromUnixTime` with a non-default format, and a +handful of others — and falls back for predicates and datetime expressions under non-default collations +rather than risking a wrong result. ## New Expression and Aggregate Support From e672af6dcce2f23fa7c8eb93933ce787796d3ef6 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 10:30:14 -0600 Subject: [PATCH 09/19] Reframe Correctness section opener Replace the generic "A 1.0 release is only as good as its results" opener with concrete framing: correctness fixes have been part of every release, and 1.0 was an extra push to clear the bulk of the known issues. --- content/blog/2026-08-07-datafusion-comet-1.0.0.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 6cae57b1..e7982a06 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -102,11 +102,12 @@ the native execution loop. ## Correctness -A 1.0 release is only as good as its results. This release fixes roughly 30 divergences from Spark, spanning -casts, ANSI error semantics, wrong-result bugs, string and encoding handling, collations, and error -reporting. Most were found by running extensive AI-assisted audit sweeps of the code base, comparing Comet's -expression implementations with all supported versions of Spark. See the [change log] for the full list; two -representative fixes give a sense of the shape of the work: +Correctness fixes have been a regular part of every Comet release, but for 1.0 we made an extra push to +clear out the bulk of the known issues before drawing a line under the stable release. This release fixes +roughly 30 divergences from Spark, spanning casts, ANSI error semantics, wrong-result bugs, string and +encoding handling, collations, and error reporting. Most were found by running extensive AI-assisted audit +sweeps of the code base, comparing Comet's expression implementations with all supported versions of Spark. +See the [change log] for the full list; two representative fixes give a sense of the shape of the work: - **Whitespace trimming in string-to-numeric casts.** Casts to `boolean`, integral types, `float`/`double`, and `decimal` now use Spark's exact trim rules. Comet's kernels had used four different trim sets, three of From d936d18ec68259d14866dacbb4de1324918d8cb9 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 10:37:26 -0600 Subject: [PATCH 10/19] Add Road to 1.0 section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sketch the arc of the project across five threads — query coverage, Spark support, ecosystem, correctness rigor, and performance — grounded in concrete facts from earlier releases (106 → 404 expressions, JVM codegen dispatcher in 0.17.0, Iceberg in 0.10.0, 9% TPC-DS gain in 0.17.0). Trim the "What 1.0 Means" intro since the arc is now covered above. --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 35 ++++++++++++++++--- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index e7982a06..de963428 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -37,14 +37,41 @@ contributors. See the [change log] for the full list of changes. [change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md -## What 1.0 Means +## The Road to 1.0 -The 1.0.0 release is the culmination of more than two years of work since the project was [donated] as an -Apache DataFusion subproject in March 2024, and is less about any single new feature than about the -accumulated maturity of the project: +Comet was [donated] to the Apache DataFusion project in March 2024 and cut its first release, 0.1.0, five +months later with 15 data types, 13 operators, 106 expressions, and a "modest performance speedup." The +nineteen releases between then and now cover a lot of ground: [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ +- **Query coverage went from a handful of operators to the shape of a real Spark query.** Native + SortMergeJoin, HashJoin, and BroadcastHashJoin landed early; native columnar and native shuffle, + broadcast nested loop joins, native window functions, native sampling, and mixed partial/final aggregation + followed. Supported expressions grew from 106 at 0.1.0 to 404 in 1.0, and the introduction of the JVM + codegen dispatcher in 0.17.0 gave Comet a way to keep unsupported expressions Arrow-native by running + Spark's own generated code inside the pipeline rather than falling back to row-based execution. +- **Spark support broadened.** The 0.1.0 release targeted Spark 3.3, 3.4, and 3.5, with experimental 4.0. The + 1.0 line drops 3.3, adds 4.1, and ships an experimental 4.2 profile. ANSI semantics — on by default in + Spark 4 — moved from partial to a supported default across the natively implemented surface. +- **The ecosystem story filled in.** Native Iceberg support arrived in 0.10.0 and has been extended through + 1.11 and format V3, native Parquet writes and CSV reads landed as experimental features and matured, + Azure joined S3 on the native cloud path, and 1.0 adds experimental accelerated PyArrow UDFs alongside the + Java and Scala UDF support that shipped in 0.17.0. +- **Correctness rigor grew alongside the surface area.** Early releases relied on fuzz testing to surface + divergences; later releases added a full Spark SQL test-suite run against every supported Spark version, + followed by AI-assisted expression audits comparing Comet's behavior against every supported Spark version + edge case by edge case. 1.0 ships with every known open correctness gap documented in the compatibility + guide. +- **Performance moved from "modest speedup" to workload-level wins.** TPC-DS at 1TB has become the reference + workload, with successive releases removing FFI round trips, caching Parquet metadata, shrinking plan + serialization, and tightening the shuffle write path. 0.17.0 alone was ~9% faster than 0.16.0 on TPC-DS + 1TB, and the per-expression optimization work in 1.0 continues that trajectory. + +## What 1.0 Means + +Beyond the accumulated features, 1.0 is where the project commits to being something you can build on: + - **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.9, 4.0.4, and 4.1.3 out of the same codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental Spark 4.2 profile for early evaluation. A published Spark version adoption and support-lifetime policy From 3012383d2fcd32595a3477452ee9e51b4c739ab1 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 4 Aug 2026 13:16:00 -0600 Subject: [PATCH 11/19] Fix release count and expressions in blog post Corrected the number of major releases from nineteen to sixteen and updated the supported expressions count to more than 400 in version 1.0.0. --- content/blog/2026-08-07-datafusion-comet-1.0.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index de963428..3c879fc5 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -41,14 +41,14 @@ contributors. See the [change log] for the full list of changes. Comet was [donated] to the Apache DataFusion project in March 2024 and cut its first release, 0.1.0, five months later with 15 data types, 13 operators, 106 expressions, and a "modest performance speedup." The -nineteen releases between then and now cover a lot of ground: +sixteen major releases between then and now cover a lot of ground: [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ - **Query coverage went from a handful of operators to the shape of a real Spark query.** Native SortMergeJoin, HashJoin, and BroadcastHashJoin landed early; native columnar and native shuffle, broadcast nested loop joins, native window functions, native sampling, and mixed partial/final aggregation - followed. Supported expressions grew from 106 at 0.1.0 to 404 in 1.0, and the introduction of the JVM + followed. Supported expressions grew from 106 at 0.1.0 to more than 400 in 1.0.0, and the introduction of the JVM codegen dispatcher in 0.17.0 gave Comet a way to keep unsupported expressions Arrow-native by running Spark's own generated code inside the pipeline rather than falling back to row-based execution. - **Spark support broadened.** The 0.1.0 release targeted Spark 3.3, 3.4, and 3.5, with experimental 4.0. The @@ -56,7 +56,7 @@ nineteen releases between then and now cover a lot of ground: Spark 4 — moved from partial to a supported default across the natively implemented surface. - **The ecosystem story filled in.** Native Iceberg support arrived in 0.10.0 and has been extended through 1.11 and format V3, native Parquet writes and CSV reads landed as experimental features and matured, - Azure joined S3 on the native cloud path, and 1.0 adds experimental accelerated PyArrow UDFs alongside the + Azure joined S3 on the native cloud path, and 1.0.0 adds experimental accelerated PyArrow UDFs alongside the Java and Scala UDF support that shipped in 0.17.0. - **Correctness rigor grew alongside the surface area.** Early releases relied on fuzz testing to surface divergences; later releases added a full Spark SQL test-suite run against every supported Spark version, From 64872f1625472a41cea31869bf0e651fc50913d2 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 10:52:03 -0600 Subject: [PATCH 12/19] start of human rewrite --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 86 ++++++++----------- 1 file changed, 34 insertions(+), 52 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 3c879fc5..67840c87 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -40,61 +40,43 @@ contributors. See the [change log] for the full list of changes. ## The Road to 1.0 Comet was [donated] to the Apache DataFusion project in March 2024 and cut its first release, 0.1.0, five -months later with 15 data types, 13 operators, 106 expressions, and a "modest performance speedup." The -sixteen major releases between then and now cover a lot of ground: +months later with support for 13 operators, and 106 expressions. [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ -- **Query coverage went from a handful of operators to the shape of a real Spark query.** Native - SortMergeJoin, HashJoin, and BroadcastHashJoin landed early; native columnar and native shuffle, - broadcast nested loop joins, native window functions, native sampling, and mixed partial/final aggregation - followed. Supported expressions grew from 106 at 0.1.0 to more than 400 in 1.0.0, and the introduction of the JVM - codegen dispatcher in 0.17.0 gave Comet a way to keep unsupported expressions Arrow-native by running - Spark's own generated code inside the pipeline rather than falling back to row-based execution. -- **Spark support broadened.** The 0.1.0 release targeted Spark 3.3, 3.4, and 3.5, with experimental 4.0. The - 1.0 line drops 3.3, adds 4.1, and ships an experimental 4.2 profile. ANSI semantics — on by default in - Spark 4 — moved from partial to a supported default across the natively implemented surface. -- **The ecosystem story filled in.** Native Iceberg support arrived in 0.10.0 and has been extended through - 1.11 and format V3, native Parquet writes and CSV reads landed as experimental features and matured, - Azure joined S3 on the native cloud path, and 1.0.0 adds experimental accelerated PyArrow UDFs alongside the - Java and Scala UDF support that shipped in 0.17.0. -- **Correctness rigor grew alongside the surface area.** Early releases relied on fuzz testing to surface - divergences; later releases added a full Spark SQL test-suite run against every supported Spark version, - followed by AI-assisted expression audits comparing Comet's behavior against every supported Spark version - edge case by edge case. 1.0 ships with every known open correctness gap documented in the compatibility - guide. -- **Performance moved from "modest speedup" to workload-level wins.** TPC-DS at 1TB has become the reference - workload, with successive releases removing FFI round trips, caching Parquet metadata, shrinking plan - serialization, and tightening the shuffle write path. 0.17.0 alone was ~9% faster than 0.16.0 on TPC-DS - 1TB, and the per-expression optimization work in 1.0 continues that trajectory. - -## What 1.0 Means - -Beyond the accumulated features, 1.0 is where the project commits to being something you can build on: - -- **Broad Spark coverage.** Comet supports Apache Spark 3.4.3, 3.5.9, 4.0.4, and 4.1.3 out of the same - codebase, with dedicated Maven profiles, shim sources, and CI matrices for each, plus an experimental - Spark 4.2 profile for early evaluation. A published Spark version adoption and support-lifetime policy - now states how long each Spark minor stays supported. -- **ANSI SQL by default.** Spark 4 enables ANSI semantics by default. Comet implements ANSI behavior for the - expressions it supports natively, including arithmetic overflow checks, ANSI cast behavior, and `try_*` - variants, so ANSI workloads keep accelerating rather than falling back. -- **A correctness harness, not just unit tests.** Comet runs the full Apache Spark SQL test suite through its - native execution path against every supported Spark version. Running Spark's own correctness tests is what - surfaces semantic shifts before they reach user workloads, and it is the foundation of the confidence behind - a 1.0 release. -- **A stable release line going forward.** With 1.0, the project commits to semantic versioning. Because Comet - is a plugin rather than a library, its configuration is its primary API surface, so `spark.comet.*` keys, an - explicitly enumerated public Java and Scala API, and query results documented as Compatible are all part of - the compatibility surface. Behavior changes in a minor release now require a `spark.comet.legacy.*` key that - restores the previous behavior, and each release records its behavior changes in a user-facing upgrade - guide. Correctness fixes are exempt from being treated as breaking changes. The first deprecations under - that policy are announced in this release (see [Deprecation Notice](#deprecation-notice) below). -- **Documented limitations.** Every open correctness issue is now surfaced in the generated compatibility - guide, down to the expression level, so you can see where Comet is known to diverge from Spark before you - hit it in production rather than after. - -The rest of this post covers what is new since 0.17.0. +Comet 1.0.0 now supports more than 400 expressions, but that isn't the only way the project has grown over this time. + +Here's a recap of some of the main advances over the past two years + +### Correctness Testing + +It is important that queries accelerated by Comet produce the same results as Spark. Correctness checking has always been a large effort in Comet development, but the approach has evolved over time. + +- Comet has always run Spark's own test suite with Comet enabled, providing more than 24,000 unit tests effectively for free. These tests run in Comet's CI for all supported Spark versions. +- Scala tests: Comet has Scala tests that run queries end to end with Comet enabled vs disabled and ensure that the results match +- Fuzz testing: Many of the scala tests use a fuzz testing approach to generate randomized data that queries run against, helping to catch regressions around edge cases such as nulls, NaN, Infinity, and timezone issues +- Comet SQL Tests: In an effort to make it easier to write tests, Comet now provides a SQL-based testing approach that is inspired by sqllogictest +- Generative AI: More recently, Comet has taken advantage of agentic skills to perform audit sweeps of ll expressions, comapring the implementation to Spark's souce code and ensuring that Comet has tests covering all important edge cases + +### Performance + +The early Comet releases provided a very modest speedup and the published benchmark results were based on running TPC workloads at small scale factors on a single node. There are now independent benchmark results published by AWS Labs that show significant speedups for TPC-DS @ 3TB running in EKS. + +### Codegen Dispatch + +Matt TBD + +### Iceberg Support + + +## Improvements since 0.17.1 + +The rest of this post covers what is new since the 0.17.1 release. + + + +# OLD AI CONTENT BELOW - NEEDS TO BE REVIEWED AND/OR REWRITTEN + ## Native Expression Performance From fd43a1c42f6605e7dd46f5694dc550d4803616d6 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:00:31 -0600 Subject: [PATCH 13/19] more rewrite --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 177 +++--------------- 1 file changed, 29 insertions(+), 148 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 67840c87..cb18478b 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -66,79 +66,25 @@ The early Comet releases provided a very modest speedup and the published benchm Matt TBD -### Iceberg Support - - ## Improvements since 0.17.1 The rest of this post covers what is new since the 0.17.1 release. +### Experimental PyArrow UDF Support +This release adds experimental support for accelerated PyArrow UDFs, allowing PyArrow-based user-defined +functions to participate in native execution instead of forcing a fallback to Spark. When the feature is +disabled, Comet now hints at the native PyArrow UDF path in its fallback reasons so users know the option +exists. This is an early-stage feature and we welcome feedback from users experimenting with it. -# OLD AI CONTENT BELOW - NEEDS TO BE REVIEWED AND/OR REWRITTEN - - -## Native Expression Performance - -A large share of this release is dedicated to making Comet's native scalar expressions faster. These kernels -run per-row or per-batch, so improvements here compound across every query that uses them. Expressions with -optimized implementations in this release include: - -- **Casts** between numeric, string, decimal, and date types, including a faster float-to-decimal cast, an - optimized integer-to-integer cast, shared no-overflow fast paths in `CheckOverflow` and - `DecimalRescaleCheckOverflow`, and a `cast_binary_to_string` that is up to 27x faster on binary-format - styles. -- **JSON, regex, and URL parsing**: `get_json_object`, `regexp_extract`, and `parse_url`. -- **Date/time and decimal kernels**: `date_trunc`, `spark_ceil`, and a vectorized `spark_unscaled_value`. -- **String and array kernels**: `lpad`, `unhex`, `size`, `arrays_overlap`, `escape_string`, and the `try_*` - arithmetic kernel. - -To make this kind of work repeatable, the release also adds a scalar expression optimization guide -documenting how to benchmark a kernel, keep its output bit-identical to Spark, and gate changes on a -no-regression check. - -Parquet reads pick up several improvements as well. Full Parquet metadata, including the page index, is now -cached via DataFusion's `CachedParquetFileReaderFactory`; identity casts are unwrapped in the schema adapter -so Parquet statistics pruning can engage; filter pushdown configuration has been revised; the native scan -passes a metadata size hint so a single read usually captures the footer; and the native Parquet scan seeds -its reader options from the session config so Parquet settings you already set take effect. - -Query planning and plan serialization also got cheaper. `QueryContext` SQL text is now interned into a -per-plan pool, which makes serialized plans up to 20x smaller on TPC-DS — every plan crosses the JNI -boundary, so this shrinks per-query overhead across the board. Plan-data injection is now an O(1) lookup by -operator kind and no longer rebuilds operators it does not touch. Comet also makes far fewer timer calls in -the native execution loop. - -## Correctness - -Correctness fixes have been a regular part of every Comet release, but for 1.0 we made an extra push to -clear out the bulk of the known issues before drawing a line under the stable release. This release fixes -roughly 30 divergences from Spark, spanning casts, ANSI error semantics, wrong-result bugs, string and -encoding handling, collations, and error reporting. Most were found by running extensive AI-assisted audit -sweeps of the code base, comparing Comet's expression implementations with all supported versions of Spark. -See the [change log] for the full list; two representative fixes give a sense of the shape of the work: - -- **Whitespace trimming in string-to-numeric casts.** Casts to `boolean`, integral types, `float`/`double`, - and `decimal` now use Spark's exact trim rules. Comet's kernels had used four different trim sets, three of - them wrong, so results diverged in both directions — returning null where Spark parses a value, and - returning a value where Spark returns null. -- **ANSI errors swallowed by null short-circuit.** The codegen dispatcher's null short-circuit no longer - swallows errors Spark raises. Spark evaluates null-intolerant expressions per node and left to right, so - short-circuiting on the union of input ordinals skipped subtrees Spark would have evaluated, losing their - errors. Roughly 70 built-in expressions route through this dispatcher and ANSI is on by default in Spark 4, - so this affected a wide surface. - -Alongside the bug fixes, Comet also adds conservative fallbacks where native behavior could not be made to -match — decimal `SUM` / `AVG` over sliding window frames, `FromUnixTime` with a non-default format, and a -handful of others — and falls back for predicates and datetime expressions under non-default collations -rather than risking a wrong result. - -## New Expression and Aggregate Support +### New Expression and Aggregate Support -This release expands the set of Spark expressions and aggregates that run natively: +This release expands the set of Spark expressions and aggregates that are accelerated by Comet: - **Aggregates**: `approx_percentile` / `percentile_approx`, exact `percentile` / `median`, `approx_count_distinct`, and native `collect_list` / `array_agg`. + **Cast**: Cast expressions where the native implementation is marked as incompatible or unsupported + are now routed through codegen dispatch. - **Grouping**: `grouping()` and `grouping_id()`. - **Intervals**: interval types via `make_ym_interval` and `make_dt_interval`, `CalendarIntervalType`, `multiply_dt_interval`, and interval codegen dispatch for nested values and native shuffle. @@ -150,100 +96,35 @@ This release expands the set of Spark expressions and aggregates that run native `PreciseTimestampConversion` for native time-window grouping. - **Windows**: extended native window function support and Spark 4 decimal window average. -## Native Sampling - -Spark's `SampleExec` now runs natively when sampling without replacement, covering `DataFrame.sample`, SQL -`TABLESAMPLE`, and `DataFrame.randomSplit`. The native operator ports Spark's `BernoulliCellSampler` on top of -`XorShiftRandom` and seeds per partition exactly as Spark does, so it selects the same rows for a given seed — -sampling stays reproducible whether or not Comet is enabled. Sampling with replacement still falls back to -Spark. - -## Experimental PyArrow UDF Support +### Faster Parquet Scans -This release adds experimental support for accelerated PyArrow UDFs, allowing PyArrow-based user-defined -functions to participate in native execution instead of forcing a fallback to Spark. When the feature is -disabled, Comet now hints at the native PyArrow UDF path in its fallback reasons so users know the option -exists. This is an early-stage feature and we welcome feedback from users experimenting with it. +Parquet reads pick up several improvements as well. Full Parquet metadata, including the page index, is now +cached via DataFusion's `CachedParquetFileReaderFactory`; identity casts are unwrapped in the schema adapter +so Parquet statistics pruning can engage; filter pushdown configuration has been revised; the native scan +passes a metadata size hint so a single read usually captures the footer; and the native Parquet scan seeds +its reader options from the session config so Parquet settings you already set take effect. -## Iceberg +### Iceberg Table Format V3 Comet now supports Iceberg 1.11 and Iceberg table format V3 (with native table decryption for V3 tables and fallback to Spark for other V3 features). The native Iceberg scan supports the `_pos`, `_spec`, `_file`, and `_partition` metadata columns, sizes delete files correctly to avoid dropped deletes, disambiguates scans that share a `metadata_location`, and dedupes residuals and delete files in the native scan serde. A prior case where Iceberg native scan exchange reuse with different pushed filters could produce wrong results is also -fixed. - -## Native Parquet I/O and Cloud - -Native Parquet writes now support gzip compression, and the native Parquet scan supports Azure authentication, -complementing the existing S3 credential support. - -## Execution and Fallback Improvements - -- **Mixed partial/final aggregation.** `sum` and non-decimal `avg` can now run with the partial and final - aggregation stages split across Spark and Comet, keeping more aggregation plans partially native. -- **Codegen dispatch for opt-in serdes.** Expressions reported as `Unsupported` can now route through JVM - codegen dispatch for opt-in serdes, and native opt-in expressions surface as compatible-by-default with a - `COMET-INFO` plan hint. Casts join this path: an incompatible or unsupported cast now runs Spark's own - generated code inside the Comet pipeline instead of pulling the whole operator back to Spark, so one - awkward cast no longer de-accelerates a query. `sort_array` under strict floating-point mode and `concat` - under non-`UTF8_BINARY` collations opt in the same way. -- **JVM columnar-to-row by default.** Isolated benchmarking showed the native columnar-to-row converter is - roughly 3.7x slower per row than the JVM implementation at the default batch size — and up to 15.7x slower - for small batches — because of a fixed per-batch JNI and FFI cost, with no end-to-end benefit in TPC-DS - runs. It is now disabled by default. -- **Clearer fallback reasons.** Unsupported scalar serde cases are reported in the support level, - mixed-execution fallback messages name the incompatible aggregate functions, unsupported metadata column - names appear in fallback reasons, and the spurious "WriteFilesExec is not supported" message is suppressed. -- **Expression coverage in extended explain.** The extended explain summary previously reported operator - coverage but said nothing about expressions. It now ends with a line such as `Comet accelerated 14 - expressions (14 native, 1 codegen dispatch)`, so you can see how much of a plan's expression evaluation - runs in native DataFusion kernels versus Spark's generated code inside the dispatcher. -- **Quieter logs and runtime cleanup.** Logging levels on frequent call sites have been lowered so Comet no - longer floods executor logs at default settings, and the Tokio runtime is now released on driver and - executor exit. - -## Shuffle Improvements - -- **Native shuffle memory cap.** A new `spark.comet.shuffle.maxBufferBytes` config caps native shuffle writer - memory to bound worst-case usage. -- **Shuffle IPC schema encoding.** The IPC schema is now encoded once per writer instead of per block, - cutting per-batch shuffle overhead. -- **BatchCoalescer bypass.** Shuffle bypasses the `BatchCoalescer` for batches that are already appropriately - sized. -- **Single-partition shuffle.** A redundant concatenation layer has been removed from the single-partition - path. - -## Configuration Changes - -Because configuration is Comet's primary API surface, 1.0 is the release where the naming gets cleaned up. -Every rename below registers the old key as an alias, so existing configurations keep working and log a -deprecation warning when an old key is read: - -- **Unified shuffle prefix.** Shuffle configs were spread across four disjoint prefixes - (`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`, `spark.comet.native.shuffle.*`, - `spark.comet.shuffle.*`). They now all live under `spark.comet.shuffle.*`, with `.jvm.` and `.native.` - sub-namespaces matching the `spark.comet.shuffle.mode` value you already set. -- **Grouped explain configs.** The orphan explain-related configs are collected under a single - `spark.comet.explain.*` prefix, and the PyArrow UDF config is renamed to `pyarrowUDF` for consistency. -- **`spark.comet.version`.** The loaded Comet build version is now exposed as a runtime config, so you can - confirm which Comet a cluster is actually running with `spark.conf.get` or `SET`. -- **Removed dead configs.** The Parquet parallel-IO knobs, `spark.comet.use.lazyMaterialization`, and - `spark.comet.exceptionOnDatetimeRebase` had no remaining effect and were misleading anyone tuning against - them. -- **Removed async columnar shuffle.** The `spark.comet.columnar.shuffle.async.*` path was off by default and - untested, and has been removed rather than carried into a stable release line. - -## Documentation - -The documentation received a substantial overhaul for 1.0: a design refresh of the docs site, versioned user -guides with captioned sidebar sections, and a set of accessibility and navigation fixes. The -supported-expressions page now shows how each expression is implemented — native, codegen dispatch, or hybrid -— generated from the planner itself rather than maintained by hand. An expanded tuning guide covers the -performance and memory configs that matter in practice, and there is an updated post-1.0 roadmap and a new -blog posts and talks page. +### Native Expression Performance + +Many native expression implementations have been optimized to more efficiently leverage Arrow kernels or to avoid per-row builders. + +- **Casts** between numeric, string, decimal, and date types, including a faster float-to-decimal cast, an + optimized integer-to-integer cast, shared no-overflow fast paths in `CheckOverflow` and + `DecimalRescaleCheckOverflow`, and a `cast_binary_to_string` that is up to 27x faster on binary-format + styles. +- **JSON, regex, and URL parsing**: `get_json_object`, `regexp_extract`, and `parse_url`. +- **Date/time and decimal kernels**: `date_trunc`, `spark_ceil`, and a vectorized `spark_unscaled_value`. +- **String and array kernels**: `lpad`, `unhex`, `size`, `arrays_overlap`, `escape_string`, and the `try_*` + arithmetic kernel. + ## Deprecation Notice With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic From 4ca0221f1cd8b085e91f83f59c64885fcda0e8b8 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:06:50 -0600 Subject: [PATCH 14/19] aws labs links, codegen dispatch --- content/blog/2026-08-07-datafusion-comet-1.0.0.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index cb18478b..2796a820 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -60,11 +60,13 @@ It is important that queries accelerated by Comet produce the same results as Sp ### Performance -The early Comet releases provided a very modest speedup and the published benchmark results were based on running TPC workloads at small scale factors on a single node. There are now independent benchmark results published by AWS Labs that show significant speedups for TPC-DS @ 3TB running in EKS. +The early Comet releases provided a very modest speedup and the published benchmark results were based on running TPC workloads at small scale factors on a single node. There are now independent benchmark results published by AWS Labs that show significant speedups for [TPC-DS @ 3TB running in EKS](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark). + +AWS Labs also published performance results [comparing Comet with Apache Gluten](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark), concluding that "Gluten + Velox v1.6.0 and DataFusion Comet v0.16.0 deliver similar overall runtime". ### Codegen Dispatch -Matt TBD +A significant innovation that landed in 0.17.0 was to change the approach to "native" acceleration. Rather than falling back to Spark row-based execution whenever Comet lacked a native Rust implementation of an expression, Comet is now able to execute Spark's own expression evaluation logic directly against Arrow data in the Comet pipeline. This immediately expanded the number of expressions that Comet could support without an expensive transition from columnar to row-based data. ## Improvements since 0.17.1 From 21688e2792fa5da2a558a5cbc2b055534f20dc12 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:14:05 -0600 Subject: [PATCH 15/19] ansi, more codegen, links to benchmarks --- content/blog/2026-08-07-datafusion-comet-1.0.0.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 2796a820..773c6abd 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -48,6 +48,10 @@ Comet 1.0.0 now supports more than 400 expressions, but that isn't the only way Here's a recap of some of the main advances over the past two years +### Support for Spark 4.0+ with ANSI mode + +Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default starting with Spark 4.0 + ### Correctness Testing It is important that queries accelerated by Comet produce the same results as Spark. Correctness checking has always been a large effort in Comet development, but the approach has evolved over time. @@ -62,12 +66,18 @@ It is important that queries accelerated by Comet produce the same results as Sp The early Comet releases provided a very modest speedup and the published benchmark results were based on running TPC workloads at small scale factors on a single node. There are now independent benchmark results published by AWS Labs that show significant speedups for [TPC-DS @ 3TB running in EKS](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark). -AWS Labs also published performance results [comparing Comet with Apache Gluten](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark), concluding that "Gluten + Velox v1.6.0 and DataFusion Comet v0.16.0 deliver similar overall runtime". +AWS Labs also published performance results [comparing Comet with Apache Gluten](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark), concluding that "Gluten + Velox v1.6.0 and DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's documentation also has a guide explaining how [Comet compares to Gluten](https://datafusion.apache.org/comet/about/gluten_comparison.html). ### Codegen Dispatch A significant innovation that landed in 0.17.0 was to change the approach to "native" acceleration. Rather than falling back to Spark row-based execution whenever Comet lacked a native Rust implementation of an expression, Comet is now able to execute Spark's own expression evaluation logic directly against Arrow data in the Comet pipeline. This immediately expanded the number of expressions that Comet could support without an expensive transition from columnar to row-based data. +Another advantage of this approch is that Comet can support certain categories of expression, such as regular expressions, with 100% compatibility with Spark, which is not practical when delegating to native code due to the many differences between Java's regular expression engine and those available in Rust or C++. + +## Scala and Java UDF Support + +The codegen dispatch approach described in the previous chapter + ## Improvements since 0.17.1 The rest of this post covers what is new since the 0.17.1 release. From 596b329aea52df983ba30ebeecec76cd1dfb04d7 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:16:51 -0600 Subject: [PATCH 16/19] ai suggestions --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 773c6abd..643d7ed0 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -40,17 +40,17 @@ contributors. See the [change log] for the full list of changes. ## The Road to 1.0 Comet was [donated] to the Apache DataFusion project in March 2024 and cut its first release, 0.1.0, five -months later with support for 13 operators, and 106 expressions. +months later with support for 13 operators and 106 expressions. [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ Comet 1.0.0 now supports more than 400 expressions, but that isn't the only way the project has grown over this time. -Here's a recap of some of the main advances over the past two years +Here's a recap of some of the main advances over the past two years. ### Support for Spark 4.0+ with ANSI mode -Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default starting with Spark 4.0 +Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default starting with Spark 4.0. ### Correctness Testing @@ -60,19 +60,17 @@ It is important that queries accelerated by Comet produce the same results as Sp - Scala tests: Comet has Scala tests that run queries end to end with Comet enabled vs disabled and ensure that the results match - Fuzz testing: Many of the scala tests use a fuzz testing approach to generate randomized data that queries run against, helping to catch regressions around edge cases such as nulls, NaN, Infinity, and timezone issues - Comet SQL Tests: In an effort to make it easier to write tests, Comet now provides a SQL-based testing approach that is inspired by sqllogictest -- Generative AI: More recently, Comet has taken advantage of agentic skills to perform audit sweeps of ll expressions, comapring the implementation to Spark's souce code and ensuring that Comet has tests covering all important edge cases +- Generative AI: More recently, Comet has taken advantage of agentic skills to perform audit sweeps of all expressions, comparing the implementation to Spark's source code and ensuring that Comet has tests covering all important edge cases ### Performance The early Comet releases provided a very modest speedup and the published benchmark results were based on running TPC workloads at small scale factors on a single node. There are now independent benchmark results published by AWS Labs that show significant speedups for [TPC-DS @ 3TB running in EKS](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark). -AWS Labs also published performance results [comparing Comet with Apache Gluten](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark), concluding that "Gluten + Velox v1.6.0 and DataFusion Comet v0.16.0 deliver similar overall runtime". Comet's documentation also has a guide explaining how [Comet compares to Gluten](https://datafusion.apache.org/comet/about/gluten_comparison.html). - ### Codegen Dispatch A significant innovation that landed in 0.17.0 was to change the approach to "native" acceleration. Rather than falling back to Spark row-based execution whenever Comet lacked a native Rust implementation of an expression, Comet is now able to execute Spark's own expression evaluation logic directly against Arrow data in the Comet pipeline. This immediately expanded the number of expressions that Comet could support without an expensive transition from columnar to row-based data. -Another advantage of this approch is that Comet can support certain categories of expression, such as regular expressions, with 100% compatibility with Spark, which is not practical when delegating to native code due to the many differences between Java's regular expression engine and those available in Rust or C++. +Another advantage of this approach is that Comet can support certain categories of expression, such as regular expressions, with 100% compatibility with Spark, which is not practical when delegating to native code due to the many differences between Java's regular expression engine and those available in Rust or C++. ## Scala and Java UDF Support @@ -95,7 +93,7 @@ This release expands the set of Spark expressions and aggregates that are accele - **Aggregates**: `approx_percentile` / `percentile_approx`, exact `percentile` / `median`, `approx_count_distinct`, and native `collect_list` / `array_agg`. - **Cast**: Cast expressions where the native implementation is marked as incompatible or unsupported +- **Cast**: Cast expressions where the native implementation is marked as incompatible or unsupported are now routed through codegen dispatch. - **Grouping**: `grouping()` and `grouping_id()`. - **Intervals**: interval types via `make_ym_interval` and `make_dt_interval`, `CalendarIntervalType`, @@ -124,7 +122,7 @@ fallback to Spark for other V3 features). The native Iceberg scan supports the ` share a `metadata_location`, and dedupes residuals and delete files in the native scan serde. A prior case where Iceberg native scan exchange reuse with different pushed filters could produce wrong results is also -### Native Expression Performance +### Native Expression Performance Many native expression implementations have been optimized to more efficiently leverage Arrow kernels or to avoid per-row builders. @@ -136,7 +134,7 @@ Many native expression implementations have been optimized to more efficiently l - **Date/time and decimal kernels**: `date_trunc`, `spark_ceil`, and a vectorized `spark_unscaled_value`. - **String and array kernels**: `lpad`, `unhex`, `size`, `arrays_overlap`, `escape_string`, and the `try_*` arithmetic kernel. - + ## Deprecation Notice With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic From c89bc02c3ee28fb8d74148461bf6bb2159a747ca Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:17:46 -0600 Subject: [PATCH 17/19] ai suggestions --- content/blog/2026-08-07-datafusion-comet-1.0.0.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 643d7ed0..91708a5b 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -40,13 +40,18 @@ contributors. See the [change log] for the full list of changes. ## The Road to 1.0 Comet was [donated] to the Apache DataFusion project in March 2024 and cut its first release, 0.1.0, five -months later with support for 13 operators and 106 expressions. +months later with support for 13 operators and 106 expressions. Since then, the project has shipped 20 +releases and drawn contributions from more than 120 developers, and the codebase now recognizes over 400 +Spark expressions. [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ -Comet 1.0.0 now supports more than 400 expressions, but that isn't the only way the project has grown over this time. +The 1.0 release marks the point at which Comet begins following semantic versioning. Users upgrading within +the 1.x line can expect backward-compatible changes only; features slated for removal will be deprecated in +a minor release before being dropped in the next major version. This is why the deprecations of JDK 11 and +Spark 3.4 announced below are scheduled for 1.1 rather than landing in 1.0 itself. -Here's a recap of some of the main advances over the past two years. +The rest of this section is a recap of the main advances since donation. ### Support for Spark 4.0+ with ANSI mode From 77239ade4d399c1d99c8935a8a3a180f48203b7e Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 11:20:52 -0600 Subject: [PATCH 18/19] ai suggestions --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 30 ++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index 91708a5b..e678dc47 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -73,13 +73,29 @@ The early Comet releases provided a very modest speedup and the published benchm ### Codegen Dispatch -A significant innovation that landed in 0.17.0 was to change the approach to "native" acceleration. Rather than falling back to Spark row-based execution whenever Comet lacked a native Rust implementation of an expression, Comet is now able to execute Spark's own expression evaluation logic directly against Arrow data in the Comet pipeline. This immediately expanded the number of expressions that Comet could support without an expensive transition from columnar to row-based data. - -Another advantage of this approach is that Comet can support certain categories of expression, such as regular expressions, with 100% compatibility with Spark, which is not practical when delegating to native code due to the many differences between Java's regular expression engine and those available in Rust or C++. - -## Scala and Java UDF Support - -The codegen dispatch approach described in the previous chapter +Comet 0.17.0 introduced a new approach to filling gaps in expression coverage. In earlier releases, +whenever Comet's planner encountered an expression that lacked a native Rust implementation, it fell back +to executing an entire subtree of the plan in Spark. That required converting Arrow columns back to Spark +rows before the expression ran and back to Arrow after, and the cost was often enough to erase the speedup +Comet had bought elsewhere in the plan. + +Codegen dispatch scopes that fallback down to the expression itself: the batch stays in the Comet pipeline +and Comet invokes Spark's own generated code for just the missing expression, leaving the rest of the query +running natively. Three consequences are worth calling out. + +- **Coverage.** Expressions that would previously have blocked native execution of a whole subtree are + now supported immediately, without a Rust port. In 1.0, this pathway also handles cast fallbacks, + several interval expressions, and additional string and timestamp functions. +- **Compatibility.** For categories where a native reimplementation would inevitably diverge from Spark's + semantics — regular expressions being the canonical case, given the gap between Java's regex engine and + any Rust or C++ equivalent — codegen dispatch delivers bit-for-bit Spark parity because it *is* Spark's + implementation. +- **Scala and Java UDFs.** User-defined functions are compiled to the same codegen surface as built-in + expressions, so they can flow through codegen dispatch without any change from the user. Queries that + were previously disqualified from acceleration only because they contained a UDF can now benefit as + long as the surrounding operators are supported. See the [Scala and Java UDF guide] for details. + +[Scala and Java UDF guide]: https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html ## Improvements since 0.17.1 From 1ddc5ca7e411263fbfc7aa8f2410918ff724ae69 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 6 Aug 2026 12:39:14 -0600 Subject: [PATCH 19/19] ai suggestions --- .../blog/2026-08-07-datafusion-comet-1.0.0.md | 35 ++++++++++--------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/content/blog/2026-08-07-datafusion-comet-1.0.0.md b/content/blog/2026-08-07-datafusion-comet-1.0.0.md index e678dc47..5317a983 100644 --- a/content/blog/2026-08-07-datafusion-comet-1.0.0.md +++ b/content/blog/2026-08-07-datafusion-comet-1.0.0.md @@ -46,26 +46,26 @@ Spark expressions. [donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/ -The 1.0 release marks the point at which Comet begins following semantic versioning. Users upgrading within -the 1.x line can expect backward-compatible changes only; features slated for removal will be deprecated in -a minor release before being dropped in the next major version. This is why the deprecations of JDK 11 and -Spark 3.4 announced below are scheduled for 1.1 rather than landing in 1.0 itself. +The 1.0 release marks the point at which Comet begins following [semantic versioning]. Users upgrading +within the 1.x line can expect backward-compatible changes only; features slated for removal will be +deprecated in a minor release before being dropped in the next major version. This is why the deprecations +of JDK 11 and Spark 3.4 announced below are scheduled for 1.1 rather than landing in 1.0 itself. -The rest of this section is a recap of the main advances since donation. +[semantic versioning]: https://datafusion.apache.org/comet/about/versioning_policy.html ### Support for Spark 4.0+ with ANSI mode -Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default starting with Spark 4.0. +Comet 1.0.0 supports Spark versions 3.4 through 4.1, with experimental support for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default starting with Spark 4.0. ### Correctness Testing It is important that queries accelerated by Comet produce the same results as Spark. Correctness checking has always been a large effort in Comet development, but the approach has evolved over time. -- Comet has always run Spark's own test suite with Comet enabled, providing more than 24,000 unit tests effectively for free. These tests run in Comet's CI for all supported Spark versions. -- Scala tests: Comet has Scala tests that run queries end to end with Comet enabled vs disabled and ensure that the results match -- Fuzz testing: Many of the scala tests use a fuzz testing approach to generate randomized data that queries run against, helping to catch regressions around edge cases such as nulls, NaN, Infinity, and timezone issues -- Comet SQL Tests: In an effort to make it easier to write tests, Comet now provides a SQL-based testing approach that is inspired by sqllogictest -- Generative AI: More recently, Comet has taken advantage of agentic skills to perform audit sweeps of all expressions, comparing the implementation to Spark's source code and ensuring that Comet has tests covering all important edge cases +- **Upstream Spark tests**: Comet runs Spark's own test suite with Comet enabled, providing more than 24,000 unit tests effectively for free. These tests run in Comet's CI for all supported Spark versions. +- **Scala tests**: end-to-end queries that run with Comet enabled versus disabled, checking that results match. +- **Fuzz testing**: many of the Scala tests generate randomized data to catch regressions around edge cases such as nulls, NaN, Infinity, and timezone issues. +- **Comet SQL tests**: a sqllogictest-inspired approach that makes end-to-end tests easier to write. +- **Generative AI audits**: agentic skills sweep every expression, comparing Comet's implementation to Spark's source and ensuring tests cover important edge cases. ### Performance @@ -79,7 +79,7 @@ to executing an entire subtree of the plan in Spark. That required converting Ar rows before the expression ran and back to Arrow after, and the cost was often enough to erase the speedup Comet had bought elsewhere in the plan. -Codegen dispatch scopes that fallback down to the expression itself: the batch stays in the Comet pipeline +Codegen dispatch narrows that fallback to the expression itself: the batch stays in the Comet pipeline and Comet invokes Spark's own generated code for just the missing expression, leaving the rest of the query running natively. Three consequences are worth calling out. @@ -97,9 +97,9 @@ running natively. Three consequences are worth calling out. [Scala and Java UDF guide]: https://datafusion.apache.org/comet/user-guide/latest/scala_java_udfs.html -## Improvements since 0.17.1 +## Improvements since 0.17.0 -The rest of this post covers what is new since the 0.17.1 release. +The rest of this post covers what is new since the 0.17.0 release. ### Experimental PyArrow UDF Support @@ -142,6 +142,7 @@ fallback to Spark for other V3 features). The native Iceberg scan supports the ` `_partition` metadata columns, sizes delete files correctly to avoid dropped deletes, disambiguates scans that share a `metadata_location`, and dedupes residuals and delete files in the native scan serde. A prior case where Iceberg native scan exchange reuse with different pushed filters could produce wrong results is also +fixed. ### Native Expression Performance @@ -158,14 +159,16 @@ Many native expression implementations have been optimized to more efficiently l ## Deprecation Notice -With the move to a stable 1.0 release line, Comet begins deprecating older platforms under semantic -versioning: +With the move to a stable 1.0 release line, Comet begins deprecating older platforms under its +[versioning policy]: - **JDK 11** is deprecated and scheduled for removal in Comet 1.1.0. - **Apache Spark 3.4** is deprecated and scheduled for removal in Comet 1.1.0. Users on these platforms should plan to move to JDK 17+ and Spark 3.5 or later before upgrading to 1.1.0. +[versioning policy]: https://datafusion.apache.org/comet/about/versioning_policy.html + ## Compatibility Supported platforms include: