From efd2c8e001304b9071855bb1a27db3a2f0344f9c Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 3 Aug 2026 17:11:17 -0600 Subject: [PATCH] docs: add 1.0.0 changelog (#5243) (cherry picked from commit 98b01810bb720bf5cf09a5d3c6ae1ced46c16bb8) --- docs/source/changelog/1.0.0.md | 313 +++++++++++++++++++++++++++++++++ docs/source/changelog/index.md | 1 + 2 files changed, 314 insertions(+) create mode 100644 docs/source/changelog/1.0.0.md diff --git a/docs/source/changelog/1.0.0.md b/docs/source/changelog/1.0.0.md new file mode 100644 index 0000000000..fbd8ca3ba7 --- /dev/null +++ b/docs/source/changelog/1.0.0.md @@ -0,0 +1,313 @@ + + +# DataFusion Comet 1.0.0 Changelog + +This release consists of 244 commits from 23 contributors. See credits at the end of this changelog for more information. + +**Fixed bugs:** + +- fix: decline native V1 scans on object_store-unsupported filesystem schemes [#4525](https://github.com/apache/datafusion-comet/pull/4525) (schenksj) +- fix: exclude release scratch dirs from RAT and license skill docs [#4685](https://github.com/apache/datafusion-comet/pull/4685) (andygrove) +- fix(shuffle): tolerate non-UTF-8 bytes in get_string (lossy decode) [#4524](https://github.com/apache/datafusion-comet/pull/4524) (schenksj) +- fix: decline CreateArray with struct-nullability-divergent children [#4533](https://github.com/apache/datafusion-comet/pull/4533) (schenksj) +- fix: propagate nested cast errors [#4675](https://github.com/apache/datafusion-comet/pull/4675) (manuzhang) +- fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x [#4357](https://github.com/apache/datafusion-comet/pull/4357) (andygrove) +- fix: gate str_to_map collations [#4701](https://github.com/apache/datafusion-comet/pull/4701) (manuzhang) +- fix: repair broken rust-test build on main (init_datasource_exec arg mismatch) [#4712](https://github.com/apache/datafusion-comet/pull/4712) (andygrove) +- fix: correct stale unix_timestamp NTZ and date_format codegen-default doc text (#4502) [#4645](https://github.com/apache/datafusion-comet/pull/4645) (andygrove) +- fix: fall back for decimal SUM/AVG over sliding window frames (window audit) [#4732](https://github.com/apache/datafusion-comet/pull/4732) (andygrove) +- fix: date_trunc schema mismatch and DST handling in non-UTC timezones [#4761](https://github.com/apache/datafusion-comet/pull/4761) (andygrove) +- fix: size Iceberg delete files in the native scan to avoid dropping deletes [#4760](https://github.com/apache/datafusion-comet/pull/4760) (mbutrovich) +- fix: gate non-default collations for Spark 4 datetime expressions [#4693](https://github.com/apache/datafusion-comet/pull/4693) (0lai0) +- fix: ArrayInsert evaluation for null source arrays [#4726](https://github.com/apache/datafusion-comet/pull/4726) (peterxcli) +- fix: support Spark 4 decimal window avg [#4749](https://github.com/apache/datafusion-comet/pull/4749) (manuzhang) +- fix: array expression audit follow-ups (#4503) [#4713](https://github.com/apache/datafusion-comet/pull/4713) (andygrove) +- fix: prevent wrong results from Iceberg native scan exchange reuse with different pushed filters [#4812](https://github.com/apache/datafusion-comet/pull/4812) (mbutrovich) +- fix: count returning zero when scan is disabled and going through CometSparkColumnarToColumnar [#4795](https://github.com/apache/datafusion-comet/pull/4795) (Dummk0pf) +- fix: handle null sub-arrays in flatten [#4822](https://github.com/apache/datafusion-comet/pull/4822) (michaelmitchell-bit) +- fix: correct user guide URL in compatibility fallback messages [#4854](https://github.com/apache/datafusion-comet/pull/4854) (andygrove) +- fix: report FromUnixTime non-default format as Unsupported [#4847](https://github.com/apache/datafusion-comet/pull/4847) (andygrove) +- fix: emit JDK libjvm search path from core build script to fix -ljvm CI link failures [#4868](https://github.com/apache/datafusion-comet/pull/4868) (andygrove) +- fix: register partitioning scalar subqueries for native shuffle to avoid "Subquery N not found" [#4869](https://github.com/apache/datafusion-comet/pull/4869) (mbutrovich) +- fix: match Spark percentile interpolation precision [#4792](https://github.com/apache/datafusion-comet/pull/4792) (manuzhang) +- fix: restrict array_filter array_compact fast path to the lambda variable [#4848](https://github.com/apache/datafusion-comet/pull/4848) (andygrove) +- fix: suppress spurious WriteFilesExec fallback reason on native writes [#4928](https://github.com/apache/datafusion-comet/pull/4928) (andygrove) +- fix: decode CAST(binary AS string) JVM-compatibly instead of unsafe reinterpret [#4763](https://github.com/apache/datafusion-comet/pull/4763) (andygrove) +- fix: widen local table scan child nullability to match native kernels [#4843](https://github.com/apache/datafusion-comet/pull/4843) (andygrove) +- fix: apply 1.11 Iceberg diff changes from #4991 to other versions [#5020](https://github.com/apache/datafusion-comet/pull/5020) (mbutrovich) +- fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [#5014](https://github.com/apache/datafusion-comet/pull/5014) (andygrove) +- fix: materialize ConstantColumnVector on Comet's serialize/export paths [#4532](https://github.com/apache/datafusion-comet/pull/4532) (schenksj) +- fix: make native pow implementation compatible with Spark [#5033](https://github.com/apache/datafusion-comet/pull/5033) (andygrove) +- fix: resolve Comet jar without hardcoding the SNAPSHOT qualifier [#5108](https://github.com/apache/datafusion-comet/pull/5108) (andygrove) +- fix: honour Spark's legacy `null IN ()` behavior [#5127](https://github.com/apache/datafusion-comet/pull/5127) (andygrove) +- fix: honor fail_on_error in native make_decimal [#5080](https://github.com/apache/datafusion-comet/pull/5080) (andygrove) +- fix: work around DataFusion 54.1.0 Parquet page-index regression [#5132](https://github.com/apache/datafusion-comet/pull/5132) (mbutrovich) +- fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode [#5084](https://github.com/apache/datafusion-comet/pull/5084) (andygrove) +- fix: seed native Parquet scan reader options from session config [#5107](https://github.com/apache/datafusion-comet/pull/5107) (mbutrovich) +- fix: raise REMAINDER_BY_ZERO for Float/Double under ANSI mode [#5081](https://github.com/apache/datafusion-comet/pull/5081) (andygrove) +- fix: reduce log verbosity at per-task logging callsites [#5155](https://github.com/apache/datafusion-comet/pull/5155) (mbutrovich) +- fix: round on Int64 with scale <= -19 now overflows correctly [#5082](https://github.com/apache/datafusion-comet/pull/5082) (andygrove) +- fix: match Spark's whitespace trim semantics for casts from string to boolean, integral, float/double and decimal [#5150](https://github.com/apache/datafusion-comet/pull/5150) (andygrove) +- fix: disambiguate Iceberg scans that share a metadata_location [#5180](https://github.com/apache/datafusion-comet/pull/5180) (mbutrovich) +- fix: use per-expression eval mode for decimal promotion [#5171](https://github.com/apache/datafusion-comet/pull/5171) (peterxcli) +- fix: codegen dispatcher null short-circuit swallowed ANSI errors, plus two latent TIME-type gaps [#5219](https://github.com/apache/datafusion-comet/pull/5219) (andygrove) +- fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain [#5206](https://github.com/apache/datafusion-comet/pull/5206) (andygrove) + +**Performance related:** + +- perf: O(1) PlanDataInjector lookup by op kind [#4535](https://github.com/apache/datafusion-comet/pull/4535) (schenksj) +- perf: cache full Parquet metadata (incl. page index) via DataFusion's CachedParquetFileReaderFactory [#4707](https://github.com/apache/datafusion-comet/pull/4707) (mbutrovich) +- perf: add metadata size hint to Parquet reader to match Iceberg path [#4717](https://github.com/apache/datafusion-comet/pull/4717) (mbutrovich) +- perf(parquet): revise filter pushdown configuration [#4722](https://github.com/apache/datafusion-comet/pull/4722) (mbutrovich) +- perf: unwrap identity casts in schema adapter to enable Parquet stats pruning [#4730](https://github.com/apache/datafusion-comet/pull/4730) (mbutrovich) +- perf: avoid excessive timer calls [#4739](https://github.com/apache/datafusion-comet/pull/4739) (wForget) +- perf: optimize spark_size in spark-expr [#4877](https://github.com/apache/datafusion-comet/pull/4877) (andygrove) +- perf: optimize spark_unhex in spark-expr [#4876](https://github.com/apache/datafusion-comet/pull/4876) (andygrove) +- perf: optimize `parse_url` (50x speedup) [#4893](https://github.com/apache/datafusion-comet/pull/4893) (andygrove) +- perf: optimize `to_json` (2x faster) [#4902](https://github.com/apache/datafusion-comet/pull/4902) (andygrove) +- perf: optimize `spark_cast_int_to_int` (100x faster) [#4920](https://github.com/apache/datafusion-comet/pull/4920) (andygrove) +- perf: optimize `try_arithmetic_kernel` (up to 4x faster) [#4910](https://github.com/apache/datafusion-comet/pull/4910) (andygrove) +- perf: optimize `spark_arrays_overlap` (up to 18x faster) [#4906](https://github.com/apache/datafusion-comet/pull/4906) (andygrove) +- perf: optimize `spark_lpad` (up to 2x faster) [#4919](https://github.com/apache/datafusion-comet/pull/4919) (andygrove) +- perf: optimize `CheckOverflow` with a shared no-overflow fast path (ANSI and non-ANSI) [#4937](https://github.com/apache/datafusion-comet/pull/4937) (andygrove) +- perf: skip the null-masking pass in `DecimalRescaleCheckOverflow` when nothing overflows [#4938](https://github.com/apache/datafusion-comet/pull/4938) (andygrove) +- perf: vectorize floating-point-to-decimal cast [#4940](https://github.com/apache/datafusion-comet/pull/4940) (andygrove) +- perf: optimize `spark_get_json_object` (4x faster) [#4907](https://github.com/apache/datafusion-comet/pull/4907) (andygrove) +- perf: optimize `spark_regexp_extract` [#4894](https://github.com/apache/datafusion-comet/pull/4894) (andygrove) +- perf: vectorize `spark_unscaled_value` (9x faster) [#4972](https://github.com/apache/datafusion-comet/pull/4972) (u70b3) +- perf: optimize `cast_binary_to_string` binary-format styles (up to 27x faster) [#4912](https://github.com/apache/datafusion-comet/pull/4912) (andygrove) +- perf: optimize `parse_string_to_decimal` (30-40% faster) [#4916](https://github.com/apache/datafusion-comet/pull/4916) (andygrove) +- perf: optimize `date_parser` / cast string to date (up to 2x faster) [#4917](https://github.com/apache/datafusion-comet/pull/4917) (andygrove) +- perf: optimize cast_decimal128_to_utf8 in datafusion-comet-spark-expr [#4924](https://github.com/apache/datafusion-comet/pull/4924) (andygrove) +- perf: optimize `date_trunc` (>2x faster) [#4915](https://github.com/apache/datafusion-comet/pull/4915) (andygrove) +- perf: optimize `spark_cast_float64_to_utf8` (~40% faster) [#4918](https://github.com/apache/datafusion-comet/pull/4918) (andygrove) +- perf: dedupe Iceberg residuals and delete files in native scan serde [#4982](https://github.com/apache/datafusion-comet/pull/4982) (mbutrovich) +- perf: optimize `spark_ceil` (3x faster) [#4926](https://github.com/apache/datafusion-comet/pull/4926) (andygrove) +- perf: bypass shuffle BatchCoalescer for already-sized batches [#5003](https://github.com/apache/datafusion-comet/pull/5003) (andygrove) +- perf: encode shuffle IPC schema once per writer instead of per block [#5006](https://github.com/apache/datafusion-comet/pull/5006) (andygrove) +- refactor: drop redundant concat layer in single-partition shuffle [#5004](https://github.com/apache/datafusion-comet/pull/5004) (andygrove) +- fix: make native cast from float/double to decimal compatible with Spark [#5136](https://github.com/apache/datafusion-comet/pull/5136) (andygrove) +- perf: intern QueryContext SQL text into a per-plan pool (up to 20x smaller serialized plans for TPC-DS) [#5204](https://github.com/apache/datafusion-comet/pull/5204) (andygrove) +- perf: avoid rebuilding untouched operators in PlanDataInjector.injectPlanData [#5220](https://github.com/apache/datafusion-comet/pull/5220) (andygrove) + +**Implemented enhancements:** + +- feat: surface native parquet read failures as FAILED_READ_FILE [#4536](https://github.com/apache/datafusion-comet/pull/4536) (schenksj) +- feat: opt concat into codegen dispatch for non-UTF8_BINARY collations [#4640](https://github.com/apache/datafusion-comet/pull/4640) (andygrove) +- feat: opt sort_array into codegen dispatch under strict floating-point mode [#4637](https://github.com/apache/datafusion-comet/pull/4637) (andygrove) +- feat: support MapType input for ElementAt [#4697](https://github.com/apache/datafusion-comet/pull/4697) (0lai0) +- feat: extend native windows support [#4209](https://github.com/apache/datafusion-comet/pull/4209) (comphead) +- feat: add array_prepend support [#4716](https://github.com/apache/datafusion-comet/pull/4716) (andygrove) +- feat: surface native opt-in expressions as compatible-by-default with a COMET-INFO plan hint [#4721](https://github.com/apache/datafusion-comet/pull/4721) (andygrove) +- feat: support StringSplitSQL for split_part [#4592](https://github.com/apache/datafusion-comet/pull/4592) (michaelmitchell-bit) +- feat: support exact percentile and median aggregates natively [#4542](https://github.com/apache/datafusion-comet/pull/4542) (andygrove) +- feat: support TimestampNTZ inputs natively for hour/minute/second [#4753](https://github.com/apache/datafusion-comet/pull/4753) (andygrove) +- feat: add unsupported metadata column names to fallback reasons [#4758](https://github.com/apache/datafusion-comet/pull/4758) (hsiang-c) +- feat: Add support for `base64` [#4778](https://github.com/apache/datafusion-comet/pull/4778) (andygrove) +- feat: support interval types and make_ym_interval / make_dt_interval [#4541](https://github.com/apache/datafusion-comet/pull/4541) (andygrove) +- feat: support PreciseTimestampConversion for native batch time-window grouping [#4784](https://github.com/apache/datafusion-comet/pull/4784) (andygrove) +- feat: support Azure authentication for native Parquet scan [#4783](https://github.com/apache/datafusion-comet/pull/4783) (Dummk0pf) +- feat: support shuffle array expression [#4797](https://github.com/apache/datafusion-comet/pull/4797) (andygrove) +- feat: rebalance associative bitwise/Add/Multiply chains to avoid protobuf recursion limit [#4588](https://github.com/apache/datafusion-comet/pull/4588) (schenksj) +- feat: support grouping() and grouping_id() indicator functions [#4815](https://github.com/apache/datafusion-comet/pull/4815) (andygrove) +- feat: Stage based fallback [#4519](https://github.com/apache/datafusion-comet/pull/4519) (karuppayya) +- feat: release tokio runtime on driver/executor exit [#4734](https://github.com/apache/datafusion-comet/pull/4734) (wForget) +- feat: route Unsupported through codegen dispatch for opt-in serdes [#4728](https://github.com/apache/datafusion-comet/pull/4728) (andygrove) +- docs: comet docs design overhaul- phase 1 [#4353](https://github.com/apache/datafusion-comet/pull/4353) (pranamya123) +- feat: name incompatible aggregate functions in mixed-execution fallback reason [#4750](https://github.com/apache/datafusion-comet/pull/4750) (andygrove) +- feat: implement native empty2null spark inner function [#4683](https://github.com/apache/datafusion-comet/pull/4683) (kazantsev-maksim) +- feat: enable mixed partial/final execution for sum and non-decimal avg [#4861](https://github.com/apache/datafusion-comet/pull/4861) (andygrove) +- feat: Add experimental support for accelerated PyArrow UDFs [#4234](https://github.com/apache/datafusion-comet/pull/4234) (andygrove) +- feat: support Iceberg 1.11, audit existing Iceberg diffs, bump iceberg-rust dep, add run-iceberg-tests CI trigger [#4840](https://github.com/apache/datafusion-comet/pull/4840) (mbutrovich) +- feat: support approx_percentile / percentile_approx aggregate [#4801](https://github.com/apache/datafusion-comet/pull/4801) (andygrove) +- feat: remove constraint on array of nested elements [#4714](https://github.com/apache/datafusion-comet/pull/4714) (hsiang-c) +- feat: core SPI for contrib leaf scans (CometScanWithPlanData) [Delta contrib split, part 1] [#4700](https://github.com/apache/datafusion-comet/pull/4700) (schenksj) +- feat: support gzip compression in native Parquet writes [#4930](https://github.com/apache/datafusion-comet/pull/4930) (andygrove) +- feat: support size() for MapType input [#4580](https://github.com/apache/datafusion-comet/pull/4580) (marvelshan) +- feat: hint at native pyarrow UDF path when the feature is disabled [#4892](https://github.com/apache/datafusion-comet/pull/4892) (andygrove) +- feat: Implement TimeType support - Infrastructure - shuffle [#4398](https://github.com/apache/datafusion-comet/pull/4398) (YutaLin) +- feat: fall back to Spark for collated predicate operands [#4948](https://github.com/apache/datafusion-comet/pull/4948) (comphead) +- feat: add withAlternative alias mechanism for CometConf [#4979](https://github.com/apache/datafusion-comet/pull/4979) (andygrove) +- feat: add spark.comet.shuffle.maxBufferBytes to cap native shuffle writer memory [#4989](https://github.com/apache/datafusion-comet/pull/4989) (andygrove) +- feat: support approx_count_distinct aggregate expression [#4819](https://github.com/apache/datafusion-comet/pull/4819) (andygrove) +- feat: Iceberg table format V3: native table decryption, fall back for other V3 features [#4991](https://github.com/apache/datafusion-comet/pull/4991) (mbutrovich) +- feat: support multiply_dt_interval with codegen dispatch [#4900](https://github.com/apache/datafusion-comet/pull/4900) (peterxcli) +- feat: Support Spark levenshtein expression in native execution [#4105](https://github.com/apache/datafusion-comet/pull/4105) (Myx778) +- feat: native collect_list / array_agg aggregate [#4720](https://github.com/apache/datafusion-comet/pull/4720) (andygrove) +- feat: add CalendarIntervalType support [#4898](https://github.com/apache/datafusion-comet/pull/4898) (peterxcli) +- feat: native randstr implementation compatible with Spark [#5035](https://github.com/apache/datafusion-comet/pull/5035) (andygrove) +- feat: support interval codegen dispatch for nested values and native shuffle [#4976](https://github.com/apache/datafusion-comet/pull/4976) (peterxcli) +- feat: add codegen dispatch fallback for CometCast incompatible/unsupported cases (including legacy config paths) [#5079](https://github.com/apache/datafusion-comet/pull/5079) (comphead) +- feat: support SampleExec natively for sampling without replacement [#5110](https://github.com/apache/datafusion-comet/pull/5110) (andygrove) +- feat: disable native columnar-to-row conversion by default [#5114](https://github.com/apache/datafusion-comet/pull/5114) (andygrove) +- feat: expose Comet version as spark.comet.version runtime config [#5049](https://github.com/apache/datafusion-comet/pull/5049) (andygrove) +- feat: native uuid() implementation compatible with Spark [#5034](https://github.com/apache/datafusion-comet/pull/5034) (andygrove) +- feat: support Iceberg metadata columns \_pos, \_spec, \_file, and \_partition [#4752](https://github.com/apache/datafusion-comet/pull/4752) (parthchandra) +- feat: report native vs codegen-dispatch expression coverage in extended explain [#5201](https://github.com/apache/datafusion-comet/pull/5201) (andygrove) + +**Documentation updates:** + +- docs: update release_process for changelog [#4668](https://github.com/apache/datafusion-comet/pull/4668) (mbutrovich) +- docs: restore 0.13 user guide dropped in 0.17 release [#4704](https://github.com/apache/datafusion-comet/pull/4704) (andygrove) +- docs: fix prettier check error [#4748](https://github.com/apache/datafusion-comet/pull/4748) (wForget) +- docs: refresh expression audit notes for resolved correctness issues [#4762](https://github.com/apache/datafusion-comet/pull/4762) (andygrove) +- docs: mark try_avg and try_sum as natively supported [#4776](https://github.com/apache/datafusion-comet/pull/4776) (andygrove) +- docs: update Iceberg docs to clarify S3 storage credentials [#4767](https://github.com/apache/datafusion-comet/pull/4767) (mbutrovich) +- docs: render captioned sidebar sections for versioned user guides [#4699](https://github.com/apache/datafusion-comet/pull/4699) (andygrove) +- docs: triage bug vs enhancement and apply type labels consistently [#4768](https://github.com/apache/datafusion-comet/pull/4768) (andygrove) +- docs: mark make_dt_interval and make_ym_interval as supported [#4790](https://github.com/apache/datafusion-comet/pull/4790) (andygrove) +- docs: correct window function support status and limitations [#4833](https://github.com/apache/datafusion-comet/pull/4833) (andygrove) +- docs: minor docs site improvements [#4855](https://github.com/apache/datafusion-comet/pull/4855) (andygrove) +- docs: remove references to closed and fixed issues from compatibility guide [#4856](https://github.com/apache/datafusion-comet/pull/4856) (andygrove) +- docs: fix sidebar drawer mis-tap, missing aria-expanded, homepage permalink, and ASF links page [#4858](https://github.com/apache/datafusion-comet/pull/4858) (pranamya123) +- docs: announce JDK 11 and Spark 3.4 deprecation for 1.1.0 removal [#4857](https://github.com/apache/datafusion-comet/pull/4857) (andygrove) +- docs: note invalid UTF-8 scan limitation and FFI import risk [#4846](https://github.com/apache/datafusion-comet/pull/4846) (andygrove) +- docs: update Gluten comparison with AWS Labs benchmark and Comet strengths [#4873](https://github.com/apache/datafusion-comet/pull/4873) (andygrove) +- docs: add scalar expression optimization guide, skill, and performance audits [#4933](https://github.com/apache/datafusion-comet/pull/4933) (andygrove) +- doc: Document scan tuning for `spark.sql.files.maxPartitionBytes` [#4931](https://github.com/apache/datafusion-comet/pull/4931) (comphead) +- docs: add 0.17.1 changelog [#4961](https://github.com/apache/datafusion-comet/pull/4961) (mbutrovich) +- docs: document Spark version adoption and support-lifetime policy [#4977](https://github.com/apache/datafusion-comet/pull/4977) (andygrove) +- docs: show implementation kind for each expression [#5028](https://github.com/apache/datafusion-comet/pull/5028) (andygrove) +- docs: add blog posts and talks page [#5043](https://github.com/apache/datafusion-comet/pull/5043) (andygrove) +- docs: avoid "not a blocker" comments in reviews for review skill [#5057](https://github.com/apache/datafusion-comet/pull/5057) (mbutrovich) +- docs: refresh stale issue references and normalize issue link format [#5062](https://github.com/apache/datafusion-comet/pull/5062) (andygrove) +- docs: fix references to configuration keys that do not exist [#5063](https://github.com/apache/datafusion-comet/pull/5063) (andygrove) +- docs: define post-1.0 versioning policy and legacy config process [#5056](https://github.com/apache/datafusion-comet/pull/5056) (andygrove) +- docs: expand tuning guide with performance and memory configs [#4908](https://github.com/apache/datafusion-comet/pull/4908) (andygrove) +- docs: document known correctness issues in the compatibility guide [#5085](https://github.com/apache/datafusion-comet/pull/5085) (andygrove) +- docs: update post 1.0.0 roadmap [#5064](https://github.com/apache/datafusion-comet/pull/5064) (mbutrovich) +- docs: drop compatibility notes for bugs that are now fixed [#5154](https://github.com/apache/datafusion-comet/pull/5154) (andygrove) +- docs: document ReusedExchange caveat and both CometSparkToColumnar names in operator-count exclusions [#5240](https://github.com/apache/datafusion-comet/pull/5240) (andygrove) + +**Other:** + +- chore: fix generate-release-docs.sh for per-Spark-version doc layout [#4662](https://github.com/apache/datafusion-comet/pull/4662) (mbutrovich) +- chore: add branch protection to release branches, update release_process.md [#4665](https://github.com/apache/datafusion-comet/pull/4665) (mbutrovich) +- chore: start 0.18.0 development [#4664](https://github.com/apache/datafusion-comet/pull/4664) (mbutrovich) +- test: cover nested complex casts [#4608](https://github.com/apache/datafusion-comet/pull/4608) (manuzhang) +- refactor: move string expression support checks to getSupportLevel [#4676](https://github.com/apache/datafusion-comet/pull/4676) (andygrove) +- refactor: move arithmetic and math support checks to getSupportLevel [#4674](https://github.com/apache/datafusion-comet/pull/4674) (andygrove) +- refactor: move aggregate expression support checks to getSupportLevel [#4678](https://github.com/apache/datafusion-comet/pull/4678) (andygrove) +- refactor: move array expression support checks to getSupportLevel [#4677](https://github.com/apache/datafusion-comet/pull/4677) (andygrove) +- chore: add array tests with NaN handling [#4686](https://github.com/apache/datafusion-comet/pull/4686) (comphead) +- chore: tweak CI execution memory params [#4687](https://github.com/apache/datafusion-comet/pull/4687) (comphead) +- chore: add ordering tests for `array_union` [#4688](https://github.com/apache/datafusion-comet/pull/4688) (comphead) +- chore: fix `ConstantFolding` rule exclusion for benchmarks [#4689](https://github.com/apache/datafusion-comet/pull/4689) (comphead) +- chore(deps): bump actions/checkout from 6 to 7 [#4690](https://github.com/apache/datafusion-comet/pull/4690) (dependabot[bot]) +- chore(deps): bump the all-other-cargo-deps group in /native with 2 updates [#4691](https://github.com/apache/datafusion-comet/pull/4691) (dependabot[bot]) +- chore(deps): bump itertools from 0.14.0 to 0.15.0 in /native [#4692](https://github.com/apache/datafusion-comet/pull/4692) (dependabot[bot]) +- chore: update documentation links for 0.17.0 release [#4698](https://github.com/apache/datafusion-comet/pull/4698) (andygrove) +- chore: add optional CI flow for parquet writes [#4696](https://github.com/apache/datafusion-comet/pull/4696) (comphead) +- deps: Upgrade to DataFusion 54.0.0 [#4062](https://github.com/apache/datafusion-comet/pull/4062) (andygrove) +- chore: apply same SBT profile for Spark 4.0 [#4709](https://github.com/apache/datafusion-comet/pull/4709) (comphead) +- refactor: remove map_contains_key serde [#4703](https://github.com/apache/datafusion-comet/pull/4703) (manuzhang) +- chore: Spark 4.0 restore `DEDICATED_JVM_SBT_TESTS` [#4711](https://github.com/apache/datafusion-comet/pull/4711) (comphead) +- refactor: share Spark 4.1+ shim sources to remove duplication [#4710](https://github.com/apache/datafusion-comet/pull/4710) (andygrove) +- chore(deps): bump object_store from 0.13.2 to 0.14.0 in /native [#4737](https://github.com/apache/datafusion-comet/pull/4737) (dependabot[bot]) +- chore(deps): bump the all-other-cargo-deps group in /native with 3 updates [#4736](https://github.com/apache/datafusion-comet/pull/4736) (dependabot[bot]) +- chore(deps): bump actions/cache from 5 to 6 [#4735](https://github.com/apache/datafusion-comet/pull/4735) (dependabot[bot]) +- deps: revert object_store to 0.13.2 [#4740](https://github.com/apache/datafusion-comet/pull/4740) (mbutrovich) +- chore: fix `4.0.2` diff to handle `CometWindowsExec` properly [#4743](https://github.com/apache/datafusion-comet/pull/4743) (comphead) +- chore: surface DataFusion 54 PruningMetrics and Ratio in CometNativeScan metrics [#4733](https://github.com/apache/datafusion-comet/pull/4733) (mbutrovich) +- refactor: gate CometUnaryMinus input types in getSupportLevel [#4759](https://github.com/apache/datafusion-comet/pull/4759) (andygrove) +- test: enable literal sha2 now that native engine handles scalar args [#4773](https://github.com/apache/datafusion-comet/pull/4773) (andygrove) +- ci: republish docs site when generated-doc sources change [#4777](https://github.com/apache/datafusion-comet/pull/4777) (andygrove) +- test: enable SparkSessionExtensionSuite with Comet [#4772](https://github.com/apache/datafusion-comet/pull/4772) (andygrove) +- chore: start 1.0.0 development [#4794](https://github.com/apache/datafusion-comet/pull/4794) (andygrove) +- chore(deps): bump rand from 0.10.1 to 0.10.2 in /native in the all-other-cargo-deps group [#4805](https://github.com/apache/datafusion-comet/pull/4805) (dependabot[bot]) +- deps: bump iceberg-rust version to a rev with `ArrowReader` fix [#4826](https://github.com/apache/datafusion-comet/pull/4826) (sandugood) +- chore: Use native impl of `soundex` function [#4824](https://github.com/apache/datafusion-comet/pull/4824) (kazantsev-maksim) +- refactor(shuffle): Introduce PartitionWriter interface to decouple shuffle partitioning from storage [#4779](https://github.com/apache/datafusion-comet/pull/4779) (wForget) +- chore: fallback for `spark.sql.mapKeyDedupPolicy` == `LAST_WIN` [#4863](https://github.com/apache/datafusion-comet/pull/4863) (comphead) +- chore: remove `hdfs` Comet crate [#4904](https://github.com/apache/datafusion-comet/pull/4904) (comphead) +- test: cover invalid unhex inputs in SQL file tests [#4890](https://github.com/apache/datafusion-comet/pull/4890) (andygrove) +- chore: Codegen fallback explain for plan [#4891](https://github.com/apache/datafusion-comet/pull/4891) (comphead) +- test: add version check on an Iceberg test for Variant type [#4935](https://github.com/apache/datafusion-comet/pull/4935) (mbutrovich) +- chore: use Datafusion `substring` [#4161](https://github.com/apache/datafusion-comet/pull/4161) (comphead) +- chore: Support native writer `Overwrite` and `ErrorIfExists` modes [#4946](https://github.com/apache/datafusion-comet/pull/4946) (comphead) +- refactor: move AttributeReference and KnownFloatingPointNormalized expression support checks to getSupportLevel [#4745](https://github.com/apache/datafusion-comet/pull/4745) (peterxcli) +- chore(deps): bump actions/setup-node from 6 to 7 [#4955](https://github.com/apache/datafusion-comet/pull/4955) (dependabot[bot]) +- chore: bump spark-4.2 profile to the released 4.2.0 [#4960](https://github.com/apache/datafusion-comet/pull/4960) (andygrove) +- chore(deps): bump the all-other-cargo-deps group across 1 directory with 6 updates [#4956](https://github.com/apache/datafusion-comet/pull/4956) (dependabot[bot]) +- chore(deps): bump lz4_flex from 0.13.1 to 0.14.0 in /native [#4958](https://github.com/apache/datafusion-comet/pull/4958) (dependabot[bot]) +- chore: codeql dependabot fix [#4962](https://github.com/apache/datafusion-comet/pull/4962) (comphead) +- chore(deps): bump the codeql-actions group with 2 updates [#4970](https://github.com/apache/datafusion-comet/pull/4970) (dependabot[bot]) +- test: promote `try_to_date`/`try_to_timestamp` SQL tests to native coverage [#4973](https://github.com/apache/datafusion-comet/pull/4973) (u70b3) +- chore: remove dead Parquet parallel-IO configs [#4981](https://github.com/apache/datafusion-comet/pull/4981) (andygrove) +- refactor: remove untested async columnar shuffle [#4985](https://github.com/apache/datafusion-comet/pull/4985) (andygrove) +- deps: bump to datafusion 54.1.0 and latest iceberg-rust [#4996](https://github.com/apache/datafusion-comet/pull/4996) (mbutrovich) +- chore: use DF `array_repeat` and `array_compact` [#4741](https://github.com/apache/datafusion-comet/pull/4741) (comphead) +- chore: remove legacy use.lazyMaterialization [#4998](https://github.com/apache/datafusion-comet/pull/4998) (kazuyukitanimura) +- chore(deps): bump actions/stale from 10.3.0 to 10.4.0 [#4879](https://github.com/apache/datafusion-comet/pull/4879) (dependabot[bot]) +- chore(deps): bump base64 from 0.22.1 to 0.23.0 in /native [#5018](https://github.com/apache/datafusion-comet/pull/5018) (dependabot[bot]) +- chore(deps): bump actions/setup-python from 6 to 7 [#5017](https://github.com/apache/datafusion-comet/pull/5017) (dependabot[bot]) +- chore(deps): bump the codeql-actions group with 2 updates [#5016](https://github.com/apache/datafusion-comet/pull/5016) (dependabot[bot]) +- chore(deps): bump actions/cache from 5 to 6 [#4882](https://github.com/apache/datafusion-comet/pull/4882) (dependabot[bot]) +- chore(deps): bump actions/checkout from 6 to 7 [#4881](https://github.com/apache/datafusion-comet/pull/4881) (dependabot[bot]) +- refactor: group orphan `spark.comet.explain.*` configs under one prefix [#5026](https://github.com/apache/datafusion-comet/pull/5026) (andygrove) +- refactor: unify shuffle configs under `spark.comet.shuffle.*` prefix [#4986](https://github.com/apache/datafusion-comet/pull/4986) (andygrove) +- test: add isolated and small-batch benchmarks for columnar-to-row conversion [#5113](https://github.com/apache/datafusion-comet/pull/5113) (andygrove) +- chore: stop enabling incompatible casts in plan stability suite [#5139](https://github.com/apache/datafusion-comet/pull/5139) (andygrove) +- chore: replace deprecated symbol literals with $"col" in tests [#5168](https://github.com/apache/datafusion-comet/pull/5168) (andygrove) +- test: cover ArrayExists three-valued logic config [#5000](https://github.com/apache/datafusion-comet/pull/5000) (manuzhang) +- chore(deps): bump actions/stale from 10.4.0 to 11.0.0 [#5163](https://github.com/apache/datafusion-comet/pull/5163) (dependabot[bot]) +- chore: remove Array -> Seq round trips flagged by the 2.13 copy deprecation [#5170](https://github.com/apache/datafusion-comet/pull/5170) (andygrove) +- chore: supply explicit empty argument lists for 2.13 [#5175](https://github.com/apache/datafusion-comet/pull/5175) (andygrove) +- chore: clean up Scala compiler warnings [#5141](https://github.com/apache/datafusion-comet/pull/5141) (andygrove) +- chore: drop deprecated .toIterable from protobuf builder calls [#5173](https://github.com/apache/datafusion-comet/pull/5173) (andygrove) +- chore: bump Spark 3.5 to 3.5.9 [#5181](https://github.com/apache/datafusion-comet/pull/5181) (andygrove) +- chore: bump Spark 4.1 to 4.1.3 [#5183](https://github.com/apache/datafusion-comet/pull/5183) (andygrove) +- refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position [#5176](https://github.com/apache/datafusion-comet/pull/5176) (peterxcli) +- chore: bump Spark 4.0 to 4.0.4 [#5182](https://github.com/apache/datafusion-comet/pull/5182) (andygrove) +- chore: remove unused spark.comet.exceptionOnDatetimeRebase config [#5221](https://github.com/apache/datafusion-comet/pull/5221) (andygrove) +- refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs [#5197](https://github.com/apache/datafusion-comet/pull/5197) (andygrove) +- chore: cargo update for 1.0 release [#5226](https://github.com/apache/datafusion-comet/pull/5226) (mbutrovich) + +## Credits + +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. + +``` + 132 Andy Grove + 28 Matt Butrovich + 19 Oleks V + 18 dependabot[bot] + 8 Scott Schenkein + 7 Manu Zhang + 7 Peter Lee + 4 Zhen Wang + 2 ChenChen Lai + 2 Kazantsev Maksim + 2 Prames Maanikam + 2 Pranamya Vadlamani + 2 hsiang-c + 2 kid + 1 Bolin Lin + 1 KAZUYUKI TANIMURA + 1 Karuppayya + 1 Mitchell + 1 Myx778 + 1 Parth Chandra + 1 Zaki + 1 alexander domenti + 1 michaelmitchell-bit +``` + +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. diff --git a/docs/source/changelog/index.md b/docs/source/changelog/index.md index 609c76ed36..e79878f7e7 100644 --- a/docs/source/changelog/index.md +++ b/docs/source/changelog/index.md @@ -24,6 +24,7 @@ Per-release change logs for Apache DataFusion Comet. ```{toctree} :maxdepth: 1 +1.0.0 <1.0.0> 0.17.1 <0.17.1> 0.17.0 <0.17.0> 0.16.0 <0.16.0>