<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion 42.0.0 Changelog

This release consists of 296 commits from 73 contributors. See credits at the end of this changelog for more information.

**Breaking changes:**

- feat: expose centroids in approx_percentile_cont fluent api [#11878](https://github.com/apache/datafusion/pull/11878) (Michael-J-Ward)
- UDAF refactor: Add PhysicalExpr trait dependency on `datafusion-expr` and remove logical expressions requirement for creating physical aggregate expression [#11845](https://github.com/apache/datafusion/pull/11845) (jayzhan211)
- `datafusion.catalog.has_headers` default value set to `true` [#11919](https://github.com/apache/datafusion/pull/11919) (korowa)
- Use `schema_name` to create the `physical_name` [#11977](https://github.com/apache/datafusion/pull/11977) (joroKr21)
- Add ability to return `LogicalPlan` by value from `TableProvider::get_logical_plan` [#12113](https://github.com/apache/datafusion/pull/12113) (askalt)
- Remove Sort expression (`Expr::Sort`) [#12177](https://github.com/apache/datafusion/pull/12177) (findepi)
- Remove TableSource::supports_filter_pushdown function [#12239](https://github.com/apache/datafusion/pull/12239) (findepi)
- Remove Box from Sort [#12207](https://github.com/apache/datafusion/pull/12207) (findepi)
- Avoid unnecessary move when setting SessionConfig [#12260](https://github.com/apache/datafusion/pull/12260) (findepi)
- Remove unused `AggregateOptions` struct and `scalar_update_factor` config setting [#12241](https://github.com/apache/datafusion/pull/12241) (jc4x4)
- Remove deprecated LogicalPlan::with_new_inputs function [#12285](https://github.com/apache/datafusion/pull/12285) (findepi)
- Fixes missing `nth_value` UDAF expr function [#12279](https://github.com/apache/datafusion/pull/12279) (jcsherin)
- Remove unnecessary `Result` from return type in `NamePreserver` [#12358](https://github.com/apache/datafusion/pull/12358) (jonahgao)
- Removed Arc wrapping for AggregateFunctionExpr [#12353](https://github.com/apache/datafusion/pull/12353) (athultr1997)

**Performance related:**

- perf: avoid repeat format in calc_func_dependencies_for_project [#12305](https://github.com/apache/datafusion/pull/12305) (haohuaijin)

**Implemented enhancements:**

- feat: Add map_extract module and function [#11969](https://github.com/apache/datafusion/pull/11969) (Weijun-H)
- feat: use Substrait's PrecisionTimestamp and PrecisionTimestampTz instead of deprecated Timestamp [#11597](https://github.com/apache/datafusion/pull/11597) (Blizzara)
- feat: support upper and lower for stringview [#12138](https://github.com/apache/datafusion/pull/12138) (tshauck)
- feat: Add DateFieldExtractStyle::Strftime support for SqliteDialect unparser [#12161](https://github.com/apache/datafusion/pull/12161) (peasee)
- feat: Enforce the uniqueness of map key name for the map/make_map function [#12153](https://github.com/apache/datafusion/pull/12153) (Weijun-H)
- feat: Add projection to FilterExec [#12281](https://github.com/apache/datafusion/pull/12281) (eejbyfeldt)
- feat: Support `FixedSizedList` in `array_distance` function [#12381](https://github.com/apache/datafusion/pull/12381) (Weijun-H)

**Fixed bugs:**

- fix: invalid sqls when unparsing derived table with columns contains calculations, limit/order/distinct [#11756](https://github.com/apache/datafusion/pull/11756) (y-f-u)
- fix: make ScalarValue::Dictionary with NULL values produce NULL arrays [#11908](https://github.com/apache/datafusion/pull/11908) (mhilton)
- fix: throw error on sub-day generate_series increments [#11907](https://github.com/apache/datafusion/pull/11907) (tshauck)
- fix: impl ordering for serialization/deserialization for AggregateUdf [#11926](https://github.com/apache/datafusion/pull/11926) (haohuaijin)
- fix: Fix various complaints from the latest nightly clippy [#11958](https://github.com/apache/datafusion/pull/11958) (itsjunetime)
- fix: move coercion of union from builder to `TypeCoercion` [#11961](https://github.com/apache/datafusion/pull/11961) (jonahgao)
- fix: incorrect aggregation result of `bool_and` [#12017](https://github.com/apache/datafusion/pull/12017) (jonahgao)
- fix: support min/max for Float16 type [#12050](https://github.com/apache/datafusion/pull/12050) (korowa)
- fix: Panic non-integer for the second argument of `nth_value` function [#12076](https://github.com/apache/datafusion/pull/12076) (Weijun-H)
- fix: ser/de fetch in CoalesceBatchesExec [#12107](https://github.com/apache/datafusion/pull/12107) (haohuaijin)
- fix: UDF, UDAF, UDWF with_alias(..) should wrap the inner function fully [#12098](https://github.com/apache/datafusion/pull/12098) (Blizzara)
- fix: Produce buffered null join row only if all joined rows are failed on join filter in SMJ full join [#12090](https://github.com/apache/datafusion/pull/12090) (viirya)
- fix: single partition in SortPreservingMergeExec don't take fetch [#12109](https://github.com/apache/datafusion/pull/12109) (haohuaijin)
- fix: set `supports_retract_batch` to false for `ApproxPercentileAccumulator` [#12132](https://github.com/apache/datafusion/pull/12132) (jonahgao)
- fix: preserve expression names when replacing placeholders [#12126](https://github.com/apache/datafusion/pull/12126) (jonahgao)
- fix: Skip buffered rows which are not joined with streamed side when checking join filter results [#12159](https://github.com/apache/datafusion/pull/12159) (viirya)
- fix: preserve qualifiers when rewriting expressions [#12341](https://github.com/apache/datafusion/pull/12341) (jonahgao)
- fix: support Substrait VirtualTables with no columns [#12339](https://github.com/apache/datafusion/pull/12339) (Blizzara)
- fix: nested loop join requires outer table to be a FusedStream [#12189](https://github.com/apache/datafusion/pull/12189) (YjyJeff)

**Documentation updates:**

- chore: Prepare 41.0.0-rc1 [#11889](https://github.com/apache/datafusion/pull/11889) (andygrove)
- Enforce sorting handle fetchable operators, add option to repartition based on row count estimates [#11875](https://github.com/apache/datafusion/pull/11875) (mustafasrepo)
- Minor: change wording for PMC membership notice [#11930](https://github.com/apache/datafusion/pull/11930) (alamb)
- Minor: fix outdated link [#11964](https://github.com/apache/datafusion/pull/11964) (austin362667)
- Minor: polish `Accumulator::state` docs [#12053](https://github.com/apache/datafusion/pull/12053) (lewiszlw)
- Fix CI check when version changes -- remove checked in file that is created by doc example [#12034](https://github.com/apache/datafusion/pull/12034) (alamb)
- Add new user doc to translate logical plan to physical plan [#12026](https://github.com/apache/datafusion/pull/12026) (jc4x4)
- Remove vestigal `datafusion-docs` module compilation [#12081](https://github.com/apache/datafusion/pull/12081) (alamb)
- Minor: Add example for configuring SessionContext [#12139](https://github.com/apache/datafusion/pull/12139) (alamb)
- Make it easier to understand datafusion-cli exclusion [#12188](https://github.com/apache/datafusion/pull/12188) (findepi)
- Add documentation on `EXPLAIN` and `EXPLAIN ANALYZE` [#12122](https://github.com/apache/datafusion/pull/12122) (devanbenz)
- Add `array_distance` function [#12211](https://github.com/apache/datafusion/pull/12211) (austin362667)
- Minor: fix `list_distinct` alias link title [#12246](https://github.com/apache/datafusion/pull/12246) (austin362667)
- Support `map_keys` & `map_values` for MAP type [#12194](https://github.com/apache/datafusion/pull/12194) (dharanad)
- Minor: Improve ExecutionMode documentation [#12214](https://github.com/apache/datafusion/pull/12214) (alamb)
- Implement `kurtosis_pop` UDAF [#12273](https://github.com/apache/datafusion/pull/12273) (goldmedal)
- Update download page to reflect latest version (v41) [#12310](https://github.com/apache/datafusion/pull/12310) (phillipleblanc)
- Fix issue with "to_date" failing to process dates later than year 2262 [#12227](https://github.com/apache/datafusion/pull/12227) (MartinKolbAtWork)
- Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [#12271](https://github.com/apache/datafusion/pull/12271) (wiedld)
- Add documentation about performance PRs, add (TBD) section on feature criteria [#12372](https://github.com/apache/datafusion/pull/12372) (alamb)
- Implement native support StringView for `CONTAINS` function [#12168](https://github.com/apache/datafusion/pull/12168) (tlm365)
- Fix parquet statistics for ListingTable and Utf8View with `schema_force_string_view`, rename config option to `schema_force_view_types` [#12232](https://github.com/apache/datafusion/pull/12232) (wiedld)
- Minor: Fix project website links [#12419](https://github.com/apache/datafusion/pull/12419) (alamb)
- doc: Update MSRV policy, shortening to max(4 months, 4 releases) [#12402](https://github.com/apache/datafusion/pull/12402) (comphead)
- Add a `version()` UDF [#12429](https://github.com/apache/datafusion/pull/12429) (samuelcolvin)
- Support timestamps and steps of less than a day for range/generate_series [#12400](https://github.com/apache/datafusion/pull/12400) (Omega359)
- Improve comments on target user and unify intro summaries [#12418](https://github.com/apache/datafusion/pull/12418) (alamb)
- Add 'Extensions List' page to the documentation [#12420](https://github.com/apache/datafusion/pull/12420) (alamb)
- Added array_any_value function [#12329](https://github.com/apache/datafusion/pull/12329) (athultr1997)

**Other:**

- Sync rust docs params for CI and dev [#11890](https://github.com/apache/datafusion/pull/11890) (findepi)
- Update ASCII scalar function to support Utf8View #11834 [#11884](https://github.com/apache/datafusion/pull/11884) (dmitrybugakov)
- Fix `Duration` vs `Interval` comparisons and `Interval` as LHS [#11876](https://github.com/apache/datafusion/pull/11876) (samuelcolvin)
- Produce clear error message when build runs with conflicting features [#11895](https://github.com/apache/datafusion/pull/11895) (findepi)
- Add tests for StringView / character functions, fix `regexp_like` and `regexp_match` to work with StringView [#11753](https://github.com/apache/datafusion/pull/11753) (alamb)
- Avoid unecessary copy when reading arrow files [#11840](https://github.com/apache/datafusion/pull/11840) (XiangpengHao)
- Support NULL literal in Min/Max [#11812](https://github.com/apache/datafusion/pull/11812) (xinlifoobar)
- Remove many `crate::` imports in listing table provider module [#11887](https://github.com/apache/datafusion/pull/11887) (findepi)
- Rename `Expr::display_name` to `Expr::schema_name`, make `UNNEST` naming conform to convention [#11797](https://github.com/apache/datafusion/pull/11797) (jayzhan211)
- Make `CommonSubexprEliminate` top-down like [#11683](https://github.com/apache/datafusion/pull/11683) (peter-toth)
- Add `generate_series` tests for arrays [#11921](https://github.com/apache/datafusion/pull/11921) (alamb)
- Minor: use `lit(true)` and `lit(false)` more [#11904](https://github.com/apache/datafusion/pull/11904) (alamb)
- Fix: panics in `approx_percentile_cont()` aggregate function [#11934](https://github.com/apache/datafusion/pull/11934) (2010YOUY01)
- Ingore shebang at top of file in `datafusion-cli` [#11927](https://github.com/apache/datafusion/pull/11927) (PsiACE)
- Parse Sqllogictest column types from physical schema [#11929](https://github.com/apache/datafusion/pull/11929) (jonahgao)
- Update INITCAP scalar function to support Utf8View [#11888](https://github.com/apache/datafusion/pull/11888) (xinlifoobar)
- Implement native support StringView for Octet Length [#11906](https://github.com/apache/datafusion/pull/11906) (PsiACE)
- Implement native support StringView for Ends With [#11924](https://github.com/apache/datafusion/pull/11924) (PsiACE)
- Implement native support StringView for Levenshtein [#11925](https://github.com/apache/datafusion/pull/11925) (PsiACE)
- Implement native stringview support for BTRIM [#11920](https://github.com/apache/datafusion/pull/11920) (Kev1n8)
- Move `LimitPushdown` to physical-optimizer crate [#11945](https://github.com/apache/datafusion/pull/11945) (lewiszlw)
- Minor: Improve comments in row_hash.rs for skipping aggregation [#11820](https://github.com/apache/datafusion/pull/11820) (alamb)
- chore: Add SessionState to MockContextProvider just like SessionContextProvider [#11940](https://github.com/apache/datafusion/pull/11940) (dharanad)
- Update labeler.yml to match crates [#11937](https://github.com/apache/datafusion/pull/11937) (alamb)
- Support tuples as types [#11896](https://github.com/apache/datafusion/pull/11896) (samuelcolvin)
- Support `convert_to_state` for `AVG` accumulator [#11734](https://github.com/apache/datafusion/pull/11734) (alamb)
- minor: Update release documentation based on 41.0.0 release [#11947](https://github.com/apache/datafusion/pull/11947) (andygrove)
- Make `Precision<usize>` copy to make it clear clones are not expensive [#11828](https://github.com/apache/datafusion/pull/11828) (alamb)
- Minor: simplify SQL number parsing and add a comment about unused [#11965](https://github.com/apache/datafusion/pull/11965) (alamb)
- Support Arrays for the Map scalar functions [#11712](https://github.com/apache/datafusion/pull/11712) (dharanad)
- Implement Utf8View for lpad scalar function [#11941](https://github.com/apache/datafusion/pull/11941) (Omega359)
- Add native stringview support for LTRIM & RTRIM [#11948](https://github.com/apache/datafusion/pull/11948) (Kev1n8)
- Move wildcard expansions to the analyzer [#11681](https://github.com/apache/datafusion/pull/11681) (goldmedal)
- Add native stringview support for RIGHT [#11955](https://github.com/apache/datafusion/pull/11955) (Kev1n8)
- Register get_field by default [#11959](https://github.com/apache/datafusion/pull/11959) (leoyvens)
- Refactor `CoalesceBatches` to use an explicit state machine [#11966](https://github.com/apache/datafusion/pull/11966) (berkaysynnada)
- Implement native support StringView for find in set [#11970](https://github.com/apache/datafusion/pull/11970) (PsiACE)
- test: re-enable window function over parquet with forced collisions [#11939](https://github.com/apache/datafusion/pull/11939) (korowa)
- Implement native support StringView for `REPEAT` [#11962](https://github.com/apache/datafusion/pull/11962) (tlm365)
- Update RPAD scalar function to support Utf8View [#11942](https://github.com/apache/datafusion/pull/11942) (Lordworms)
- Improve lpad udf by using a GenericStringBuilder [#11987](https://github.com/apache/datafusion/pull/11987) (Omega359)
- Implement native support StringView for substr_index [#11974](https://github.com/apache/datafusion/pull/11974) (PsiACE)
- Add native stringview support for LEFT [#11976](https://github.com/apache/datafusion/pull/11976) (Kev1n8)
- Minor: Improve function documentation [#11996](https://github.com/apache/datafusion/pull/11996) (alamb)
- Implement native support StringView for overlay [#11968](https://github.com/apache/datafusion/pull/11968) (PsiACE)
- Keep the existing default catalog for `SessionStateBuilder::new_from_existing` [#11991](https://github.com/apache/datafusion/pull/11991) (goldmedal)
- Use tracked-consumers memory pool be the default. [#11949](https://github.com/apache/datafusion/pull/11949) (wiedld)
- Update REVERSE scalar function to support Utf8View [#11973](https://github.com/apache/datafusion/pull/11973) (Omega359)
- Support partial aggregation skip for boolean functions [#11847](https://github.com/apache/datafusion/pull/11847) (2010YOUY01)
- feat/11953: Support StringView for TRANSLATE() fn [#11967](https://github.com/apache/datafusion/pull/11967) (devanbenz)
- Update SPLIT_PART scalar function to support Utf8View [#11975](https://github.com/apache/datafusion/pull/11975) (Lordworms)
- Handle arguments checking of `min`/`max` function to avoid crashes [#12016](https://github.com/apache/datafusion/pull/12016) (tlm365)
- Fix: support NULL input for regular expression comparison operations [#11985](https://github.com/apache/datafusion/pull/11985) (HuSen8891)
- Remove physical sort parameters on aggregate window functions [#12009](https://github.com/apache/datafusion/pull/12009) (timsaucer)
- Minor: Use execution error in ScalarValue::iter_to_array for incorrect usage [#11999](https://github.com/apache/datafusion/pull/11999) (jayzhan211)
- Fix: support NULL input for like operations [#12025](https://github.com/apache/datafusion/pull/12025) (HuSen8891)
- Minor: Add error tests for min/max with 2 arguments [#12024](https://github.com/apache/datafusion/pull/12024) (alamb)
- Improve performance of REPEAT functions [#12015](https://github.com/apache/datafusion/pull/12015) (tlm365)
- Update SUBSTR scalar function to support Utf8View [#12019](https://github.com/apache/datafusion/pull/12019) (dmitrybugakov)
- Minor: Remove wrong comment on `Accumulator::evaluate` and `Accumulator::state` [#12001](https://github.com/apache/datafusion/pull/12001) (lewiszlw)
- Minor: cleanup `.gitignore` [#12035](https://github.com/apache/datafusion/pull/12035) (alamb)
- Improve documentation about `ParquetExec` / Parquet predicate pushdown [#11994](https://github.com/apache/datafusion/pull/11994) (alamb)
- refactor: Move `LimitedDistinctAggregation` to `physical-optimizer` crate [#12036](https://github.com/apache/datafusion/pull/12036) (Weijun-H)
- Convert built-in `row_number` to user-defined window function [#12030](https://github.com/apache/datafusion/pull/12030) (jcsherin)
- Fix projection name with DataFrame::with_column and window functions [#12000](https://github.com/apache/datafusion/pull/12000) (devanbenz)
- Update to `sqlparser-rs` v0.50.0 [#12014](https://github.com/apache/datafusion/pull/12014) (samuelcolvin)
- Minor: make some physical-plan properties public [#12022](https://github.com/apache/datafusion/pull/12022) (emgeee)
- chore: improve variable naming conventions [#12042](https://github.com/apache/datafusion/pull/12042) (caicancai)
- Fix: handle NULL input for regex match operations [#12028](https://github.com/apache/datafusion/pull/12028) (HuSen8891)
- Fix compilation, change row_number() expr_fn to 0 args [#12043](https://github.com/apache/datafusion/pull/12043) (alamb)
- Minor: Remove warning when building `datafusion-cli` from `Dockerfile` [#12018](https://github.com/apache/datafusion/pull/12018) (tlm365)
- Minor: add getter method for LogicalPlanBuilder.plan [#12038](https://github.com/apache/datafusion/pull/12038) (emgeee)
- Window UDF signature check [#12045](https://github.com/apache/datafusion/pull/12045) (jayzhan211)
- Fix: generate_series function support string type [#12002](https://github.com/apache/datafusion/pull/12002) (getChan)
- Do not add redundant subquery ordering into plan [#12003](https://github.com/apache/datafusion/pull/12003) (mertak-synnada)
- Fix: Remove Unrelated Fields When Expanding Wildcards in Functional Dependency Projections [#12060](https://github.com/apache/datafusion/pull/12060) (berkaysynnada)
- Update async-trait in CLI and catalog crates [#12061](https://github.com/apache/datafusion/pull/12061) (findepi)
- Minor: remove clones in `coerce_plan_expr_for_schema` [#12051](https://github.com/apache/datafusion/pull/12051) (jonahgao)
- implement utf8_view for replace [#12004](https://github.com/apache/datafusion/pull/12004) (thinh2)
- Minor: update sqllogictest to treat Utf8View as text [#12033](https://github.com/apache/datafusion/pull/12033) (alamb)
- [MINOR] correct document mistakes [#12068](https://github.com/apache/datafusion/pull/12068) (FANNG1)
- Plan `LATERAL` subqueries [#11456](https://github.com/apache/datafusion/pull/11456) (aalexandrov)
- Faster random() scalar function [#12078](https://github.com/apache/datafusion/pull/12078) (2010YOUY01)
- functions: support strpos with mixed string types [#12072](https://github.com/apache/datafusion/pull/12072) (nrc)
- Update to `clap` 4.5.16 [#12064](https://github.com/apache/datafusion/pull/12064) (findepi)
- Fix the schema mismatch between logical and physical for aggregate function, add `AggregateUDFImpl::is_null` [#11989](https://github.com/apache/datafusion/pull/11989) (jayzhan211)
- minor: Remove unused create_row_hashes [#12083](https://github.com/apache/datafusion/pull/12083) (andygrove)
- Improve rpad udf by using a GenericStringBuilder [#12070](https://github.com/apache/datafusion/pull/12070) (Lordworms)
- Add test to verify count aggregate function should not be nullable [#12100](https://github.com/apache/datafusion/pull/12100) (HuSen8891)
- Minor: Extract `BatchCoalescer` to its own module [#12047](https://github.com/apache/datafusion/pull/12047) (alamb)
- Add Utf8View support to STRPOS function [#12087](https://github.com/apache/datafusion/pull/12087) (dmitrybugakov)
- Update itertools requirement from 0.12 to 0.13 [#10556](https://github.com/apache/datafusion/pull/10556) (dependabot[bot])
- Fix wildcard expansion for `HAVING` clause [#12046](https://github.com/apache/datafusion/pull/12046) (goldmedal)
- Convert LogicalPlanBuilder to use Arc<LogicalPlan> [#12040](https://github.com/apache/datafusion/pull/12040) (jc4x4)
- Minor: rename `dictionary_coercion` to `dictionary_comparison_coercion`, add comments [#12102](https://github.com/apache/datafusion/pull/12102) (alamb)
- Improve documentation on `StringArrayType` trait [#12027](https://github.com/apache/datafusion/pull/12027) (alamb)
- Improve split_part udf by using a GenericStringBuilder [#12093](https://github.com/apache/datafusion/pull/12093) (Lordworms)
- Fix compilation on main [#12108](https://github.com/apache/datafusion/pull/12108) (alamb)
- minor: SortExec measure elapsed_compute time when sorting [#12099](https://github.com/apache/datafusion/pull/12099) (mhilton)
- Support string concat `||` for StringViewArray [#12063](https://github.com/apache/datafusion/pull/12063) (dharanad)
- Minor: make RowNumber public [#12110](https://github.com/apache/datafusion/pull/12110) (berkaysynnada)
- Add benchmark for SUBSTR to evaluate improvements using StringView [#12111](https://github.com/apache/datafusion/pull/12111) (Kev1n8)
- [minor] Use Vec instead of primitive builders [#12121](https://github.com/apache/datafusion/pull/12121) (Dandandan)
- Fix thread panic when "unreachable" SpawnedTask code is reachable. [#12086](https://github.com/apache/datafusion/pull/12086) (wiedld)
- Improve `CombinePartialFinalAggregate` code readability [#12128](https://github.com/apache/datafusion/pull/12128) (lewiszlw)
- Use `LexRequirement` alias as much as possible [#12130](https://github.com/apache/datafusion/pull/12130) (lewiszlw)
- `array_has` avoid row converter for string type [#12097](https://github.com/apache/datafusion/pull/12097) (jayzhan211)
- Throw `not_impl_error` for `approx_percentile_cont` parameters validation [#12133](https://github.com/apache/datafusion/pull/12133) (goldmedal)
- minor: Add comments for `GroupedHashAggregateStream` struct [#12127](https://github.com/apache/datafusion/pull/12127) (2010YOUY01)
- fix concat dictionary(int32, utf8) bug [#12143](https://github.com/apache/datafusion/pull/12143) (thinh2)
- `array_has` with eq kernel [#12125](https://github.com/apache/datafusion/pull/12125) (jayzhan211)
- Check for overflow in substring with negative start [#12141](https://github.com/apache/datafusion/pull/12141) (findepi)
- Minor: add test for panic propagation [#12134](https://github.com/apache/datafusion/pull/12134) (alamb)
- Add benchmark for STDDEV and VAR to Clickbench extended [#12146](https://github.com/apache/datafusion/pull/12146) (alamb)
- Use Result.unwrap_or_else where applicable [#12166](https://github.com/apache/datafusion/pull/12166) (findepi)
- Provide documentation of expose APIs to enable handling of type coercion at UNION plan construction. [#12142](https://github.com/apache/datafusion/pull/12142) (wiedld)
- Implement groups accumulator for stddev and variance [#12095](https://github.com/apache/datafusion/pull/12095) (eejbyfeldt)
- Minor: refine Partitioning documentation [#12145](https://github.com/apache/datafusion/pull/12145) (alamb)
- Minor: allow to build RuntimeEnv from RuntimeConfig [#12151](https://github.com/apache/datafusion/pull/12151) (theirix)
- benches: add lower benches for stringview [#12152](https://github.com/apache/datafusion/pull/12152) (tshauck)
- Replace Arc::try_unwrap with Arc::unwrap_or_clone where cloning anyway [#12173](https://github.com/apache/datafusion/pull/12173) (findepi)
- Enable the test for creating empty map [#12176](https://github.com/apache/datafusion/pull/12176) (goldmedal)
- Remove unwrap_arc helper [#12172](https://github.com/apache/datafusion/pull/12172) (findepi)
- Fix typo [#12169](https://github.com/apache/datafusion/pull/12169) (findepi)
- Minor: remove vestigal github workflow `pr_comment.yml` [#12182](https://github.com/apache/datafusion/pull/12182) (alamb)
- Remove `AggregateExpr` trait [#12096](https://github.com/apache/datafusion/pull/12096) (lewiszlw)
- Deduplicate sort unparsing logic [#12175](https://github.com/apache/datafusion/pull/12175) (findepi)
- Require sort expressions to be of type Sort [#12171](https://github.com/apache/datafusion/pull/12171) (findepi)
- Remove `parse_vec_expr` helper [#12178](https://github.com/apache/datafusion/pull/12178) (findepi)
- Reuse bulk serialization helpers for protobuf [#12179](https://github.com/apache/datafusion/pull/12179) (findepi)
- Remove unnecessary clones from `.../logical_plan/builder.rs` [#12196](https://github.com/apache/datafusion/pull/12196) (findepi)
- Remove unnecessary clones with clippy [#12197](https://github.com/apache/datafusion/pull/12197) (findepi)
- Make RuntimeEnvBuilder rather than RuntimeConfig [#12157](https://github.com/apache/datafusion/pull/12157) (devanbenz)
- Minor: Fix grouping set typo [#12216](https://github.com/apache/datafusion/pull/12216) (lewiszlw)
- Unbounded SortExec (and Top-K) Implementation When Req's Are Satisfied [#12174](https://github.com/apache/datafusion/pull/12174) (berkaysynnada)
- Remove normalize_with_schemas function [#12233](https://github.com/apache/datafusion/pull/12233) (findepi)
- Update AWS dependencies in CLI [#12229](https://github.com/apache/datafusion/pull/12229) (findepi)
- Avoid Arc::clone when serializing physical expressions [#12235](https://github.com/apache/datafusion/pull/12235) (findepi)
- Confirming UDF aliases are serialized correctly [#12219](https://github.com/apache/datafusion/pull/12219) (edmondop)
- Minor: Reuse `NamePreserver` in `SimplifyExpressions` [#12238](https://github.com/apache/datafusion/pull/12238) (jonahgao)
- Remove redundant argument and improve error message [#12217](https://github.com/apache/datafusion/pull/12217) (findepi)
- Remove deprecated from_plan function [#12265](https://github.com/apache/datafusion/pull/12265) (findepi)
- Remove redundant result of `AggregateFunctionExpr::field` [#12258](https://github.com/apache/datafusion/pull/12258) (lewiszlw)
- Define current arrow_cast behavior for BinaryView [#12200](https://github.com/apache/datafusion/pull/12200) (wiedld)
- Update prost dependency [#12237](https://github.com/apache/datafusion/pull/12237) (findepi)
- Bump webpack from 5.88.2 to 5.94.0 in /datafusion/wasmtest/datafusion-wasm-app [#12236](https://github.com/apache/datafusion/pull/12236) (dependabot[bot])
- Avoid redundant pass-by-value in physical optimizer [#12261](https://github.com/apache/datafusion/pull/12261) (findepi)
- Remove FileScanConfig::repartition_file_groups function [#12242](https://github.com/apache/datafusion/pull/12242) (findepi)
- Make group expressions nullable more accurate [#12256](https://github.com/apache/datafusion/pull/12256) (lewiszlw)
- Avoid redundant pass-by-value in optimizer [#12262](https://github.com/apache/datafusion/pull/12262) (findepi)
- Support alternate format for Date32 unparsing (TEXT/SQLite) [#12282](https://github.com/apache/datafusion/pull/12282) (sgrebnov)
- Extract drive-by fixes from PR 12135 for easier reviewing [#12240](https://github.com/apache/datafusion/pull/12240) (itsjunetime)
- Move `CombinePartialFinalAggregate` rule into physical-optimizer crate [#12167](https://github.com/apache/datafusion/pull/12167) (lewiszlw)
- Minor: Add `RuntimeEnvBuilder::build_arc() [#12213](https://github.com/apache/datafusion/pull/12213) (alamb)
- Introduce `Signature::Coercible` [#12275](https://github.com/apache/datafusion/pull/12275) (jayzhan211)
- fix hash-repartition panic [#12297](https://github.com/apache/datafusion/pull/12297) (thinh2)
- Remove unsafe Send impl from PriorityMap [#12289](https://github.com/apache/datafusion/pull/12289) (findepi)
- test: check record count and types in parquet window test [#12277](https://github.com/apache/datafusion/pull/12277) (korowa)
- Optimize `struct` and `named_struct` functions [#11688](https://github.com/apache/datafusion/pull/11688) (Rafferty97)
- Update the CONCAT scalar function to support Utf8View [#12224](https://github.com/apache/datafusion/pull/12224) (devanbenz)
- chore: Fix warnings produced by shellcheck on bench.sh [#12303](https://github.com/apache/datafusion/pull/12303) (eejbyfeldt)
- test: re-enable merge join test with forced collisions [#12276](https://github.com/apache/datafusion/pull/12276) (korowa)
- Fix various typos in aggregation doc [#12301](https://github.com/apache/datafusion/pull/12301) (lewiszlw)
- Improve binary scalars display [#12192](https://github.com/apache/datafusion/pull/12192) (lewiszlw)
- Minor: Reduce string allocations in ScalarValue::binary display [#12322](https://github.com/apache/datafusion/pull/12322) (alamb)
- minor: Add PartialEq, Eq traits to StatsType [#12327](https://github.com/apache/datafusion/pull/12327) (andygrove)
- Update to `arrow`/`parquet` `53.0.0`, `tonic`, `prost`, `object_store`, `pyo3` [#12032](https://github.com/apache/datafusion/pull/12032) (alamb)
- Minor: Update Sanity Checker Error Messages [#12333](https://github.com/apache/datafusion/pull/12333) (berkaysynnada)
- Improve & unify validation in LogicalPlan::with_new_exprs [#12264](https://github.com/apache/datafusion/pull/12264) (findepi)
- Support the custom terminator for the CSV file format [#12263](https://github.com/apache/datafusion/pull/12263) (goldmedal)
- Support try_from_array and eq_array for ScalarValue::Union [#12208](https://github.com/apache/datafusion/pull/12208) (joroKr21)
- Fix some clippy warnings [#12346](https://github.com/apache/datafusion/pull/12346) (mbrobbel)
- minor: reuse SessionStateBuilder methods for default builder [#12330](https://github.com/apache/datafusion/pull/12330) (comphead)
- Push down null filters for more join types [#12348](https://github.com/apache/datafusion/pull/12348) (Dandandan)
- Move `TopKAggregation` rule into `physical-optimizer` crate [#12334](https://github.com/apache/datafusion/pull/12334) (lewiszlw)
- Support Utf8View and BinaryView in substrait serialization. [#12199](https://github.com/apache/datafusion/pull/12199) (wiedld)
- Fix Possible Congestion Scenario in `SortPreservingMergeExec` [#12302](https://github.com/apache/datafusion/pull/12302) (berkaysynnada)
- Minor: Re-export variable provider [#12351](https://github.com/apache/datafusion/pull/12351) (lewiszlw)
- Support protobuf encoding and decoding of `UnnestExec` [#12344](https://github.com/apache/datafusion/pull/12344) (joroKr21)
- Fix subquery alias table definition unparsing for SQLite [#12331](https://github.com/apache/datafusion/pull/12331) (sgrebnov)
- Remove deprecated ScalarValue::get_datatype [#12361](https://github.com/apache/datafusion/pull/12361) (findepi)
- Improve StringView support for SUBSTR [#12044](https://github.com/apache/datafusion/pull/12044) (Kev1n8)
- Minor: improve performance of `ScalarValue::Binary*` debug [#12323](https://github.com/apache/datafusion/pull/12323) (alamb)
- Apply non-nested kernel for non-nested in `array_has` and `inlist` [#12164](https://github.com/apache/datafusion/pull/12164) (jayzhan211)
- Faster `character_length()` string function for ASCII-only case [#12356](https://github.com/apache/datafusion/pull/12356) (2010YOUY01)
- Unparse TableScan with projections, filters or fetch to SQL string [#12158](https://github.com/apache/datafusion/pull/12158) (goldmedal)
- Minor: Support protobuf serialization for Utf8View and BinaryView [#12165](https://github.com/apache/datafusion/pull/12165) (Lordworms)
- Minor: Add tests for using FilterExec when parquet was pushed down [#12362](https://github.com/apache/datafusion/pull/12362) (alamb)
- Minor: Add getter for logical optimizer rules [#12379](https://github.com/apache/datafusion/pull/12379) (maronavenue)
- Update sqllogictest requirement from 0.21.0 to 0.22.0 [#12388](https://github.com/apache/datafusion/pull/12388) (dependabot[bot])
- Support StringView for binary operators [#12212](https://github.com/apache/datafusion/pull/12212) (tlm365)
- Support for SIMILAR TO for physical plan [#12350](https://github.com/apache/datafusion/pull/12350) (theirix)
- Remove deprecated expression optimizer's utils [#12390](https://github.com/apache/datafusion/pull/12390) (findepi)
- Minor: Remove redundant usage of clone [#12392](https://github.com/apache/datafusion/pull/12392) (waruto210)
- Introduce the `DynamicFileCatalog` in `datafusion-catalog` [#11035](https://github.com/apache/datafusion/pull/11035) (goldmedal)
- tests: enable fuzz for filtered anti-semi NLJoin [#12360](https://github.com/apache/datafusion/pull/12360) (korowa)
- Refactor `SqlToRel::sql_expr_to_logical_expr_internal` to reduce stack size [#12384](https://github.com/apache/datafusion/pull/12384) (Jefffrey)
- Reuse `spill_record_batch_by_size` function [#12389](https://github.com/apache/datafusion/pull/12389) (lewiszlw)
- minor: improve join fuzz tests debug kit [#12397](https://github.com/apache/datafusion/pull/12397) (comphead)
- Fix invalid ref in UserDefinedLogicalNodeCore doc [#12396](https://github.com/apache/datafusion/pull/12396) (dttung2905)
- Don't push down IsNotNull for `null_equals_null` case [#12404](https://github.com/apache/datafusion/pull/12404) (Dandandan)
- Fix: `substr()` on StringView column's behavior is inconsistent with the old version [#12383](https://github.com/apache/datafusion/pull/12383) (2010YOUY01)
- validate and adjust Substrait NamedTable schemas (#12223) [#12245](https://github.com/apache/datafusion/pull/12245) (vbarua)
- Bump rstest from 0.17.0 to 0.22.0 in /datafusion-cli [#12413](https://github.com/apache/datafusion/pull/12413) (dependabot[bot])
- fix tpc-h parquet setting to respect global options [#12405](https://github.com/apache/datafusion/pull/12405) (XiangpengHao)
- Bump dirs from 4.0.0 to 5.0.1 in /datafusion-cli [#12411](https://github.com/apache/datafusion/pull/12411) (dependabot[bot])
- Allow using dictionary arrays as filters [#12382](https://github.com/apache/datafusion/pull/12382) (adriangb)
- Add support for Utf8View, Boolean, Date32/64, int32/64 for writing hive style partitions [#12283](https://github.com/apache/datafusion/pull/12283) (Omega359)
- Bump env_logger from 0.9.3 to 0.11.5 in /datafusion-cli [#12410](https://github.com/apache/datafusion/pull/12410) (dependabot[bot])
- Check window functions by str for with_column [#12431](https://github.com/apache/datafusion/pull/12431) (timsaucer)
- Fix incorrect OFFSET during LIMIT pushdown. [#12399](https://github.com/apache/datafusion/pull/12399) (wiedld)
- Fix: upper case qualifier wildcard bug [#12426](https://github.com/apache/datafusion/pull/12426) (JasonLi-cn)
- Fix: Internal error in regexp_replace() for some StringView input [#12203](https://github.com/apache/datafusion/pull/12203) (devanbenz)
- Automate sqllogictest for StringView (for one function, `substr`) [#12433](https://github.com/apache/datafusion/pull/12433) (2010YOUY01)
- Update concat_ws scalar function to support Utf8View [#12309](https://github.com/apache/datafusion/pull/12309) (devanbenz)
- Bump serve-static and express in /datafusion/wasmtest/datafusion-wasm-app [#12434](https://github.com/apache/datafusion/pull/12434) (dependabot[bot])
- Minor: add err on `create` `temporary` table [#12439](https://github.com/apache/datafusion/pull/12439) (hailelagi)
- Minor: Add a test for version() function [#12441](https://github.com/apache/datafusion/pull/12441) (alamb)
- Handle case-sensitive identifier when decorrelating predicate subquery [#12443](https://github.com/apache/datafusion/pull/12443) (goldmedal)
- Bump send and express in /datafusion/wasmtest/datafusion-wasm-app [#12447](https://github.com/apache/datafusion/pull/12447) (dependabot[bot])
- Add PartialOrd for the DF subfields/structs for the WindowFunction expr [#12421](https://github.com/apache/datafusion/pull/12421) (ngli-me)
- Making avro_to_arrow::schema::to_arrow_schema public [#12452](https://github.com/apache/datafusion/pull/12452) (ameyc)
- Bump rustyline from 11.0.0 to 14.0.0 in /datafusion-cli [#12407](https://github.com/apache/datafusion/pull/12407) (dependabot[bot])
- Minor: add `ListingOptions::with_file_extension_opt` [#12461](https://github.com/apache/datafusion/pull/12461) (alamb)
- Improve PhysicalExpr and Column documentation [#12457](https://github.com/apache/datafusion/pull/12457) (alamb)
- fix length error with `array_has` [#12459](https://github.com/apache/datafusion/pull/12459) (samuelcolvin)

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
    39	Andrew Lamb
    35	Piotr Findeisen
    15	张林伟
    10	Jax Liu
     9	Jay Zhan
     9	Jonah Gao
     9	dependabot[bot]
     8	wiedld
     7	Chojan Shang
     7	WeblWabl
     7	Yongting You
     6	Berkay Şahin
     6	Eduard Karacharov
     6	Tai Le Manh
     6	kf zheng
     5	Alex Huang
     5	Bruce Ritchie
     5	Lordworms
     5	Samuel Colvin
     4	Andy Grove
     4	Dharan Aditya
     4	HuSen
     4	Huaijin
     3	Arttu
     3	Austin Liu
     3	Daniël Heres
     3	Dmitry Bugakov
     3	Emil Ejbyfeldt
     3	Georgi Krastev
     3	JC
     3	Oleks V
     3	Trent Hauck
     3	iamthinh
     2	Athul T R
     2	June
     2	Liang-Chi Hsieh
     2	Martin Hilton
     2	Matt Green
     2	Sergei Grebnov
     2	Tim Saucer
     2	Xiangpeng Hao
     2	Xin Li
     2	jcsherin
     2	theirix
     1	Adrian Garcia Badaracco
     1	Albert Skalt
     1	Alexander Alexandrov
     1	Alexander Rafferty
     1	Amey Chaugule
     1	Cancai Cai
     1	Dao Thanh Tung
     1	Edmondo Porcu
     1	FANNG
     1	Haile
     1	JasonLi
     1	Jeffrey Vo
     1	Leonardo Yvens
     1	Maron Montano
     1	Martin Kolb
     1	Matthijs Brobbel
     1	Michael J Ward
     1	Mustafa Akur
     1	Namgung Chan
     1	Nick Cameron
     1	Peter Toth
     1	Phillip LeBlanc
     1	Victor Barua
     1	YjyJeff
     1	mertak-synnada
     1	ngli-me
     1	peasee
     1	waruto
     1	yfu
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.
