<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

## [31.0.0](https://github.com/apache/datafusion/tree/31.0.0) (2023-09-08)

[Full Changelog](https://github.com/apache/datafusion/compare/30.0.0...31.0.0)

**Breaking changes:**

- Specialize Avg and Sum accumulators (#6842) [#7358](https://github.com/apache/datafusion/pull/7358) (tustvold)
- Use datum arithmetic scalar value [#7375](https://github.com/apache/datafusion/pull/7375) (tustvold)

**Implemented enhancements:**

- feat: `array-empty` [#7313](https://github.com/apache/datafusion/pull/7313) (Weijun-H)
- Support `REPLACE` SQL alias syntax [#7368](https://github.com/apache/datafusion/pull/7368) (berkaysynnada)
- feat: support primary key alternate syntax [#7160](https://github.com/apache/datafusion/pull/7160) (parkma99)
- docs: Add `Expr` library developer page [#7359](https://github.com/apache/datafusion/pull/7359) (tshauck)
- feat: support Binary for `min/max` [#7397](https://github.com/apache/datafusion/pull/7397) (Weijun-H)
- feat: Add memory pool configuration to `datafusion-cli` [#7424](https://github.com/apache/datafusion/pull/7424) (Weijun-H)
- Support Configuring Arrow RecordBatch Writers via SQL Statement Options [#7390](https://github.com/apache/datafusion/pull/7390) (devinjdangelo)
- Add ROLLUP and GROUPING SETS substrait support [#7382](https://github.com/apache/datafusion/pull/7382) (nseekhao)
- feat: Allow creating a ValuesExec from record batches [#7444](https://github.com/apache/datafusion/pull/7444) (scsmithr)
- minor: Add ARROW to `CREATE EXTERNAL TABLE` docs and add example of `COMPRESSION TYPE` [#7489](https://github.com/apache/datafusion/pull/7489) (alamb)
- Support Configuring Parquet Column Specific Options via SQL Statement Options [#7466](https://github.com/apache/datafusion/pull/7466) (devinjdangelo)
- Write Multiple Parquet Files in Parallel [#7483](https://github.com/apache/datafusion/pull/7483) (devinjdangelo)
- feat: explain with statistics [#7459](https://github.com/apache/datafusion/pull/7459) (korowa)

**Fixed bugs:**

- fix: union_distinct shouldn't remove child distinct [#7346](https://github.com/apache/datafusion/pull/7346) (jackwener)
- fix: inconsistent scalar types in `DistinctArrayAggAccumulator` state [#7385](https://github.com/apache/datafusion/pull/7385) (korowa)
- fix: incorrect nullability calculation of `InListExpr` [#7496](https://github.com/apache/datafusion/pull/7496) (jonahgao)

**Merged pull requests:**

- Remove redundant type check in Avg [#7374](https://github.com/apache/datafusion/pull/7374) (viirya)
- feat: `array-empty` [#7313](https://github.com/apache/datafusion/pull/7313) (Weijun-H)
- Minor: add `WriteOp::name` and `DmlStatement::name` [#7329](https://github.com/apache/datafusion/pull/7329) (alamb)
- Specialize Median Accumulator [#7376](https://github.com/apache/datafusion/pull/7376) (tustvold)
- Specialize Avg and Sum accumulators (#6842) [#7358](https://github.com/apache/datafusion/pull/7358) (tustvold)
- Change error type of invalid argument to PlanError rather than InternalError, remove misleading comments [#7355](https://github.com/apache/datafusion/pull/7355) (alamb)
- Implement `array_pop_back` function [#7348](https://github.com/apache/datafusion/pull/7348) (tanruixiang)
- Add `exec_err!` error macro [#7361](https://github.com/apache/datafusion/pull/7361) (comphead)
- Update sqlparser requirement from 0.36.1 to 0.37.0 [#7387](https://github.com/apache/datafusion/pull/7387) (viirya)
- DML documentation [#7362](https://github.com/apache/datafusion/pull/7362) (spaydar)
- Support `REPLACE` SQL alias syntax [#7368](https://github.com/apache/datafusion/pull/7368) (berkaysynnada)
- Bug-fix/next_value() of Min/Max Scalar Values [#7384](https://github.com/apache/datafusion/pull/7384) (berkaysynnada)
- Prepare 30.0.0 release [#7372](https://github.com/apache/datafusion/pull/7372) (andygrove)
- fix: union_distinct shouldn't remove child distinct [#7346](https://github.com/apache/datafusion/pull/7346) (jackwener)
- feat: support primary key alternate syntax [#7160](https://github.com/apache/datafusion/pull/7160) (parkma99)
- Merge hash table implementations and remove leftover utilities [#7366](https://github.com/apache/datafusion/pull/7366) (metesynnada)
- Minor: remove stray `println!` from `array_expressions.rs` [#7389](https://github.com/apache/datafusion/pull/7389) (alamb)
- Projection Order Propagation [#7364](https://github.com/apache/datafusion/pull/7364) (berkaysynnada)
- Document and `scratch` directory for sqllogictest and make test specific [#7312](https://github.com/apache/datafusion/pull/7312) (alamb)
- Minor: Move test for `Distribution` and `Partitioning` with code [#7392](https://github.com/apache/datafusion/pull/7392) (alamb)
- Minor: move datasource statistics code into its own module [#7391](https://github.com/apache/datafusion/pull/7391) (alamb)
- Use datum arithmetic scalar value [#7375](https://github.com/apache/datafusion/pull/7375) (tustvold)
- Fix IN expr for NaN [#7378](https://github.com/apache/datafusion/pull/7378) (sarutak)
- Unnecessary to list all files during partition pruning [#7395](https://github.com/apache/datafusion/pull/7395) (smallzhongfeng)
- Optimize `Unnest` and implement `skip_nulls=true` if specified [#7371](https://github.com/apache/datafusion/pull/7371) (smiklos)
- Docs: Add query syntax to `COPY` docs [#7388](https://github.com/apache/datafusion/pull/7388) (alamb)
- Clean up clippy for Rust 1.72 release [#7399](https://github.com/apache/datafusion/pull/7399) (alamb)
- fix: inconsistent scalar types in `DistinctArrayAggAccumulator` state [#7385](https://github.com/apache/datafusion/pull/7385) (korowa)
- Fix python CI [#7416](https://github.com/apache/datafusion/pull/7416) (tustvold)
- docs: Add `Expr` library developer page [#7359](https://github.com/apache/datafusion/pull/7359) (tshauck)
- Update ObjectStore 0.7.0 and Arrow 46.0.0 [#7282](https://github.com/apache/datafusion/pull/7282) (tustvold)
- Fix Decimal256 scalar display string in sqllogictest [#7404](https://github.com/apache/datafusion/pull/7404) (viirya)
- feat: support Binary for `min/max` [#7397](https://github.com/apache/datafusion/pull/7397) (Weijun-H)
- Make sqllogictest distinguish NaN from -NaN [#7403](https://github.com/apache/datafusion/pull/7403) (sarutak)
- Replace lazy_static with OnceLock [#7409](https://github.com/apache/datafusion/pull/7409) (sarutak)
- Minor: Remove the unreached simplification rule for `0 / 0` [#7405](https://github.com/apache/datafusion/pull/7405) (jonahgao)
- feat: Add memory pool configuration to `datafusion-cli` [#7424](https://github.com/apache/datafusion/pull/7424) (Weijun-H)
- Minor: Debug log when FairPool is created [#7431](https://github.com/apache/datafusion/pull/7431) (alamb)
- Support Configuring Arrow RecordBatch Writers via SQL Statement Options [#7390](https://github.com/apache/datafusion/pull/7390) (devinjdangelo)
- Add ROLLUP and GROUPING SETS substrait support [#7382](https://github.com/apache/datafusion/pull/7382) (nseekhao)
- Refactor sort_fuzz test to clarify what is covered [#7439](https://github.com/apache/datafusion/pull/7439) (alamb)
- Use DateTime::from_naive_utc_and_offset instead of DateTime::from_utc [#7451](https://github.com/apache/datafusion/pull/7451) (sarutak)
- Update substrait requirement from 0.12.0 to 0.13.1 [#7443](https://github.com/apache/datafusion/pull/7443) (viirya)
- [minior fix]: adjust the projection statistics [#7428](https://github.com/apache/datafusion/pull/7428) (liukun4515)
- Add new known users: Arroyo and Restate [#7464](https://github.com/apache/datafusion/pull/7464) (jychen7)
- ScalarFunctionExpr Maintaining Order [#7417](https://github.com/apache/datafusion/pull/7417) (berkaysynnada)
- Bug-fix/find_orderings_of_exprs [#7457](https://github.com/apache/datafusion/pull/7457) (berkaysynnada)
- Update prost-derive requirement from 0.11 to 0.12 [#7468](https://github.com/apache/datafusion/pull/7468) (dependabot[bot])
- Revert "Update prost-derive requirement from 0.11 to 0.12 (#7468)" [#7476](https://github.com/apache/datafusion/pull/7476) (viirya)
- Return error if spill file does not exist in ExternalSorter [#7479](https://github.com/apache/datafusion/pull/7479) (viirya)
- [minor fix]: Remove unused duplicate `file_type.rs` [#7478](https://github.com/apache/datafusion/pull/7478) (sarutak)
- Minor: more flexible pool size setting for datafusion-cli [#7454](https://github.com/apache/datafusion/pull/7454) (yjshen)
- Bump actions/checkout from 3 to 4 [#7480](https://github.com/apache/datafusion/pull/7480) (dependabot[bot])
- Support Write Options in DataFrame::write\_\* methods [#7435](https://github.com/apache/datafusion/pull/7435) (devinjdangelo)
- cp_solver, Duration vs Interval cases [#7475](https://github.com/apache/datafusion/pull/7475) (berkaysynnada)
- feat: Allow creating a ValuesExec from record batches [#7444](https://github.com/apache/datafusion/pull/7444) (scsmithr)
- Make `LogicalPlan::with_new_exprs,` deprecate `from_plan` [#7396](https://github.com/apache/datafusion/pull/7396) (alamb)
- refactor: change file type logic for create table [#7477](https://github.com/apache/datafusion/pull/7477) (tshauck)
- minor: do not fail analyzer if subquery plan contains extension [#7455](https://github.com/apache/datafusion/pull/7455) (waynexia)
- Make IN expr work with multiple items [#7449](https://github.com/apache/datafusion/pull/7449) (sarutak)
- Minor: Add doc comments and example for `ScalarVaue::to_scalar` [#7491](https://github.com/apache/datafusion/pull/7491) (alamb)
- minor: Add ARROW to `CREATE EXTERNAL TABLE` docs and add example of `COMPRESSION TYPE` [#7489](https://github.com/apache/datafusion/pull/7489) (alamb)
- Add backtrace to error messages [#7434](https://github.com/apache/datafusion/pull/7434) (comphead)
- Make sqllogictest platform-independent for the sign of NaN [#7462](https://github.com/apache/datafusion/pull/7462) (sarutak)
- Support Configuring Parquet Column Specific Options via SQL Statement Options [#7466](https://github.com/apache/datafusion/pull/7466) (devinjdangelo)
- Minor: improve error message [#7498](https://github.com/apache/datafusion/pull/7498) (alamb)
- Write Multiple Parquet Files in Parallel [#7483](https://github.com/apache/datafusion/pull/7483) (devinjdangelo)
- `PrimitiveGroupsAccumulator` should propagate timestamp timezone information properly [#7494](https://github.com/apache/datafusion/pull/7494) (sunchao)
- Minor: Add `ScalarValue::data_type()` for consistency with other APIs [#7492](https://github.com/apache/datafusion/pull/7492) (alamb)
- feat: explain with statistics [#7459](https://github.com/apache/datafusion/pull/7459) (korowa)
- fix: incorrect nullability calculation of `InListExpr` [#7496](https://github.com/apache/datafusion/pull/7496) (jonahgao)
