<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

## [32.0.0](https://github.com/apache/datafusion/tree/32.0.0) (2023-10-07)

[Full Changelog](https://github.com/apache/datafusion/compare/31.0.0...32.0.0)

**Breaking changes:**

- Remove implicit interval type coercion from ScalarValue comparison [#7514](https://github.com/apache/datafusion/pull/7514) (tustvold)
- Remove get_scan_files and ExecutionPlan::file_scan_config (#7357) [#7487](https://github.com/apache/datafusion/pull/7487) (tustvold)
- Move `FileCompressionType` out of `common` and into `core` [#7596](https://github.com/apache/datafusion/pull/7596) (haohuaijin)
- Update arrow 47.0.0 in DataFusion [#7587](https://github.com/apache/datafusion/pull/7587) (tustvold)
- Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [#7723](https://github.com/apache/datafusion/pull/7723) (alamb)

**Implemented enhancements:**

- Parallelize Stateless (CSV/JSON) File Write Serialization [#7452](https://github.com/apache/datafusion/pull/7452) (devinjdangelo)
- Create a Priority Queue based Aggregation with `limit` [#7192](https://github.com/apache/datafusion/pull/7192) (avantgardnerio)
- feat: add guarantees to simplification [#7467](https://github.com/apache/datafusion/pull/7467) (wjones127)
- [Minor]: Produce better plan when group by contains all of the ordering requirements [#7542](https://github.com/apache/datafusion/pull/7542) (mustafasrepo)
- Make AvroArrowArrayReader possible to scan Avro backed table which contains nested records [#7525](https://github.com/apache/datafusion/pull/7525) (sarutak)
- feat: Support spilling for hash aggregation [#7400](https://github.com/apache/datafusion/pull/7400) (kazuyukitanimura)
- Parallelize Parquet Serialization [#7562](https://github.com/apache/datafusion/pull/7562) (devinjdangelo)
- feat: natively support more data types for the `abs` function. [#7568](https://github.com/apache/datafusion/pull/7568) (jonahgao)
- feat: Parallel collecting parquet files statistics #7573 [#7595](https://github.com/apache/datafusion/pull/7595) (hengfeiyang)
- Support hashing List columns [#7616](https://github.com/apache/datafusion/pull/7616) (jonmmease)
- feat: Better large output display in datafusion-cli with --maxrows option [#7617](https://github.com/apache/datafusion/pull/7617) (2010YOUY01)
- feat: make parse_float_as_decimal work on negative numbers [#7648](https://github.com/apache/datafusion/pull/7648) (jonahgao)
- Update Default Parquet Write Compression [#7692](https://github.com/apache/datafusion/pull/7692) (devinjdangelo)
- Support all the codecs supported by Avro [#7718](https://github.com/apache/datafusion/pull/7718) (sarutak)
- Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator [#7721](https://github.com/apache/datafusion/pull/7721) (Dandandan)

**Fixed bugs:**

- fix: inconsistent behaviors when dividing floating numbers by zero [#7503](https://github.com/apache/datafusion/pull/7503) (jonahgao)
- fix: skip EliminateCrossJoin rule if inner join with filter is found [#7529](https://github.com/apache/datafusion/pull/7529) (epsio-banay)
- fix: check for precision overflow when parsing float as decimal [#7627](https://github.com/apache/datafusion/pull/7627) (jonahgao)
- fix: substrait limit when fetch is None [#7669](https://github.com/apache/datafusion/pull/7669) (waynexia)
- fix: coerce text to timestamps with timezones [#7720](https://github.com/apache/datafusion/pull/7720) (mhilton)
- fix: avro_to_arrow: Handle avro nested nullable struct (union) [#7663](https://github.com/apache/datafusion/pull/7663) (Samrose-Ahmed)

**Documentation updates:**

- Documentation Updates for New Write Related Features [#7520](https://github.com/apache/datafusion/pull/7520) (devinjdangelo)
- Create 2023 Q4 roadmap [#7551](https://github.com/apache/datafusion/pull/7551) (graydenshand)
- docs: add section on supports_filters_pushdown [#7680](https://github.com/apache/datafusion/pull/7680) (tshauck)
- Add LanceDB to the list of Known Users [#7716](https://github.com/apache/datafusion/pull/7716) (alamb)
- Document crate feature flags [#7713](https://github.com/apache/datafusion/pull/7713) (alamb)

**Merged pull requests:**

- Prepare 31.0.0 release [#7508](https://github.com/apache/datafusion/pull/7508) (andygrove)
- Minor(proto): Implement `TryFrom<&DFSchema>` for `protobuf::DfSchema` [#7505](https://github.com/apache/datafusion/pull/7505) (jonahgao)
- fix: inconsistent behaviors when dividing floating numbers by zero [#7503](https://github.com/apache/datafusion/pull/7503) (jonahgao)
- Parallelize Stateless (CSV/JSON) File Write Serialization [#7452](https://github.com/apache/datafusion/pull/7452) (devinjdangelo)
- Minor: Remove stray comment markings from encoding error message [#7512](https://github.com/apache/datafusion/pull/7512) (devinjdangelo)
- Remove implicit interval type coercion from ScalarValue comparison [#7514](https://github.com/apache/datafusion/pull/7514) (tustvold)
- Minor: deprecate ScalarValue::get_datatype() [#7507](https://github.com/apache/datafusion/pull/7507) (Weijun-H)
- Propagate error from spawned task reading spills [#7510](https://github.com/apache/datafusion/pull/7510) (viirya)
- Refactor the EnforceDistribution Rule [#7488](https://github.com/apache/datafusion/pull/7488) (mustafasrepo)
- Remove get_scan_files and ExecutionPlan::file_scan_config (#7357) [#7487](https://github.com/apache/datafusion/pull/7487) (tustvold)
- Simplify ScalarValue::distance (#7517) [#7519](https://github.com/apache/datafusion/pull/7519) (tustvold)
- typo: change `delimeter` to `delimiter` [#7521](https://github.com/apache/datafusion/pull/7521) (Weijun-H)
- Fix some simplification rules for floating-point arithmetic operations [#7515](https://github.com/apache/datafusion/pull/7515) (jonahgao)
- Documentation Updates for New Write Related Features [#7520](https://github.com/apache/datafusion/pull/7520) (devinjdangelo)
- [MINOR]: Move tests from repartition to enforce_distribution file [#7539](https://github.com/apache/datafusion/pull/7539) (mustafasrepo)
- Update the async-trait crate to resolve clippy bug [#7541](https://github.com/apache/datafusion/pull/7541) (metesynnada)
- Fix flaky `test_sort_fetch_memory_calculation` test [#7534](https://github.com/apache/datafusion/pull/7534) (viirya)
- Move common code to utils [#7545](https://github.com/apache/datafusion/pull/7545) (mustafasrepo)
- Minor: Add comments and clearer constructors to `Interval` [#7526](https://github.com/apache/datafusion/pull/7526) (alamb)
- fix: skip EliminateCrossJoin rule if inner join with filter is found [#7529](https://github.com/apache/datafusion/pull/7529) (epsio-banay)
- Create a Priority Queue based Aggregation with `limit` [#7192](https://github.com/apache/datafusion/pull/7192) (avantgardnerio)
- feat: add guarantees to simplification [#7467](https://github.com/apache/datafusion/pull/7467) (wjones127)
- [Minor]: Produce better plan when group by contains all of the ordering requirements [#7542](https://github.com/apache/datafusion/pull/7542) (mustafasrepo)
- Minor: beautify interval display [#7554](https://github.com/apache/datafusion/pull/7554) (Weijun-H)
- replace ptree with termtree [#7560](https://github.com/apache/datafusion/pull/7560) (avantgardnerio)
- Make AvroArrowArrayReader possible to scan Avro backed table which contains nested records [#7525](https://github.com/apache/datafusion/pull/7525) (sarutak)
- Fix a race condition issue on reading spilled file [#7538](https://github.com/apache/datafusion/pull/7538) (sarutak)
- [MINOR]: Add is single method [#7558](https://github.com/apache/datafusion/pull/7558) (mustafasrepo)
- Fix `describe <table>` to work without SessionContext [#7441](https://github.com/apache/datafusion/pull/7441) (alamb)
- Make the tests in SHJ faster [#7543](https://github.com/apache/datafusion/pull/7543) (metesynnada)
- feat: Support spilling for hash aggregation [#7400](https://github.com/apache/datafusion/pull/7400) (kazuyukitanimura)
- Make backtrace as a cargo feature [#7527](https://github.com/apache/datafusion/pull/7527) (comphead)
- Minor: Fix `clippy` by switching to `timestamp_nanos_opt` instead of (deprecated) `timestamp_nanos` [#7572](https://github.com/apache/datafusion/pull/7572) (alamb)
- Update sqllogictest requirement from 0.15.0 to 0.16.0 [#7569](https://github.com/apache/datafusion/pull/7569) (dependabot[bot])
- extract `datafusion-physical-plan` to its own crate [#7432](https://github.com/apache/datafusion/pull/7432) (alamb)
- First and Last Accumulators should update with state row excluding is_set flag [#7565](https://github.com/apache/datafusion/pull/7565) (viirya)
- refactor: simplify code of eliminate_cross_join.rs [#7561](https://github.com/apache/datafusion/pull/7561) (jackwener)
- Update release instructions for datafusion-physical-plan crate [#7576](https://github.com/apache/datafusion/pull/7576) (alamb)
- Minor: Update chrono pin to `0.4.31` [#7575](https://github.com/apache/datafusion/pull/7575) (alamb)
- [feat] Introduce cacheManager in session ctx and make StatisticsCache share in session [#7570](https://github.com/apache/datafusion/pull/7570) (Ted-Jiang)
- Enhance/Refactor Ordering Equivalence Properties [#7566](https://github.com/apache/datafusion/pull/7566) (mustafasrepo)
- fix misplaced statements in sqllogictest [#7586](https://github.com/apache/datafusion/pull/7586) (jonahgao)
- Update substrait requirement from 0.13.1 to 0.14.0 [#7585](https://github.com/apache/datafusion/pull/7585) (dependabot[bot])
- chore: use the `create_udwf` function in `simple_udwf`, consistent with `simple_udf` and `simple_udaf` [#7579](https://github.com/apache/datafusion/pull/7579) (tanruixiang)
- Implement protobuf serialization for AnalyzeExec [#7574](https://github.com/apache/datafusion/pull/7574) (adhish20)
- chore: fix catalog's usage docs error and add docs about `CatalogList` trait [#7582](https://github.com/apache/datafusion/pull/7582) (tanruixiang)
- Implement `CardinalityAwareRowConverter` while doing streaming merge [#7401](https://github.com/apache/datafusion/pull/7401) (JayjeetAtGithub)
- Parallelize Parquet Serialization [#7562](https://github.com/apache/datafusion/pull/7562) (devinjdangelo)
- feat: natively support more data types for the `abs` function. [#7568](https://github.com/apache/datafusion/pull/7568) (jonahgao)
- implement string_to_array [#7577](https://github.com/apache/datafusion/pull/7577) (casperhart)
- Create 2023 Q4 roadmap [#7551](https://github.com/apache/datafusion/pull/7551) (graydenshand)
- chore: reduce `physical-plan` dependencies [#7599](https://github.com/apache/datafusion/pull/7599) (crepererum)
- Minor: add githubs start/fork buttons to documentation page [#7588](https://github.com/apache/datafusion/pull/7588) (alamb)
- Minor: add more examples for `CREATE EXTERNAL TABLE` doc [#7594](https://github.com/apache/datafusion/pull/7594) (comphead)
- Update nix requirement from 0.26.1 to 0.27.1 [#7438](https://github.com/apache/datafusion/pull/7438) (dependabot[bot])
- Update sqllogictest requirement from 0.16.0 to 0.17.0 [#7606](https://github.com/apache/datafusion/pull/7606) (dependabot[bot])
- Fix panic in TopK [#7609](https://github.com/apache/datafusion/pull/7609) (avantgardnerio)
- Move `FileCompressionType` out of `common` and into `core` [#7596](https://github.com/apache/datafusion/pull/7596) (haohuaijin)
- Expose contents of Constraints [#7603](https://github.com/apache/datafusion/pull/7603) (tv42)
- Change the unbounded_output API default [#7605](https://github.com/apache/datafusion/pull/7605) (metesynnada)
- feat: Parallel collecting parquet files statistics #7573 [#7595](https://github.com/apache/datafusion/pull/7595) (hengfeiyang)
- Support hashing List columns [#7616](https://github.com/apache/datafusion/pull/7616) (jonmmease)
- [MINOR] Make the sink input aware of its plan [#7610](https://github.com/apache/datafusion/pull/7610) (metesynnada)
- [MINOR] Reduce complexity on SHJ [#7607](https://github.com/apache/datafusion/pull/7607) (metesynnada)
- feat: Better large output display in datafusion-cli with --maxrows option [#7617](https://github.com/apache/datafusion/pull/7617) (2010YOUY01)
- Minor: add examples for `arrow_cast` and `arrow_typeof` to user guide [#7615](https://github.com/apache/datafusion/pull/7615) (alamb)
- [MINOR]: Fix stack overflow bug for get field access expr [#7623](https://github.com/apache/datafusion/pull/7623) (mustafasrepo)
- Group By All [#7622](https://github.com/apache/datafusion/pull/7622) (berkaysynnada)
- Implement protobuf serialization for `(Bounded)WindowAggExec`. [#7557](https://github.com/apache/datafusion/pull/7557) (vrongmeal)
- Make it possible to compile datafusion-common without default features [#7625](https://github.com/apache/datafusion/pull/7625) (jonmmease)
- Minor: Adding backtrace documentation [#7628](https://github.com/apache/datafusion/pull/7628) (comphead)
- fix(5975/5976): timezone handling for timestamps and `date_trunc`, `date_part` and `date_bin` [#7614](https://github.com/apache/datafusion/pull/7614) (wiedld)
- Minor: remove unecessary `Arc`s in datetime_expressions [#7630](https://github.com/apache/datafusion/pull/7630) (alamb)
- fix: check for precision overflow when parsing float as decimal [#7627](https://github.com/apache/datafusion/pull/7627) (jonahgao)
- Update arrow 47.0.0 in DataFusion [#7587](https://github.com/apache/datafusion/pull/7587) (tustvold)
- Add test crate to compile DataFusion with wasm-pack [#7633](https://github.com/apache/datafusion/pull/7633) (jonmmease)
- Minor: Update documentation of case expression [#7646](https://github.com/apache/datafusion/pull/7646) (ongchi)
- Minor: improve docstrings on `SessionState` [#7654](https://github.com/apache/datafusion/pull/7654) (alamb)
- Update example in the DataFrame documentation. [#7650](https://github.com/apache/datafusion/pull/7650) (jsimpson-gro)
- Add HTTP object store example [#7602](https://github.com/apache/datafusion/pull/7602) (pka)
- feat: make parse_float_as_decimal work on negative numbers [#7648](https://github.com/apache/datafusion/pull/7648) (jonahgao)
- Minor: add doc comments to `ExtractEquijoinPredicate` [#7658](https://github.com/apache/datafusion/pull/7658) (alamb)
- [MINOR]: Do not add unnecessary hash repartition to the physical plan [#7667](https://github.com/apache/datafusion/pull/7667) (mustafasrepo)
- Minor: add ticket references to parallel parquet writing code [#7592](https://github.com/apache/datafusion/pull/7592) (alamb)
- Minor: Add ticket reference and add test comment [#7593](https://github.com/apache/datafusion/pull/7593) (alamb)
- Support Avro's Enum type and Fixed type [#7635](https://github.com/apache/datafusion/pull/7635) (sarutak)
- Minor: Migrate datafusion-proto tests into it own binary [#7668](https://github.com/apache/datafusion/pull/7668) (ongchi)
- Upgrade apache-avro to 0.16 [#7674](https://github.com/apache/datafusion/pull/7674) (sarutak)
- Move window analysis to the window method [#7672](https://github.com/apache/datafusion/pull/7672) (mustafasrepo)
- Don't add filters to projection in TableScan [#7670](https://github.com/apache/datafusion/pull/7670) (Dandandan)
- Minor: Improve `TableProviderFilterPushDown` docs [#7685](https://github.com/apache/datafusion/pull/7685) (alamb)
- FIX: Test timestamp with table [#7701](https://github.com/apache/datafusion/pull/7701) (jayzhan211)
- Fix bug in `SimplifyExpressions` [#7699](https://github.com/apache/datafusion/pull/7699) (Dandandan)
- Enhance Enforce Dist capabilities to fix, sub optimal bad plans [#7671](https://github.com/apache/datafusion/pull/7671) (mustafasrepo)
- docs: add section on supports_filters_pushdown [#7680](https://github.com/apache/datafusion/pull/7680) (tshauck)
- Improve cache usage in CI [#7678](https://github.com/apache/datafusion/pull/7678) (sarutak)
- fix: substrait limit when fetch is None [#7669](https://github.com/apache/datafusion/pull/7669) (waynexia)
- minor: revert parsing precedence between Aggr and UDAF [#7682](https://github.com/apache/datafusion/pull/7682) (waynexia)
- Minor: Move hash utils to common [#7684](https://github.com/apache/datafusion/pull/7684) (jayzhan211)
- Update Default Parquet Write Compression [#7692](https://github.com/apache/datafusion/pull/7692) (devinjdangelo)
- Stop using cache for the benchmark job [#7706](https://github.com/apache/datafusion/pull/7706) (sarutak)
- Change rust.yml to run benchmark [#7708](https://github.com/apache/datafusion/pull/7708) (sarutak)
- Extend infer_placeholder_types to support BETWEEN predicates [#7703](https://github.com/apache/datafusion/pull/7703) (andrelmartins)
- Minor: Add comment explaining why verify benchmark results uses release mode [#7712](https://github.com/apache/datafusion/pull/7712) (alamb)
- Support all the codecs supported by Avro [#7718](https://github.com/apache/datafusion/pull/7718) (sarutak)
- Update substrait requirement from 0.14.0 to 0.15.0 [#7719](https://github.com/apache/datafusion/pull/7719) (dependabot[bot])
- fix: coerce text to timestamps with timezones [#7720](https://github.com/apache/datafusion/pull/7720) (mhilton)
- Add LanceDB to the list of Known Users [#7716](https://github.com/apache/datafusion/pull/7716) (alamb)
- Enable avro reading/writing in datafusion-cli [#7715](https://github.com/apache/datafusion/pull/7715) (alamb)
- Document crate feature flags [#7713](https://github.com/apache/datafusion/pull/7713) (alamb)
- Minor: Consolidate UDF tests [#7704](https://github.com/apache/datafusion/pull/7704) (alamb)
- Minor: fix CI failure due to Cargo.lock in datafusioncli [#7733](https://github.com/apache/datafusion/pull/7733) (yjshen)
- MINOR: change file to column index in page_filter trace log [#7730](https://github.com/apache/datafusion/pull/7730) (mapleFU)
- preserve array type / timezone in `date_bin` and `date_trunc` functions [#7729](https://github.com/apache/datafusion/pull/7729) (mhilton)
- Remove redundant is_numeric for DataType [#7734](https://github.com/apache/datafusion/pull/7734) (qrilka)
- fix: avro_to_arrow: Handle avro nested nullable struct (union) [#7663](https://github.com/apache/datafusion/pull/7663) (Samrose-Ahmed)
- Rename `SessionContext::with_config_rt` to `SessionContext::new_with_config_from_rt`, etc [#7631](https://github.com/apache/datafusion/pull/7631) (alamb)
- Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [#7723](https://github.com/apache/datafusion/pull/7723) (alamb)
- Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator [#7721](https://github.com/apache/datafusion/pull/7721) (Dandandan)
- Minor: Improve crate docs [#7740](https://github.com/apache/datafusion/pull/7740) (alamb)
- [MINOR]: Resolve linter errors in the main [#7753](https://github.com/apache/datafusion/pull/7753) (mustafasrepo)
- Minor: Build concat_internal() with ListArray construction instead of ArrayData [#7748](https://github.com/apache/datafusion/pull/7748) (jayzhan211)
- Minor: Add comment on input_schema from AggregateExec [#7727](https://github.com/apache/datafusion/pull/7727) (viirya)
- Fix column name for COUNT(\*) set by AggregateStatistics [#7757](https://github.com/apache/datafusion/pull/7757) (qrilka)
- Add documentation about type signatures, and export `TIMEZONE_WILDCARD` [#7726](https://github.com/apache/datafusion/pull/7726) (alamb)
- [feat] Support cache ListFiles result cache in session level [#7620](https://github.com/apache/datafusion/pull/7620) (Ted-Jiang)
- Support `SHOW ALL VERBOSE` to show settings description [#7735](https://github.com/apache/datafusion/pull/7735) (comphead)
