<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

## [18.0.0](https://github.com/apache/datafusion/tree/18.0.0) (2023-02-10)

[Full Changelog](https://github.com/apache/datafusion/compare/17.0.0...18.0.0)

**Breaking changes:**

- Use DataFusionError instead of ArrowError in SendableRecordBatchStream [\#5101](https://github.com/apache/datafusion/pull/5101) ([comphead](https://github.com/comphead))
- Update to arrow 32 and Switch to RawDecoder for JSON [\#5056](https://github.com/apache/datafusion/pull/5056) [[sql](https://github.com/apache/datafusion/labels/sql)] ([tustvold](https://github.com/tustvold))

**Implemented enhancements:**

- DiskManager to create a spill folder if doesn't exist [\#5186](https://github.com/apache/datafusion/issues/5186)
- cast expression may cause duplicate column name error [\#5174](https://github.com/apache/datafusion/issues/5174)
- Add type coercion from Dictionary to string for regular expressions [\#5154](https://github.com/apache/datafusion/issues/5154)
- Unnecessary `Filter` on Parquet datasources [\#5149](https://github.com/apache/datafusion/issues/5149)
- \[sqllogictest\] Support `pg_typeof` for Postgres compatibility tests [\#5147](https://github.com/apache/datafusion/issues/5147)
- Supporting Grafana global variables [\#5144](https://github.com/apache/datafusion/issues/5144)
- add example for standalone DataFusion server which supports Arrow Flight SQL JDBC driver [\#5139](https://github.com/apache/datafusion/issues/5139)
- Support for InList in datafusion-substrait [\#5134](https://github.com/apache/datafusion/issues/5134)
- Pipeline file opening in `FileStream` [\#5129](https://github.com/apache/datafusion/issues/5129)
- Make `parse_physical_expr` public [\#5107](https://github.com/apache/datafusion/issues/5107)
- Use DataFusionError in SendableRecordBatchStream [\#5039](https://github.com/apache/datafusion/issues/5039)
- Interval coercion:`date_bin('1 hour',...)` does not work but `date_bin(interval '1 hour', ...` does [\#4853](https://github.com/apache/datafusion/issues/4853)
- `Explain <query>` should not fail if meeting errors when optimizing the query [\#4766](https://github.com/apache/datafusion/issues/4766)
- Add option to determine whether to convert identifiers [\#4551](https://github.com/apache/datafusion/issues/4551)
- Replace `&Option<T>` with `Option<&T>`. [\#4424](https://github.com/apache/datafusion/issues/4424)
- Error type in `RecordBatchStream` [\#4172](https://github.com/apache/datafusion/issues/4172)
- Support non-equi join \(e.g. `ON` clause\) in Dataframe API [\#1254](https://github.com/apache/datafusion/issues/1254)
- Allow ParquetExec to parallelize work based on row groups [\#137](https://github.com/apache/datafusion/issues/137)

**Fixed bugs:**

- Confusing schema errors when using window partition [\#5229](https://github.com/apache/datafusion/issues/5229)
- Propagating empty_relation generate an illegal plan [\#5218](https://github.com/apache/datafusion/issues/5218)
- The test `in_list_types_struct_literal` fails when setting `skip_failed_rules` as `false` [\#5217](https://github.com/apache/datafusion/issues/5217)
- Placeholder values are not replaced in ScalarSubqueries [\#5215](https://github.com/apache/datafusion/issues/5215)
- Querying against delta lake table does not seem to work [\#5202](https://github.com/apache/datafusion/issues/5202)
- Arithmetic operation doesn't work with DictionaryArray [\#5193](https://github.com/apache/datafusion/issues/5193)
- simplify_expr\(\) invoke nullable\(\) exist bug [\#5191](https://github.com/apache/datafusion/issues/5191)
- CI is currently broken on git diff: Not a git repository [\#5180](https://github.com/apache/datafusion/issues/5180)
- `write_csv/json/parquet` isn't cancel safe [\#5178](https://github.com/apache/datafusion/issues/5178)
- no hyperlink to blaze-rs \[doc: README-"Use Cases"\] [\#5175](https://github.com/apache/datafusion/issues/5175)
- Arithmetic scalar operation doesn't work with DictionaryArray [\#5150](https://github.com/apache/datafusion/issues/5150)
- Sort operator disappear in physical_plan [\#5100](https://github.com/apache/datafusion/issues/5100)
- Window function error: InvalidArgumentError\("number of columns\(27\) must match number of fields\(35\) in schema" [\#5090](https://github.com/apache/datafusion/issues/5090)
- `INSERT` statements without target column list are not working [\#5078](https://github.com/apache/datafusion/issues/5078)
- fix file stream time scanning metrics bug [\#5019](https://github.com/apache/datafusion/issues/5019)
- Date before `1678` causes panic [\#4875](https://github.com/apache/datafusion/issues/4875)
- Can not ORDER BY an aliased group column [\#4854](https://github.com/apache/datafusion/issues/4854)
- The `filters` expressions in `TableScan` may contain fields not included in `schema`. [\#4793](https://github.com/apache/datafusion/issues/4793)
- Comparing a `Timestamp` to a `Date32` fails [\#4644](https://github.com/apache/datafusion/issues/4644)
- String --\> TableReference parsing does not properly handle `"` and `.` [\#4532](https://github.com/apache/datafusion/issues/4532)
- can't compare NULL type with NULL type [\#4335](https://github.com/apache/datafusion/issues/4335)
- Add ambiguous check when generate selection plan [\#4196](https://github.com/apache/datafusion/issues/4196)
- Internal error in CAST from Timestamp\[us\] [\#3922](https://github.com/apache/datafusion/issues/3922)
- Run median expr on parquet file column got error [\#3805](https://github.com/apache/datafusion/issues/3805)
- aliasing a field renders it missing in the order by clause [\#669](https://github.com/apache/datafusion/issues/669)
- Querying datetime data in DataFusion with an embedded timezone always fails [\#153](https://github.com/apache/datafusion/issues/153)

**Documentation updates:**

- Update README.md fix \[welcoming community\] links [\#5232](https://github.com/apache/datafusion/pull/5232) ([jiangzhx](https://github.com/jiangzhx))
- Update README.md update blaze-rs link to https://github.com/blaze-init/blaze [\#5190](https://github.com/apache/datafusion/pull/5190) ([jiangzhx](https://github.com/jiangzhx))
- Typo of greptimedb [\#5103](https://github.com/apache/datafusion/pull/5103) ([fengjiachun](https://github.com/fengjiachun))
- chore: change `DataBend` to `Databend` [\#5096](https://github.com/apache/datafusion/pull/5096) ([xudong963](https://github.com/xudong963))

**Closed issues:**

- Change coerced type for comparison between timestamp with date to timestamp [\#4761](https://github.com/apache/datafusion/issues/4761)

**Merged pull requests:**

- fix: correct expected error in test [\#5224](https://github.com/apache/datafusion/pull/5224) ([jackwener](https://github.com/jackwener))
- bugfix: fix propagating empty_relation generates an illegal plan [\#5219](https://github.com/apache/datafusion/pull/5219) ([yukkit](https://github.com/yukkit))
- Replace placeholders in ScalarSubqueries [\#5216](https://github.com/apache/datafusion/pull/5216) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- Dataframe join_on method [\#5210](https://github.com/apache/datafusion/pull/5210) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jefffrey](https://github.com/Jefffrey))
- bugfix: fix eval `nullalbe()` in `simplify_exprs` [\#5208](https://github.com/apache/datafusion/pull/5208) ([jackwener](https://github.com/jackwener))
- minor: remove unnecessary clone [\#5207](https://github.com/apache/datafusion/pull/5207) ([Ted-Jiang](https://github.com/Ted-Jiang))
- minor: extract `merge_schema()` function. [\#5203](https://github.com/apache/datafusion/pull/5203) ([jackwener](https://github.com/jackwener))
- minor: remove unnecessary `continue` [\#5200](https://github.com/apache/datafusion/pull/5200) ([xiaoyong-z](https://github.com/xiaoyong-z))
- fix\(MemTable\): make it cancel-safe and fix parallelism [\#5197](https://github.com/apache/datafusion/pull/5197) ([DDtKey](https://github.com/DDtKey))
- fix: make `write_csv/json/parquet` cancel-safe [\#5196](https://github.com/apache/datafusion/pull/5196) ([DDtKey](https://github.com/DDtKey))
- Support arithmetic operation on DictionaryArray [\#5194](https://github.com/apache/datafusion/pull/5194) ([viirya](https://github.com/viirya))
- sqllogicaltest: add cleanup and use rowsort. [\#5189](https://github.com/apache/datafusion/pull/5189) ([jackwener](https://github.com/jackwener))
- bugfix: fix `TableScan` may contain fields not included in `schema` [\#5188](https://github.com/apache/datafusion/pull/5188) ([jackwener](https://github.com/jackwener))
- Create disk manager spill folder if doesn't exist [\#5185](https://github.com/apache/datafusion/pull/5185) ([comphead](https://github.com/comphead))
- Parse identifiers properly for TableReferences [\#5183](https://github.com/apache/datafusion/pull/5183) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jefffrey](https://github.com/Jefffrey))
- Fix decimal scalar dyn kernels [\#5179](https://github.com/apache/datafusion/pull/5179) ([viirya](https://github.com/viirya))
- Patch git Safe Paths in CI [\#5177](https://github.com/apache/datafusion/pull/5177) ([tustvold](https://github.com/tustvold))
- Add initial support for serializing physical plans with Substrait [\#5176](https://github.com/apache/datafusion/pull/5176) ([andygrove](https://github.com/andygrove))
- Bump tokio from 1.24.1 to 1.24.2 in /datafusion-cli [\#5172](https://github.com/apache/datafusion/pull/5172) ([dependabot[bot]](https://github.com/apps/dependabot))
- Make EnforceSorting global sort aware, fix sort mis-optimizations involving unions, support parallel sort + merge transformations [\#5171](https://github.com/apache/datafusion/pull/5171) ([mustafasrepo](https://github.com/mustafasrepo))
- Update substrait README.md [\#5168](https://github.com/apache/datafusion/pull/5168) ([jiangzhx](https://github.com/jiangzhx))
- Switch to use sum kernel from arrow-rs for Decimal128 [\#5167](https://github.com/apache/datafusion/pull/5167) ([sunchao](https://github.com/sunchao))
- FileStream: Open next file in parallel while decoding [\#5161](https://github.com/apache/datafusion/pull/5161) ([thinkharderdev](https://github.com/thinkharderdev))
- Fix FairSpillPool try_grow for non-spillable consumers [\#5160](https://github.com/apache/datafusion/pull/5160) ([tustvold](https://github.com/tustvold))
- fix: treat unsupported SQL plans as "not implemented" [\#5159](https://github.com/apache/datafusion/pull/5159) ([crepererum](https://github.com/crepererum))
- Compare NULL types [\#5158](https://github.com/apache/datafusion/pull/5158) ([melgenek](https://github.com/melgenek))
- chore: add object_name_to_table_reference in SqlToRel [\#5155](https://github.com/apache/datafusion/pull/5155) [[sql](https://github.com/apache/datafusion/labels/sql)] ([jiacai2050](https://github.com/jiacai2050))
- Ambiguity check for where selection [\#5153](https://github.com/apache/datafusion/pull/5153) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jefffrey](https://github.com/Jefffrey))
- feat: Type coercion for Dictionary\(\_, \_\) to Utf8 for regex conditions [\#5152](https://github.com/apache/datafusion/pull/5152) ([stuartcarnie](https://github.com/stuartcarnie))
- Support arithmetic scalar operation with DictionaryArray [\#5151](https://github.com/apache/datafusion/pull/5151) ([viirya](https://github.com/viirya))
- \[sqllogictest\] Support `pg_typeof` [\#5148](https://github.com/apache/datafusion/pull/5148) ([melgenek](https://github.com/melgenek))
- Date to Timestamp cast [\#5140](https://github.com/apache/datafusion/pull/5140) ([comphead](https://github.com/comphead))
- add example for Flight SQL server that supports JDBC driver [\#5138](https://github.com/apache/datafusion/pull/5138) ([kmitchener](https://github.com/kmitchener))
- Add in-list test [\#5135](https://github.com/apache/datafusion/pull/5135) ([nseekhao](https://github.com/nseekhao))
- Bug fix: Empty Record Batch handling [\#5131](https://github.com/apache/datafusion/pull/5131) ([mustafasrepo](https://github.com/mustafasrepo))
- Add option to control whether to normalize ident [\#5124](https://github.com/apache/datafusion/pull/5124) [[sql](https://github.com/apache/datafusion/labels/sql)] ([jiacai2050](https://github.com/jiacai2050))
- Make `parse_physical_expr` public [\#5118](https://github.com/apache/datafusion/pull/5118) ([comphead](https://github.com/comphead))
- Support coercing `utf8` to `interval` and `timestamp` \(including arguments to `date_bin`\) [\#5117](https://github.com/apache/datafusion/pull/5117) ([alamb](https://github.com/alamb))
- Fix release issues [\#5116](https://github.com/apache/datafusion/pull/5116) [[sql](https://github.com/apache/datafusion/labels/sql)] ([andygrove](https://github.com/andygrove))
- minor: port date_bin tests to sqllogictests [\#5115](https://github.com/apache/datafusion/pull/5115) ([alamb](https://github.com/alamb))
- Minor: reduce code duplication using `rewrite_expr` [\#5114](https://github.com/apache/datafusion/pull/5114) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- Replace &Option\<T\> with Option\<&T\> [\#5113](https://github.com/apache/datafusion/pull/5113) ([gaoxinge](https://github.com/gaoxinge))
- Improve `get_meet_of_orderings` to check for common prefixes [\#5111](https://github.com/apache/datafusion/pull/5111) ([ozankabak](https://github.com/ozankabak))
- \[sqllogictest\] Apply rowsort when there is no explicit order by [\#5110](https://github.com/apache/datafusion/pull/5110) ([melgenek](https://github.com/melgenek))
- Add unnest_column to DataFrame [\#5106](https://github.com/apache/datafusion/pull/5106) ([vincev](https://github.com/vincev))
- Minor: reduce indent level in page filter pruning code [\#5105](https://github.com/apache/datafusion/pull/5105) ([alamb](https://github.com/alamb))
- Replace &Option\<T\> with Option\<&T\> [\#5102](https://github.com/apache/datafusion/pull/5102) ([gaoxinge](https://github.com/gaoxinge))
- Minor: remove unused methods in datafusion/optimizer/src/utils.rs [\#5098](https://github.com/apache/datafusion/pull/5098) ([ygf11](https://github.com/ygf11))
- ci: don't trigger rust ci for doc changes [\#5097](https://github.com/apache/datafusion/pull/5097) ([xudong963](https://github.com/xudong963))
- sqllogicaltest: fix unstable slt case. [\#5095](https://github.com/apache/datafusion/pull/5095) ([jackwener](https://github.com/jackwener))
- chore: update cranelift-module [\#5094](https://github.com/apache/datafusion/pull/5094) ([jackwener](https://github.com/jackwener))
- refactor: Add `rewrite_expr` convenience method for rewriting `Expr`s [\#5092](https://github.com/apache/datafusion/pull/5092) ([alamb](https://github.com/alamb))
- Minor: extract sort col rewrite into its own module, add unit tests [\#5088](https://github.com/apache/datafusion/pull/5088) ([alamb](https://github.com/alamb))
- \[sqllogictest\] Move `decimal.rs` tests [\#5086](https://github.com/apache/datafusion/pull/5086) ([melgenek](https://github.com/melgenek))
- Insert target columns empty fix [\#5079](https://github.com/apache/datafusion/pull/5079) [[sql](https://github.com/apache/datafusion/labels/sql)] ([gruuya](https://github.com/gruuya))
- sqllogicaltest: move union.rs [\#5075](https://github.com/apache/datafusion/pull/5075) ([jackwener](https://github.com/jackwener))
- Support ORDER BY an aliased column [\#5067](https://github.com/apache/datafusion/pull/5067) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- Parquet parallel scan [\#5057](https://github.com/apache/datafusion/pull/5057) ([korowa](https://github.com/korowa))
- \[BugFix\] fix file stream time scanning metrics bug [\#5020](https://github.com/apache/datafusion/pull/5020) ([xiaoyong-z](https://github.com/xiaoyong-z))
- Show optimization errors in explain [\#4819](https://github.com/apache/datafusion/pull/4819) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jefffrey](https://github.com/Jefffrey))
