<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

## [17.0.0](https://github.com/apache/datafusion/tree/17.0.0) (2023-01-27)

[Full Changelog](https://github.com/apache/datafusion/compare/17.0.0-rc1...17.0.0)

**Breaking changes:**

- Implemented a ReadOptions trait for cleaner code. [\#5025](https://github.com/apache/datafusion/pull/5025) ([saikrishna1-bidgely](https://github.com/saikrishna1-bidgely))

**Implemented enhancements:**

- Add null-equals-null JOIN support in Substrait producer/consumer [\#5084](https://github.com/apache/datafusion/issues/5084)
- Cleaner code for Read Options in reader methdos. [\#5024](https://github.com/apache/datafusion/issues/5024)
- Substrait donation follow-on work [\#4897](https://github.com/apache/datafusion/issues/4897)
- Add `len` method to `DataFrame` [\#1926](https://github.com/apache/datafusion/issues/1926)

**Fixed bugs:**

- Clippy failures in master branch and in PRs \(due to new nightly Rust\) [\#5080](https://github.com/apache/datafusion/issues/5080)

**Merged pull requests:**

- Add null-equals-null join support [\#5085](https://github.com/apache/datafusion/pull/5085) ([nseekhao](https://github.com/nseekhao))
- Optimize returned plan in roundtrip_fill_na function [\#5083](https://github.com/apache/datafusion/pull/5083) ([nseekhao](https://github.com/nseekhao))
- fix clippy failures [\#5081](https://github.com/apache/datafusion/pull/5081) [[sql](https://github.com/apache/datafusion/labels/sql)] ([andygrove](https://github.com/andygrove))
- Add NULL literal support for decimal and integers [\#5077](https://github.com/apache/datafusion/pull/5077) ([nseekhao](https://github.com/nseekhao))
- DataFrame count method [\#5071](https://github.com/apache/datafusion/pull/5071) ([Jefffrey](https://github.com/Jefffrey))
- \[sqllogictests\] Port orderby.rs to sqllogictests [\#5062](https://github.com/apache/datafusion/pull/5062) ([alamb](https://github.com/alamb))

## [17.0.0-rc1](https://github.com/apache/datafusion/tree/17.0.0-rc1) (2023-01-26)

[Full Changelog](https://github.com/apache/datafusion/compare/16.1.0...17.0.0-rc1)

**Breaking changes:**

- Change ExecutionPlan::maintains_input_order to return vector \(to support multi children executors better\) [\#5035](https://github.com/apache/datafusion/pull/5035) ([mustafasrepo](https://github.com/mustafasrepo))
- Allow overriding error type in DataFusion Result [\#5000](https://github.com/apache/datafusion/pull/5000) ([tustvold](https://github.com/tustvold))
- Add dictionary_expresions feature \(\#4386\) [\#4999](https://github.com/apache/datafusion/pull/4999) ([tustvold](https://github.com/tustvold))

**Implemented enhancements:**

- Retain the ordering of fields in the table schema when creating the projection for an update plan [\#5052](https://github.com/apache/datafusion/issues/5052)
- \[sqllogictest\] Remove `integration-tests` directory [\#5011](https://github.com/apache/datafusion/issues/5011)
- \[sqllogictest\] Consolidate normalization code for the postgres and non-postgres paths [\#5010](https://github.com/apache/datafusion/issues/5010)
- \[sqllogictest\] Don't orchestrate the postgres containers with rust / docker [\#5009](https://github.com/apache/datafusion/issues/5009)
- check external table exist before creating a table [\#4997](https://github.com/apache/datafusion/issues/4997)
- Implement `std::error::Error` for DataFusionError [\#4991](https://github.com/apache/datafusion/issues/4991)
- Return Vec\<bool\> instead of bool in ExecutionPlan::maintains_input_order [\#4980](https://github.com/apache/datafusion/issues/4980)
- Add support for linear range search [\#4979](https://github.com/apache/datafusion/issues/4979)
- Add support for bounded execution when window query involves UNBOUNDED PRECEDING [\#4978](https://github.com/apache/datafusion/issues/4978)
- Infer prepared statement parameter types for insert queries with values clauses [\#4976](https://github.com/apache/datafusion/issues/4976)
- The filter of outer table happens multiple time after optimizing in-subquery to join [\#4914](https://github.com/apache/datafusion/issues/4914)
- Support Describe FILE in datafusion-cli [\#4913](https://github.com/apache/datafusion/issues/4913)
- Release DataFusion 16 [\#4776](https://github.com/apache/datafusion/issues/4776)
- Support writing lists in the arrow csv writer [\#4502](https://github.com/apache/datafusion/issues/4502)
- Replace python based integration test with sqllogictest [\#4462](https://github.com/apache/datafusion/issues/4462)
- Support CREATE TABLE table_name\(...schema_fields\) [\#4396](https://github.com/apache/datafusion/issues/4396)
- Make Binary Dictionary Operations Optional [\#4386](https://github.com/apache/datafusion/issues/4386)
- Improve / Cleanup DataFusion CI [\#3045](https://github.com/apache/datafusion/issues/3045)
- More frequent DataFusion releases to crates.io \(discussion\) [\#2327](https://github.com/apache/datafusion/issues/2327)

**Fixed bugs:**

- UPDATE statment for non existent column doesn't error out [\#5068](https://github.com/apache/datafusion/issues/5068)
- Limit doesn't drop on first batch when limit size == fetch size. [\#5064](https://github.com/apache/datafusion/issues/5064)
- Performance regressions since DataFusion 15.x [\#5060](https://github.com/apache/datafusion/issues/5060)
- Quoted schema and table names result in double-quoted names in logical plan. [\#5058](https://github.com/apache/datafusion/issues/5058)
- Homebrew release script has the amount of arguments being incorrect [\#5043](https://github.com/apache/datafusion/issues/5043)
- CI Failing with Out of Disk [\#5040](https://github.com/apache/datafusion/issues/5040)
- Doc links to LogicalPlan in the core package need updating. [\#5036](https://github.com/apache/datafusion/issues/5036)
- explain analyze can not see csvexec execution time metrics [\#5014](https://github.com/apache/datafusion/issues/5014)
- AVG\(nulls\) returns 0 rather than NULL [\#5007](https://github.com/apache/datafusion/issues/5007)
- Invalid Placeholders return internal error \(rather than Plan error\) [\#5005](https://github.com/apache/datafusion/issues/5005)
- select \* from csv error [\#4996](https://github.com/apache/datafusion/issues/4996)
- Incorrect nested error wrapped to `ArrowError:External` variant for joins [\#4981](https://github.com/apache/datafusion/issues/4981)

**Documentation updates:**

- MINOR: Add Substrait to feature list in README [\#4955](https://github.com/apache/datafusion/pull/4955) ([andygrove](https://github.com/andygrove))
- Minor: comma engineering in Readme [\#4954](https://github.com/apache/datafusion/pull/4954) ([alamb](https://github.com/alamb))
- Update main DataFusion README [\#4903](https://github.com/apache/datafusion/pull/4903) ([alamb](https://github.com/alamb))
- Docs: Add known user - Kamu [\#4899](https://github.com/apache/datafusion/pull/4899) ([sergiimk](https://github.com/sergiimk))

**Closed issues:**

- Support sub directories in sqllogictest runner [\#4709](https://github.com/apache/datafusion/issues/4709)
- Bug displaying fractional seconds in `IntervalMonthDayNano` [\#4220](https://github.com/apache/datafusion/issues/4220)

**Merged pull requests:**

- Add `release-crates.sh` script [\#5070](https://github.com/apache/datafusion/pull/5070) ([iajoiner](https://github.com/iajoiner))
- Validate assignment target column existence for UPDATE statements [\#5069](https://github.com/apache/datafusion/pull/5069) [[sql](https://github.com/apache/datafusion/labels/sql)] ([gruuya](https://github.com/gruuya))
- Fix limit when size of batch to poll == skip/fetch value [\#5066](https://github.com/apache/datafusion/pull/5066) ([Dandandan](https://github.com/Dandandan))
- Fix CREATE SCHEMA schema name double quoting issue. [\#5059](https://github.com/apache/datafusion/pull/5059) [[sql](https://github.com/apache/datafusion/labels/sql)] ([neumark](https://github.com/neumark))
- Minor: Move some aggregate error tests to sqllogictests [\#5055](https://github.com/apache/datafusion/pull/5055) ([alamb](https://github.com/alamb))
- Add decimal support to substrait serde [\#5054](https://github.com/apache/datafusion/pull/5054) ([andygrove](https://github.com/andygrove))
- Retain schema order in projection [\#5053](https://github.com/apache/datafusion/pull/5053) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- Improve join type support in substrait [\#5051](https://github.com/apache/datafusion/pull/5051) ([andygrove](https://github.com/andygrove))
- \[Substrait\] ReadRel. Get column names from TableScan source [\#5050](https://github.com/apache/datafusion/pull/5050) ([andygrove](https://github.com/andygrove))
- Ensure insert projections are of correct type [\#5049](https://github.com/apache/datafusion/pull/5049) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- Remove unnecessary pyo3 dependency from datafusion crate [\#5048](https://github.com/apache/datafusion/pull/5048) ([tustvold](https://github.com/tustvold))
- Cleanup CI \(\#5040\) [\#5047](https://github.com/apache/datafusion/pull/5047) ([tustvold](https://github.com/tustvold))
- Fix homebrew publish script [\#5044](https://github.com/apache/datafusion/pull/5044) ([iajoiner](https://github.com/iajoiner))
- Update docs links to logical plans module. [\#5037](https://github.com/apache/datafusion/pull/5037) ([vincev](https://github.com/vincev))
- \[sqllogictest\] Read subdirectories in `test_files` [\#5033](https://github.com/apache/datafusion/pull/5033) ([melgenek](https://github.com/melgenek))
- minor: Fix docs for create_default_catalog_and_schema [\#5032](https://github.com/apache/datafusion/pull/5032) ([alamb](https://github.com/alamb))
- Remove python based posgres comparsion `integration-test` [\#5031](https://github.com/apache/datafusion/pull/5031) ([alamb](https://github.com/alamb))
- \[sqllogictest\] Create empty tables [\#5026](https://github.com/apache/datafusion/pull/5026) [[sql](https://github.com/apache/datafusion/labels/sql)] ([melgenek](https://github.com/melgenek))
- Simplify the `PushDownLimit`. [\#5021](https://github.com/apache/datafusion/pull/5021) ([HaoYang670](https://github.com/HaoYang670))
- \[BugFix\] fix explain csv/json/avro exec can not see metrics bug [\#5018](https://github.com/apache/datafusion/pull/5018) ([xiaoyong-z](https://github.com/xiaoyong-z))
- Check placeholder \_\_timeTo and return Datafusion::Plan error [\#5017](https://github.com/apache/datafusion/pull/5017) [[sql](https://github.com/apache/datafusion/labels/sql)] ([matthias-Q](https://github.com/matthias-Q))
- \[sqllogictets\] Remove postgres container orchestration [\#5015](https://github.com/apache/datafusion/pull/5015) ([alamb](https://github.com/alamb))
- Sqllogictest: use the same normalization for all tests [\#5013](https://github.com/apache/datafusion/pull/5013) ([melgenek](https://github.com/melgenek))
- Minor: Remove invalid comments [\#5012](https://github.com/apache/datafusion/pull/5012) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- AVG\(null\) is NULL \(not zero\) [\#5008](https://github.com/apache/datafusion/pull/5008) ([alamb](https://github.com/alamb))
- Minor: improve internal error message [\#5006](https://github.com/apache/datafusion/pull/5006) ([alamb](https://github.com/alamb))
- Support for bounded execution when window frame involves UNBOUNDED PRECEDING [\#5003](https://github.com/apache/datafusion/pull/5003) ([mustafasrepo](https://github.com/mustafasrepo))
- Bump sqllogictest to v0.11.1 [\#5002](https://github.com/apache/datafusion/pull/5002) ([xudong963](https://github.com/xudong963))
- Minor: Document how to create `ListingTables` [\#5001](https://github.com/apache/datafusion/pull/5001) ([alamb](https://github.com/alamb))
- \[Enhancement\] early check table exist before create [\#4998](https://github.com/apache/datafusion/pull/4998) ([xiaoyong-z](https://github.com/xiaoyong-z))
- \[Feature\] support describe file [\#4995](https://github.com/apache/datafusion/pull/4995) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xiaoyong-z](https://github.com/xiaoyong-z))
- Implement `std::error::Error::source()` for `DataFusionError`, make `DataFusionError::find_root` more generic [\#4992](https://github.com/apache/datafusion/pull/4992) ([alamb](https://github.com/alamb))
- Add support for linear range calculation in WINDOW functions [\#4989](https://github.com/apache/datafusion/pull/4989) ([mustafasrepo](https://github.com/mustafasrepo))
- re-export substrait crate [\#4988](https://github.com/apache/datafusion/pull/4988) ([jdye64](https://github.com/jdye64))
- minor: Update data type support documentation [\#4984](https://github.com/apache/datafusion/pull/4984) ([alamb](https://github.com/alamb))
- fix\(4981\): incorrect error wrapping in `OnceFut` [\#4983](https://github.com/apache/datafusion/pull/4983) ([DDtKey](https://github.com/DDtKey))
- Infer values for inserts [\#4977](https://github.com/apache/datafusion/pull/4977) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- Simplify GroupByHash implementation \(to prepare for more work\) [\#4972](https://github.com/apache/datafusion/pull/4972) ([alamb](https://github.com/alamb))
- Add DataFusionError::Substrait variant to DataFusionError enum [\#4971](https://github.com/apache/datafusion/pull/4971) ([jdye64](https://github.com/jdye64))
- refactor: display input partitions for `RepartitionExec` [\#4969](https://github.com/apache/datafusion/pull/4969) ([crepererum](https://github.com/crepererum))
- Upgrade to Substrait 0.4.0 [\#4966](https://github.com/apache/datafusion/pull/4966) ([mbrobbel](https://github.com/mbrobbel))
- Expose `sql_to_statement` and `statement_to_plan` on `SessionState` [\#4958](https://github.com/apache/datafusion/pull/4958) ([avantgardnerio](https://github.com/avantgardnerio))
- Minor: Make messages consistent for LogicalPlan::Dml [\#4953](https://github.com/apache/datafusion/pull/4953) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- Do not resort inputs to `UnionExec` if they are already sorted [\#4946](https://github.com/apache/datafusion/pull/4946) ([alamb](https://github.com/alamb))
- Minor: Reduce even more redundancy creating window_agg in sort_enforcement tests [\#4945](https://github.com/apache/datafusion/pull/4945) ([alamb](https://github.com/alamb))
- Only add outer filter once when transforming exists/in subquery to join [\#4944](https://github.com/apache/datafusion/pull/4944) ([ygf11](https://github.com/ygf11))
- fix: `FieldNotFound` error message without valid fields [\#4942](https://github.com/apache/datafusion/pull/4942) [[sql](https://github.com/apache/datafusion/labels/sql)] ([DDtKey](https://github.com/DDtKey))
- Propagate planning error back to user [\#4940](https://github.com/apache/datafusion/pull/4940) ([fsdvh](https://github.com/fsdvh))
- Make it able to specify a session id for SessionState [\#4933](https://github.com/apache/datafusion/pull/4933) ([yahoNanJing](https://github.com/yahoNanJing))
- SUPPORT SEMI/ANTI JOIN SQL syntax in DataFusion [\#4932](https://github.com/apache/datafusion/pull/4932) [[sql](https://github.com/apache/datafusion/labels/sql)] ([mingmwang](https://github.com/mingmwang))
- Support gs:// as GCS schema [\#4930](https://github.com/apache/datafusion/pull/4930) ([jychen7](https://github.com/jychen7))
- Upgrade object_store from 0.5.0 to 0.5.3 [\#4929](https://github.com/apache/datafusion/pull/4929) ([jychen7](https://github.com/jychen7))
- Reduce redundancy in sort_enforcement tests [\#4928](https://github.com/apache/datafusion/pull/4928) ([alamb](https://github.com/alamb))
- Update to arrow 31 [\#4927](https://github.com/apache/datafusion/pull/4927) [[sql](https://github.com/apache/datafusion/labels/sql)] ([tustvold](https://github.com/tustvold))
- Unify Row hash and hash implementation [\#4924](https://github.com/apache/datafusion/pull/4924) ([mustafasrepo](https://github.com/mustafasrepo))
- Support join-filter pushdown for semi/anti join [\#4923](https://github.com/apache/datafusion/pull/4923) ([ygf11](https://github.com/ygf11))
- Minor add ticket link to broken test [\#4919](https://github.com/apache/datafusion/pull/4919) ([alamb](https://github.com/alamb))
- Improve documentation for ExprVisitor, port simple uses to new walking function [\#4916](https://github.com/apache/datafusion/pull/4916) ([alamb](https://github.com/alamb))
- Add substrait label to PRs [\#4915](https://github.com/apache/datafusion/pull/4915) ([alamb](https://github.com/alamb))
- Executing ProjectionExec with no column should not return an Err [\#4912](https://github.com/apache/datafusion/pull/4912) ([viirya](https://github.com/viirya))
- Refactor: `Add LogicalPlan::observe_expressions` to walk expressions [\#4906](https://github.com/apache/datafusion/pull/4906) ([alamb](https://github.com/alamb))
- Minor: Port information schema tests to sqllogictest [\#4905](https://github.com/apache/datafusion/pull/4905) ([alamb](https://github.com/alamb))
- Add insert/update/delete to LogicalPlan and add SQL planner support [\#4902](https://github.com/apache/datafusion/pull/4902) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- fix: Visit subqueries in `Expr::Alias` [\#4900](https://github.com/apache/datafusion/pull/4900) ([askoa](https://github.com/askoa))
- \[Substrait\] Change API to return LogicalPlan instead of DataFrame [\#4896](https://github.com/apache/datafusion/pull/4896) ([andygrove](https://github.com/andygrove))
- Upgrade to substrait 0.3 [\#4895](https://github.com/apache/datafusion/pull/4895) ([andygrove](https://github.com/andygrove))
- Add datafusion-substrait crate to workspace [\#4893](https://github.com/apache/datafusion/pull/4893) ([andygrove](https://github.com/andygrove))
- refactor and add simple function to deserialize and serialize proto b… [\#4892](https://github.com/apache/datafusion/pull/4892) ([jdye64](https://github.com/jdye64))
- Update `optimize_children` to return `Result<Option<LogicalPlan>>` [\#4888](https://github.com/apache/datafusion/pull/4888) ([HaoYang670](https://github.com/HaoYang670))
- Do not repartition inputs whose sort order is required [\#4885](https://github.com/apache/datafusion/pull/4885) ([alamb](https://github.com/alamb))
- Minor: Add docstrings to UnionExec [\#4884](https://github.com/apache/datafusion/pull/4884) ([alamb](https://github.com/alamb))
- Update datafusion-substrait crate to build against repo version of DataFusion [\#4879](https://github.com/apache/datafusion/pull/4879) ([andygrove](https://github.com/andygrove))
- Fix column indices in EnforceDistribution optimizer in Partial AggregateMode [\#4878](https://github.com/apache/datafusion/pull/4878) ([jonmmease](https://github.com/jonmmease))
- refactor: improve repartition buffering [\#4867](https://github.com/apache/datafusion/pull/4867) ([crepererum](https://github.com/crepererum))
- Rewrite coerce_plan_expr_for_schema to fix union type coercion [\#4862](https://github.com/apache/datafusion/pull/4862) ([ygf11](https://github.com/ygf11))
- \(\#4462\) Postgres compatibility tests using sqllogictest [\#4834](https://github.com/apache/datafusion/pull/4834) ([melgenek](https://github.com/melgenek))
- Support non-tuple expression for in-subquery to join [\#4826](https://github.com/apache/datafusion/pull/4826) ([ygf11](https://github.com/ygf11))
- Update to arrow `30.0.1` [\#4818](https://github.com/apache/datafusion/pull/4818) [[sql](https://github.com/apache/datafusion/labels/sql)] ([tustvold](https://github.com/tustvold))
- Refine the statistics estimation for the limit and aggregate operator [\#4716](https://github.com/apache/datafusion/pull/4716) ([yahoNanJing](https://github.com/yahoNanJing))
- Infer prepared statement parameter types [\#4701](https://github.com/apache/datafusion/pull/4701) [[sql](https://github.com/apache/datafusion/labels/sql)] ([avantgardnerio](https://github.com/avantgardnerio))
- Add datafusion-substrait crate [\#4543](https://github.com/apache/datafusion/pull/4543) ([andygrove](https://github.com/andygrove))
- Refactor loser tree code in SortPreservingMerge per PR comments [\#4407](https://github.com/apache/datafusion/pull/4407) ([alamb](https://github.com/alamb))
