<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

## [7.0.0](https://github.com/apache/datafusion/tree/7.0.0) (2022-02-14)

[Full Changelog](https://github.com/apache/datafusion/compare/6.0.0...7.0.0)

**Breaking changes:**

- Consolidate various configurations options, remove unrelated `batch_size` [\#1565](https://github.com/apache/datafusion/issues/1565)
- Extract logical plans in LogicalPlan as independent struct [\#1228](https://github.com/apache/datafusion/issues/1228)
- Update `ExecutionPlan` to know about sortedness and repartitioning optimizer pass respect the invariants [\#1776](https://github.com/apache/datafusion/pull/1776) ([alamb](https://github.com/alamb))
- Update to `arrow 8.0.0` [\#1673](https://github.com/apache/datafusion/pull/1673) ([alamb](https://github.com/alamb))
- Remove non idiomatic `DataFusionError::into_arrow_external_error` in favor of From conversion [\#1645](https://github.com/apache/datafusion/pull/1645) ([alamb](https://github.com/alamb))
- Remove `Accumulator::update` and `Accumulator::merge` [\#1582](https://github.com/apache/datafusion/pull/1582) ([Jimexist](https://github.com/Jimexist))
- implement `Hash` for various types and replace `PartialOrd` [\#1580](https://github.com/apache/datafusion/pull/1580) ([Jimexist](https://github.com/Jimexist))
- Replace `DatafusionError` with `GenericError` in `ObjectStore` interface [\#1541](https://github.com/apache/datafusion/pull/1541) ([matthewmturner](https://github.com/matthewmturner))
- Make `FLOAT` SQL type map to `Float32` rather than `Float64` [\#1423](https://github.com/apache/datafusion/pull/1423) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- Map `REAL` SQL type to `Float32` rather than `Float64` to be consistent with pg [\#1390](https://github.com/apache/datafusion/pull/1390) [[sql](https://github.com/apache/datafusion/labels/sql)] ([hntd187](https://github.com/hntd187))

**Implemented enhancements:**

- Create new `datafusion_expr` crate [\#1753](https://github.com/apache/datafusion/issues/1753)
- Create new `datafusion_common` crate [\#1752](https://github.com/apache/datafusion/issues/1752)
- API to get Expr's type and nullability without a `DFSchema` [\#1725](https://github.com/apache/datafusion/issues/1725)
- Cleaner API to create `Expr::ScalarFunction` programatically [\#1718](https://github.com/apache/datafusion/issues/1718)
- Introduce a `Vec<u8>` based row-wise representation for DataFusion [\#1708](https://github.com/apache/datafusion/issues/1708)
- Simplify creating new `ListingTable` [\#1705](https://github.com/apache/datafusion/issues/1705)
- Implement TableProvider for DataFrameImpl to allow registration of logical plans [\#1698](https://github.com/apache/datafusion/issues/1698)
- Public Expr simplification API [\#1694](https://github.com/apache/datafusion/issues/1694)
- Query Optimizer: Add OUTER --\> INNER join conversion [\#1670](https://github.com/apache/datafusion/issues/1670)
- Support reading from CSV, Avro and Json files that have mergeable/compatible, but not identical schemas [\#1669](https://github.com/apache/datafusion/issues/1669)
- Remove `DataFusionError::into_arrow_external_error` in favor of `From` conversion [\#1644](https://github.com/apache/datafusion/issues/1644)
- Include join type in display implementation for logical plan [\#1620](https://github.com/apache/datafusion/issues/1620)
- Switch datafusion to using `eq_dyn_scalar`, etc kernels [\#1610](https://github.com/apache/datafusion/issues/1610)
- Proposal: Remove `Accumulator::update` and `Accumulator::merge` [\#1549](https://github.com/apache/datafusion/issues/1549)
- Replace DataFusionError/Result with impl Error for ObjectStore and Reader [\#1540](https://github.com/apache/datafusion/issues/1540)
- Add `approx_quantile` support [\#1538](https://github.com/apache/datafusion/issues/1538)
- support sorting decimal data type [\#1522](https://github.com/apache/datafusion/issues/1522)
- Keep all datafusion's packages up to date with Dependabot [\#1472](https://github.com/apache/datafusion/issues/1472)
- ExecutionContext support init ExecutionContextState with `new(state: Arc<Mutex<ExecutionContextState>>)` method [\#1439](https://github.com/apache/datafusion/issues/1439)
- support the decimal scalar value [\#1393](https://github.com/apache/datafusion/issues/1393)
- Documentation for using scalar functions with the DataFrame API [\#1364](https://github.com/apache/datafusion/issues/1364)
- Support `boolean == boolean` and `boolean != boolean` operators [\#1159](https://github.com/apache/datafusion/issues/1159)
- Support DataType::Decimal\(15, 2\) in TPC-H benchmark [\#174](https://github.com/apache/datafusion/issues/174)
- Make `MemoryStream` public [\#150](https://github.com/apache/datafusion/issues/150)
- Add support for Parquet schema merging [\#132](https://github.com/apache/datafusion/issues/132)
- Add SQL support for IN expression [\#118](https://github.com/apache/datafusion/issues/118)
- Add logging to datafusion-cli [\#1789](https://github.com/apache/datafusion/pull/1789) ([alamb](https://github.com/alamb))
- Add `approx_median()` aggregate function [\#1729](https://github.com/apache/datafusion/pull/1729) ([realno](https://github.com/realno))
- Add join type for logical plan display [\#1674](https://github.com/apache/datafusion/pull/1674) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Fix null comparison for Parquet pruning predicate [\#1595](https://github.com/apache/datafusion/pull/1595) ([viirya](https://github.com/viirya))
- Add `corr` aggregate function [\#1561](https://github.com/apache/datafusion/pull/1561) ([realno](https://github.com/realno))
- Add `covar`, `covar_pop` and `covar_samp` aggregate functions [\#1551](https://github.com/apache/datafusion/pull/1551) ([realno](https://github.com/realno))
- Add `approx_quantile()` aggregation function [\#1539](https://github.com/apache/datafusion/pull/1539) ([domodwyer](https://github.com/domodwyer))
- Initial MemoryManager and DiskManager APIs for query execution + External Sort implementation [\#1526](https://github.com/apache/datafusion/pull/1526) ([yjshen](https://github.com/yjshen))
- Add `stddev` and `variance` [\#1525](https://github.com/apache/datafusion/pull/1525) ([realno](https://github.com/realno))
- Add `rem` operation for Expr [\#1467](https://github.com/apache/datafusion/pull/1467) ([liukun4515](https://github.com/liukun4515))
- support decimal data type in create table [\#1431](https://github.com/apache/datafusion/pull/1431) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- Ordering by index in select expression [\#1419](https://github.com/apache/datafusion/pull/1419) [[sql](https://github.com/apache/datafusion/labels/sql)] ([hntd187](https://github.com/hntd187))
- Add support for `ORDER BY` on unprojected columns [\#1415](https://github.com/apache/datafusion/pull/1415) ([viirya](https://github.com/viirya))
- Support decimal for `min` and `max` aggregate [\#1407](https://github.com/apache/datafusion/pull/1407) ([liukun4515](https://github.com/liukun4515))
- Consolidate `ConstantFolding` and `SimplifyExpression` [\#1375](https://github.com/apache/datafusion/pull/1375) ([alamb](https://github.com/alamb))
- Datafusion cli quiet mode command to contain option bool [\#1345](https://github.com/apache/datafusion/pull/1345) ([Jimexist](https://github.com/Jimexist))
- Implement `array_agg` aggregate function [\#1300](https://github.com/apache/datafusion/pull/1300) ([viirya](https://github.com/viirya))
- Add a command to switch output format in cli [\#1284](https://github.com/apache/datafusion/pull/1284) ([capkurmagati](https://github.com/capkurmagati))
- Support `=`, `<`, `<=`, `>`, `>=`, `!=`, `is distinct from`, `is not distinct from` for `BooleanArray` [\#1163](https://github.com/apache/datafusion/pull/1163) ([alamb](https://github.com/alamb))

**Fixed bugs:**

- Unsupported data type in hasher: Timestamp\(Second, None\) [\#1768](https://github.com/apache/datafusion/issues/1768)
- SQL column identifiers should be converted to lowercase when unquoted [\#1746](https://github.com/apache/datafusion/issues/1746)
- Data type Dictionary\(Int32, Utf8\) not supported for binary operation 'eq' on dyn arrays [\#1605](https://github.com/apache/datafusion/issues/1605)
- datafusion doesn't process predicate pushdown correctly when there is outer join [\#1586](https://github.com/apache/datafusion/issues/1586)
- casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail [\#1576](https://github.com/apache/datafusion/issues/1576)
- CTE/WITH .. UNION ALL confuses name resolution in WHERE [\#1509](https://github.com/apache/datafusion/issues/1509)
- ORDER BY min\(x\) results in error `Plan("No field named 'foo.x'. Valid fields are 'MIN(foo.x)'.")` [\#1479](https://github.com/apache/datafusion/issues/1479)
- Sort discards field metadata on the output schema [\#1476](https://github.com/apache/datafusion/issues/1476)
- Datafusion should not strip out timezone information from existing types [\#1454](https://github.com/apache/datafusion/issues/1454)
- Error on some queries: "column types must match schema types, expected XXX but found YYY" [\#1447](https://github.com/apache/datafusion/issues/1447)
- Query failing to return any results when filter is an equality check on strings \(bad statistics in parquet\) [\#1433](https://github.com/apache/datafusion/issues/1433)
- Field names containing period such as `f.c1` cannot be named in SQL query [\#1432](https://github.com/apache/datafusion/issues/1432)
- `Select *` returns an unexpected result [\#1412](https://github.com/apache/datafusion/issues/1412)
- Turn off unused default features of chrono and ahash [\#1398](https://github.com/apache/datafusion/issues/1398)
- real data type is float32 in PG database, but in the datafusion it is as float64 [\#1380](https://github.com/apache/datafusion/issues/1380)
- TPC-H q10 performance regression \(expression for filter with added alias is not pushed down\) [\#1367](https://github.com/apache/datafusion/issues/1367)
- ProjectionExec Loses Field Metadata [\#1361](https://github.com/apache/datafusion/issues/1361)
- Support Filter on unprojected columns [\#1351](https://github.com/apache/datafusion/issues/1351)
- NULLS ORDER is inconsistent with postgres [\#1343](https://github.com/apache/datafusion/issues/1343)
- Fix bug while merging `RecordBatch`, add `SortPreservingMerge` fuzz tester [\#1678](https://github.com/apache/datafusion/pull/1678) ([alamb](https://github.com/alamb))
- fix a cte block with same name for many times [\#1639](https://github.com/apache/datafusion/pull/1639) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- fix: casting Int64 to Float64 unsuccessfully caused tpch8 to fail [\#1601](https://github.com/apache/datafusion/pull/1601) ([xudong963](https://github.com/xudong963))
- Fix single_distinct_to_groupby for arbitrary expressions [\#1519](https://github.com/apache/datafusion/pull/1519) ([james727](https://github.com/james727))
- Fix SortExec discards field metadata on the output schema [\#1477](https://github.com/apache/datafusion/pull/1477) ([alamb](https://github.com/alamb))
- fix calculate in many_to_many_hash_partition test. [\#1463](https://github.com/apache/datafusion/pull/1463) ([Ted-Jiang](https://github.com/Ted-Jiang))
- Add Timezone to Scalar::Time\* types, and better timezone awareness to Datafusion's time types [\#1455](https://github.com/apache/datafusion/pull/1455) ([maxburke](https://github.com/maxburke))
- Support identifiers with `.` in them [\#1449](https://github.com/apache/datafusion/pull/1449) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- Fixes for working with functions in dataframes, additional documentation [\#1430](https://github.com/apache/datafusion/pull/1430) ([tobyhede](https://github.com/tobyhede))
- \[Minor\] Fix `send_time` metric for hash-repartition [\#1421](https://github.com/apache/datafusion/pull/1421) ([Dandandan](https://github.com/Dandandan))
- fix: Select \* returns an unexpected result [\#1413](https://github.com/apache/datafusion/pull/1413) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Make cli handle multiple whitespaces [\#1388](https://github.com/apache/datafusion/pull/1388) ([capkurmagati](https://github.com/capkurmagati))
- Metadata is kept in projections for non-derived columns [\#1378](https://github.com/apache/datafusion/pull/1378) ([hntd187](https://github.com/hntd187))
- Fix Predicate Pushdown: split_members should be able to split aliased predicate [\#1368](https://github.com/apache/datafusion/pull/1368) ([viirya](https://github.com/viirya))
- Change the arg names and make parameters more meaningful [\#1357](https://github.com/apache/datafusion/pull/1357) ([liukun4515](https://github.com/liukun4515))
- collect table stats by default for listing table [\#1347](https://github.com/apache/datafusion/pull/1347) ([houqp](https://github.com/houqp))
- fix: make nulls-order consistent with postgres [\#1344](https://github.com/apache/datafusion/pull/1344) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Avoid changing expression names during constant folding [\#1319](https://github.com/apache/datafusion/pull/1319) ([viirya](https://github.com/viirya))
- improve error message for invalid create table statement [\#1294](https://github.com/apache/datafusion/pull/1294) [[sql](https://github.com/apache/datafusion/labels/sql)] ([houqp](https://github.com/houqp))
- Forbid creating the table with the same name [\#1288](https://github.com/apache/datafusion/pull/1288) ([liukun4515](https://github.com/liukun4515))

**Documentation updates:**

- Clarify docs about `Accumulator::update` and `Accumulator::update_batch` [\#1542](https://github.com/apache/datafusion/pull/1542) ([alamb](https://github.com/alamb))
- Fix duplicated `cargo run --example parquet_sql` [\#1482](https://github.com/apache/datafusion/pull/1482) ([sergey-melnychuk](https://github.com/sergey-melnychuk))
- add documentation to Datafusion cli's new commands [\#1348](https://github.com/apache/datafusion/pull/1348) ([liukun4515](https://github.com/liukun4515))
- fix some clippy warnings from nightly channel [\#1277](https://github.com/apache/datafusion/pull/1277) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jimexist](https://github.com/Jimexist))

**Performance improvements:**

- Parquet pruning predicate for `IS NULL` [\#1591](https://github.com/apache/datafusion/issues/1591)
- Fix predicate pushdown for outer joins [\#1618](https://github.com/apache/datafusion/pull/1618) ([james727](https://github.com/james727))
- fix: sql planner creates cross join instead of inner join from select predicates [\#1566](https://github.com/apache/datafusion/pull/1566) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Split fetch_metadata into fetch_statistics and fetch_schema [\#1365](https://github.com/apache/datafusion/pull/1365) ([Dandandan](https://github.com/Dandandan))
- Optimize the performance queries with a single distinct aggregate [\#1315](https://github.com/apache/datafusion/pull/1315) ([ic4y](https://github.com/ic4y))
- Left join could use bitmap for left join instead of Vec\<bool\> [\#1291](https://github.com/apache/datafusion/pull/1291) ([boazberman](https://github.com/boazberman))

**Closed issues:**

- Add `release compile` to CI [\#1728](https://github.com/apache/datafusion/issues/1728)
- DiskManager and TempFiles getting created several times per query [\#1690](https://github.com/apache/datafusion/issues/1690)
- Add a test for the `pyarrow` feature in CI [\#1635](https://github.com/apache/datafusion/issues/1635)
- SQL tests for when sorting exceeded available memory and had to spill to disk [\#1573](https://github.com/apache/datafusion/issues/1573)
- Consolidate the N-way merging code and `SortPreservingMergeStream` \(which has quite good tests of what is often quite tricky code, and it will be performance critical\) [\#1572](https://github.com/apache/datafusion/issues/1572)
- Consolidate the `SortExec` code \(so there is only a single sort operator that does in memory sorting if it has enough memory budget but then spills to disk if needed\). [\#1571](https://github.com/apache/datafusion/issues/1571)
- Track memory usage in Non Limited Operators [\#1569](https://github.com/apache/datafusion/issues/1569)
- \[Question\] Why does ballista store tables in the client instead of in the SchedulerServer [\#1473](https://github.com/apache/datafusion/issues/1473)
- Consolidate Projection for Schema and RecordBatch [\#1425](https://github.com/apache/datafusion/issues/1425)
- Support Sort on unprojected columns [\#1372](https://github.com/apache/datafusion/issues/1372)
- Unused code in hash_aggregate [\#1362](https://github.com/apache/datafusion/issues/1362)
- Why use the expr types before coercion to get the result type? [\#1358](https://github.com/apache/datafusion/issues/1358)
- A problem about the projection_push_down optimizer gathers valid columns [\#1312](https://github.com/apache/datafusion/issues/1312)
- apply constant folding to `LogicalPlan::Values` [\#1170](https://github.com/apache/datafusion/issues/1170)
- reduce usage of `IntoIterator<Item = Expr>` in logical plan builder window fn [\#372](https://github.com/apache/datafusion/issues/372)
- Why does DataFusion throw a Tokio 0.2 runtime error? [\#176](https://github.com/apache/datafusion/issues/176)
- TPC-H Query 14 [\#165](https://github.com/apache/datafusion/issues/165)
- Length kernel returns bytes not character length [\#156](https://github.com/apache/datafusion/issues/156)
- Split the logical operators out into separate source files [\#115](https://github.com/apache/datafusion/issues/115)

**Merged pull requests:**

- Fixup some doc warnings [\#1811](https://github.com/apache/datafusion/pull/1811) ([alamb](https://github.com/alamb))
- Ensure most of links in docs are correct [\#1808](https://github.com/apache/datafusion/pull/1808) [[sql](https://github.com/apache/datafusion/labels/sql)] ([HaoYang670](https://github.com/HaoYang670))
- Update CHANGELOG.md, update release scripts [\#1807](https://github.com/apache/datafusion/pull/1807) ([alamb](https://github.com/alamb))
- Update versions for split crates [\#1803](https://github.com/apache/datafusion/pull/1803) ([matthewmturner](https://github.com/matthewmturner))
- Improve the error message and UX of tpch benchmark program [\#1800](https://github.com/apache/datafusion/pull/1800) ([alamb](https://github.com/alamb))
- rename references of expr in logical plan module after datafusion-expr split [\#1797](https://github.com/apache/datafusion/pull/1797) ([Jimexist](https://github.com/Jimexist))
- Update to sqlparser 0.14 [\#1796](https://github.com/apache/datafusion/pull/1796) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- \[split/13\] move rest of expr to expr_fn in datafusion-expr module [\#1794](https://github.com/apache/datafusion/pull/1794) ([Jimexist](https://github.com/Jimexist))
- Update datafusion versions [\#1793](https://github.com/apache/datafusion/pull/1793) ([matthewmturner](https://github.com/matthewmturner))
- Less verbose plans in debug logging [\#1787](https://github.com/apache/datafusion/pull/1787) ([alamb](https://github.com/alamb))
- \[split/11\] split expr type and null info to be expr-schemable [\#1784](https://github.com/apache/datafusion/pull/1784) ([Jimexist](https://github.com/Jimexist))
- Introduce `Row` format backed by raw bytes [\#1782](https://github.com/apache/datafusion/pull/1782) ([yjshen](https://github.com/yjshen))
- rewrite predicates before pushing to union inputs [\#1781](https://github.com/apache/datafusion/pull/1781) ([korowa](https://github.com/korowa))
- Update datafusion to use arrow 9.0.0 [\#1775](https://github.com/apache/datafusion/pull/1775) ([alamb](https://github.com/alamb))
- \[split/10\] split up expr for rewriting, visiting, and simplification traits [\#1774](https://github.com/apache/datafusion/pull/1774) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jimexist](https://github.com/Jimexist))
- \#1768 Support TimeUnit::Second in hasher [\#1769](https://github.com/apache/datafusion/pull/1769) ([jychen7](https://github.com/jychen7))
- TPC-H benchmark can optionally write JSON output file with benchmark summary [\#1766](https://github.com/apache/datafusion/pull/1766) ([andygrove](https://github.com/andygrove))
- \[split/8\] move `Accumulator` and `ColumnarValue` to datafusion-expr [\#1765](https://github.com/apache/datafusion/pull/1765) ([Jimexist](https://github.com/Jimexist))
- \[split/7\] move built-in scalar function to datafusion-expr [\#1764](https://github.com/apache/datafusion/pull/1764) ([Jimexist](https://github.com/Jimexist))
- \[split/6\] move signature, type signature, volatility to datafusion-expr [\#1763](https://github.com/apache/datafusion/pull/1763) ([Jimexist](https://github.com/Jimexist))
- \[split/9+12\] move udf, udaf, `Expr` to datafusion-expr module [\#1762](https://github.com/apache/datafusion/pull/1762) [[sql](https://github.com/apache/datafusion/labels/sql)] ([Jimexist](https://github.com/Jimexist))
- \[split/5\] move window frame and operator to datafusion-expr module [\#1761](https://github.com/apache/datafusion/pull/1761) ([Jimexist](https://github.com/Jimexist))
- \[split/4\] move scalar value to datafusion-common [\#1760](https://github.com/apache/datafusion/pull/1760) ([Jimexist](https://github.com/Jimexist))
- \[split/3\] split datafusion expr module and move aggregate and window function expr [\#1759](https://github.com/apache/datafusion/pull/1759) ([Jimexist](https://github.com/Jimexist))
- \[split/2\] move column and dfschema to datafusion-common module [\#1758](https://github.com/apache/datafusion/pull/1758) ([Jimexist](https://github.com/Jimexist))
- Use ordered-float 2.10 [\#1756](https://github.com/apache/datafusion/pull/1756) ([andygrove](https://github.com/andygrove))
- \[split/1\] split datafusion-common module [\#1751](https://github.com/apache/datafusion/pull/1751) ([Jimexist](https://github.com/Jimexist))
- use clap 3 style args parsing for datafusion cli [\#1749](https://github.com/apache/datafusion/pull/1749) ([Jimexist](https://github.com/Jimexist))
- fix: Case insensitive unquoted identifiers in SQL [\#1747](https://github.com/apache/datafusion/pull/1747) [[sql](https://github.com/apache/datafusion/labels/sql)] ([mkmik](https://github.com/mkmik))
- Move more tests out of context.rs [\#1743](https://github.com/apache/datafusion/pull/1743) ([alamb](https://github.com/alamb))
- Move optimize test out of context.rs [\#1742](https://github.com/apache/datafusion/pull/1742) ([alamb](https://github.com/alamb))
- Fix typos in crate documentation [\#1739](https://github.com/apache/datafusion/pull/1739) ([r4ntix](https://github.com/r4ntix))
- add `cargo check --release` to ci [\#1737](https://github.com/apache/datafusion/pull/1737) ([xudong963](https://github.com/xudong963))
- Update parking_lot requirement from 0.11 to 0.12 [\#1735](https://github.com/apache/datafusion/pull/1735) ([dependabot[bot]](https://github.com/apps/dependabot))
- Create built-in scalar functions programmatically [\#1734](https://github.com/apache/datafusion/pull/1734) ([HaoYang670](https://github.com/HaoYang670))
- Prevent repartitioning of certain operator's direct children \(\#1731\) [\#1732](https://github.com/apache/datafusion/pull/1732) ([tustvold](https://github.com/tustvold))
- API to get Expr's type and nullability without a `DFSchema` [\#1726](https://github.com/apache/datafusion/pull/1726) ([alamb](https://github.com/alamb))
- minor: fix `cargo run --release` error [\#1723](https://github.com/apache/datafusion/pull/1723) ([xudong963](https://github.com/xudong963))
- substitute `parking_lot::Mutex` for `std::sync::Mutex` [\#1720](https://github.com/apache/datafusion/pull/1720) ([xudong963](https://github.com/xudong963))
- Convert boolean case expressions to boolean logic [\#1719](https://github.com/apache/datafusion/pull/1719) ([tustvold](https://github.com/tustvold))
- Add Expression Simplification API [\#1717](https://github.com/apache/datafusion/pull/1717) ([alamb](https://github.com/alamb))
- Create ListingTableConfig which includes file format and schema inference [\#1715](https://github.com/apache/datafusion/pull/1715) ([matthewmturner](https://github.com/matthewmturner))
- make `select_to_plan` clearer [\#1714](https://github.com/apache/datafusion/pull/1714) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Add upper bound for public function `signature` [\#1713](https://github.com/apache/datafusion/pull/1713) ([HaoYang670](https://github.com/HaoYang670))
- Add tests and CI for optional pyarrow module [\#1711](https://github.com/apache/datafusion/pull/1711) ([wjones127](https://github.com/wjones127))
- Create SchemaAdapter trait to map table schema to file schemas [\#1709](https://github.com/apache/datafusion/pull/1709) ([thinkharderdev](https://github.com/thinkharderdev))
- refine test in repartition.rs & coalesce_batches.rs [\#1707](https://github.com/apache/datafusion/pull/1707) ([xudong963](https://github.com/xudong963))
- Fuzz test for spillable sort [\#1706](https://github.com/apache/datafusion/pull/1706) ([yjshen](https://github.com/yjshen))
- Support `create_physical_expr` and `ExecutionContextState` or `DefaultPhysicalPlanner` for faster speed [\#1700](https://github.com/apache/datafusion/pull/1700) ([alamb](https://github.com/alamb))
- Implement TableProvider for DataFrameImpl [\#1699](https://github.com/apache/datafusion/pull/1699) ([cpcloud](https://github.com/cpcloud))
- Move timestamp related tests out of context.rs and into sql integration test [\#1696](https://github.com/apache/datafusion/pull/1696) ([alamb](https://github.com/alamb))
- Lazy TempDir creation in DiskManager [\#1695](https://github.com/apache/datafusion/pull/1695) ([alamb](https://github.com/alamb))
- Add `MemTrackingMetrics` to ease memory tracking for non-limited memory consumers [\#1691](https://github.com/apache/datafusion/pull/1691) ([yjshen](https://github.com/yjshen))
- \(minor\) Reduce memory manager and disk manager logs from `info!` to `debug!` [\#1689](https://github.com/apache/datafusion/pull/1689) ([alamb](https://github.com/alamb))
- Make `SortPreservingMergeStream` stable on input stream order [\#1687](https://github.com/apache/datafusion/pull/1687) ([alamb](https://github.com/alamb))
- Incorporate dyn scalar kernels [\#1685](https://github.com/apache/datafusion/pull/1685) ([matthewmturner](https://github.com/matthewmturner))
- Move `information_schema` tests out of execution/context.rs to `sql_integration` tests [\#1684](https://github.com/apache/datafusion/pull/1684) ([alamb](https://github.com/alamb))
- Add a new metric type: `Gauge` + `CurrentMemoryUsage` to metrics [\#1682](https://github.com/apache/datafusion/pull/1682) ([yjshen](https://github.com/yjshen))
- refactor array_agg to not to have `update` and `merge` [\#1681](https://github.com/apache/datafusion/pull/1681) ([Jimexist](https://github.com/Jimexist))
- Use NamedTempFile rather than `String` in DiskManager [\#1680](https://github.com/apache/datafusion/pull/1680) ([alamb](https://github.com/alamb))
- upgrade clap to version 3 [\#1672](https://github.com/apache/datafusion/pull/1672) ([Jimexist](https://github.com/Jimexist))
- Improve configuration and resource use of `MemoryManager` and `DiskManager` [\#1668](https://github.com/apache/datafusion/pull/1668) ([alamb](https://github.com/alamb))
- feat: Support quarter granularity in date_trunc function [\#1667](https://github.com/apache/datafusion/pull/1667) ([ovr](https://github.com/ovr))
- Fix can not load parquet table form spark in datafusion-cli. [\#1665](https://github.com/apache/datafusion/pull/1665) ([Ted-Jiang](https://github.com/Ted-Jiang))
- Make `MemoryManager` and `MemoryStream` public [\#1664](https://github.com/apache/datafusion/pull/1664) ([yjshen](https://github.com/yjshen))
- \[Cleanup\] Move `AggregatedMetricsSet` to `metrics` for further reuse [\#1663](https://github.com/apache/datafusion/pull/1663) ([yjshen](https://github.com/yjshen))
- fix: substr - correct behaivour with negative start pos [\#1660](https://github.com/apache/datafusion/pull/1660) ([ovr](https://github.com/ovr))
- suppport bitwise and as an example [\#1653](https://github.com/apache/datafusion/pull/1653) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- refine match pattern related code [\#1650](https://github.com/apache/datafusion/pull/1650) ([xudong963](https://github.com/xudong963))
- update md-5, sha2, blake2 [\#1647](https://github.com/apache/datafusion/pull/1647) ([xudong963](https://github.com/xudong963))
- Add `DataFusionError` -\> `ArrowError` conversion [\#1643](https://github.com/apache/datafusion/pull/1643) ([alamb](https://github.com/alamb))
- Add `spill_count` and `spilled_bytes` to `BaselineMetrics`, test sort with spill [\#1641](https://github.com/apache/datafusion/pull/1641) ([yjshen](https://github.com/yjshen))
- support hash decimal array and group by [\#1640](https://github.com/apache/datafusion/pull/1640) ([liukun4515](https://github.com/liukun4515))
- Consolidate Schema and RecordBatch projection [\#1638](https://github.com/apache/datafusion/pull/1638) ([alamb](https://github.com/alamb))
- Update hashbrown requirement from 0.11 to 0.12 [\#1631](https://github.com/apache/datafusion/pull/1631) ([dependabot[bot]](https://github.com/apps/dependabot))
- Update pyo3 requirement from 0.14 to 0.15 [\#1627](https://github.com/apache/datafusion/pull/1627) ([dependabot[bot]](https://github.com/apps/dependabot))
- Optimize `SortPreservingMergeStream` to avoid `SortKeyCursor` sharing [\#1624](https://github.com/apache/datafusion/pull/1624) ([yjshen](https://github.com/yjshen))
- Handle merging of evolved schemas in ParquetExec [\#1622](https://github.com/apache/datafusion/pull/1622) ([thinkharderdev](https://github.com/thinkharderdev))
- feat: Support Substring\(str \[from int\] \[for int\]\) [\#1621](https://github.com/apache/datafusion/pull/1621) [[sql](https://github.com/apache/datafusion/labels/sql)] ([ovr](https://github.com/ovr))
- feat: Support complex interval via IntervalMonthDayNano [\#1615](https://github.com/apache/datafusion/pull/1615) [[sql](https://github.com/apache/datafusion/labels/sql)] ([ovr](https://github.com/ovr))
- consolidate binary_expr coercion rule code into `binary_rule.rs` module [\#1607](https://github.com/apache/datafusion/pull/1607) ([alamb](https://github.com/alamb))
- Fix comparison of dictionary arrays [\#1606](https://github.com/apache/datafusion/pull/1606) ([alamb](https://github.com/alamb))
- add test for decimal to decimal [\#1603](https://github.com/apache/datafusion/pull/1603) ([liukun4515](https://github.com/liukun4515))
- update nightly version [\#1597](https://github.com/apache/datafusion/pull/1597) ([Jimexist](https://github.com/Jimexist))
- Consolidate sort and external_sort [\#1596](https://github.com/apache/datafusion/pull/1596) ([yjshen](https://github.com/yjshen))
- support from_slice for binary, string, and boolean array types [\#1589](https://github.com/apache/datafusion/pull/1589) ([Jimexist](https://github.com/Jimexist))
- add from_slice trait to ease arrow2 migration [\#1588](https://github.com/apache/datafusion/pull/1588) ([Jimexist](https://github.com/Jimexist))
- Implement ARRAY_AGG\(DISTINCT ...\) [\#1579](https://github.com/apache/datafusion/pull/1579) ([james727](https://github.com/james727))
- Rename sql integration tests from `mod` to `sql_integration` [\#1575](https://github.com/apache/datafusion/pull/1575) ([alamb](https://github.com/alamb))
- minor: improve the benchmark readme [\#1567](https://github.com/apache/datafusion/pull/1567) ([xudong963](https://github.com/xudong963))
- Consolidate `batch_size` configuration in `ExecutionConfig`, `RuntimeConfig` and `PhysicalPlanConfig` [\#1562](https://github.com/apache/datafusion/pull/1562) ([yjshen](https://github.com/yjshen))
- Update to rust 1.58 [\#1557](https://github.com/apache/datafusion/pull/1557) ([xudong963](https://github.com/xudong963))
- support mathematics operation for decimal data type [\#1554](https://github.com/apache/datafusion/pull/1554) ([liukun4515](https://github.com/liukun4515))
- Address clippy warnings [\#1553](https://github.com/apache/datafusion/pull/1553) ([sergey-melnychuk](https://github.com/sergey-melnychuk))
- enhance arithmetic operation for array with scalar [\#1552](https://github.com/apache/datafusion/pull/1552) ([liukun4515](https://github.com/liukun4515))
- Remove unused `update` and `merge` implementations from Aggregates and supporting `ScalarValue` arithmetic [\#1550](https://github.com/apache/datafusion/pull/1550) ([alamb](https://github.com/alamb))
- Add batch operations to stddev [\#1547](https://github.com/apache/datafusion/pull/1547) ([realno](https://github.com/realno))
- Mark ARRAY_AGG\(DISTINCT ...\) not implemented [\#1534](https://github.com/apache/datafusion/pull/1534) ([james727](https://github.com/james727))
- Update to arrow-7.0.0 [\#1523](https://github.com/apache/datafusion/pull/1523) ([alamb](https://github.com/alamb))
- Fix ORDER BY on aggregate [\#1506](https://github.com/apache/datafusion/pull/1506) ([viirya](https://github.com/viirya))
- Add example on how to query multiple parquet files [\#1497](https://github.com/apache/datafusion/pull/1497) ([nitisht](https://github.com/nitisht))
- Refactor testing modules [\#1491](https://github.com/apache/datafusion/pull/1491) ([hntd187](https://github.com/hntd187))
- add rfcs for datafusion [\#1490](https://github.com/apache/datafusion/pull/1490) ([xudong963](https://github.com/xudong963))
- support comparison for decimal data type and refactor the binary coercion rule [\#1483](https://github.com/apache/datafusion/pull/1483) ([liukun4515](https://github.com/liukun4515))
- Minor: Rename `predicate_builder` --\> `pruning_predicate` for consistency [\#1481](https://github.com/apache/datafusion/pull/1481) ([alamb](https://github.com/alamb))
- Tests for support try_cast/cast decimal to numeric [\#1465](https://github.com/apache/datafusion/pull/1465) ([liukun4515](https://github.com/liukun4515))
- Avoid send empty batches for Hash partitioning. [\#1459](https://github.com/apache/datafusion/pull/1459) ([Ted-Jiang](https://github.com/Ted-Jiang))
- Planner code cleanup [\#1450](https://github.com/apache/datafusion/pull/1450) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- Fix bug in projection: "column types must match schema types, expected XXX but found YYY" [\#1448](https://github.com/apache/datafusion/pull/1448) ([alamb](https://github.com/alamb))
- Update arrow-rs to 6.4.0 and replace boolean comparison in datafusion with arrow compute kernel [\#1446](https://github.com/apache/datafusion/pull/1446) ([xudong963](https://github.com/xudong963))
- support cast/try_cast for decimal: signed numeric to decimal [\#1442](https://github.com/apache/datafusion/pull/1442) ([liukun4515](https://github.com/liukun4515))
- Consolidate decimal error checking and improve error messages [\#1438](https://github.com/apache/datafusion/pull/1438) [[sql](https://github.com/apache/datafusion/labels/sql)] ([alamb](https://github.com/alamb))
- use 0.13 sql parser [\#1435](https://github.com/apache/datafusion/pull/1435) ([Jimexist](https://github.com/Jimexist))
- Minor Code cleanups [\#1428](https://github.com/apache/datafusion/pull/1428) ([alamb](https://github.com/alamb))
- Clarify communication on bi-weekly sync [\#1427](https://github.com/apache/datafusion/pull/1427) ([alamb](https://github.com/alamb))
- support sum/avg agg for decimal, change sum\(float32\) --\> float64 [\#1408](https://github.com/apache/datafusion/pull/1408) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- Fix bugs with nullability during rewrites: Combine `simplify` and `Simplifier` [\#1401](https://github.com/apache/datafusion/pull/1401) ([alamb](https://github.com/alamb))
- Minimize features [\#1399](https://github.com/apache/datafusion/pull/1399) ([carols10cents](https://github.com/carols10cents))
- Update rust vesion to 1.57 [\#1395](https://github.com/apache/datafusion/pull/1395) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- support decimal scalar value [\#1394](https://github.com/apache/datafusion/pull/1394) ([liukun4515](https://github.com/liukun4515))
- Add coercion rules for AggregateFunctions [\#1387](https://github.com/apache/datafusion/pull/1387) ([liukun4515](https://github.com/liukun4515))
- upgrade the arrow-rs version [\#1385](https://github.com/apache/datafusion/pull/1385) ([liukun4515](https://github.com/liukun4515))
- add array agg name [\#1382](https://github.com/apache/datafusion/pull/1382) ([liukun4515](https://github.com/liukun4515))
- Make tests for `simplify` and `Simplifer` consistent [\#1376](https://github.com/apache/datafusion/pull/1376) ([alamb](https://github.com/alamb))
- Refactor: Consolidate expression simplification code in `simplify_expression.rs` [\#1374](https://github.com/apache/datafusion/pull/1374) ([alamb](https://github.com/alamb))
- remove unused code in hash_aggregate [\#1370](https://github.com/apache/datafusion/pull/1370) ([ic4y](https://github.com/ic4y))
- Use `BufReader` for LocalFileReader to revert performance regression in parquet reading [\#1366](https://github.com/apache/datafusion/pull/1366) ([Dandandan](https://github.com/Dandandan))
- Add unit test for constant folding on values [\#1355](https://github.com/apache/datafusion/pull/1355) ([viirya](https://github.com/viirya))
- Extract logical plan: rename the plan name \(follow up\) [\#1354](https://github.com/apache/datafusion/pull/1354) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- Moved aggr_test_schema to test_utils [\#1338](https://github.com/apache/datafusion/pull/1338) ([rdettai](https://github.com/rdettai))
- upgrade arrow-rs to 6.2.0 [\#1334](https://github.com/apache/datafusion/pull/1334) ([liukun4515](https://github.com/liukun4515))
- Update release instructions [\#1331](https://github.com/apache/datafusion/pull/1331) ([alamb](https://github.com/alamb))
- \#1268: allow datafusion-cli to toggle quiet flag within CLI [\#1330](https://github.com/apache/datafusion/pull/1330) ([jgoday](https://github.com/jgoday))
- Extract Aggregate, Sort, and Join to struct from AggregatePlan [\#1326](https://github.com/apache/datafusion/pull/1326) ([matthewmturner](https://github.com/matthewmturner))
- Extract `EmptyRelation`, `Limit`, `Values` from `LogicalPlan` [\#1325](https://github.com/apache/datafusion/pull/1325) ([liukun4515](https://github.com/liukun4515))
- Extract CrossJoin, Repartition, Union in LogicalPlan [\#1322](https://github.com/apache/datafusion/pull/1322) ([liukun4515](https://github.com/liukun4515))
- Fifth batch of updating sql tests to use assert_batches_eq [\#1318](https://github.com/apache/datafusion/pull/1318) ([matthewmturner](https://github.com/matthewmturner))
- Extract Explain, Analyze, Extension in LogicalPlan as independent struct [\#1317](https://github.com/apache/datafusion/pull/1317) [[sql](https://github.com/apache/datafusion/labels/sql)] ([xudong963](https://github.com/xudong963))
- Extract CreateMemoryTable, DropTable, CreateExternalTable in LogicalPlan as independent struct [\#1311](https://github.com/apache/datafusion/pull/1311) [[sql](https://github.com/apache/datafusion/labels/sql)] ([liukun4515](https://github.com/liukun4515))
- Extract Projection, Filter, Window in LogicalPlan as independent struct [\#1309](https://github.com/apache/datafusion/pull/1309) ([ic4y](https://github.com/ic4y))
- Add PSQL comparison tests for except, intersect [\#1292](https://github.com/apache/datafusion/pull/1292) ([mrob95](https://github.com/mrob95))
- Extract logical plans in LogicalPlan as independent struct: TableScan [\#1290](https://github.com/apache/datafusion/pull/1290) ([xudong963](https://github.com/xudong963))
- Add statement helper command to cli [\#1285](https://github.com/apache/datafusion/pull/1285) ([matthewmturner](https://github.com/matthewmturner))
- Python bindings for window functions [\#819](https://github.com/apache/datafusion/pull/819) [[sql](https://github.com/apache/datafusion/labels/sql)] ([jgoday](https://github.com/jgoday))
