<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Comet Roadmap

Comet is an open-source project and contributors are welcome to work on any issues at any time, but we find it
helpful to have a roadmap for some of the major items that require coordination between contributors.

## Major Initiatives

### Iceberg Integration

Iceberg tables reads are now fully native, powered by a scan operator backed by Iceberg-rust ([#2528]). We anticipate
major improvements expected in the next few releases, including bringing Iceberg table format V3 features (_e.g._,
encryption) to the reader.

[#2528]: https://github.com/apache/datafusion-comet/pull/2528

### Spark 4.0 Support

Comet has experimental support for Spark 4.0, but there is more work to do ([#1637]), such as enabling
more Spark SQL tests and fully implementing ANSI support ([#313]) for all supported expressions.

[#313]: https://github.com/apache/datafusion-comet/issues/313
[#1637]: https://github.com/apache/datafusion-comet/issues/1637

### Dynamic Partition Pruning

Iceberg table scans support Dynamic Partition Pruning (DPP) filters generated by Spark's `PlanDynamicPruningFilters`
optimizer rule ([#3349]). However, we still need to bring this functionality to our Parquet reader. Furthermore,
Spark's `PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's rules, so DPP with Adaptive Query
Execution requires a redesign of Comet's plan translation. We are focused on implementing DPP to keep Comet competitive
with benchmarks that benefit from this feature like TPC-DS. This effort can be tracked at [#3510].

[#3349]: https://github.com/apache/datafusion-comet/pull/3349
[#3510]: https://github.com/apache/datafusion-comet/issues/3510

## Ongoing Improvements

In addition to the major initiatives above, we have the following ongoing areas of work:

- Adding support for more Spark expressions
- Moving more expressions to the `datafusion-spark` crate in the core DataFusion repository
- Performance tuning
- Nested type support improvements
