Use sort ordering on timestamp array #443
Conversation
server/src/storage/localfs.rs
Outdated
| infinite_source: false, | ||
| format: Arc::new(file_format), | ||
| table_partition_cols: vec![], | ||
| collect_stat: true, | ||
| target_partitions: 1, | ||
| target_partitions: 32, |
There was a problem hiding this comment.
What the significance of changing this field target_partitions here?
There was a problem hiding this comment.
Roughly the partition here is number of parallel streams that is generated by datafusion during execution. Having this 1 was causing all files to be grouped in one partition and datafusion is unable to use external sort information for files in a group as it cannot infer order between grouped files and if they are overlapping in time range or not.
There was a problem hiding this comment.
So at max datafusion will hold 32 streams to calculate the output and merge them back using SortPreservingMerge
server/src/storage/s3.rs
Outdated
| infinite_source: false, | ||
| format: Arc::new(file_format), | ||
| table_partition_cols: vec![], | ||
| collect_stat: true, | ||
| target_partitions: 1, | ||
| target_partitions: 32, |
There was a problem hiding this comment.
This section is repeated for local and s3 mode. Can we move it to the common abstraction?
There was a problem hiding this comment.
Yes, this needs refactoring
Fixes #430.
Description
Write timestamp sortedness metadata to parquet and provide external sort information to datafusion. This way the SortExec can be avoided in execution plan with most queries which use
order by p_timestamp.Example
explain select p_timestamp from {{stream_name}} order by p_timestamp ascIn physical plan it is visible that SortExec is eliminated as output_ordering is pushed to ParquetExec node
Note:
This is still not the most optimized version of this query as SortPreservingExec is not really needed here. The issue here is that the Datafusion is not aware that the partitions / files are non overlapping when considering timestamp
Also if the target partition limit is crossed then datafusion again adds SortExec to physical plan.
This PR has: