Enable using multithreaded setup_page_index in hybrid scan reader
#20721
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
| }; | ||
|
|
||
| // Use parallel processing only if we have enough columns to justify the overhead | ||
| constexpr std::size_t parallel_threshold = 256; |
There was a problem hiding this comment.
@vuule did you pick 512 for this empirically or did you run any experiments. I am seeing more benefit with 256 for the hybrid scan example but can revert.
mhaseeb123
added
3 - Ready for Review
| return; | ||
| } | ||
|
|
||
| auto& row_groups = per_file_metadata.front().row_groups; |
There was a problem hiding this comment.
No need for all this anymore. Just call metadata.setup_page_index(..) method.
| all_column_chunks.emplace_back(std::ref(col)); | ||
| } | ||
| } | ||
| if (max_offset > min_offset) { |
There was a problem hiding this comment.
Not sure why we were checking max_offset > 0, max_offset > min_offset should be a better check unless I am missing something here?
| int64_t const length = max_offset - min_offset; | ||
| auto const idx_buf = source->host_read(min_offset, length); | ||
| // Flatten all columns into a single vector for easier task distribution | ||
| std::vector<std::reference_wrapper<ColumnChunk>> all_column_chunks; |
There was a problem hiding this comment.
Moved everything beyond this point as-is into the setup_page_index function.
| { | ||
| CUDF_FUNC_RANGE(); | ||
|
|
||
| // Flatten all columns into a single vector for easier task distribution |
There was a problem hiding this comment.
Moved as-is from the constructor
| metadata& operator=(metadata const& other) = delete; | ||
| metadata& operator=(metadata&& other) = default; | ||
|
|
||
| ~metadata() = default; |
There was a problem hiding this comment.
Use base reader's destructor that uses multiple threads
1 participant
Description
This PR refactors the multithreaded page index setup code in main parquet reader so that the child class used in hybrid scan reader can also benefit from it. No new code is introduced.
Checklist