Skip to content

[PERF]: Establish a default for max_concurrent_io_tasks on local scans #23864

Description

@Matt711

max_concurrent_io_tasks defaults to 2 for all scan types. PR #23847 adds separate defaults for local and remote paths but does not establish what the local value should be.

We need a sweep at SF300–SF1000 with data on NVMe, varying max_concurrent_io_tasks across 2, 4, and 8. Naively, more concurrency should help or at least be neutral, but we need to confirm this and understand the penalty if it is not. Any query that regresses should be profiled with nsys so we can identify where the contention is coming from.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    PerformancePerformance related issuecudf-polarsIssues specific to cudf-polars

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions