Limit the chunked read memory claim to CSV results

With Polars 1.39.0, scan_parquet(...).collect_batches() read a whole
676 MB UNLOAD object before the first batch, so a result-size-independent
memory bound holds only for CSV. The benchmark README note about the
Polars file cache now applies only to Polars before 1.39.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: laughingman7743

benchmarks/README.md

@@ -225,7 +225,7 @@ API row conversion and native DataFrame/Table access appear as separate cases.
 Sampled RSS can miss short peaks, and the constructor suite's RSS includes its subsequent validation read.
 If the operating system denies access to per-thread CPU times, `thread_cpu_available` is false; thread counts and RSS are still recorded.
 Each trial process receives its own `POLARS_TEMP_DIR`, which the parent samples with RSS and removes after the trial.
-Polars lazy CSV scans of S3 objects download the whole object into a file cache in this directory before yielding batches; in the recorded runs, those files remained after the process exited.
+Polars before 1.39.0 downloaded the whole S3 object of a lazy CSV scan into a file cache in this directory before yielding batches and left the file after the process exited; PyAthena requires Polars 1.39.0 or later, whose scans do not write that cache.
 When the temporary directory is a tmpfs, as `/tmp` is on Amazon Linux 2023, this storage uses memory that RSS does not include.
 
 | Capability | Treatment |

docs/polars.md

@@ -311,9 +311,9 @@ for chunk in cursor.iter_chunks():
 ```
 
 This method uses Polars' `scan_csv()` and `scan_parquet()` with `collect_batches()`,
-which stream the result from S3 without writing it to local storage.
-Polars reads ahead a bounded amount of data, so memory usage does not grow with the result size,
-but it is determined by Polars' read-ahead buffer rather than by `chunksize`.
+which read the result from S3 without writing it to local storage.
+Memory usage depends on how far Polars reads ahead, not on `chunksize`.
+For CSV results, Polars limits the read-ahead by the number of CPU cores, so memory usage does not grow with the result size.
 If reading a chunk fails, iteration raises `OperationalError` instead of ending early.
 
 The chunked iteration also works with the unload option:

tests/pyathena/polars/test_result_set.py

@@ -30,7 +30,7 @@ def _chunked_result_set() -> AthenaPolarsResultSet:
 
 class TestAthenaPolarsResultSet:
     def test_iter_csv_chunks_raises_when_read_fails_partway(self, tmp_path):
-        """A CSV read that fails after some batches raises instead of ending the iteration."""
+        """A CSV read that fails partway through the data raises instead of ending early."""
         path = tmp_path / "result.csv"
         path.write_text(
             "a\n" + "".join(f"{i}\n" for i in range(_ROWS_BEFORE_FAILURE)) + "not-a-number\n"
@@ -61,7 +61,7 @@ def test_iter_csv_chunks_raises_when_read_fails_partway(self, tmp_path):
             list(result_set._iter_csv_chunks())
 
     def test_iter_parquet_chunks_raises_when_read_fails_partway(self, tmp_path):
-        """A Parquet read that fails after some batches raises instead of ending the iteration."""
+        """A Parquet read that fails partway through the data raises instead of ending early."""
         pl.DataFrame({"a": range(_ROWS_BEFORE_FAILURE)}).write_parquet(
             tmp_path / "0.parquet", row_group_size=10_000
         )