You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark tests cost about $20 of the September bill (58 DPU-hours; see #834).
The spark_cursor and async_spark_cursor fixtures in tests/pyathena/conftest.py and aio_spark_cursor in tests/pyathena/aio/conftest.py are function-scoped.
Every Spark test starts a new Athena Spark session and waits for it to become idle.
The cursor's close() terminates the session.
So the DPU cost of starting a session is paid once per test.
Since #837, Spark tests run on pull requests only when Spark code or tests change, plus weekly and on dispatch. The Spark lifecycle work tracked in #791 changes those files, so they run often while that work is in progress.
Proposed change
Reuse Spark sessions across tests where isolation allows it:
Provide a module- or worker-scoped Spark connection/session fixture for tests that only run calculations.
Keep function-scoped sessions for tests that exercise session lifecycle: creation failure, termination, cancellation, and close() behavior.
Use case
Spark tests cost about $20 of the September bill (58 DPU-hours; see #834).
spark_cursorandasync_spark_cursorfixtures intests/pyathena/conftest.pyandaio_spark_cursorintests/pyathena/aio/conftest.pyare function-scoped.close()terminates the session.So the DPU cost of starting a session is paid once per test.
Since #837, Spark tests run on pull requests only when Spark code or tests change, plus weekly and on dispatch. The Spark lifecycle work tracked in #791 changes those files, so they run often while that work is in progress.
Proposed change
Reuse Spark sessions across tests where isolation allows it:
close()behavior.terminate_session_on_close) so the tests still cover the lifecycle contracts.Validation plan (if implementing)
tests/pyathena/spark/,tests/pyathena/aio/spark/) against thepyathena-sparkwork group, and confirm the same results as before.ListSessionsfor the work group during the run) and the DPU-hours before and after.