Skip to content

Reuse Athena Spark sessions across Spark tests #849

Description

@laughingman7743

Use case

Spark tests cost about $20 of the September bill (58 DPU-hours; see #834).

  • The spark_cursor and async_spark_cursor fixtures in tests/pyathena/conftest.py and aio_spark_cursor in tests/pyathena/aio/conftest.py are function-scoped.
  • Every Spark test starts a new Athena Spark session and waits for it to become idle.
  • The cursor's close() terminates the session.

So the DPU cost of starting a session is paid once per test.

Since #837, Spark tests run on pull requests only when Spark code or tests change, plus weekly and on dispatch. The Spark lifecycle work tracked in #791 changes those files, so they run often while that work is in progress.

Proposed change

Reuse Spark sessions across tests where isolation allows it:

  • Provide a module- or worker-scoped Spark connection/session fixture for tests that only run calculations.
  • Keep function-scoped sessions for tests that exercise session lifecycle: creation failure, termination, cancellation, and close() behavior.
  • Check how this interacts with the Improve Spark cursor session ownership and lifecycle handling #791 changes (for example, terminate_session_on_close) so the tests still cover the lifecycle contracts.

Validation plan (if implementing)

  • Run the Spark tests (tests/pyathena/spark/, tests/pyathena/aio/spark/) against the pyathena-spark work group, and confirm the same results as before.
  • Compare the number of sessions started (ListSessions for the work group during the run) and the DPU-hours before and after.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions