Skip to content

Run the three test suites in parallel - #814

Merged
laughingman7743 merged 1 commit into
masterfrom
ci/parallel-test-stages
Sep 25, 2026
Merged

laughingman7743 merged 1 commit into
masterfrom
ci/parallel-test-stages

Conversation

@laughingman7743

Copy link
Copy Markdown
Member

WHAT

.github/workflows/test.yaml no longer chains test → test-sqla → test-sqla-async with needs:. The three suites start together. Each stays a separate job, so "Re-run failed jobs" still works per suite.

WHY

The chain came from 8c3be28 ("Split CI test jobs for independent retries"). It split one job into three so that failed jobs could be re-run separately; it was not there to limit load. The suites share no named AWS resources:

  • tests/pyathena creates a random pyathena_test_<id> schema per process, with S3 data under it.
  • The SQLAlchemy suites create <ident>, <ident>_test_schema and <ident>_test_schema_2 from the plugin's follower ident.
  • Tests in the shared S3 Tables namespace use a random table-name suffix.

Measured on run 36023393076 (#803, before this change):

  • Total: 46.5 min.
  • test took 10–13 min, test-sqla 13–15 min, test-sqla-async 16–17 min, each stage 5 Python versions × 8 pytest workers.
  • In parallel, the wall time should approach the longest suite plus setup.

Load: with the suites in parallel, about 120 pytest workers run at once instead of 40. The account's adjustable Athena quotas are:

  • Active DDL queries: 100
  • Active DML queries: 150

TooManyRequestsException from StartQueryExecution is retried by the default RetryConfig. Metadata throttling is answered from Glue since #803.

TEST

  • just lint passed, and the workflow YAML parses.
  • This PR's own CI run is the measurement. It will be recorded here: total and per-job durations compared with 46.5 min, failures, and any TooManyRequestsException or rerun seen in the logs.

🤖 Generated with Claude Code

The needs chain from test to test-sqla to test-sqla-async came from
splitting one job into three for per-suite reruns; the suites share no
named AWS resources, so they start together and each remains its own
job for "Re-run failed jobs".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@laughingman7743
laughingman7743 marked this pull request as ready for review September 25, 2026 07:23
@laughingman7743
laughingman7743 merged commit 76ccd10 into master Sep 25, 2026
31 of 32 checks passed
@laughingman7743
laughingman7743 deleted the ci/parallel-test-stages branch September 25, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant