Run all unit tests in a single Buildkite job #22259

wzieba · 2025-10-06T13:15:16Z

Closes: AINFRA-1369: Merge unit tests into a single Buildkite job in WordPress/Jetpack Android

Description

This PR merges 5 separate unit tests Buildkite jobs into one.

Before	After

Motivation

What motivated me to propose this change:

Resources: with 5 separate jobs, we allocate a BK runner that builds modules from the ground just to run a few seconds of unit tests. With 1 job, we build the app once. It's especially visible when running tests on WordPress module, which builds all submodules but runs only WordPress unit tests.
No significant performance regression: The signle unit tests step will still be shorter than the longest step of the pipeline (UI tests). So overall PR checks job won't be longer. It's true, that unit tests step on its own will be longer, but this is a very small (few seconds) difference.
Single report: With single Buildkite step, we also gain a single Buildkite Test Engine report (example). This is better, because if we look for flaky or long unit tests, there's no benefit to look for them in 5 different "test suites".
Easier automation: Lastly, this change was motivated by the work on integrating Buildkite Claude plugin. It's easier to design automations if there's one trigger, rather than 5 of them.

Testing instructions

Test reports: On this PR checks, verify that you can see 6373 tests in Test Engine (button in top-right corner of logs)
Code coverage reports: See that code coverage wasn't changed meaningfully by this PR (a slight change comes from the fact it's outdated branch). To double check, see in Buildkite logs that 5 reports have been uploaded to Codecov
Failed tests report: Open PR [DNM] Unit tests in single job failure #22263. Verify, that tests have been reported to Test Engine, even though one of them failed.

wpmobilebot · 2025-10-06T13:34:49Z

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Flavor	Jalapeno
	Build Type	Debug
	Version	pr22259-454737c
	Commit	`454737c`
	Direct Download	`wordpress-prototype-build-pr22259-454737c.apk`

Note: Google Login is not supported on these builds.

wpmobilebot · 2025-10-06T13:37:34Z

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Flavor	Jalapeno
	Build Type	Debug
	Version	pr22259-454737c
	Commit	`454737c`
	Direct Download	`jetpack-prototype-build-pr22259-454737c.apk`

Note: Google Login is not supported on these builds.

… of `jalapeno`

As Codecov have difficulties finding these reports on its own

codecov · 2025-10-07T08:17:07Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 39.89%. Comparing base (33d22f3) to head (454737c).
⚠️ Report is 6 commits behind head on trunk.

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk   #22259      +/-   ##
==========================================
- Coverage   39.91%   39.89%   -0.03%     
==========================================
  Files        2165     2165              
  Lines      102511   102560      +49     
  Branches    14771    14791      +20     
==========================================
- Hits        40922    40920       -2     
- Misses      58122    58171      +49     
- Partials     3467     3469       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR consolidates five separate Buildkite unit test jobs into a single job to improve resource efficiency and simplify CI pipeline management. The change reduces build overhead by running all unit tests in one job instead of building modules separately for each test suite.

Key changes:

Merged WordPress, processors, image-editor, fluxc, and login unit test jobs into one
Updated test collection and reporting to handle multiple modules in a single job
Modified code coverage upload to handle multiple coverage reports simultaneously

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
.buildkite/pipeline.yml	Consolidated 5 separate unit test jobs into single job configuration
.buildkite/commands/run-unit-tests.sh	Refactored script to run all unit tests together and collect results from multiple modules
.buildkite/commands/upload-code-coverage.sh	Enhanced to upload multiple coverage files in one execution

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.buildkite/commands/run-unit-tests.sh

ParaskP7 · 2025-10-08T09:46:29Z

FYI @wzieba that we might want to add post-merge instructions to not forget to remove these 3 pending and soon-to-be outdated checks...

...from within branch protections rules of this repo:

ParaskP7

Nicely done @wzieba , much better and we gain so much out of this change, awesome! ❤️ 🚀

What motivated me to propose this change:

💯 🎯 🥇

Test reports: On this PR checks, verify that you can see 6373 tests in Test Engine (button in top-right corner of logs)

✅

Code coverage reports: See that code coverage wasn't changed meaningfully by this PR (a slight change comes from the fact it's outdated branch). To double check, see in Buildkite logs that 5 reports have been uploaded to Codecov

✅

Failed tests report: Open PR #22263. Verify, that tests have been reported to Test Engine, even though one of them failed.

✅

.buildkite/commands/run-unit-tests.sh

ParaskP7 · 2025-10-08T10:21:55Z

.buildkite/commands/run-unit-tests.sh

-results_file="$test_results_dir/merged-test-results.xml"
+if [[ "$TESTS_EXIT_STATUS" -eq 0 ]]; then
+  echo "--- ⚒️ Uploading code coverage"
+  # Find all kover XML reports and upload them


.buildkite/commands/run-unit-tests.sh

ParaskP7 · 2025-10-08T10:24:05Z

.buildkite/commands/run-unit-tests.sh

+temp_test_results_dir=$(mktemp -d)

+# Define test result directories for each module
+declare -A TEST_RESULT_DIRS=(


Praise (❤️): Love this approach, much thing so much simpler, although, as per my previous comment, it is now harder to make this too dynamic... 💭

ParaskP7 · 2025-10-08T10:24:27Z

.buildkite/commands/upload-code-coverage.sh

+done
+
+# Upload all coverage reports in a single execution
+./codecov -t "$CODECOV_TOKEN" "${coverage_args[@]}"


Praise (❤️): Nice and TIL! 🥇

sonarqubecloud · 2025-10-08T10:48:29Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Run all unit tests in a single Buildkite job

87e3eeb

wzieba added 5 commits October 6, 2025 16:23

Fix results file path

3212163

Refactor and fix merging junit reports

4b5829b

Fix test collector token name

3105725

Remove Jetpack variant when runnig unit tests. Use wasabi instead…

0308568

… of `jalapeno`

Provide Kover reports to Codecov

a90bbd0

As Codecov have difficulties finding these reports on its own

wzieba mentioned this pull request Oct 7, 2025

[DNM] Unit tests in single job failure #22263

Closed

wzieba requested a review from ParaskP7 October 7, 2025 11:29

wzieba marked this pull request as ready for review October 7, 2025 11:29

wzieba added this to the 26.3 milestone Oct 7, 2025

wzieba added the internal label Oct 7, 2025

ParaskP7 requested a review from Copilot October 8, 2025 09:33

Copilot AI reviewed Oct 8, 2025

View reviewed changes

.buildkite/commands/run-unit-tests.sh Show resolved Hide resolved

ParaskP7 approved these changes Oct 8, 2025

View reviewed changes

Move temp_test_results_dir declaration closer to its usage

454737c

wzieba modified the milestones: 26.3, 26.4 Oct 8, 2025

wzieba enabled auto-merge (squash) October 8, 2025 10:48

wzieba merged commit 783d50e into trunk Oct 8, 2025
23 of 24 checks passed

wzieba deleted the tests_in_single_job branch October 8, 2025 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run all unit tests in a single Buildkite job #22259

Run all unit tests in a single Buildkite job #22259

wzieba commented Oct 6, 2025 •

edited

Loading

Uh oh!

wpmobilebot commented Oct 6, 2025 •

edited

Loading

Uh oh!

wpmobilebot commented Oct 6, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

ParaskP7 commented Oct 8, 2025

Uh oh!

ParaskP7 left a comment

Uh oh!

Uh oh!

ParaskP7 Oct 8, 2025

Uh oh!

Uh oh!

ParaskP7 Oct 8, 2025

Uh oh!

ParaskP7 Oct 8, 2025

Uh oh!

sonarqubecloud bot commented Oct 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Run all unit tests in a single Buildkite job #22259

Run all unit tests in a single Buildkite job #22259

Conversation

wzieba commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Testing instructions

Uh oh!

wpmobilebot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

ParaskP7 commented Oct 8, 2025

Uh oh!

ParaskP7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ParaskP7 Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ParaskP7 Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ParaskP7 Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Oct 8, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wzieba commented Oct 6, 2025 •

edited

Loading

wpmobilebot commented Oct 6, 2025 •

edited

Loading

wpmobilebot commented Oct 6, 2025 •

edited

Loading

codecov bot commented Oct 7, 2025 •

edited

Loading