test(e2e): run demo suites one at a time - #296
Merged
Merged
Conversation
📦 Bundle size impactComparing No size changes detected. 🟢 |
TarikGul
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Runs the demo end to end suites one package at a time instead of four at once, so the E2E job reports a usable result again.
Changes
product-sdk/package.json, one line, thetest:e2escript.--workspace-concurrency=1runs one demo package at a time, which is what avoids the failure.--no-bailkeeps every package reporting. pnpm stops at the first failing package by default, so run one at a time a hang incontracts, third of nine alphabetically, would leave the six after it unrun.Why these changes
The E2E job is failing often enough on
mainand the failure only reproduces when demo packages run at the same time. A demo on its own passes 8 runs of 8, the full suite four at once fails 4 of 5, the same suite one at a time passes 6 of 6.Running them one at a time is therefore the smallest change that makes the job report a usable result.
Testing
Run from
product-sdk/:All nine demo packages should report one after another, in about 2.5 minutes on a clean pass. On a failing run, check that all nine still report, which is what
--no-bailis for.Test results
Measured locally on
mainplus the host-api-test-sdk 0.12.0 bump.This branch has not been measured on CI yet.
Known limitations
This lowers the chance of hitting the hang, it does not remove it. In 2 of the 6 passing runs a package reported one fewer pass and ran about 90 seconds long, which is what Playwright absorbing a failure as a flaky retry looks like. That was never directly observed, so treat it as indicated, not proven.
Two reasons to expect residual flakiness. The bad window varies in length, and some specs burn both Playwright attempts at about 3 minutes each, which no retry policy survives. And a separate known flake, the public Paseo RPC, produces the same
"connecting"and 90 second signature.Six runs is thin, consistent with a residual failure rate between 1 in 4 and 1 in 6. A single red run after this lands does not mean the change failed.