An HTTP response can no longer be lost to the next request's reset #52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # T1 test tier: the host-driven tests of the phyphox test strategy. They need a running app, | |
| # so they build for the simulator, install, and drive the app through the remote-access API - | |
| # which the -phyphoxRemote launch argument (AppDelegate, AutomationLaunchOptions) enables | |
| # headlessly. The tools live in phyphox-docs, checked out as a sibling like the T0 corpus. | |
| # | |
| # The tier is BUILT ONCE and then run in parallel: the build job hands its products over as an | |
| # artifact and each test job unpacks them and runs its slice with test-without-building. Split | |
| # without that hand-off, every job would rebuild and the wall clock would stay where it was. | |
| # What each job needs beforehand is the same, so it lives in .github/actions/t1-prepare. | |
| # | |
| # FIVE test jobs OF EQUAL LENGTH: the account runs at most five macOS jobs at a time, ACROSS | |
| # WORKFLOWS, and the t0 workflow - which starts on the same push - holds one of those five for | |
| # its first ten minutes. So four t1 jobs start immediately and one waits, and WHICH one waits is | |
| # the scheduler's choice, not ours: it was the shortest job on 2026-08-27 at 07:17 and the | |
| # longest at 08:20, which cost that run ten minutes. Packing a short job for the late slot | |
| # therefore does not work; jobs of the same length do, because it stops mattering which one it | |
| # picks. | |
| # | |
| # Measured minutes (the run of 2026-08-27, which took 29 with five equal jobs; 67 serially | |
| # before any of this): | |
| # | |
| # language sweep 19.7 experiment sweep 12.7 UI suites 13.5 (chrome 2.7, save-to-collection | |
| # 2.5, lifecycle 2.0, view behavior 1.5, accessibility 1.3, switch-bypassed 1.3) | |
| # tablet chrome 3.1 graph goldens 2.2 network fixtures 1.3 contract test 1.1 | |
| # and about four minutes per job for the checkout, the build artifact, a booted simulator and | |
| # the warm-up launch | |
| # | |
| # The two sweeps are split with the shard mechanism the test matrix defines (--shard i/n for the | |
| # experiments driver, PHYPHOX_TEST_LANGUAGE_SHARD for the language sweep), which is what makes a | |
| # balanced packing possible at all - each of them alone was longer than the target. What the | |
| # capacity allows is about 19 minutes: four jobs running the whole time plus one starting seven | |
| # minutes late, against 52 minutes of work. | |
| # | |
| # NOT used: xcodebuild's own -parallel-testing-enabled. Measured on a 12-core development Mac, | |
| # the UI suite went from 460 s to 321 s with two workers - but it parallelises by test CLASS, so | |
| # it does nothing for the language sweep, which is one method looping over 22 languages and is | |
| # the longest slice of the tier. It would only speed up a job that is no longer the critical | |
| # path, on runners with a fraction of the cores and two simulator clones competing for them. | |
| # Measured, written down, dropped. | |
| # | |
| # Deliberately NOT scheduled: T1 runs on pushes to the development and release branches and on | |
| # demand. It costs macOS runner minutes and needs no nightly cadence. | |
| # | |
| # What the simulator can and cannot do: it has no motion sensors, no barometer, no proximity | |
| # and no ambient light, so about half the shipped collection cannot open there - the app | |
| # reports the missing sensor and returns to the collection. The experiments driver classifies | |
| # those as not loadable on this target rather than failed, and skips microphone experiments | |
| # (AVAudioEngine's input can abort the app in a simulator), so the end-to-end job covers the | |
| # loadable subset. Real coverage of the sensor experiments is a T2 lab-phone job, not this one. | |
| name: t1 | |
| on: | |
| push: | |
| branches: | |
| - development | |
| - 'release/**' | |
| workflow_dispatch: | |
| # A push supersedes the run of the push before it: the tier costs the better part of an hour of | |
| # macOS runner time, and an outdated answer is worth none of it. | |
| concurrency: | |
| group: t1-${{ github.ref }} | |
| cancel-in-progress: true | |
| env: | |
| BUNDLE: de.rwth-aachen.physics.phyphox | |
| # The port the app is told to serve remote access on (-phyphoxRemotePort); the tools use | |
| # the same one, the driver passes it to every launch itself. | |
| PORT: '8080' | |
| jobs: | |
| build: | |
| runs-on: macos-latest | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| submodules: recursive | |
| # The package checkouts are the same for every run that does not change Package.resolved, | |
| # and resolving them is network-bound rather than CPU-bound | |
| - uses: actions/cache@v4 | |
| with: | |
| path: ${{ github.workspace }}/spm | |
| key: spm-${{ runner.os }}-${{ hashFiles('phyphox-ios/phyphox-iOS/phyphox.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }} | |
| restore-keys: spm-${{ runner.os }}- | |
| - name: Build for testing | |
| run: > | |
| cd phyphox-ios/phyphox-iOS && xcodebuild build-for-testing -project phyphox.xcodeproj | |
| -scheme phyphox -destination 'generic/platform=iOS Simulator' | |
| -derivedDataPath "$GITHUB_WORKSPACE/dd" | |
| -clonedSourcePackagesDirPath "$GITHUB_WORKSPACE/spm" CODE_SIGNING_ALLOWED=NO | |
| - name: Pack the build | |
| run: | | |
| set -euo pipefail | |
| # tar, not the artifact's own layout: upload-artifact does not preserve the executable | |
| # bit, and an app bundle that cannot be executed fails in ways that look like the app | |
| # is broken. The archive carries the .xctestrun as well, whose paths are relative to it. | |
| tar -czf "$GITHUB_WORKSPACE/products.tar.gz" -C "$GITHUB_WORKSPACE/dd/Build" Products | |
| ls -lh "$GITHUB_WORKSPACE/products.tar.gz" | |
| - uses: actions/upload-artifact@v4 | |
| with: | |
| name: t1-build | |
| path: products.tar.gz | |
| retention-days: 1 | |
| compression-level: 0 | |
| # phyphox-test: translations-ui | |
| # phyphox-test: remote-contract-t1 | |
| languages-1: | |
| needs: build | |
| runs-on: macos-latest | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| - id: prepare | |
| uses: ./phyphox-ios/.github/actions/t1-prepare | |
| - name: Test dependencies | |
| run: | | |
| set -euo pipefail | |
| pip3 install --break-system-packages pyyaml jsonschema | |
| - name: Half of the shipped languages | |
| run: | | |
| set -euo pipefail | |
| # One launch per language: the experiment screen, its menu and the collection, checked | |
| # for text that lands outside the screen and for a menu that lost entries. Which | |
| # languages is decided by the shard convention of the test matrix - round robin over | |
| # the sorted language list, so both halves cost the same and Android's shards cover | |
| # the same languages. A test bundle does not inherit the environment of the xcodebuild | |
| # that starts it, so the variable goes into the .xctestrun. | |
| mkdir -p t1-diagnostics | |
| python3 phyphox-ios/.github/scripts/xctestrun_env.py \ | |
| "${{ steps.prepare.outputs.xctestrun }}" phyphoxUITests \ | |
| PHYPHOX_TEST_LANGUAGE_SHARD=1/2 | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -resultBundlePath "$GITHUB_WORKSPACE/t1-diagnostics/languages-1-2.xcresult" \ | |
| -only-testing:phyphoxUITests/TranslationsUITests | |
| - name: Remote interface contract test | |
| run: | | |
| set -euo pipefail | |
| UDID="${{ steps.prepare.outputs.udid }}" | |
| BASE="http://127.0.0.1:$PORT" | |
| # The tone generator needs no sensor, no camera and no microphone - the least the | |
| # simulator can get wrong. Swap it for camera-luminance.phyphox if a runner ever | |
| # lacks an audio output device. | |
| xcrun simctl launch --terminate-running-process "$UDID" "$BUNDLE" \ | |
| -phyphoxUrl "phyphox://asset=tone_generator.phyphox" \ | |
| -phyphoxRemote -phyphoxRemotePort "$PORT" | |
| for i in $(seq 30); do | |
| curl -fs -m 2 "$BASE/config" >/dev/null && break | |
| sleep 1 | |
| done | |
| curl -fs -m 5 "$BASE/config" >/dev/null | |
| # Probes are uninformative on an experiment that has never measured | |
| curl -fs -m 5 "$BASE/control?cmd=start" >/dev/null | |
| sleep 3 | |
| curl -fs -m 5 "$BASE/control?cmd=stop" >/dev/null | |
| python3 phyphox-docs/tools/contract_test.py --ios "$BASE" --allow-control | |
| xcrun simctl terminate "$UDID" "$BUNDLE" || true | |
| - name: Diagnostics | |
| if: failure() | |
| run: bash phyphox-ios/.github/scripts/t1_diagnostics.sh "${{ steps.prepare.outputs.udid }}" | |
| - uses: actions/upload-artifact@v4 | |
| if: always() | |
| with: | |
| name: t1-results-languages-1 | |
| path: t1-diagnostics/ | |
| if-no-files-found: ignore | |
| # phyphox-test: translations-ui | |
| # phyphox-test: network-http | |
| # phyphox-test: network-mqtt | |
| # phyphox-test: graph-snapshots | |
| languages-2: | |
| needs: build | |
| runs-on: macos-latest | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| - id: prepare | |
| uses: ./phyphox-ios/.github/actions/t1-prepare | |
| - name: Test dependencies | |
| run: | | |
| set -euo pipefail | |
| pip3 install --break-system-packages pyyaml jsonschema | |
| # the broker the mqtt network fixture talks to | |
| brew install mosquitto | |
| - name: The other half of the shipped languages | |
| run: | | |
| set -euo pipefail | |
| # One launch per language: the experiment screen, its menu and the collection, checked | |
| # for text that lands outside the screen and for a menu that lost entries. Which | |
| # languages is decided by the shard convention of the test matrix - round robin over | |
| # the sorted language list, so both halves cost the same and Android's shards cover | |
| # the same languages. A test bundle does not inherit the environment of the xcodebuild | |
| # that starts it, so the variable goes into the .xctestrun. | |
| mkdir -p t1-diagnostics | |
| python3 phyphox-ios/.github/scripts/xctestrun_env.py \ | |
| "${{ steps.prepare.outputs.xctestrun }}" phyphoxUITests \ | |
| PHYPHOX_TEST_LANGUAGE_SHARD=2/2 | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -resultBundlePath "$GITHUB_WORKSPACE/t1-diagnostics/languages-2-2.xcresult" \ | |
| -only-testing:phyphoxUITests/TranslationsUITests | |
| - name: Network fixtures | |
| run: | | |
| set -euo pipefail | |
| UDID="${{ steps.prepare.outputs.udid }}" | |
| # The runner starts phyphox-docs' fixture server and the broker itself, substitutes | |
| # the placeholders and serves the experiments over http, which is how they reach the | |
| # app. | |
| xcrun simctl terminate "$UDID" "$BUNDLE" >/dev/null 2>&1 || true | |
| # The helpers' own logs go to the artifact: a fixture that cannot reach its server | |
| # produces empty buffers, which reads like an app failure until you can see whether | |
| # the server was even up. | |
| mkdir -p t1-diagnostics | |
| python3 -u phyphox-ios/.github/scripts/network_fixtures.py --udid "$UDID" \ | |
| --fixtures phyphox-docs/fixtures/network --port "$PORT" \ | |
| --log-dir t1-diagnostics --keep-going | |
| - name: Graph goldens | |
| run: | | |
| set -euo pipefail | |
| mkdir -p t1-diagnostics | |
| # The plot is drawn by OpenGL into a GLKView, which renders nothing in a layer-based | |
| # capture: the goldens need the view in a visible window on a real context, which is | |
| # what makes this row T1 while the other view goldens are T0. | |
| # The result bundle carries the reference and the actual image of a failed golden, | |
| # which is the only way to see WHY one did not match on a machine you cannot open | |
| xcrun simctl terminate "${{ steps.prepare.outputs.udid }}" "$BUNDLE" >/dev/null 2>&1 || true | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -resultBundlePath "$GITHUB_WORKSPACE/t1-diagnostics/graph-goldens.xcresult" \ | |
| -only-testing:phyphoxTests/GraphSnapshotTests | |
| - name: Diagnostics | |
| if: failure() | |
| run: bash phyphox-ios/.github/scripts/t1_diagnostics.sh "${{ steps.prepare.outputs.udid }}" | |
| - uses: actions/upload-artifact@v4 | |
| if: always() | |
| with: | |
| name: t1-results-languages-2 | |
| path: t1-diagnostics/ | |
| if-no-files-found: ignore | |
| # phyphox-test: experiments-e2e-t1 | |
| # phyphox-test: switch-bypassed-ui | |
| # phyphox-test: accessibility-smoke | |
| experiments-1: | |
| needs: build | |
| runs-on: macos-latest | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| # the experiment collection the driver sweeps | |
| submodules: recursive | |
| - id: prepare | |
| uses: ./phyphox-ios/.github/actions/t1-prepare | |
| - name: Test dependencies | |
| run: | | |
| set -euo pipefail | |
| pip3 install --break-system-packages pyyaml jsonschema | |
| - name: Experiments end to end, shard 1/2 | |
| run: | | |
| set -euo pipefail | |
| UDID="${{ steps.prepare.outputs.udid }}" | |
| # The driver opens, starts, stops and exports every experiment of its shard - it never | |
| # presses a button, which matters for sensordb.phyphox: its connection has no interval | |
| # and only fires on the Submit button, so nothing is posted to phyphox.org from here. | |
| # Do not point a trigger probe (contract_test --allow-control) at that experiment. | |
| # --shard is round robin over the sorted collection. | |
| xcrun simctl terminate "$UDID" "$BUNDLE" >/dev/null 2>&1 || true | |
| # -u: unbuffered, so each experiment is timestamped in the step log as it happens | |
| # instead of the whole sweep arriving at once when the pipe is flushed | |
| python3 -u phyphox-docs/tools/t1_experiments.py --platform ios --serial "$UDID" \ | |
| --port "$PORT" --collection "$GITHUB_WORKSPACE/phyphox-ios/phyphox-experiments" \ | |
| --shard 1/2 --out t1-experiments-1-2.json | |
| - name: What the switches bypass, and the accessibility audit | |
| run: | | |
| set -euo pipefail | |
| # Everything -phyphoxAutoConfirm and -phyphoxRemote skip, driven by hand instead, plus a report-only audit | |
| xcrun simctl terminate "${{ steps.prepare.outputs.udid }}" "$BUNDLE" >/dev/null 2>&1 || true | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -only-testing:phyphoxUITests/SwitchBypassedUITests \ | |
| -only-testing:phyphoxUITests/AccessibilitySmokeTests | |
| - name: Diagnostics | |
| if: failure() | |
| run: bash phyphox-ios/.github/scripts/t1_diagnostics.sh "${{ steps.prepare.outputs.udid }}" | |
| - uses: actions/upload-artifact@v4 | |
| if: always() | |
| with: | |
| name: t1-results-experiments-1 | |
| path: | | |
| t1-*.json | |
| t1-diagnostics/ | |
| if-no-files-found: ignore | |
| # phyphox-test: experiments-e2e-t1 | |
| # phyphox-test: app-chrome | |
| # phyphox-test: view-behavior | |
| experiments-2: | |
| needs: build | |
| runs-on: macos-latest | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| # the experiment collection the driver sweeps | |
| submodules: recursive | |
| - id: prepare | |
| uses: ./phyphox-ios/.github/actions/t1-prepare | |
| - name: Test dependencies | |
| run: | | |
| set -euo pipefail | |
| pip3 install --break-system-packages pyyaml jsonschema | |
| - name: Experiments end to end, shard 2/2 | |
| run: | | |
| set -euo pipefail | |
| UDID="${{ steps.prepare.outputs.udid }}" | |
| # The driver opens, starts, stops and exports every experiment of its shard - it never | |
| # presses a button, which matters for sensordb.phyphox: its connection has no interval | |
| # and only fires on the Submit button, so nothing is posted to phyphox.org from here. | |
| # Do not point a trigger probe (contract_test --allow-control) at that experiment. | |
| # --shard is round robin over the sorted collection. | |
| xcrun simctl terminate "$UDID" "$BUNDLE" >/dev/null 2>&1 || true | |
| # -u: unbuffered, so each experiment is timestamped in the step log as it happens | |
| # instead of the whole sweep arriving at once when the pipe is flushed | |
| python3 -u phyphox-docs/tools/t1_experiments.py --platform ios --serial "$UDID" \ | |
| --port "$PORT" --collection "$GITHUB_WORKSPACE/phyphox-ios/phyphox-experiments" \ | |
| --shard 2/2 --out t1-experiments-2-2.json | |
| - name: Chrome and the view elements | |
| run: | | |
| set -euo pipefail | |
| # What surrounds an experiment, and what the fixtures' controls write | |
| xcrun simctl terminate "${{ steps.prepare.outputs.udid }}" "$BUNDLE" >/dev/null 2>&1 || true | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -only-testing:phyphoxUITests/AppChromeTests \ | |
| -only-testing:phyphoxUITests/ViewBehaviorTests | |
| - name: Diagnostics | |
| if: failure() | |
| run: bash phyphox-ios/.github/scripts/t1_diagnostics.sh "${{ steps.prepare.outputs.udid }}" | |
| - uses: actions/upload-artifact@v4 | |
| if: always() | |
| with: | |
| name: t1-results-experiments-2 | |
| path: | | |
| t1-*.json | |
| t1-diagnostics/ | |
| if-no-files-found: ignore | |
| # phyphox-test: lifecycle | |
| # phyphox-test: save-to-collection | |
| # phyphox-test: app-chrome | |
| interaction: | |
| needs: build | |
| runs-on: macos-latest | |
| timeout-minutes: 40 | |
| steps: | |
| - uses: actions/checkout@v5 | |
| with: | |
| path: phyphox-ios | |
| - id: prepare | |
| uses: ./phyphox-ios/.github/actions/t1-prepare | |
| - name: Interruptions and saving to the collection | |
| run: | | |
| set -euo pipefail | |
| # What an interruption does to a run, and saving a container the app opened | |
| xcrun simctl terminate "${{ steps.prepare.outputs.udid }}" "$BUNDLE" >/dev/null 2>&1 || true | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=${{ steps.prepare.outputs.udid }}" \ | |
| -only-testing:phyphoxUITests/LifecycleTests \ | |
| -only-testing:phyphoxUITests/SaveToCollectionTests | |
| - name: App chrome on a tablet | |
| run: | | |
| set -euo pipefail | |
| # The layouts differ there, and an action sheet becomes a popover, which is a | |
| # different dismissal path. The tablet is this job's second act rather than a job of | |
| # its own - a sixth job would wait for a runner slot. | |
| export FAMILY=iPad | |
| IPAD=$(xcrun simctl list devices available -j | python3 -c "import json,sys,os;d=json.load(sys.stdin)['devices'];f=os.environ['FAMILY'];print(([x['udid'] for r in sorted(d) for x in d[r] if x.get('isAvailable') and x['name'].startswith(f)] or [''])[-1])") | |
| if [ -z "$IPAD" ]; then echo "no iPad simulator on this runner - tablet layouts not covered"; exit 0; fi | |
| # Booted here rather than in the preparation: two simulators booting at once starve | |
| # the runner, and by now the phone is done with its work. Bounded, so a device that | |
| # never comes up costs this step and not the job. | |
| xcrun simctl boot "$IPAD" || true | |
| if ! bash phyphox-ios/.github/scripts/wait_for_boot.sh "$IPAD" 420; then | |
| echo "::warning::the iPad simulator did not boot - tablet layouts not covered" | |
| exit 0 | |
| fi | |
| xcrun simctl install "$IPAD" "$GITHUB_WORKSPACE/dd/Build/Products/Debug-iphonesimulator/phyphox.app" | |
| xcrun simctl privacy "$IPAD" grant all "$BUNDLE" || true | |
| # Installing on a struggling device took 5.4 minutes once (run 33388729717), and the | |
| # device was gone by the end of it - xcodebuild then listed no simulator at all and the | |
| # job failed on a runner problem rather than on the app. The tablet is the optional half | |
| # of this job, so a device that does not survive its own setup skips the tests it was | |
| # for. | |
| if ! xcrun simctl list devices | grep -q "$IPAD.*Booted"; then | |
| echo "::warning::the iPad simulator was lost while the app was installed - tablet layouts not covered" | |
| exit 0 | |
| fi | |
| # The same warm-up the preparation gives the phone (see t1-prepare): the first launch | |
| # of a freshly installed app on a freshly booted device is the slow one, and on the | |
| # tablet it comes right after a boot that has just starved the runner. Run 33595623632 | |
| # lost that first launch to XCUITest's timeout, the next test then attached to the | |
| # late arrival of the previous one, and the third spent a minute failing to terminate | |
| # it - four failures from one slow start. So the app is launched here once, with an | |
| # experiment so that remote access can say when it is really up. Its own port: the | |
| # simulators share the host's loopback, and a phone-side leftover on 8080 must not | |
| # answer for the tablet. | |
| WARMUP_PORT=8085 | |
| xcrun simctl launch "$IPAD" "$BUNDLE" \ | |
| -phyphoxUrl "phyphox://asset=tone_generator.phyphox" -phyphoxAutoConfirm \ | |
| -phyphoxRemote -phyphoxRemotePort "$WARMUP_PORT" >/dev/null 2>&1 || true | |
| for i in $(seq 120); do | |
| curl -fs -m 2 "http://127.0.0.1:$WARMUP_PORT/config" >/dev/null && break | |
| sleep 1 | |
| done | |
| if curl -fs -m 5 "http://127.0.0.1:$WARMUP_PORT/config" >/dev/null; then | |
| echo "the app is up on the tablet after $i s" | |
| else | |
| # Not fatal: what matters is that the tests get a warm app, and a genuinely broken | |
| # launch fails them loudly enough | |
| echo "::warning::the tablet warm-up launch did not answer within 120 s" | |
| fi | |
| xcrun simctl terminate "$IPAD" "$BUNDLE" >/dev/null 2>&1 || true | |
| xcodebuild test-without-building -xctestrun "${{ steps.prepare.outputs.xctestrun }}" \ | |
| -destination "id=$IPAD" -only-testing:phyphoxUITests/AppChromeTests | |
| - name: Diagnostics | |
| if: failure() | |
| run: bash phyphox-ios/.github/scripts/t1_diagnostics.sh "${{ steps.prepare.outputs.udid }}" | |
| - uses: actions/upload-artifact@v4 | |
| if: always() | |
| with: | |
| name: t1-results-interaction | |
| path: t1-diagnostics/ | |
| if-no-files-found: ignore |