chore(web): enable testing against legacy devices via BrowserStack

[jahorton]

Fixes #11501.

Well, not completely... but close enough to merit closing the issue, I suppose.

My progress has been impeded somewhat due to issues I documented in modernweb-dev/web#2756. There are some limitations I'd rather not have, but at least things are far enough along to unblock progress.

Some neat things: we actually get more data about the test runs within the BrowserStack dashboard now, along with greater control of test-run labeling. A rough glimpse of that:

1. As our test runs do have separate test sets for specific modules, we can label which module each run is for while keeping them all associated within the same overall build version label.

   

2. How about some neat network logs? This way, we can see what sort of load times different parts of the test suites are costing to have better data for setting timeouts and/or determining sources of related instability.

   

3. Even the "Text Logs" provide us a little more information than they used to.

4. Console logs? Sure, why not?

   

   Messages were temporarily added while I was analyzing the issues that led to [@web/test-runner-browserstack] + [@web/dev-server-legacy]: issues when using BrowserStack launcher for targeting certain older devices modernweb-dev/web#2756.

Note that a new package had to be added in order to support the older version of Chrome used here as a test target - @web/dev-server-legacy. It seems simple on the surface, but its dependencies are the source of the monstrous set of changes to package-lock.json.

@keymanapp-test-bot skip

[keymanapp-test-bot]

User Test Results

Test specification and instructions

User tests are not required

[jahorton]

Seems small on the surface, but comes with a very significant dependency cost - just look at the "lines changed" count for package-lock.json!

[mcdurdin]

Okay, so I am not keen on this. Yeah, it's a dev dependency, but it's gonna cost us significantly in dependabot and more. Adding dependencies needs to go through review and cost/benefit analysis.

[jahorton]

So... should I put this back in draft, pending such a review? Or just abandon this and close out the issue as a wontfix?

We can't reasonably test against older Chrome versions without it, which kinda defeats our main reason to do remote testing against BrowserStack.

[mcdurdin]

It is relatively easy to download older versions of Chromium and test on those -- which I think would be sufficient. That then resolves the need to drag in 5,000 new lines of dependencies into package-lock.json and probably removes most of our BrowserStack requirement altogether?

[jahorton]

To be clear, for this older version of Chrome. Supposedly, version 63 or 64 would allow us to drop it, but I'd want to check that experimentally first.

[mcdurdin]

force-push means I cannot see the context here... But I think you mean version 57 right?

[jahorton]

Test (Keyman - iOS) build failure cause:

15:38:43   /opt/homebrew/Cellar/fastlane/2.220.0/libexec/gems/fastlane-2.220.0/spaceship/lib/spaceship/connect_api/api_client.rb:250:in `handle_error': [!] The request could not be completed because: (Spaceship::ProgramLicenseAgreementUpdated)
15:38:43     A required agreement is missing or has expired. - This request requires an in-effect agreement that has not been signed or has expired.

From https://build.palaso.org/buildConfiguration/Keyman_iOS_TestPullRequests/474686?buildTab=log&linesState=2357.2448&logView=flowAware&focusLine=2565

[jahorton]

Note: I haven't gotten it working perfectly yet. Over the course of a day, the test targets aside from the first - Chrome on Windows - tend to stop connecting, then later start working again. It appears that the test-runner will sometimes just... force-quit when this happens, with no further info provided or obtainable. (Our custom reporter's stop method is even bypassed completely.)

So, it's not stable, thus not "good enough" to connect to CI... but it gets us much closer to where we were before we shifted from using Karma to BrowserStack.

[ermshiperete]

no named() for this one?

[jahorton]

I feel like I had it working before, but the names aren't actually showing up from the reporter. So, I didn't bother. 🤷

[jahorton]

Oof, that was quite the merge conflict; this branch started before @ermshiperete's recent reorganization work. I'm pushing now and will use the "changed files" view as a guide for further merge cleanup.

[mcdurdin]

Okay, so I am not keen on this. Yeah, it's a dev dependency, but it's gonna cost us significantly in dependabot and more. Adding dependencies needs to go through review and cost/benefit analysis.