-
Notifications
You must be signed in to change notification settings - Fork 192
BrowserStack Tests
Updated configurations and instructions are in PR #1432
While the tests are running, you can see failures both in the terminal and in the BrowserStack UI (there are some additional screenshots and text errors)
-
The Outreachy goal was to get the travis tests running on the BrowserStack environment (for forked repos) but I was unable to resolve the issue where ideally the BrowserStack access_key should remain secure (travis has docs for encrypting sensitive data but encrypted vars are unavailable to forked repos)
-
So the solution above is just an interim one where any user should be able to run tests (triggered locally) against the BrowserStack environment but there is a security issue of how to share the access_key...
-
I believe that running the BrowserStack tests with the current solution will add another timing variable that would probably result in additional test failures
- The number of tests that fail locally (running w/o BrowserStack vs w/ BrowserStack) are different
-
From the terminal output, it is not always obvious which test environment (OS/browser version) is associated with a failing test. So maybe unit tests should be modified to also print this info with error messages.
-
For the free BrowserStack account, there is a limit of running 2 tests in parallel. In the intern config file (currently browserstack.js), you could create a test grid with 6 different test environments but BrowserStack will run tests 3x.
-
FWIW I have some feeling that a bunch of the locally failing tests (even w/o BrowserStack) have some timing issue involved. Also that a bunch of the failing tests (specific to travis) have some missing fixture file issue involved. So if those can be cleaned up, I believe a good majority of failing tests will be resolved.
-
There is a new BrowserStack account with owner: webcompat_at_gmail.com
-
The new BrowserStack account will be free for another year (exp 03/30/2018). When the subscription is close to expiring, there will be a layer in the UI with a warning and a renewal request can be made to support_at_browserstack.com (the current contact is Mark Gomes).