closeSession and stopWebDriver failures cause cucumber formatter to fail

[tosborne1215]

There have been previous issues opened about a similar issue, but I think I've seen enough to be able to explain what the problem is.

I've noticed that the failure of closeSession and stopWebDriver can cause the status of a scenario to be undefined. When this happens the cucumber summary formatter then fails. For me that means my pipeline fails and fails to report which makes this a big issue for us.

I do not have steps to reproduce as this is not something that happens due to execution flow. However, when there are connection issues I have seen the issue. Here is an example stack trace that caused the issue.

11:09:55  �POST http://hub-cloud.browserstack.com /wd/hub/session/78ad5a74fed85c1538d86ea2e44efe62acf8c288/elements - ECONNRESET
11:09:55 Error: socket hang up
11:09:55    at createHangUpError (_http_client.js:331:15)
11:09:55    at Socket.socketCloseListener (_http_client.js:363:23)�[0m
11:09:55 �  at Socket.socketCloseListener (_http_client.js:363:9)

Here is the cucumber SummaryFormatter exception

11:09:55 TypeError: Cannot read property 'status' of undefined
11:09:55     at SummaryFormatter.isTestCaseFailure (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:25:106)
11:09:55     at _lodash.default.each.testCase (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:37:16)
11:09:55     at /data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:4911:15
11:09:55     at baseForOwn (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:2996:24)
11:09:55     at /data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:4880:18
11:09:55     at Function.forEach (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:9344:14)
11:09:55     at SummaryFormatter.logSummary (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:36:21)
11:09:55     at EventEmitter.PrettyFormatter.options.eventBroadcaster.on (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber-pretty/index.js:116:17)
11:09:55     at emitOne (events.js:121:20)
11:09:55     at EventEmitter.emit (events.js:211:7)
11:09:55     at Master.onSlaveClose (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/runtime/parallel/master.js:96:29)
11:09:55     at ChildProcess.slave.process.on (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/runtime/parallel/master.js:83:12)
11:09:55     at emitTwo (events.js:126:13)
11:09:55     at ChildProcess.emit (events.js:214:7)
11:09:55     at maybeClose (internal/child_process.js:925:16)
11:09:55     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

Your Environment

• Version used:
  "cucumber": "5.1.0",
  "nightwatch": "1.1.12",
  "nightwatch-api": "2.3.0",
• Browser Name and version: windows8Ie11 windows10Edge chrome safari pixel3
• Operating System and version (desktop or mobile):

[spnraju]

Hi @tosborne1215 , could you confirm if you still see this issue?

[tosborne1215]

@spnraju Yes I still experience this. Currently our workaround is hacking the cucumber formatter class to treat undefined statuses as failures. While developing my hack I think I was able to repro the issue or at least cause an issue that was similar.

1. Run 2 failing scenarios on BrowserStack
2. After the first failure stop the session through automate.browserstack.com before the 2nd scenario fails.
3. You will see the error after your last scenario times out.

I think this is just simulating a disconnect from BrowserStack, but I know that this is not the only thing that causes this error. Regardless, cucumber's default reporters expect every scenario to have a status. The fact that I can get an error where the status is undefined is a problem because if that occurs then there is no report.

[spnraju]

@tosborne1215 thank you for the details.