Feature proposal: Notification

[jmgarnier]

I read this blog post "Flaky Tests: Are You Sure You Want to Rerun Them?" by @semaphoreci and it was inspiring.

The cost of fixing of flickering tests is sometimes very high. Hence, I am using rspec-retry for chrome headless feature tests as sometimes, this is the only way to get a green build. Pragmatic way :)

With this issue, I'd like to discuss a new feature that I can implement:

Feature: Notification
  When a test is silently failing and passes after being re-run by `rspec-retry` - it's flacky / flickering / randomly failing - I want to be notified, So I can address it and make my build faster.

  Background:
  * Number of retries: 2

  Scenario: "Real failure" => Do not notify
    When a real failure occurs 🔴🔴, (2 failed attempts)
    Then it should not notify the developer as there could be hundreds

  Scenario: 1 "Flickering"" => Notify
    When a test is flickering 🔴💚, (1 failed attempt followed by a "passed")
    Then it should notify the developer with the BUILD url and details about the test which failed

I have already implemented a Slack notification so it's so much work.

[ngsikai]

Have similar issues dealing with flaky tests, this is a great idea! 👍