Content loaded via BigPipe is sometimes not available by the time steps are executed, leading to random failures

[bkosborne]

I have tests that check if a status message contains some text or click a link in the tasks menu. Both of this content is loaded dynamically via BigPipe (if enabled) and happens via JS (if enabled). This creates a race condition, because BigPipe's JS needs time to actual replace the placeholders with the actual contents.

Here's an example:

Scenario: Content Author role is assigned default permissions                   # features/general/content_author_permissions.feature:4

    Given I am logged in as a "admin"                                             # Drupal\DrupalExtension\Context\DrupalContext::assertAuthenticatedByRole()

    And I am on "/admin/structure/types/add"                                      # Drupal\DrupalExtension\Context\MinkContext::visit()

    And I fill in "Behat Test Content Type" for "Name"                            # Drupal\DrupalExtension\Context\MinkContext::fillField()

    And I press "Edit" in the content region                                      # Drupal\DrupalExtension\Context\MinkContext::assertRegionPressButton()

    And I fill in "behat_test_content_type" for "Machine-readable name"           # Drupal\DrupalExtension\Context\MinkContext::fillField()

    And I press the "Save and manage fields" button                               # Drupal\DrupalExtension\Context\MinkContext::pressButton()

    Then I should see "The content type Behat Test Content Type has been added."  # Drupal\DrupalExtension\Context\MinkContext::assertPageContainsText()

    Given I am logged in as a "author"                                            # Drupal\DrupalExtension\Context\DrupalContext::assertAuthenticatedByRole()

    And I am on "/node/add/behat_test_content_type"                               # Drupal\DrupalExtension\Context\MinkContext::visit()

    And I fill in "Behat Test node" for "Title"                                   # Drupal\DrupalExtension\Context\MinkContext::fillField()

    And I press the "Save" button                                                 # Drupal\DrupalExtension\Context\MinkContext::pressButton()

    Then I should see "Behat Test Content Type Behat Test node has been created." # Drupal\DrupalExtension\Context\MinkContext::assertPageContainsText()

      The text "Behat Test Content Type Behat Test node has been created." was not found anywhere in the text of the current page. (Behat\Mink\Exception\ResponseTextException)

I use chrome for the driver.

I have things setup to take a screenshot and dump the page HTML on failure. The screenshot indeed shows no status message. However, when I look at the HTML of the page, I see this:

<div class="region region-highlighted">
    <div data-drupal-messages-fallback="" class="hidden"></div><span data-big-pipe-placeholder-id="callback=Drupal%5CCore%5CRender%5CElement%5CStatusMessages%3A%3ArenderMessages&amp;args%5B0%5D&amp;token=_HAdUpwWmet0TOTe2PSiJuMntExoshbm1kh2wQzzzAA"></span>
</div>

And I see that the actual message is available in the pending (not yet processed by BigPipe) script tag:

<script type="application/vnd.drupal-ajax" data-big-pipe-replacement-for-placeholder-with-id="callback=Drupal%5CCore%5CRender%5CElement%5CStatusMessages%3A%3ArenderMessages&amp;args%5B0%5D&amp;token=_HAdUpwWmet0TOTe2PSiJuMntExoshbm1kh2wQzzzAA">
    [{"command":"insert","method":"replaceWith","selector":"[data-big-pipe-placeholder-id=\u0022callback=Drupal%5CCore%5CRender%5CElement%5CStatusMessages%3A%3ArenderMessages\u0026args%5B0%5D\u0026token=_HAdUpwWmet0TOTe2PSiJuMntExoshbm1kh2wQzzzAA\u0022]","data":"    \u003Cdiv role=\u0022contentinfo\u0022 aria-label=\u0022Status message\u0022 class=\u0022alert alert-status\u0022\u003E\n                  \u003Ch2 class=\u0022visually-hidden\u0022\u003EStatus message\u003C\/h2\u003E\n                    Behat Test Content Type \u003Cem class=\u0022placeholder\u0022\u003E\u003Ca href=\u0022\/node\/634\u0022 hreflang=\u0022en\u0022\u003EBehat Test node\u003C\/a\u003E\u003C\/em\u003E has been created.\n            \u003C\/div\u003E\n    ","settings":null}]
    </script>

This problem as mentioned by Wim Leers over in PR #325:

Finally: when does Behat run the assertions? Does it wait for the load event to complete? I'm asking because otherwise tests using BigPipe could run into random failures depending on how fast the server responded, and depending on how fast the JS was executed.

So I wonder what the best solution is? For now, I've been just adding 1 second delays before my steps for checking messages and local task links, but this is not a good long term solution. It's also not a good long term solution to add specific delays to the step definitions themselves.

I wonder if there's a way to, after each page load if JS is enabled, pause until the "load" event is run as Wim suggested?