Most of the story tests failing when used with storybook/[email protected]

[certainlyakey]

Hey, thanks for the great tool!

I noticed that when it is used with a latest @storybook/addon-a11y (8.5.0), most of the tests fail with this error:

page.evaluate: Error: Axe is already running. Use `await axe.run()` to wait for the previous run to finish before starting a new run.
    at Cv (http://127.0.0.1:8000/assets/axe-BHni_kjN.js:18:55391)
    at Object.oT [as run] (http://127.0.0.1:8000/assets/axe-BHni_kjN.js:24:93604)
    at eval (eval at evaluate (:234:30), <anonymous>:17:27)
    at async <anonymous>:260:30

I discovered this comment. Considering that case, I found out when I go back to 8.4.7 for @storybook/addon-a11y (as well as all the other Storybook-related packages), the issue goes away.

Not sure if this will lead somewhere, and there's no documented issue yet for @storybook/addon-a11y itself.

[mpelham]

Thanks for reporting @certainlyakey!

Just some notes:
Locally, I was able to get my tests to pass (most of the time), by setting --maxWorkers=1 and by reducing the amount of calls to AXE. Note: This did not help when running it via Github Actions / etc.

For example: https://storybook.js.org/docs/writing-tests/test-runner#helpers

async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);
    // Do not run a11y tests on disabled stories.
    if (storyContext.parameters?.a11y?.disable) {
      return;
    }
    await checkA11y(page, '#storybook-root', {
      ...
}

vs

async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);
    // Apply story-level a11y rules
    await configureAxe(page, {
      rules: storyContext.parameters?.a11y?.config?.rules,
    });

    const element = storyContext.parameters?.a11y?.element ?? '#storybook-root';
    await checkA11y(page, element, {
   ...
}

In the second example, we are much more likely to experience the axe.run() issue you described because we have multiple asynchronous await methods in each postVisit. I found that I experienced less flakey tests by moving to the first example instead of the second.

This is not meant as a solution, just hopefully sheds some light on the issue.

[ahuth]

Thanks for reporting! I haven't tried using React 19 or storybook 8.5 in this project, yet. I'll look into this tomorrow my time.

[booc0mtaco]

@ahuth Also having a look today. Thanks all for the reports and notes! Will keep this updated with any results from investigations :-)

[booc0mtaco]

I believe I have found the issue:

• axe-storybook-testing will inject a version of axe-core into the page, living at window.axe
• with the latest version of the storybook packages, it will ALSO inject a version on top of the same location (somewhere?)
• after the first test run completes, things get out of sync, as some of the state used by the version we inject remains, but the object on window has been (partially?) overwritten (We are now in the bad state)

Will spend some time confirming and testing whether this is fully the case. Discovered this locally by opening a test on previous storybook packages and newest ones, navigating to the iframe view of the test, and checking for window.axe in the console in each.

• https://github.com/storybookjs/storybook/pull/29643/files#diff-0ca25392be00b3f87ce15e4a68ac228f0a6a4ccedf4e43a80b76743cf04de45aR42
• https://github.com/dequelabs/axe-core/blob/develop/lib/core/index.js#L24

[ahuth]

Good finds! Confirmed that updating storybook to 8.5.0 and including storybook-addon-a11y results in this issue. Also that not including storybook-addon-a11y "fixes" the issue.

Haven't found a good way to resolve it yet, though 😐

[booc0mtaco]

Ideally there would be a way to intercept/modify the stories being tested. But, since we run storybook build, then run axe-storybook-testing, the built tests already include the usage of @storybook/addons-a11y. It's very tricky and probably not possible to stop window.axe from being overwritten.

I did find a solution that we can document here for the time-being. We can use storybook build --test to disable the addon (and maybe other things...).

• update the command to run storybook build --test && axe-storybook # or similar
• in your main.(js|ts) file, add in the config you'd like to disable. In our case build: { test: { disabledAddons: ['@storybook/addon-a11y'] } }
• re-run the script

It took me a second to understand the storybook documentation on this; you need to add --test which won't do anything until you ALSO add some changes to main.(js|ts)

@mpelham @certainlyakey given your projects, would you be able to try this approach?

[certainlyakey]

Sure, I will try it out soon. Thank you for the research! On 23. Jan 2025, at 18:22, Andrew Holloway ***@***.***> wrote: Ideally there would be a way to intercept/modify the stories being tested. But, since we run storybook build, then run axe-storybook-testing, the built tests already include the usage of @storybook/addons-a11y. It's very tricky and probably not possible to stop window.axe from being overwritten. I did find a solution that we can document here for the time-being. We can use storybook build --test to disable the addon (and maybe other things...). update the command to run storybook build --test && axe-storybook # or similar in your main.(js|ts) file, add in the config you'd like to disable. In our case build: { test: { disabledAddons: ***@***.***/addon-a11y'] } } re-run the script It took me a second to understand the storybook documentation on this; you need to add --test which won't do anything until you ALSO add some changes to main.(js|ts) @mpelham @certainlyakey given your projects, would you be able to try this approach? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[valentinpalkovic]

Hi folks,

The Storybook Core Maintaner here who broke your integration for Storybook 8.5 🙃👋

Via query parameters, the a11y addon can be deactivated by appending the following information to the URL when visiting a particular story

http://localhost:6006/<story>?globals=a11y.manual:!true

Then axe from addon-a11y will not run! Can you try this out?

[booc0mtaco]

Hi folks,

The Storybook Core Maintaner here which broke your integration for Storybook 8.5 🙃👋

Via query parameters, the a11y addon can be deactivated by appending the following information to the URL when visiting a particular story

http://localhost:6006/?globals=a11y.manual:!true
Then axe from addon-a11y will not run! Can you try this out?

Hi @valentinpalkovic, thanks for the note! I'll try this out; sounds like it should work, even with stories that were built with the addon enabled. It's nice to have a few paths forward 💯 .

Will update with results once I get another 2 seconds

[booc0mtaco]

Documented using --test in a new wiki page to unblock folks needing to complete an upgrade: https://github.com/chanzuckerberg/axe-storybook-testing/wiki/FAQ-&-Troubleshooting

I have a local patch that can work without any config changes, using the tip from @valentinpalkovic (thanks again!). Will test that and work toward a PR including this change.

cc @ahuth

[certainlyakey]

@booc0mtaco sorry for the delay, the --test "hack" did work for me. Although i reckon it's better to update to the latest version of axe-storybook-testing anyway now.

[certainlyakey]

And the latest version also resolved it for me. So i guess the issue can be closed now.

[booc0mtaco]

Good to hear @certainlyakey . I wanted to add that documentation in case some other plugins (custom or otherwise) have a similar symptom (and to build something that may appear in search engine results).

Let us know if you run into any other issues.