UI comparison in PR using github action

[synox]

Story

As a developer, I want to be sure that my PR does not break anything major on an unrelated page.

Idea

When a Pull Request created, a bot automatically compares the main with the branch rendering of each page based on a list of the most important pages and the block library.

There is no need for storing historical screenshots, as we can load the main and branch pages at any time and directly compare them.

A visual difference has been detected for the following pages. To acknowledge/ignore, press the checkbox next to it.
[ ] /product/a {link to diff image}
[ ] /block-library/teaser {link to diff image}

Block the PR until all the checkboxes are checked.

How

Ideally, an existing solution can be customized to fit this need. Or Playwright and Github actions can be used to compare screenshots.

[dylandepass]

I like the idea, I think if visual regression testing becomes a common ask from customers than using helix bot would be a nice experience.

Worth noting that if customers really need this and they use storybook then they can use Chromatic today with Franklin.. For example, it would only be a couple lines of code to get visual regression testing on Rosalind which is setup with storybook for blocks and templates.

[synox]

I created a simple PoC, see PR. It compares a list of paths, and also fetches a list of indexes to get more paths.

Here is an example output after comparing 34 paths: https://github.com/hlxsites/vg-volvotrucks-us/pull/412#issuecomment-1536944669

See also hlxsites/24life#198

[sdmcraft]

We are using a variant of the playwright based visual tests suggested by @synox above for a customer project and it is working reasonable well for us. Though couple of feedback points:

1. It tends to make the PR conversations a bit too noisy if there are lot of syncs happening in it. @dnbute recently made a few enhancements in the associated Github workflow to reuse the same comment.
2. As the number of test paths keep getting added, the duration of the visual tests keeps increasing. And these may not cover situation where I may want to run the tests against a broader/customized set of webpages on one-off basis. To this end, I extracted this in a separate standalone tool to run the visual tests. More details can be found here.

[synox]

Thank you for this valuable feedback. More hands-on experience is definitely needed to make this tool more broadly usable.

[synox]

In additionally to the diff screenshots, I was thinking of forking https://github.com/pianomister/diffsite and using it to display pages side-by-side. This view is often easier to understand than the diff images.