-
-
Notifications
You must be signed in to change notification settings - Fork 21
Description
Describe the Bug
Hi, first of all huge thanks for your work on this great plugin.
I've noticed [...].actual.png files don't have FRSOURCE_CPVRD_V set, which causes problems when they're used to update the baseline images.
In our workflow we mostly use Cypress on the command line (cypress run). We "accept" a new baseline image by overwriting the old baseline with the new [...].actual.png file (see "Additional context" for why we do this).
This mostly works fine, but this .actual.png file doesn't have the FRSOURCE_CPVRD_V field set. This means that when we run the same tests later, FRSOURCE_CPVRD_V gets set by the plugin, and therefore there is a diff according to git, but not visually. This can show up later in the diff for an unrelated change, which is difficult for a reviewer to understand, and will happen repeatedly as baselines are updated over time.
Steps to reproduce
Developer A:
- Create a test and run it, generating an initial baseline
- Modify the component under test such that the test will fail
- Run the test, which fails, producing
.actual.pngand.diff.pngfiles - Rename
.actual.pngto overwrite the baseline - Commit, push etc.
Developer B:
- Run the test, which passes, but which automatically updates the baseline to include
FRSOURCE_CPVRD_V - Now any commits/PR etc. this developer makes may include a confusing rewrite of that image
- Any subsequent baseline updates (performed like in step 4) will unset
FRSOURCE_CPVRD_Vand cause this cycle to happen again!
Expected behavior
I would expect to see no diff after step 6. The implication of step 8 is that seemingly random results are introduced into diffs, including PRs. I think setting FRSOURCE_CPVRD_V when generating the .actual.png files would solve the issue.
Additional context
I'm aware that updateImages does not have this problem, but it doesn't work for our GHA workflow: we run the tests and would like indicate which are failing (not possible with updateImages) and then automatically open a PR with new baselines. It is these PRs which contain many irrelevant diffs.
Despite our (potentially unusual) use case, I imagine all users who rename .actual.png to update their baselines are running into similar problems.
Let me know if I'm missing anything, I could look into opening a PR otherwise.
Package version
3.3.10
Device
Any
OS and version
Any
Browser and version
Any
Node version
Any
I can work on it by myself
- Yes
- No
Code of Conduct
- I agree to follow this project's Code of Conduct