Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image comparison test failures should explain how to investigate them #4065

Open
kpreid opened this issue Aug 16, 2023 · 0 comments
Open

Image comparison test failures should explain how to investigate them #4065

kpreid opened this issue Aug 16, 2023 · 0 comments
Labels
area: documentation Documentation for crate items, public or private

Comments

@kpreid
Copy link
Contributor

kpreid commented Aug 16, 2023

If an image comparison test fails, the error text is like:

        FAIL [   4.647s]                     wgpu-msaa-line-example::bin/msaa-line msaa_line

--- STDOUT:                                  wgpu-msaa-line-example::bin/msaa-line msaa_line ---

running 1 test
Starting image comparison test with reference image "/Users/kpreid/Projects/rust/contributing/wgpu/examples/common/../..//examples/msaa-line/screenshot.png"
Mean: 0.075663
Min Value: 0.000000
      25%: 0.219296
      50%: 0.371955
      75%: 0.475240
      95%: 0.623761
      99%: 0.762079
Max Value: 0.897098
Expected Mean (0.075663) to be under expected maximum (0.065): FAIL
Expected 50% (0.371955) to be under expected maximum (0.29): FAIL
test msaa_line ... FAILED

failures:

failures:
    msaa_line

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.19s


--- STDERR:                                  wgpu-msaa-line-example::bin/msaa-line msaa_line ---
thread 'msaa_line' panicked at 'Image data mismatch!', /Users/kpreid/Projects/rust/contributing/wgpu/tests/src/image.rs:243:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'msaa_line' panicked at 'UNEXPECTED TEST FAILURE DUE TO PANIC', /Users/kpreid/Projects/rust/contributing/wgpu/tests/src/lib.rs:344:9

None of this tells me where to find the actual-output image to diagnose whether the image is wrong or the threshold is wrong.

@teoxoy teoxoy added the area: documentation Documentation for crate items, public or private label Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: documentation Documentation for crate items, public or private
Projects
None yet
Development

No branches or pull requests

2 participants