Discussion: Implementing Visual Testing for Processing Core

[Vaivaswat2244]

Background

As part of the PR05 grant project, development of visual-regression-engine has been completed and the npm package published.
This engine provides the core functionality for comparing images and detecting visual regressions.

Phase 2 is now ready to begin: implementing a comprehensive visual testing framework for the Processing Core library.
This will help us catch rendering bugs and ensure consistent visual output across all supported platforms and future versions.

Hence before implementing, I was hoping to get a thorough discussion of possible implementations and how will they benefit the contributors.

[Vaivaswat2244]

Proposed Approach

We will implement visual testing primarily for the core module, focusing on foundational elements like:

• Shapes
• Color
• Typography
• Basic transformations

The fundamental workflow for each test will be:

1. A dedicated Java test class creates a small, focused Processing sketch.
2. The test runner executes the sketch and captures the rendered output as a PNG image.
3. A script invokes the visual-regression-engine to compare the newly generated image against a pre-approved reference image.
4. The test fails if the images do not match within a defined threshold, indicating a visual regression.

---

Proposed Test Structure

To keep tests modular and easy to manage, we propose co-locating each test’s source code with its corresponding reference image.
This structure makes it intuitive to find, add, or update tests.

core/
└── src/
└── test/
└── java/
└── processing/
└── core/
└── visual/
├── P2D/
│ ├── shape/
│ │ ├── TestLine.java
│ │ └── line.png <-- Reference image
│ └── color/
│ ├── TestFill.java
│ └── fill.png <-- Reference image
└── FX2D/
└── ...

markdown
Copy
Edit

In this model, TestLine.java would generate an image that is then compared against line.png.

---

Java ↔ Node.js Communication Architecture

Option A: Direct Process Execution

• Flow:

  1. Java test generates the output image (e.g., build/visual-test/output/P2D/shape/line.png).
  2. Uses Java's ProcessBuilder to execute a Node.js script.
  3. Passes the paths to the new image and the reference image as command-line arguments.
  4. Node.js script performs the comparison and returns success/fail via exit code or JSON output.

• Pros:

  • Simple to implement
  • Minimal overhead
  • Few dependencies

• Cons:

  • Can be brittle
  • Limited error handling
  • Less scalable for complex interactions

---

Option B: Reusable Bridge Architecture

• Flow:

  1. Gradle plugin manages the Node.js environment.
  2. Java sends structured JSON commands over stdin.
  3. A long-running Node.js process listens for commands and returns typed JSON results over stdout.
  4. Java deserializes JSON responses for type-safe results and detailed error messages.

• Pros:

  • Highly robust
  • Great error handling
  • Extensible for future needs

• Cons:

  • More complex initial setup

---

Key Questions for Discussion

1. Initial Scope

   • Should we start with:
     • Basic 2D primitives (line, rect, ellipse) and colors?
     • Include typography, transformations, or 3D elements from the outset?

2. Communication Architecture

   • Is simplicity of Option A enough?
   • Is robustness of Option B worth added complexity?

3. Reference Image Management

   • Should reference images be committed directly to the Git repository?
   • Process for updating references when a change is intentional? (e.g., --update-references flag)

4. Handling Platform Differences

   • Allow a global diff tolerance threshold?
   • Maintain separate reference images per platform (e.g., line-macos.png, line-windows.png)?
   • Other strategies?

5. CI/CD Integration

   • Run tests:
     • On every PR?
     • Only on main branch pushes?
     • As separate, non-blocking workflow?

@SableRaf @Stefterv @mingness @catilac, would really appreciate your views on this...

[Vaivaswat2244]

I realized the earlier comment might’ve felt like too many questions at once, so I’ll break them down and ask just a few at a time. With higher priority, the first decision we need is around how Java communicates with Node.js for the visual tests.

We’ve got two approaches:

Option A: Direct Process Execution

• Java spawns a fresh Node.js process for each image comparison
• Simple ProcessBuilder call with file paths as arguments
• Returns success/fail via exit code

Easy to set up, but slower and less flexible

Option B: Reusable Bridge Architecture

• Keep a long-running Node.js process (managed by Gradle)
• Java sends JSON commands over stdin, receives structured responses
• Faster and richer error handling

More complex initial setup, better error handling and faster

Personally, I’m leaning towards Option B, since the performance and error handling benefits seem worthwhile for a testing framework we’ll rely on heavily.

This is the main question I’d like to settle first, since the broader architecture decisions won’t block us from kicking off implementation.

cc @SableRaf @mingness @Stefterv @catilac

[mingness]

Hi @Vaivaswat2244 ! Thanks for this initial analysis and design into the testing suite we want to build. I recommend to build a POC for the option that you are leaning towards, which you said was Option B, and then we can reevaluate.

Just generally, for under-the-hood type design decisions, I suggest to build what makes sense to you, with guidance from me, and whatever happens, we'll learn from the process, and can report back and ask questions in the group calls. For questions about developer experience, actual integration into existing processes, and anything that I can't help with, let's ask this larger group.

1. For initial scope, let's just create a few simple tests to test the flow, for instance, a couple tests for 2D primitives. After that, let's duplicate the tests that exist in p5.js. After that, let's you and I talk, and then ask for feedback in the group calls, or here again.
2. It's a little hard for me to know if option a is too simple or not - if it's not too much trouble to implement, and not too much extra work to do both, the best way forward is to try one way as a POC, and then iterate and pivot from there.
3. What happens in p5 currently? I think the images are in the repo, so let's stick with that for now. We can always change as needed.
4. It's hard for me to know what the answer to platform differences is without seeing what's happening. my general suggestion is to mirror what p5 does at the moment. How does p5 handle platform differences?
5. I suggest the tests run more often than not, so they would run for every PR, and also locally via pre-commit hooks.

[Vaivaswat2244]

Thanks a lot @mingness for the review,
I've raised a draft pr for the proof of concept which was for the Option B which will explain the architecture better.

[catilac]

Thank you so much for your feedback @mingness and all your hard work @Vaivaswat2244.

One thing I would say architecture-wise is that I don't want to introduce the node/npm ecosystem as a requirement. I think that it could be helpful if you were able to ship a binary of your project that we could download and fold into our testing process in a way that makes sense for us.

The other feedback about this project, is I'm wondering about the feasibility of the regression process taking screenshots for us. It could be better if we were to use native tooling to take our own screen shots and then pass them over to the engine.