Add PROOF ballots to the report #5

eventualbuddha · 2026-02-10T22:36:17Z

We currently generate votes and mark ballots accordingly, then sum up those votes and compare the totals against the totals of all the CVRs. This ensures that the totals are correct for the given marking patterns, but it doesn't protect against candidate bubbles being flipped or reordered if both candidates got the same total.

As a first step in protecting against that case, this change makes VxQA produce PROOF ballots, i.e. copies of the ballots annotated to aid in validating their correctness. They have the candidate/contest option associated with the bubble shown with labels in a green box, the locations we expect the bubbles to be marked with a red X, and the area we inspect for write-in marks in a tan box.

The implementation is based on a previous version that was only ever used on a New Hampshire-specific branch: votingworks/vxsuite#5267. It's modified a bit for style and to suit the slightly different tooling in vx-qa.

Tests pull in jest-image-snapshot to do PDF snapshotting in a similar manner to how vxsuite does it.

Added: Ballot Proof Gallery (screenshot)

Example PROOF PDF:

PROOF-ballot-2_en-s3y4ua5am2y7-official-absentee.pdf

Copilot

Pull request overview

This PR adds “PROOF” ballot PDFs to VxQA runs and exposes them in the generated HTML report to help visually verify bubble-to-contest/option mapping (and catch issues like candidate flip/reorder when totals match).

Changes:

Generate PROOF ballot PDFs during runQAWorkflow and include them in the report output directory.
Update HTML report generation to render a “Ballot Proof Gallery” pairing each base ballot PDF with its PROOF counterpart.
Add PDF snapshot testing utilities (Vitest + jest-image-snapshot) and snapshot fixtures for proof ballot rendering.

Reviewed changes

Copilot reviewed 9 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vitest.config.ts	Adds a Vitest setup file to register custom matchers.
src/test/setup.ts	Extends Vitest `expect` with `toMatchImageSnapshot`.
src/test/pdf-snapshot.ts	Adds a helper to render PDF pages to images and snapshot-test them.
src/report/pdf-thumbnail.ts	Makes PDF thumbnail scale configurable.
src/report/html-generator.ts	Builds base/proof ballot pairs and renders a new “Ballot Proof Gallery”.
src/cli/config-runner.ts	Generates `PROOF-*.pdf` alongside each base ballot PDF.
src/ballots/proof-ballot.ts	Implements proof ballot annotation overlay generation with `pdf-lib`.
src/ballots/proof-ballot.test.ts	Adds unit + fixture-based snapshot tests for proof ballots.
src/ballots/image_snapshots/*	Adds snapshot PNGs for proof ballot fixture rendering.
package.json	Adds dev deps for image snapshot testing.
pnpm-lock.yaml	Locks new snapshot-testing dependencies.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-10T22:42:31Z

src/ballots/proof-ballot.test.ts

+describe('generateProofBallot with real election fixture', async () => {
+  const tmp = await mkdtemp(join(tmpdir(), 'vx-qa-proof-test-'));
+  afterAll(() => rm(tmp, { recursive: true, force: true }));
+
+  const { electionPackage } = await loadElectionPackage(FIXTURE_PATH, tmp);
+  const { election } = electionPackage.electionDefinition;


describe blocks should be synchronous in Vitest. Using an async callback here means the awaited setup runs outside Vitest's normal suite-definition phase and can lead to tests not being registered or running in an unexpected order. Move the async setup (mkdtemp/loadElectionPackage/fixture ballot lookup) into beforeAll (or beforeEach) inside a synchronous describe, and store the loaded election/ballots in variables for the tests to use.

Copilot · 2026-02-10T22:42:31Z

src/ballots/proof-ballot.test.ts

+const FIXTURE_PATH = join(
+  import.meta.dirname,
+  '../../test-fixtures/election-package-and-ballots-e71c80e-c4446e7.zip',
+);


import.meta.dirname is only available in newer Node versions, but this repo's engines.node allows any >=20.0.0. To avoid test failures on Node 20.x versions that don't support it, derive the directory from import.meta.url (via fileURLToPath(new URL('.', import.meta.url))) or tighten the Node engine range to a version that guarantees import.meta.dirname.

I believe this will be set up by vitest.

src/report/html-generator.ts

Copilot · 2026-02-10T22:42:32Z

src/report/html-generator.ts

+  async function makeBallotGalleryThumbnail(
+    file: (typeof ballotFiles)[number],
+  ): Promise<string | null> {
+    return file.name.endsWith('.pdf')
+      ? await generatePdfThumbnail(file.path, { scale: 2 })
+      : `data:image/png;base64,${await readFileAsBase64(file.path)}`;


Generating ballot gallery thumbnails for PDFs at scale: 2 can be significantly more CPU/memory intensive than the previous thumbnail behavior, especially if there are many ballots. Consider using a smaller default scale (or making it configurable via CLI/config) so report generation stays responsive while still being readable.

For the number of ballots we have right now, this is likely to be acceptable. A lot of this is going to be untenable if our ballot count goes beyond ~30 or so.

Overlay contest/candidate labels on each ballot PDF so a human can visually verify that bubble positions are correctly mapped. Proof PDFs are saved alongside base ballots and now appear in the HTML report's new Ballot Gallery section.

Add jest-image-snapshot with a lightweight helper that renders PDF pages via pdf-to-img and compares them pixel-by-pixel, similar to vxsuite's toMatchPdfSnapshot.

Export pure geometry/label functions for direct unit testing. Add PDF snapshot test that renders a synthetic proof ballot and compares against stored baseline images.

Filter gallery to ballots that have a PROOF- counterpart, excluding marked/scanned variants. Display each pair side-by-side for easy comparison. Render gallery thumbnails at 2x scale for readability.

Replace synthetic snapshot test with fixture-based tests that generate proof ballots from the real election package, covering both ballot styles (2-page and 4-page).

eventualbuddha requested a review from Copilot February 10, 2026 22:36

Copilot started reviewing on behalf of eventualbuddha February 10, 2026 22:36 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

eventualbuddha added 5 commits February 10, 2026 15:29

chore: add PDF snapshot test infrastructure

4d4076c

Add jest-image-snapshot with a lightweight helper that renders PDF pages via pdf-to-img and compares them pixel-by-pixel, similar to vxsuite's toMatchPdfSnapshot.

test: add unit and snapshot tests for proof ballot generation

44295db

Export pure geometry/label functions for direct unit testing. Add PDF snapshot test that renders a synthetic proof ballot and compares against stored baseline images.

fix: ballot gallery shows only base+proof pairs at higher resolution

106d765

Filter gallery to ballots that have a PROOF- counterpart, excluding marked/scanned variants. Display each pair side-by-side for easy comparison. Render gallery thumbnails at 2x scale for readability.

test: use real election fixture for proof ballot snapshot tests

7c75f3e

Replace synthetic snapshot test with fixture-based tests that generate proof ballots from the real election package, covering both ballot styles (2-page and 4-page).

eventualbuddha force-pushed the vx-qa-proof-ballots branch from e165436 to 7c75f3e Compare February 10, 2026 23:30

eventualbuddha requested review from jonahkagan and removed request for jonahkagan February 10, 2026 23:30

eventualbuddha added 2 commits February 10, 2026 16:11

test: use vitest's context API for proof tests

b48b79e

refactor: consolidate "PROOF-" instances into a constant

71154f2

eventualbuddha requested a review from jonahkagan February 11, 2026 00:12

fix formatting

fbd3d4d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PROOF ballots to the report #5

Add PROOF ballots to the report #5

Uh oh!

eventualbuddha commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

eventualbuddha Feb 10, 2026

Uh oh!

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

eventualbuddha Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Add PROOF ballots to the report #5

Are you sure you want to change the base?

Add PROOF ballots to the report #5

Uh oh!

Conversation

eventualbuddha commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

eventualbuddha Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

eventualbuddha Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant