Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions apps/worker/src/run-task/__tests__/proof-runner-prompt.test.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions apps/worker/src/run-task/proof-runner-prompt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ export function createProofRunnerAgentPrompt(browserTarget: string): string {
'1. Decide the smallest concrete capture plan that proves the claim for the requested package, covering every checklist item from the brief.',
'2. Reach the required product state, including authentication when needed.',
'3. Capture the final screenshots and screencasts under `/tmp/capture-visual-proof/`.',
'4. Self-review each captured screenshot and screencast keyframe against its proof sentence before upload. If your model cannot inspect images directly, say so explicitly in the final report so the parent can validate the captures with the visual subagent instead.',
'4. Self-review each captured screenshot and screencast keyframe before upload. Verify both the specific proof sentence and the full captured frame for obvious visual regressions, including inconsistent light/dark theme treatment, unreadable contrast, clipping or overflow, broken layout, unintended loading or error states, and styling that conflicts with the surrounding UI. If your model cannot inspect images directly, say so explicitly in the final report so the parent can validate the captures with the visual subagent instead.',
'5. Upload each approved screenshot, screencast, and keyframe with the `manage_artifacts` MCP tool using the `upload` action and `type` set to `visual-proof`. Treat the `artifactId`, `viewUrl`, and `rawUrl` values returned by each upload tool result as the only canonical artifact references. Never invent, guess, or reconstruct artifact IDs or URLs.',
'6. Return one final report containing the uploaded artifact IDs and URLs from those tool results.',
'',
Expand Down Expand Up @@ -49,7 +49,8 @@ export function createProofRunnerAgentPrompt(browserTarget: string): string {
'- For screencasts, start recording before the interaction that matters, stop as soon as the proof is visible, validate the clip with `ffprobe`, and extract 3 to 5 keyframes under `/tmp/capture-visual-proof/`.',
'- Retry a screenshot or screencast once when the first honest capture is obviously blank, clipped, or misses the required visible state.',
'- If you capture only partial supporting evidence and the remaining checklist items cannot be shown honestly, return the run as blocked instead of reporting a narrowed success.',
'- If the UI is plainly wrong because of the implementation itself, report that as blocked instead of inventing proof.',
'- Do not approve or upload an artifact just because its focal element satisfies the proof sentence. Treat an obvious visual defect anywhere in the captured frame as a failed self-review.',
'- If the failed self-review is caused by the capture itself, retry the artifact once. If the UI is plainly wrong because of the implementation, report that as blocked instead of uploading it as successful proof.',
'',
'Final report contract:',
'',
Expand All @@ -58,7 +59,7 @@ export function createProofRunnerAgentPrompt(browserTarget: string): string {
'- `Summary`: what was captured and what it proves, or the blocker.',
'- `Blocked`: `true` or `false`, with a short `Blocker type` and blocker description when blocked. Set blocked to true when any materially distinct checklist item remains unproved.',
'- `Coverage`: the claim scope, covered checklist items, and missing checklist items.',
'- `Screenshots` and `Screencasts` (when present): one entry per artifact with its name, proof sentence, local file path, self-review outcome, and the `artifactId`, `viewUrl`, and `rawUrl` returned by its `manage_artifacts` upload result. Include keyframe artifact IDs and URLs for each screencast.',
'- `Screenshots` and `Screencasts` (when present): one entry per artifact with its name, proof sentence, local file path, self-review outcome covering both claim accuracy and full-frame visual quality, and the `artifactId`, `viewUrl`, and `rawUrl` returned by its `manage_artifacts` upload result. Include keyframe artifact IDs and URLs for each screencast.',
'- `Sharing note`: only when the report contains at least one uploaded artifact, end with this guidance for the parent: "Visual-proof uploads are not posted to chat automatically. If `send_chat_reply` with `imageArtifactIds` is available and this proof may be relevant to the user in the originating thread, share the screenshot artifact IDs listed above. For non-image proof, include the `viewUrl` links in the reply text." Omit the section entirely when no artifacts were uploaded.',
'',
'Report only artifact URLs that appeared verbatim in `manage_artifacts` upload tool results. When an upload fails, retry it once with the verified local artifact path, then report the exact upload failure instead of treating a local path as final output.',
Expand Down
Loading