Playwright PR commenter - false-positives

[webfiltered]

The PR comment bot announced playwright test success when all tests were skipped.

Appears to check if they didn't fail, rather than that they actually passed?

• [Cleanup] Remove TS migration lines from tsconfig #5330 (comment)

┆Issue is synchronized with this Notion page by Unito

[christian-byrne]

let sno make a PR to check if they playwright tests passed rather than didn't fail. If they were skipped, the comment should reflect that.

[snomiao]

✅ Investigation Complete

📊 Summary of Findings

The PR comment bot incorrectly reports "All tests passed" when Playwright tests are skipped or don't run. This false-positive occurs because:

1. Missing deployment-info artifacts: The test-ui.yaml workflow doesn't create deployment-info-* artifacts with test exit codes
2. Default to success: When no deployment-info files exist, the pr-playwright-deploy.yaml workflow defaults ALL_PASSED=true
3. No test status tracking: The workflows don't capture or pass the actual Playwright test exit codes between jobs

📋 Investigation Steps Completed

1. ✅ Viewed issue Playwright PR commenter - false-positives #5338 and all comments for context
2. ✅ Analyzed GitHub workflow files for Playwright test reporting
3. ✅ Understood how PR comments are generated for test results
4. ✅ Identified the false-positive scenario when tests are skipped
5. ✅ Researched Playwright test status handling
6. ✅ Prepared investigation report with findings
7. ✅ Updated the comment with investigation steps
8. ✅ Created draft PR with necessary changes

🔍 Technical Analysis

Root Cause

In pr-playwright-deploy.yaml:

• Line 216: ALL_PASSED=true is set by default
• Lines 217-226: Loop checks deployment-info files for failures
• Problem: If no deployment-info files exist, ALL_PASSED remains true

Missing Component

The test-ui.yaml workflow runs tests but doesn't:

1. Capture test exit codes (steps.playwright.outcome)
2. Create deployment-info artifacts with status
3. Upload these artifacts for the deploy workflow

Example False Positive

PR #5330 showed "✅ All tests passed" when tests were actually skipped, with no test report links displayed.

🛠️ Solution Implemented

Created PR #5341: Fix Playwright PR commenter false-positives when tests are skipped

📝 Changes Made

1. Modified test-ui.yaml

• Added steps to capture Playwright test outcomes
• Creates deployment-info artifacts with format: browser|exit_code|url
• Uploads artifacts for each browser test result

2. Modified pr-playwright-deploy.yaml

• Added detection for missing test results
• Shows "Tests were skipped or not run" instead of false success
• Preserves existing pass/fail logic when results exist

✨ Result

The PR comment bot will now correctly:

• Report "Tests were skipped or not run" when no test results are available

• Show actual pass/fail status based on test exit codes

• Provide clear messaging for all test scenarios

• Agent ID: 266

• Status: Complete

• PR: Fix Playwright PR commenter false-positives when tests are skipped #5341 (Draft)

[snomiao]

this is completed,

we have super detailed failed/pass/skip/flaky counts now.