🏥 Safe Output Health Report - November 8, 2025

[github-actions[bot]]

Over the past 24 hours, I analyzed 87 workflow runs to assess the health of safe output jobs in our agentic workflows. The analysis focused exclusively on safe output job failures (create_discussion, create_issue, add_comment, create_pull_request, etc.) and identified 7 failures across 43 total safe output job executions.

Overall, safe output jobs maintained an 83.72% success rate, with most failures clustering around two main issues: GitHub App workflow permissions and token permission scopes.

Full Report Details

Executive Summary

• Period: Last 24 hours (Nov 7-8, 2025)
• Runs Analyzed: 87
• Workflows Active: 17 workflows used safe outputs
• Safe Output Jobs Executed: 43
• Safe Output Jobs Failed: 7
• Error Clusters Identified: 3

Safe Output Job Statistics

Job Type	Total Executions	Failures	Success Rate
create_issue	25	2	92.0%
create_discussion	5	0	100.0%
add_comment	5	0	100.0%
create_pull_request	6	4	33.33%
push_to_pull_request_branch	2	1	50.0%
Total	43	7	83.72%

Error Clusters

Cluster 1: GitHub App Permissions - Workflow Write Access 🔴

• Count: 4 occurrences (57% of failures)
• Affected Jobs: create_pull_request
• Affected Workflows: Q, Daily Documentation Updater
• Severity: High - Blocks all PR creation attempts involving workflow files

Sample Error:

To https://github.com/githubnext/gh-aw.git
 ! [remote rejected] q/add-pr-creation-instructions-082105a9b7993799 -> q/add-pr-creation-instructions-082105a9b7993799
   (refusing to allow a GitHub App to create or update workflow `.github/workflows/q.lock.yml` without `workflows` permission)
error: failed to push some refs to 'https://github.com/githubnext/gh-aw.git'

Root Cause:

When agents modify workflow files (.md files in .github/workflows/), the safe output system creates a pull request that includes both the modified .md files and the compiled .lock.yml files. GitHub requires the workflows permission on the GitHub App token to push changes that touch .lock.yml files in the .github/workflows/ directory.

The current GitHub App token configuration lacks this permission, causing all PR creation attempts involving workflow files to fail.

Impact:

• Workflow optimization agents (like Q) cannot create PRs with their changes
• Documentation update workflows cannot commit workflow-related documentation changes
• 67% of create_pull_request jobs failed due to this issue

Affected Runs:

• §19158681351 - Q workflow
• §19159820108 - Daily Documentation Updater
• §19161200282 - Q workflow
• §19181242436 - Q workflow

Cluster 2: Issue Assignment - Personal Access Token Permissions ⚠️

• Count: 2 occurrences (29% of failures)
• Affected Jobs: create_issue
• Affected Workflows: Duplicate Code Detector
• Severity: Medium - Issues are created successfully but assignment fails

Sample Error:

failed to update https://github.com/githubnext/gh-aw/issues/3320:
GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)
failed to update 1 issue
##[error]Failed to assign issue: The process '/usr/bin/gh' failed with exit code 1
##[error]Failed to assign issue #3320 to `@copilot`: The process '/usr/bin/gh' failed with exit code 1

Root Cause:

The Duplicate Code Detector workflow attempts to assign created issues to @copilot using a personal access token. However, assigning issues to GitHub Copilot requires organization-level permissions that the current token lacks. The replaceActorsForAssignable GraphQL mutation requires elevated permissions beyond what a standard PAT provides.

Impact:

• Issues are created successfully but remain unassigned
• Manual intervention required to assign issues
• 8% of create_issue jobs experienced this failure
• Workflow functionality is partially degraded but not completely broken

Affected Runs:

• §19126079361 - Duplicate Code Detector
• §19181082887 - Duplicate Code Detector

Cluster 3: Missing File - staged_preview.cjs ⚠️

• Count: 1 occurrence (14% of failures)
• Affected Jobs: push_to_pull_request_branch
• Affected Workflows: Changeset Generator
• Severity: Medium - Workflow-specific issue, not systemic

Sample Error:

Error: Cannot find module '/home/runner/work/gh-aw/gh-aw/staged_preview.cjs'
Require stack:
- /home/runner/work/_actions/actions/github-script/ed597411d8f924073f98dfc5c65a23a2325f34cd/dist/index.js

Root Cause:

The Changeset Generator workflow's push_to_pull_request_branch safe output job attempts to require a Node.js module staged_preview.cjs that doesn't exist in the repository. This appears to be leftover code or a reference to a file that was removed or renamed.

Impact:

• Changeset Generator cannot push changes to PR branches
• 50% of push_to_pull_request_branch jobs failed
• Specific to one workflow, not a systemic issue

Affected Runs:

• §19142064434 - Changeset Generator

Root Cause Analysis

Permission-Related Issues (85% of failures)

The majority of safe output job failures (6 out of 7) stem from insufficient permissions:

1. GitHub App Token Scope (Cluster 1): The GitHub App token used by agentic workflows lacks the workflows permission, preventing any push operations that modify workflow files. This is a critical blocker for workflow optimization agents.

2. PAT Scope Limitation (Cluster 2): Personal access tokens used for issue assignment lack organization-level permissions required to assign issues to GitHub Copilot bot accounts.

Both issues indicate a need for permission scope review and adjustment in the workflow configuration.

Code Quality Issues (15% of failures)

One failure (Cluster 3) resulted from code referencing a non-existent file. This suggests:

• Incomplete cleanup after code refactoring
• Missing file validation in safe output job scripts
• Need for better error handling when requiring external modules

Recommendations

Critical Issues (Immediate Action Required)

1. Grant Workflows Permission to GitHub App Token

• Priority: Critical
• Root Cause: GitHub App lacks workflows permission scope
• Recommended Action:
  • Update GitHub App settings to grant workflows: write permission
  • This will allow agents to create PRs that modify workflow files
  • Coordinate with repository administrators to update app permissions
• Affected: Q workflow, Daily Documentation Updater, and any workflow that modifies .github/workflows/ files
• Expected Impact: Will resolve 57% of current safe output failures

2. Remove or Fix @copilot Assignment Logic

• Priority: High
• Root Cause: PAT cannot assign issues to GitHub Copilot bot accounts
• Recommended Action:
  • Option A: Remove the assignment step entirely (issues will be unassigned)
  • Option B: Assign to human users instead of @copilot
  • Option C: Investigate if GitHub App token (with proper permissions) can assign to bots
• Affected: Duplicate Code Detector workflow
• Expected Impact: Will resolve 29% of current safe output failures

Bug Fixes Required

3. Fix staged_preview.cjs Reference in Changeset Generator

• File/Location: .github/workflows/changeset.md - push_to_pull_request_branch safe output configuration
• Problem: Code attempts to require a non-existent file staged_preview.cjs
• Fix:
  • Remove the require statement if the file is no longer needed
  • If the functionality is still needed, restore the missing file or update the path
• Affected Jobs: push_to_pull_request_branch
• Expected Impact: Will resolve 14% of current safe output failures

Process Improvements

4. Implement Pre-Flight Permission Checks

• Current State: Safe output jobs fail at runtime when permissions are insufficient
• Proposed: Add permission validation before attempting operations
• Benefits:
  • Faster failure with clearer error messages
  • Ability to gracefully degrade (e.g., create issue but skip assignment if permission lacking)
  • Better user experience and debugging

5. Add File Existence Validation

• Current State: Safe output jobs assume external files exist
• Proposed: Validate file existence before requiring modules
• Benefits:
  • Prevent runtime crashes from missing files
  • Better error messages for troubleshooting
  • Fail fast with actionable feedback

6. Separate Workflow File Updates from Regular PR Creation

• Current State: All PR creation uses the same GitHub App token
• Proposed: Use different token or job for workflow file modifications
• Benefits:
  • Can use token with appropriate permissions for workflow changes
  • Reduces risk of permission issues for non-workflow PRs
  • Better separation of concerns

Work Item Plans

Work Item 1: Grant Workflows Permission to GitHub App

• Type: Configuration Change
• Priority: Critical
• Description: Update GitHub App settings to grant workflows: write permission, enabling agents to create pull requests that modify workflow files
• Acceptance Criteria:
  • GitHub App has workflows: write permission enabled
  • Q workflow can successfully create PRs with workflow file changes
  • Daily Documentation Updater can commit workflow-related documentation
  • Test with a sample workflow modification to verify permission works
• Technical Approach:
  1. Access GitHub App settings at https://github.com/settings/apps
  2. Navigate to the app used by gh-aw workflows
  3. Under "Permissions & events", locate "Repository permissions"
  4. Set "Workflows" to "Read and write"
  5. Save changes and accept the new permission prompt
  6. Test with a workflow run
• Estimated Effort: Small (15 minutes configuration + 10 minutes testing)
• Dependencies: Requires GitHub organization admin access

Work Item 2: Fix Issue Assignment Logic in Duplicate Code Detector

• Type: Bug Fix / Enhancement

• Priority: High

• Description: Remove or update the @copilot issue assignment logic to use a supported assignment method

• Acceptance Criteria:

  • Duplicate Code Detector workflow completes without assignment errors
  • Issues are either unassigned or assigned to valid assignees
  • Safe output job success rate for create_issue reaches 100%
  • Documentation updated to reflect assignment behavior

• Technical Approach:
  Option A (Recommended): Remove assignment step

  1. Edit .github/workflows/duplicate-code-detector.md
  2. Remove or comment out the assignee configuration in safe-outputs
  3. Update documentation to note issues will be unassigned
  4. Test workflow run

  Option B: Assign to human users

  1. Replace @copilot with a valid GitHub username or team
  2. Update workflow configuration
  3. Test assignment with new assignee

• Estimated Effort: Small (30 minutes implementation + 15 minutes testing)

• Dependencies: None

Work Item 3: Fix staged_preview.cjs Reference in Changeset Generator

• Type: Bug Fix
• Priority: Medium
• Description: Remove or fix the reference to non-existent staged_preview.cjs file in the push_to_pull_request_branch safe output job
• Acceptance Criteria:
  • Changeset Generator workflow's push_to_pull_request_branch job succeeds
  • No MODULE_NOT_FOUND errors in job logs
  • Workflow functionality is fully restored
• Technical Approach:
  1. Investigate why staged_preview.cjs was referenced
  2. If functionality is no longer needed, remove the require statement
  3. If functionality is still needed, create the missing file or update the path
  4. Test the Changeset Generator workflow end-to-end
  5. Verify push_to_pull_request_branch completes successfully
• Estimated Effort: Medium (1 hour investigation + 30 minutes fix + 30 minutes testing)
• Dependencies: May require understanding original intent of the module

Work Item 4: Add Permission Pre-Flight Checks to Safe Output Jobs

• Type: Enhancement
• Priority: Medium
• Description: Implement permission validation before attempting operations in safe output jobs, providing better error messages and graceful degradation
• Acceptance Criteria:
  • Safe output jobs check token permissions before operations
  • Clear error messages when permissions are insufficient
  • Jobs gracefully skip optional operations (like assignment) if permissions lacking
  • Documentation updated with permission requirements
• Technical Approach:
  1. Add permission check utilities to safe output job scripts
  2. For PR creation jobs: Check for workflows permission before pushing workflow files
  3. For issue assignment: Check assignee permissions before attempting assignment
  4. Implement graceful degradation: create issue even if assignment fails
  5. Update error messages to guide users to permission documentation
• Estimated Effort: Medium (2 hours implementation + 1 hour testing)
• Dependencies: None

Work Item 5: Improve Safe Output Job Error Handling

• Type: Enhancement
• Priority: Low
• Description: Add file existence checks and better error handling to prevent runtime crashes from missing dependencies
• Acceptance Criteria:
  • Safe output jobs validate file existence before requiring modules
  • Clear error messages identify missing files with suggested fixes
  • Jobs fail fast with actionable error messages
  • Error handling covers common failure scenarios
• Technical Approach:
  1. Add file existence checks using fs.existsSync() before require()
  2. Wrap external module loading in try-catch with descriptive errors
  3. Add validation for required environment variables
  4. Implement consistent error message formatting
  5. Update safe output job templates with new error handling patterns
• Estimated Effort: Medium (2 hours implementation + 1 hour testing)
• Dependencies: None

Historical Context

This is the first comprehensive safe output health audit. Future audits will track:

• Trends in success rates over time
• Recurring vs. one-time failures
• Impact of remediation efforts
• New failure patterns

Metrics and KPIs

• Overall Safe Output Success Rate: 83.72%
• Most Reliable Job Type: add_comment and create_discussion (100% success)
• Most Problematic Job Type: create_pull_request (33.33% success)
• Top Failure Root Cause: Insufficient GitHub App permissions (57% of failures)
• Workflows with Perfect Record: 13 out of 17 workflows had no failures

Next Steps

• Grant workflows permission to GitHub App (Critical - Work Item 1)
• Fix or remove @copilot assignment in Duplicate Code Detector (High - Work Item 2)
• Investigate and fix staged_preview.cjs issue (Medium - Work Item 3)
• Schedule follow-up audit in 1 week to measure improvement
• Implement permission pre-flight checks (Medium - Work Item 4)
• Document permission requirements for safe output jobs

---

References:

• §19158681351
• §19126079361
• §19142064434

AI generated by Safe Output Health Monitor

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.