Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Dec 9, 2025

Task: Build Safe Outputs from Safe Outputs Schema

Goal: Use the schema of safe output types (schemas/agent-output.json) to drive the code generation of custom actions folder.

Implementation Complete ✅

Schema-driven generation now works with industry-standard tooling and automatic github-script global detection:

  1. Schema-driven: All safe output types discovered from schemas/agent-output.json
  2. esbuild bundling: Uses esbuild with proper static path resolution and automatic dependency detection
  3. @actions/core, @actions/github, @actions/exec: Actions automatically import required libraries based on usage
  4. npm ci workflow: Reproducible builds with package-lock.json
  5. Minification support: Release mode enables minification for production deployments

Key Components

  • pkg/workflow/safe_output_schema.go: Single source of truth for schema parsing, used by both generator and compiler
  • actions/package.json: Declares @actions/* dependencies and build scripts
    • npm run build: Dev mode - readable, non-minified output for debugging
    • npm run build:release: Release mode - minified output for production
  • actions/package-lock.json: Committed for reproducible builds (npm ci)
  • actions/build.js: esbuild configuration with conditional minification based on RELEASE_MODE
  • make generate-action-metadata: Generates action metadata from schema with automatic global detection
  • make actions-build: Builds actions in dev mode (readable)
  • make actions-build-release: Builds actions in release mode (minified)

Auto-Detection of github-script Globals

The build system now automatically scans generated source files for usage of github-script globals and imports them as needed:

  • core. → imports @actions/core and makes global.core available
  • github. → imports @actions/github and makes global.github available
  • context. → imports @actions/github, extracts github.context, and makes global.context available

This ensures all actions have exactly the dependencies they need without manual configuration.

esbuild Fixes

  1. Static path resolution: Generated src/index.js files use static relative paths (../../../pkg/workflow/js/file.cjs) that esbuild can resolve at build time
  2. Automatic global detection: Scans for core., github., and context. usage and imports/exports them automatically
  3. Global availability: Makes detected globals available via global.* assignments so bundled .cjs files can access them

Generated Actions (11 from schema)

All actions are generated from the schema and bundled with esbuild:

  • actions/noop/
  • actions/minimize-comment/
  • actions/close-issue/
  • actions/close-pull-request/
  • actions/close-discussion/
  • actions/add-comment/
  • actions/create-issue/
  • actions/add-labels/
  • actions/create-discussion/
  • actions/update-issue/
  • actions/update-pull-request/

Plus 2 existing actions (also rebundled with esbuild):

  • actions/setup-safe-inputs/
  • actions/setup-safe-outputs/

Each directory contains:

  • action.yml - Action metadata (schema-driven)
  • README.md - Documentation (schema-driven)
  • index.js - Bundled by esbuild with all dependencies inlined (committed)

Generated but not committed:

  • src/index.js - Generated source file (gitignored, regenerated by make generate-action-metadata)
  • src/*.cjs - Intermediate build artifacts (gitignored)

All committed generated files are marked as linguist-generated=true in .gitattributes.

Building Actions

Generated actions can be rebuilt using:

make generate-action-metadata     # Regenerate from schema (creates src/index.js with auto-detected imports)
make actions-build                # Bundle with esbuild (dev mode - readable)
make actions-build-release        # Bundle with esbuild (release mode - minified)

The build process:

  1. make generate-action-metadata generates action.yml, README.md, and src/index.js from schema with automatic detection of github-script globals
  2. make actions-build or make actions-build-release runs npm ci and esbuild
  3. esbuild bundles src/index.js → index.js with all dependencies properly inlined including @actions/* libraries
    • Dev mode: Readable code for debugging and development
    • Release mode: Minified code for production deployment

<issue_title>build safe outputs from safe outputs schema</issue_title>
><issue_description>Use the schema of safe output types to drive the code generation of custom actions folder.
></issue_description>

Original prompt

This section details on the original issue you should resolve

<issue_title>build safe outputs from safe outputs schema</issue_title>
<issue_description>Use the schema of safe output types to drive the code generation of custom actions folder.
</issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 3 commits December 9, 2025 21:33
- Add schema_parser.go to parse agent-output.json schema
- Update generate_action_metadata_command.go to use schema instead of hardcoded list
- Extract safe output types from schema definitions
- Generate 11 custom actions from schema (was 5 hardcoded)
- Actions now include proper descriptions from schema

Co-authored-by: pelikhan <[email protected]>
- Add comprehensive unit tests for schema parsing functions
- Fix gosimple linter errors (unnecessary fmt.Sprintf)
- Remove unused extractActionMetadata function
- All tests pass, linter passes

Co-authored-by: pelikhan <[email protected]>
@@ -0,0 +1,384 @@
// @ts-check
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot actions//src/.js should be gitignored

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit ea114d4. All generated action directories are now in .gitignore, including their src/*.js files. The files remain on disk for local development but are not committed to the repository.

@@ -0,0 +1,69 @@
# Add Issue Comment Output
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot actions/*/README.md should be gitignored

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit ea114d4. All generated README.md files for the 11 schema-driven actions are now gitignored. The actions can still be generated locally with make generate-action-metadata.

Copilot AI and others added 2 commits December 9, 2025 22:04
- Add t.Cleanup() to restore registry state after tests modify it
- Fix TestCustomActionModeCompilation to only check create_issue job section
- Prevents test failures caused by other jobs using actions/github-script

The tests were failing because:
1. Tests modified global DefaultScriptRegistry without proper cleanup
2. Test checked entire workflow file instead of just the create_issue job
3. Other jobs (activation, detection) legitimately use actions/github-script
- Update scripts.go to use RegisterWithAction for types with custom actions
- Add GetActionDirectoryName to convert snake_case to kebab-case
- Regenerate all custom action directories with hyphenated names
- 11 safe output types now have custom actions: noop, minimize-comment,
  close-issue, close-pull-request, close-discussion, add-comment,
  create-issue, add-labels, create-discussion, update-issue, update-pull-request
- Custom actions follow GitHub Actions naming convention (hyphens)

Co-authored-by: pelikhan <[email protected]>
- Add all 11 generated action directories to .gitignore
- Untrack previously committed generated files (README.md, action.yml, src/index.js)
- Generated actions are created by 'make generate-action-metadata'
- Files remain on disk for local development but are not committed

Co-authored-by: pelikhan <[email protected]>
Copilot AI requested a review from pelikhan December 9, 2025 22:34
// Safe output scripts

// Helper to register scripts that have custom actions
registerWithAction := func(name, source string) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot move action output schema parser into workflow package (own file) and use the schema to drive definition so that the code generator and the compiler use the same source of truth.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit 5a25e71. Moved schema parser to pkg/workflow/safe_output_schema.go and added GetCustomActionTypes() as the single source of truth. Both the action generator and compiler now use the same schema-driven list, eliminating the duplication between scripts.go and the CLI generator.

…tion

- Move schema_parser.go from pkg/cli to pkg/workflow/safe_output_schema.go
- Add GetCustomActionTypes() as single source of truth for custom actions
- Update scripts.go to use schema-driven registration via GetCustomActionTypes()
- Update generate_action_metadata_command.go to use workflow.* functions
- Move and update tests to pkg/workflow/safe_output_schema_test.go
- Both code generator and compiler now use the same schema source

Co-authored-by: pelikhan <[email protected]>
Copilot AI changed the title [WIP] Build safe outputs from safe outputs schema Build safe outputs from safe outputs schema Dec 9, 2025
Copilot AI requested a review from pelikhan December 9, 2025 23:57
@pelikhan
Copy link
Contributor

@copilot the files under "actions/" have to be commited to the repo AND marked as autogenerated in the gitattributes.

- Remove generated action directories from .gitignore
- Add patterns to .gitattributes to mark generated actions as linguist-generated
- Regenerate all 11 custom actions (noop, minimize-comment, close-issue,
  close-pull-request, close-discussion, add-comment, create-issue, add-labels,
  create-discussion, update-issue, update-pull-request)
- Generated files now committed to repo but marked as auto-generated for GitHub

Co-authored-by: pelikhan <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot the files under "actions/" have to be commited to the repo AND marked as autogenerated in the gitattributes.

Done in commit 9b20f08. All 11 generated action directories are now committed to the repository and marked as linguist-generated=true in .gitattributes. GitHub will treat them as auto-generated code.

@pelikhan
Copy link
Contributor

Add a "feature" "agent-mode" in the front matter that controls the agent mode for a single agentic workflow. This flag overrides the default inferred mode. Set the value to "dev" for dev.md.

@pelikhan
Copy link
Contributor

@copilot comment above

- Add AgentMode field to WorkflowData struct
- Add extractAgentMode() function to parse agent-mode from frontmatter
- Apply agent-mode in CompileWorkflowData to override default action mode
- Update main_workflow_schema.json to include agent-mode field (enum: inline, dev)
- Set agent-mode: dev in .github/workflows/dev.md
- Allows per-workflow control of JavaScript embedding (inline vs custom actions)

Co-authored-by: pelikhan <[email protected]>
@pelikhan
Copy link
Contributor

@copilot update dev.md to create an issue with a poem. Remove safeinputs import. Safe outputs in stage mode

@github-actions
Copy link
Contributor

✅ Dev Hawk Report - Success

Workflow Run: #3090

  • Status: ✅ success
  • Commit: 669c4f6
  • Branch: copilot/build-safe-outputs-schema
  • Execution Time: ~5m 45s (06:12:46Z → 06:18:31Z)
  • Triggered By: @pelikhan via workflow_dispatch

The Dev workflow completed successfully! 🎉

Commit Details:

Gitignore generated src/index.js files

  • Add actions/*/src/index.js to .gitignore
  • These are generated files, not source files
  • Only bundled index.js files (in action root) are needed for actions
  • Keeps repository cleaner with only essential files committed

All builds, tests, and validations passed on this branch. The PR is ready for review.

🦅 Observed from above by Dev Hawk

- Change workflow to create a poem about GitHub instead of PR summary
- Remove imports section (was importing shared/gh.md)
- Remove manual steps section
- Keep safe-outputs in staged mode
- Simplify prompt to write poem and create issue
- Recompiled dev.lock.yml with updated workflow

Co-authored-by: pelikhan <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot update dev.md to create an issue with a poem. Remove safeinputs import. Safe outputs in stage mode

Done in commit 6ff1d44. Updated dev.md to:

  • Create a poem about GitHub instead of summarizing PRs
  • Removed the imports section (was importing shared/gh.md)
  • Removed manual steps section
  • Kept safe-outputs in staged mode (staged: true)
  • Simplified prompt to write a poem and create an issue with it

@github-actions
Copy link
Contributor

⚠️ Dev Hawk Report - Failure Analysis

Workflow Run: #3091

  • Status: failure ❌
  • Commit: 6ff1d44
  • Conclusion: The create_issue safe output job failed during staged mode processing

Root Cause Analysis

The Dev workflow failed in the create_issue job (step 5: "Create Output Issue"). This workflow is configured with safe-outputs: create-issue: staged: true, which means it should only generate a preview without actually creating issues.

Failure Location

  • Job: create_issue (Job ID: 57634524024)
  • Failed Step: "Create Output Issue" (step 5)
  • Job Conclusion: failure
  • Prior Jobs: ✅ activation, ✅ agent, ✅ detection (all successful)

What Happened

The workflow successfully completed:

  1. activation job - Workflow validation passed
  2. agent job - Copilot agent execution completed successfully (ran for ~70 seconds)
  3. detection job - Threat detection passed
  4. create_issue job - Failed while processing agent output in staged mode

Likely Causes

Since the agent job completed successfully but the safe output processing failed, the issue is likely one of:

  1. Invalid Agent Output Format: The agent generated create-issue safe output that doesn't match the expected schema

    • Missing required fields (e.g., title)
    • Invalid data types in fields
    • Malformed JSON structure
  2. Staged Mode Processing Error: The preview generation logic encountered an unexpected condition

    • The code tries to access createIssueItem.body.length at line 7269 but body might be undefined
    • Other field validation issues in the staging preview renderer
  3. GitHub Script Execution Error: The inline JavaScript in the workflow lock file has a runtime error

    • Could be accessing undefined properties
    • Type coercion issues

Investigation Artifacts

The following artifacts were generated and may contain more details:

  • agent_output.json - Contains the safe outputs generated by the agent
  • safe_output.jsonl - JSONL format of safe outputs
  • agent-stdio.log - Agent execution logs
  • mcp-logs - MCP server logs
  • firewall-logs-dev - Network firewall logs

Recommended Actions

  • Download and inspect agent_output.json artifact to see what the agent generated for create-issue
  • Check if the agent output includes required fields (title, body)
  • Review the agent prompt to ensure it instructs proper safe output format
  • Add defensive null checks in the staged preview generation code (around line 7269 in dev.lock.yml)
  • Consider adding validation before calling .length on potentially undefined fields

Workflow Configuration

The workflow was configured to:

safe-outputs:
  create-issue:
  staged: true  # Preview mode - should not actually create issues

This means the failure occurred during preview generation, not actual issue creation, which suggests a problem with how the agent output is being rendered for the preview.

Quick Fix

If this is blocking, you can:

  1. Temporarily remove staged: true to see the actual error when trying to create the issue
  2. Or add better error handling in the staged preview generation
  3. Or ensure the agent prompt explicitly requires all fields (title and body)

This analysis was automatically generated by Dev Hawk monitoring the Dev workflow completion.

🦅 Observed from above by Dev Hawk

@pelikhan
Copy link
Contributor

@copilot the error is a bundling issue. Make sure esbuild resolves all local require into index.js

Error: Cannot find module '/home/runner/work/gh-aw/gh-aw/pkg/workflow/js/sanitize_label_content.cjs'

- Change transformSourceForEsbuild to use static relative paths instead of path.join(__dirname)
- Use ../../../pkg/workflow/js/ paths that esbuild can resolve at build time
- Remove dynamic path construction with path.join that breaks bundling
- Regenerated all action src/index.js with static paths
- Rebuilt all 13 actions successfully with dependencies properly inlined
- Fixes "Cannot find module" error at runtime

Co-authored-by: pelikhan <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot the error is a bundling issue. Make sure esbuild resolves all local require into index.js

...

Fixed in commit 9d0823b. Changed the code generation to use static relative paths (../../../pkg/workflow/js/file.cjs) instead of dynamic path.join(__dirname, ...) construction. This allows esbuild to properly resolve and bundle all local requires at build time. All 13 actions now build successfully with dependencies fully inlined.

@github-actions
Copy link
Contributor

⚠️ Dev Hawk Report - Failure Analysis

Workflow Run: #3092

  • Status: failure
  • Conclusion: failure
  • Commit: 9d0823b
  • Duration: ~2 minutes (07:10:17 - 07:12:30 UTC)
  • Branch: copilot/build-safe-outputs-schema

Root Cause Analysis

The workflow failed in the create_issue job during the "Create Output Issue" step. This indicates that the agent attempted to create an issue as a safe output, but the action failed during execution.

Failed Job Details

Job: create_issue (View Job)

Execution Flow

The workflow successfully completed these phases before failure:

  1. activation - Timestamp validation passed
  2. agent - Copilot agent execution completed successfully (47 seconds)
    • All setup steps succeeded
    • Agent executed and produced safe outputs
    • Artifacts uploaded successfully
  3. detection - Threat detection passed (18 seconds)
  4. create_issue - Failed while processing safe outputs
  5. conclusion - Cleanup completed despite failure

Error Category

Safe Output Processing Failure - The agent completed successfully and produced safe outputs, but the downstream action that processes create_issue outputs encountered an error.

Likely Causes

Based on the workflow structure and failure point:

  1. Action Implementation Issue: The generated actions/create-issue/index.js may have a runtime error

    • Missing dependencies that weren't bundled by esbuild
    • Incorrect path resolution in the bundled code
    • JavaScript syntax or logic errors in the generated action
  2. Missing Required Fields: The safe output data structure may be missing required fields expected by the create-issue action

  3. Permission Issues: The action may lack necessary GitHub token permissions to create issues

  4. API Rate Limiting: GitHub API may have rejected the request (less likely given the quick failure)

Investigation Notes

  • Logs expired: The detailed job logs are no longer available (Azure blob storage returned Forbidden)
  • Agent succeeded: The agent phase completed successfully, indicating the issue is in the safe output processing phase
  • Quick failure: The step failed in <1 second, suggesting an immediate error rather than a timeout or network issue
  • Recent changes: This PR involves esbuild bundling changes to action files, which could have introduced bundling/runtime issues

Recommended Actions

  • Check the bundled action code: Review actions/create-issue/index.js for runtime errors

    cat actions/create-issue/index.js | head -100
  • Test action locally: Run the create-issue action with sample inputs to reproduce the error

    cd actions/create-issue
    node index.js
  • Verify esbuild configuration: Ensure actions/build.js properly bundles all dependencies

    make actions-build
  • Check safe output schema: Validate the agent's create_issue output matches the expected schema

    • Download artifact: safe_output.jsonl (artifact ID: 4821007894)
    • Verify all required fields are present
  • Re-run workflow: Trigger another Dev workflow run to see if the issue persists

Available Artifacts

The following artifacts are available for analysis (retained for 28 days):

  • agent_output.json - Agent's raw output
  • safe_output.jsonl - Processed safe outputs
  • agent-stdio.log - Agent execution logs
  • mcp-logs - MCP server logs
  • firewall-logs-dev - Network firewall logs

💡 Tip: Download the safe_output.jsonl artifact to see exactly what the agent tried to create.

🦅 Observed from above by Dev Hawk

@pelikhan
Copy link
Contributor

@copilot the error message is

ReferenceError: core is not defined

Make sure esbuild bundles the @actions/core package as well.

- Add global.core assignment in generated src/index.js
- The .cjs files expect 'core' as a global variable (github-script context)
- esbuild now properly bundles @actions/core and makes it available globally
- Regenerated all 13 actions with global core assignment
- Rebuilt all actions successfully with working @actions/core
- Fixes "ReferenceError: core is not defined" error

Co-authored-by: pelikhan <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot the error message is

...

Done in commit ef210a4. Added global.core = core; in the generated src/index.js files to make @actions/core available globally. The bundled .cjs files expect core as a global variable (they were designed for github-script context), and esbuild now properly bundles @actions/core and makes it accessible throughout the action.

@pelikhan
Copy link
Contributor

@copilot the build script needs to scan for usage of githubscriot globals and import them as needed.

core, github, context

Error: ReferenceError: context is not defined

@github-actions
Copy link
Contributor

⚠️ Dev Hawk Report - Failure Analysis

Workflow Run: #3093

  • Status: failure
  • Commit: ef210a4
  • Branch: copilot/build-safe-outputs-schema
  • Duration: ~3 minutes (12:00:23 - 12:03:17 UTC)

Root Cause Analysis

The workflow failed in the create_issue job at step 5 "Create Output Issue". This failure occurred during the safe outputs processing phase, after the agent successfully completed execution.

Failure Context

Job Sequence:

  1. activation - Completed successfully
  2. agent - Completed successfully (agent execution finished)
  3. detection - Completed successfully (threat detection passed)
  4. create_issue - FAILED (safe output processing)
  5. conclusion - Completed successfully (cleanup)

Analysis

The agent job completed successfully, meaning:

  • The agent executed without errors
  • Safe outputs were generated and uploaded
  • The agent requested to create an issue as a safe output

However, the create_issue job failed when attempting to process this request. Given that this PR implements schema-driven generation of safe output actions with esbuild bundling, this points to an issue with the newly generated actions/create-issue/ action.

Possible Root Causes

Based on the PR's implementation of schema-driven safe outputs:

  1. Generated Action Code Issue: The actions/create-issue/index.js bundled by esbuild may have:

    • Missing or incorrect @actions/core integration
    • Issues with the global core variable availability
    • Bundling problems with required dependencies
  2. Action Metadata Problem: The generated action.yml may have:

    • Incorrect input/output definitions
    • Missing required inputs
    • Schema mismatch between generation and runtime
  3. Safe Output Format Issue: The agent's safe output JSON may not match:

    • The expected schema format
    • Required field validations
    • Input parameter requirements

Recommended Actions

  • Review generated create-issue action: Check actions/create-issue/index.js for bundling issues
  • Verify action.yml schema: Ensure the generated action metadata matches schema requirements
  • Test action locally: Run the create-issue action with sample inputs to reproduce the failure
  • Check safe output format: Verify the agent's safe_output.jsonl matches the schema expectations
  • Review esbuild configuration: Ensure actions/build.js properly bundles all dependencies

Investigation Notes

⚠️ Logs are no longer accessible - The workflow logs expired and could not be retrieved for detailed error analysis. To prevent this in the future, workflow failures should be investigated promptly or log retention should be extended.

The failure pattern suggests this is related to the schema-driven generation implementation in this PR, not an infrastructure or transient issue. The other safe output types may have similar issues.

Next Steps

  1. Test the generated actions manually with sample inputs
  2. Add validation for generated action code before committing
  3. Consider adding integration tests for safe output actions
  4. Verify the esbuild bundling process produces working actions

Note: This analysis is based on workflow structure and job status since the detailed logs are no longer available. A full root cause investigation would require reproducing the failure or reviewing the logs when fresh.

🦅 Observed from above by Dev Hawk

- Scan generated source for usage of core, github, and context globals
- Automatically import @actions/core when core. is detected
- Automatically import @actions/github when github. or context. is detected
- Make all detected globals available via global.* assignments
- Regenerated all 13 actions with proper global detection
- Rebuilt all actions with complete @actions dependencies
- Fixes "ReferenceError: context is not defined" and similar errors

Co-authored-by: pelikhan <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot the build script needs to scan for usage of githubscriot globals and import them as needed.

core, github, context...

Done in commit d66a2ef. The build script now automatically scans generated source for usage of github-script globals (core., github., context.) and imports the required @actions packages:

  • Detects core. → imports @actions/core and makes global.core available
  • Detects github. → imports @actions/github and makes global.github available
  • Detects context. → imports @actions/github, extracts github.context, and makes global.context available

All 13 actions have been regenerated and rebuilt with proper automatic dependency detection.

@github-actions
Copy link
Contributor

✅ Dev Hawk Report - Success

Workflow Run: #3094

  • Status: ✅ success
  • Branch: copilot/build-safe-outputs-schema
  • Commit: d66a2ef
  • Duration: 7m 38s (started 12:15:46 UTC, completed 12:23:24 UTC)
  • Triggered by: @pelikhan via workflow_dispatch

Commit Summary

Auto-detect and import github-script globals (core, github, context) - fixes "ReferenceError: context is not defined" and similar errors by automatically scanning generated source for usage and importing required @actions dependencies.

The Dev workflow completed successfully! 🎉 All checks passed for the safe outputs schema build implementation.

🦅 Observed from above by Dev Hawk

@pelikhan pelikhan closed this Dec 10, 2025
@pelikhan pelikhan deleted the copilot/build-safe-outputs-schema branch December 11, 2025 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build safe outputs from safe outputs schema

2 participants