[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - November 2025

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns, success rates, and optimization opportunities.

Summary

Analysis Period: Last 30 days of copilot-created PRs
Total Tasks Analyzed: 1101
Clusters Identified: 6
Overall Success Rate: 75.3%

Key Findings

1. Most Common Task Type: New Features (41.9% of tasks)

   • Keywords: update, add, json, github, file
   • Success rate: 73.3%

2. Highest Success Rate: Updates (84.6% merge rate)

   • Average files changed: 15.8
   • Keywords: cli, code, version

3. Lowest Success Rate: Bug Fixes (64.4% merge rate)

   • May indicate more complex or challenging tasks
   • Average comments per PR: 2.5 (higher iteration)

4. Most Complex Tasks: New Features

   • Average files changed: 21.8 vs 16.1 overall
   • Average additions: 929 lines

Full Analysis Report

Cluster Visualization

A 2D PCA projection visualization has been generated showing semantic grouping of copilot agent tasks. The chart is available in the workflow artifacts.

Detailed Cluster Analysis

Cluster 1: New Features

Size: 461 tasks (41.9% of total)

Success Rate: 73.3% (338 merged, 123 not merged)

Top Keywords: update, add, json, github, file, fix, mcp

Average Metrics:

• Files changed: 21.8
• Lines added: 929
• Comments per PR: 1.7

Example Tasks:

1. PR Add ASCII logo and original prompt to compiled workflow headers #4501: Add minimal path format syntax reference to imports documentation

   Update the frontmatter "imports" documentation under /docs with all the supported URL and path formats

2. PR Add assign-milestone safe output type #4334: Add directory creation for copilot engine --add-dir paths

   The copilot engine should review the --add-dir folders used in the args list and make sure those folders exist before running the cli.
   Emit "mkdir -p ..." to prepare those directories.

3. PR Add close-pull-request safe output with filtering #4568: [WIP] Migrate JavaScript memory server to Wasm component

   Migrate this JavaScript memory server to Wasm Component as one of the examples in wassette...

---

Cluster 2: Gh Aw Tasks

Size: 259 tasks (23.5% of total)

Success Rate: 76.4% (198 merged, 61 not merged)

Top Keywords: gh aw, aw, gh, githubnext, githubnext gh aw, githubnext gh, workflow

Average Metrics:

• Files changed: 9.7
• Lines added: 359
• Comments per PR: 1.0

Example Tasks:

1. PR Implement golden file testing for compiler output validation #3779: [WIP] Comment on issue [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 regarding recurrence failure
2. PR Rename frontmatter_helpers.go to map_helpers.go #3850: Extract 22 YAML generation functions from compiler.go to yaml_generation.go
3. PR Add test coverage for critical CLI utilities #3733: Extract extraction functions from compiler.go to frontmatter_extraction.go

---

Cluster 3: Workflow Automation

Size: 167 tasks (15.2% of total)

Success Rate: 74.9% (125 merged, 42 not merged)

Top Keywords: agentic, agentic workflow, workflow, workflows, daily, update, add

Average Metrics:

• Files changed: 9.3
• Lines added: 1639
• Comments per PR: 2.0

Example Tasks:

1. PR Add dispatch-workflow safe output type for triggering workflows via AI agents #4285: Spread scheduled agentic workflows across 24 hours and add 6-hour schedules to smoke tests
2. PR [WIP] Add firewall feature to all agentic workflows #2430: Add smoke-outpost workflow for investigating failed smoke test runs
3. PR Add shared github-context.md import for comprehensive GitHub invocation context #4086: Add semantic function refactoring workflow for Go code analysis

---

Cluster 4: Updates

Size: 104 tasks (9.4% of total)

Success Rate: 84.6% (88 merged, 16 not merged)

Top Keywords: cli, code, version, changes, comments, pkg, update

Average Metrics:

• Files changed: 15.8
• Lines added: 488
• Comments per PR: 1.0

Example Tasks:

1. PR Refactor: Extract duplicate log parser formatting to shared module #4514: Refactor duplicate MCP code patterns for improved maintainability
2. PR Refactor: Extract duplicate agent output handler boilerplate #3503: Update cli-version-checker workflow: add node ecosystem and enforce issue creation
3. PR Refactor duplicate JSON/JSONL parsing logic into shared utility #4574: Change max-turns to 75 in cli-version-checker workflow

---

Cluster 5: Agent Tasks

Size: 65 tasks (5.9% of total)

Success Rate: 78.5% (51 merged, 14 not merged)

Top Keywords: agent, copilot, instructions, docs, mcp, make, context

Average Metrics:

• Files changed: 15.6
• Lines added: 760
• Comments per PR: 0.9

Example Tasks:

1. PR Add glossary maintainer workflow with daily incremental and weekly full scans #4336: [WIP] Update variable name to 'log' and modify instructions
2. PR Add Issue Monster workflow for automated issue triage and assignment #4684: [WIP] Address feedback on adding debug logging in Go files
3. PR Add repository quality improvement workflow with focus area rotation #3666: Migrate verbose logging to use logger package in run_command.go

---

Cluster 6: Bug Fixes

Size: 45 tasks (4.1% of total)

Success Rate: 64.4% (29 merged, 16 not merged)

Top Keywords: comments, issue, section, issue_title, issue_description, author, comment_new

Average Metrics:

• Files changed: 21.8
• Lines added: 863
• Comments per PR: 2.5

Example Tasks:

1. PR Refactor: Extract duplicate log parser markdown generation to shared module #4363: Fix: Raise error when agentic workflow hits max-turns limit
2. PR Add create-commit-status safe output type with pending/final status lifecycle #3505: [WIP] Update GITHUB_PERSONAL_ACCESS_TOKEN to GITHUB_MCP_SERVER_TOKEN
3. PR Use BurntSushi/toml encoder for Codex engine TOML configuration generation #2996: Parse firewall logs and show blocked requests in step summary

---

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Avg Additions	Top Keywords
3	Updates	104	84.6%	15.8	488	cli, code, version
4	Agent Tasks	65	78.5%	15.6	760	agent, copilot, instructions
5	Gh Aw Tasks	259	76.4%	9.7	359	gh aw, aw, gh
6	Workflow Automation	167	74.9%	9.3	1639	agentic, agentic workflow, workflow
1	New Features	461	73.3%	21.8	929	update, add, json
2	Bug Fixes	45	64.4%	21.8	863	comments, issue, section

Full Data Table

Sample of clustered tasks (showing first 50):

PR #	Title	Cluster	Outcome	Files	Keywords
#2097	Add minimal path format syntax reference to imports document...	New Features	Merged	1	update, add
#2099	Add directory creation for copilot engine --add-dir paths	New Features	Merged	25	update, add
#2101	[WIP] Migrate JavaScript memory server to Wasm component	New Features	Closed	0	update, add
#2102	Add workflow status badges documentation page	New Features	Merged	6	update, add
#2104	Add edit tool to commit-changes-analyzer workflow	New Features	Merged	2	update, add
#2108	Remove js-yaml dependency from badge generator, use regex-ba...	New Features	Merged	2	update, add
#2111	Refactor permissions parsing from compiler.go to permissions...	New Features	Closed	34	update, add
#2112	Restore AI instructions in README.md wrapped in XML comments	New Features	Merged	1	update, add
#2235	Fix: Raise error when agentic workflow hits max-turns limit	Gh Aw Tasks	Merged	54	gh aw, aw
#2254	[WIP] Update GITHUB_PERSONAL_ACCESS_TOKEN to GITHUB_MCP_SERV...	Gh Aw Tasks	Closed	64	gh aw, aw
#2313	Parse firewall logs and show blocked requests in step summar...	Gh Aw Tasks	Merged	7	gh aw, aw
#2317	Fix permission denied error when moving Copilot logs in fire...	Gh Aw Tasks	Merged	3	gh aw, aw
#2347	Validate repository features (discussions/issues) during wor...	Gh Aw Tasks	Merged	5	gh aw, aw
#2453	Fix safe-outputs directory creation when GH_AW_SAFE_OUTPUTS ...	Gh Aw Tasks	Merged	54	gh aw, aw
#2480	[WIP] Fix push to pull request folder error	Gh Aw Tasks	Closed	0	gh aw, aw
#2608	[WIP] Fix confusion between stop-after and timeout settings	Gh Aw Tasks	Closed	6	gh aw, aw
#2171	Refactor duplicate MCP code patterns for improved maintainab...	Workflow Automation	Merged	10	agentic, agentic workflow
#2216	Update cli-version-checker workflow: add node ecosystem and ...	Workflow Automation	Merged	3	agentic, agentic workflow
#2224	Change max-turns to 75 in cli-version-checker workflow	Workflow Automation	Merged	5	agentic, agentic workflow
#2249	Refactor: Extract duplicate GitHub MCP remote config renderi...	Workflow Automation	Merged	5	agentic, agentic workflow
#2256	Pass MCP configuration as CLI argument for Claude engine ins...	Workflow Automation	Closed	35	agentic, agentic workflow
#2325	Extend action SHA pinning to custom steps and imported jobs	Workflow Automation	Merged	17	agentic, agentic workflow
#2386	Optimize regex compilation in hot paths - 17-22x performance...	Workflow Automation	Merged	9	agentic, agentic workflow
#2409	Update cli-version-checker workflow to use copilot engine	Workflow Automation	Merged	3	agentic, agentic workflow
#2107	[WIP] Update variable name to 'log' and modify instructions	Updates	Merged	7	cli, code
#2126	[WIP] Address feedback on adding debug logging in Go files	Updates	Merged	1	cli, code
#2130	Migrate verbose logging to use logger package in run_command...	Updates	Closed	1	cli, code
#2141	Update reporting instructions to require overview before det...	Updates	Merged	8	cli, code
#2176	docs: add Test Failure Investigation example to quality-test...	Updates	Merged	1	cli, code
#2187	[WIP] Add AWF firewall to Copilot agent	Updates	Closed	35	cli, code
#2214	Document GH_AW_COPILOT_TOKEN authentication for Copilot oper...	Updates	Merged	1	cli, code
#2219	Add tip about enabling agentic-workflows tool in MCP Server ...	Updates	Merged	1	cli, code
#2209	[WIP] Comment on issue #2157 regarding recurrence failure	Agent Tasks	Merged	3	agent, copilot
#2282	Extract 22 YAML generation functions from compiler.go to yam...	Agent Tasks	Closed	4	agent, copilot
#2283	Extract extraction functions from compiler.go to frontmatter...	Agent Tasks	Merged	6	agent, copilot
#2284	Consolidate generic validation functions into validation.go	Agent Tasks	Merged	6	agent, copilot
#2285	Merge engine_helpers.go into engine_shared_helpers.go and re...	Agent Tasks	Merged	6	agent, copilot
#2286	Move generateGitConfigurationSteps from git.go to yaml_gener...	Agent Tasks	Merged	3	agent, copilot
#2382	Add boolean type support to bash tool in included_file_schem...	Agent Tasks	Merged	2	agent, copilot
#2383	Add missing MCP fields (entrypointArgs, headers, url) to inc...	Agent Tasks	Merged	2	agent, copilot
#2100	Spread scheduled agentic workflows across 24 hours and add 6...	Bug Fixes	Merged	30	comments, issue
#2103	Add smoke-outpost workflow for investigating failed smoke te...	Bug Fixes	Merged	2	comments, issue
#2109	Add semantic function refactoring workflow for Go code analy...	Bug Fixes	Merged	3	comments, issue
#2110	[WIP] Refactor clusters of functions in Go files	Bug Fixes	Closed	0	comments, issue
#2115	Add scheduling best practices guidance for daily workflows	Bug Fixes	Merged	1	comments, issue
#2118	Merge documentation unbloat style guidance and add import to...	Bug Fixes	Merged	3	comments, issue
#2129	Rename smoke-outpost workflow to smoke-detector	Bug Fixes	Merged	2	comments, issue
#2134	Add reporting instructions for HTML details/summary formatti...	Bug Fixes	Merged	17	comments, issue

Full dataset contains 1101 tasks

Insights & Recommendations

What Works Well

Updates (84.6% success):

• These tasks have clear, focused objectives
• Keywords: cli, code, version, changes

Agent Tasks (78.5% success):

• These tasks have clear, focused objectives
• Keywords: agent, copilot, instructions, docs

Gh Aw Tasks (76.4% success):

• These tasks have clear, focused objectives
• Keywords: gh aw, aw, gh, githubnext

Areas for Improvement

Bug Fixes (64.4% success):

• Higher comment rate (2.5 vs 1.5 avg) suggests more iteration needed
• Consider providing more context or breaking into smaller tasks

New Features (73.3% success):

• Higher comment rate (1.7 vs 1.5 avg) suggests more iteration needed
• Consider providing more context or breaking into smaller tasks

General Recommendations

1. Task Clarity: Tasks with specific keywords (like 'update', 'add', 'fix') tend to have higher success rates
2. Scope Management: Tasks changing fewer files (< 15) have better outcomes
3. Documentation Tasks: Very high success rate - good candidate for automation
4. Complex Refactoring: May benefit from breaking into smaller, incremental PRs

---

Analysis generated on 2025-11-25 19:38 UTC

Methodology: TF-IDF vectorization + K-means clustering (K=6) with PCA visualization

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #5325.
Fair winds, matey! 🏴‍☠️