[prompt-analysis] Copilot PR Prompt Analysis - 2025-11-21

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2025-11-21

Summary

Over the past 30 days, Copilot has created 1000 pull requests in this repository. Of the completed PRs, 767 were merged (76.9%) while 230 were closed without merging (23.1%). This analysis examines prompt patterns to identify what leads to successful merges.

The data shows that removal and documentation prompts have the highest success rates (88.9% and 86.5% respectively), while feature additions have a slightly lower success rate (73.5%). Prompts that include specific file references and detailed context show marginally better outcomes.

Full Analysis Report

Prompt Categories and Success Rates

Analysis of 997 completed PRs reveals distinct patterns across different prompt types:

Category	Total	Merged	Closed	Success Rate
Removal	18	16	2	88.9%
Documentation	37	32	5	86.5%
Bug Fix	588	456	132	77.6%
Refactor	13	10	3	76.9%
Update	68	51	17	75.0%
Test	20	15	5	75.0%
Other	71	54	17	76.1%
Feature Addition	181	133	48	73.5%

Prompt Analysis

✅ Successful Prompt Patterns (Merged PRs)

Common characteristics in the 767 merged PRs:

• Average prompt length: 164 words
• File references: 49.0% of prompts mention specific files
• Most common action verbs: update (359), add (353), fix (199), create (195), modify (187)
• Top categories: Bug fixes (456), Features (133), Updates (51)

Example successful prompts:

1. PR Add debug-agentic-workflow custom agent for workflow troubleshooting #4449 (Bug Fix): Generate a "custom agent" prompt (.github/agents/debug-agentic-workflow.md) file that will help a user refine and debug ... → ✓ Merged
2. PR Add --progress flag to run command for agent CLI keepalive #4450 (Feature): Add a --progress flag to the "run" command that emits regular points progress messages so that an agent cli running the ... → ✓ Merged
3. PR Review and update GitHub agentic workflows instruction files #4414 (Documentation): Review and update GitHub agentic workflows instruction files with respect to the current state of documentation and the ... → ✓ Merged

❌ Unsuccessful Prompt Patterns (Closed PRs)

Common characteristics in the 230 closed PRs:

• Average prompt length: 165 words (similar to merged)
• File references: 40.6% of prompts mention specific files (lower than merged)
• Most common action verbs: add (104), update (87), create (80), change (48), modify (46)
• Top categories: Bug fixes (132), Features (48)

Observations: Closed PRs show similar prompt lengths but have fewer file references and specific location mentions compared to merged PRs.

Example unsuccessful prompts:

1. PR Remove Claude hooks support #4432 (Bug Fix): `> ----

This section details on the original issue you should resolve

(issue_title)Remove Claude hooks(/issue...` → ✗ Closed

2. PR Update activation footer: add workflow URL, source link, and run ID #4440 (Feature): `Update the "Agentic ... started" message footer added to the message with the following:
   )

• add a link to the agenti...` → ✗ Closed

3. PR [WIP] Update README documentation to reflect latest features #4349 (Refactor): Improve the readme documentation and make it is up to date with the latest features.... → ✗ Closed

Key Insights

Based on analysis of nearly 1,000 Copilot-generated PRs, three clear patterns emerge:

1. Simpler tasks succeed more: Removal (88.9%) and documentation (86.5%) tasks have the highest merge rates, likely because they have clear, well-defined scopes and lower risk of unintended side effects.

2. Bug fixes perform well: With a 77.6% success rate across 588 PRs, bug fix prompts are consistently successful. This suggests Copilot excels at targeted fixes when given clear problem descriptions.

3. Feature additions are more challenging: At 73.5% success rate, new feature implementations have the lowest merge rate. This likely reflects the complexity and subjective nature of feature development, where requirements may evolve or need human refinement.

Specificity correlation: Merged PRs showed 49% file reference rate vs 40.6% for closed PRs, suggesting that prompts with specific file mentions have a slight advantage (though the difference is modest at ~8 percentage points).

Recommendations

Based on patterns in successful PRs, consider these best practices when writing Copilot prompts:

✅ DO:

1. Be specific about files and locations - Mention exact filenames, functions, or modules when possible
2. Start with clear action verbs - Use "fix", "add", "update", "remove" to set clear intent
3. For bug fixes, include context - Describe what's broken and expected behavior
4. Keep scope focused - Single-purpose prompts (like removals) have higher success rates
5. Reference existing patterns - Point to similar code or established conventions in the repo

⚠️ AVOID:

1. Vague improvements - Generic prompts like "improve the code" lack concrete deliverables
2. Large multi-step features - Break complex features into smaller, focused prompts
3. Ambiguous requirements - If the desired outcome isn't clear to you, it won't be clear to Copilot

📝 Prompt Template Suggestions

For bug fixes (77.6% success rate):

Fix [specific issue] in [filename/function]. 
Current behavior: [what's wrong]
Expected behavior: [what should happen]

For feature additions (73.5% success rate):

Add [specific feature] to [component/file].
Implementation: [brief approach]
Files to modify: [list specific files]

For documentation (86.5% success rate):

Update documentation in [file] to [specific change].
Add/clarify: [what needs documenting]

Statistical Summary

• Sample size: 996 completed PRs (767 merged, 229 closed)
• Overall merge rate: 77.0%
• Highest performing category: Removal tasks (88.9%)
• Most common category: Bug fixes (588 PRs, 59% of sample)
• Prompt length: ~164 words average (no significant difference between merged/closed)

---

Analysis period: Last 30 days | Data source: githubnext/gh-aw Copilot PRs

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.