Copilot PR Prompt Analysis - 2025-12-26

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2025-12-26

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 734 (73.4%) | Closed: 260 (26.0%) | Open: 6
Overall Success Rate: 73.8% (merged / completed)

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Open	Success Rate*
Testing	346	248	95	3	72.3%
Bug Fix	238	170	68	0	71.4%
Other	137	103	32	2	76.3%
Feature Addition	118	87	30	1	74.4%
Documentation	68	52	16	0	76.5%
Update	52	40	12	0	76.9%
Refactoring	24	19	5	0	79.2%
Removal	17	15	2	0	88.2%

*Success Rate = Merged / (Merged + Closed)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 862 characters (125 words)
• Include URL references: 45.6% of merged PRs
• Reference specific files: 26.3% of merged PRs
• Include error context: 20.0% of merged PRs
• Average specificity score: 3.3 (inline code/file references)

Most common keywords in merged PRs:

• includes_url (335 occurrences)
• go (304 occurrences)
• includes_code (296 occurrences)
• add (260 occurrences)
• update (259 occurrences)
• test (244 occurrences)
• resolve (209 occurrences)
• fix (205 occurrences)

Example successful prompts:

7700. #7700: Merged main branch into the security fix PR while preserving the backslash escaping fix for CodeQL Alert Proposal: Add gemini as agentic engine #83. - **Merge ... → ✅ Merged

7701. #7696: Fix is tests → ✅ Merged

7702. #7695: Added comprehensive test coverage for the incomplete string escaping vulnerability (Alert Proposal: Add gemini as agentic engine #83, CWE-116) in `formatBashCo... → ✅ Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 1126 characters (165 words)
• Include URL references: 54.6% of closed PRs
• Reference specific files: 31.2% of closed PRs
• Include error context: 17.7% of closed PRs
• Average specificity score: 4.1 (inline code/file references)

Most common keywords in closed PRs:

• includes_url (142 occurrences)
• go (135 occurrences)
• add (123 occurrences)
• resolve (120 occurrences)
• includes_code (114 occurrences)
• update (92 occurrences)
• md (91 occurrences)
• test (89 occurrences)

Example unsuccessful prompts:

7702. #7702: Fix tests → ❌ Closed

7703. #7701: Thanks for assigning this issue to me. I'm starting to work on it and will keep this PR's description up to date as I fo... → ❌ Closed

7704. #7677: > ----

This section details on the original issue you should resolve

(issue_title)[plan] Add retry and error... → ❌ Closed

Key Insights

📊 Pattern Analysis

1. Prompt Length Paradox: Closed PRs have LONGER prompts (avg 165 words) vs merged PRs (125 words), suggesting conciseness correlates with success.

2. Category Success Hierarchy:

   • Highest Success: Testing (72.3% success rate)
   • Needs Improvement: Removal (88.2% success rate)

3. URL References: 54.6% of closed PRs include URLs vs 45.6% of merged PRs - URLs are associated with MORE complex/problematic tasks.

4. Specificity Score: Closed PRs have HIGHER specificity (4.1) vs merged (3.3), indicating over-specification or attempting overly complex changes.

💡 Success Factors

Testing prompts perform best (72.3% success) - clear, focused scope makes them easier to implement correctly.

Bug fixes are moderately successful (71.4% success) - benefit from error context but may involve complex debugging.

Generic "Other" prompts struggle (76.3% success) - lack of clear categorization suggests ambiguous requirements.

Recommendations

Based on today's analysis:

✅ DO:

1. Keep prompts concise - Aim for ~125 words or less. Merged PRs average 125 words vs 165 words for closed PRs.

2. Focus on testing and bug fixes - These categories have the highest success rates (71.7% and 71.4% respectively).

3. Be specific about scope - Reference specific files when appropriate, but avoid over-specifying implementation details.

4. Include error context for bug fixes - 20% of merged PRs include error messages, helping provide clear reproduction steps.

⚠️ AVOID:

1. Overly long explanations - More words don't equal better results. Closed PRs are 31% longer on average.

2. Vague or uncategorizable requests - "Other" category has lower success rate. Be explicit about the type of change.

3. Over-specification - Higher specificity scores in closed PRs suggest micromanaging implementation reduces success.

4. Complex multi-part changes - Break large changes into smaller, focused PRs.

Historical Trends

Historical trend tracking will be available after multiple daily runs.

---

Methodology: Analyzed 1000 Copilot-generated PRs from the last 30 days in githubnext/gh-aw repository. Prompts extracted from PR bodies, categorized by intent, and correlated with merge outcomes.

References:

• §20522016885

AI generated by Copilot PR Prompt Pattern Analysis

[pelikhan]

/q update agentic workflow to maintain an issue template to create issues that will work well with copilot. Use safe output create-pull-request title prefix "[ca] ", draft: true.

Only update template if needed.

[github-actions[bot]]

🎩 Mission equipment ready! Q has optimized your workflow. Use wisely, 007! 🔫

[github-actions[bot]]

✅ Issue Template Optimizer Workflow Created

I've created a new agentic workflow to maintain GitHub issue templates based on the Copilot PR success patterns from your analysis.

What Was Created

New Workflow: .github/workflows/issue-template-optimizer.md

This workflow will:

• ✅ Analyze issue templates against Copilot success patterns (~125 word optimal length)
• ✅ Identify opportunities to promote conciseness and specificity
• ✅ Create draft PRs with [ca] prefix when improvements are needed
• ✅ Run weekly on Mondays to keep templates optimized

Key Implementation Details

Based on Analysis Findings:

• Successful prompts average 125 words (vs 165 for closed PRs)
• Current templates have 348 and 366 words - may encourage verbosity
• Conciseness and specificity correlate with higher success rates

Workflow Features:

• Uses Copilot engine as requested
• Safe-outputs configured: title-prefix: "[ca]", draft: true
• Cache memory to track optimization history
• Only updates templates when improvements are actually needed

Tools Configured:

• cache-memory: Track optimization history
• github: Analyze templates and search for patterns
• edit: Make surgical changes to templates
• bash: Measure word counts and validate changes

Pull Request

A draft PR has been created with the new workflow file for review. The workflow:

• ✅ Compiles successfully (gh-aw compile)
• ✅ Follows repository patterns (similar to glossary-maintainer.md)
• ✅ Implements exactly what was requested in your comment
• ⚠️ Note: The .lock.yml file was not included per repository guidelines

Next Steps

Once the PR is merged:

1. The workflow will run weekly on Mondays at 9am UTC
2. It will analyze the current issue templates against success patterns
3. If improvements are needed, it will create a draft PR with optimizations
4. Human review ensures template changes preserve functionality

References

• Analysis: Copilot PR Prompt Pattern Analysis - 2025-12-26
• Success Pattern: ~125 words, specific scope, technical tone
• Request: Copilot PR Prompt Analysis - 2025-12-26 #7728 (comment)

🎩 Equipped by Q fer issue #discussion #7728 🗺️