[prompt-analysis] Copilot PR Prompt Analysis - 2026-06-01

[github-actions[bot]]

Summary

Analysis Period: Last 30 days
Total PRs: 1,000 | Merged: 802 (80.2%) | Closed: 195 (19.8%) | Open: 3

Overall success rate is 80.2%, consistent with historical trend (~80.5% avg over the past 2 weeks).

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Refactor	11	10	1	91% ✅
Feature Addition	17	14	3	82% ✅
Bug Fix	787	630	154	80%
Test	151	121	30	80%
Documentation	29	22	7	76%
Other	5	5	0	100% (tiny sample)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 443 words (significantly shorter than closed PRs)
• Most common keywords (exclusive to merged): changes, setup, docs, updates, prompt, updated, explicit, repo
• Prompts tend to be concise and action-oriented

Example successful PRs:

1. PR Fix validate-yaml failure by regenerating code-simplifier lock workflow with dev metadata #36316: Fix validate-yaml failure by regenerating code-simplifier lock workflow with dev → Merged

   "validate-yaml was failing because .github/workflows/code-simplifier.lock.yml contained release-c..."

2. PR Make timeout-minutes a templatable integer in schema + custom job compilation #36314: Make timeout-minutes a templatable integer in schema + custom job compilation → Merged

   "timeout-minutes needed to validate as a templatable_integer and behave consistently..."

3. PR Optimize Code Simplifier workflow token footprint via deterministic preprocessing, inline skills, and small-model sub-agents #36311: Optimize Code Simplifier workflow token footprint via deterministic preprocessing → Merged

   "This updates code-simplifier to reduce token consumption by shifting repeatable work..."

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 603 words (36% longer than merged PRs)
• Most common keywords (exclusive to closed): engine, issue, codex, default, schema, proxy, config
• Many closed PRs contain [WIP] in the title — confirmed pattern from historical data
• Closed PRs often involve broad scope (engine changes, schema rewrites) or are abandoned early

Example closed PRs:

1. PR Harden create_pull_request guidance to block placeholder/WIP PR emissions #36328: Harden create_pull_request guidance to block placeholder/WIP PR emissions → Closed

   "Copilot sessions were opening short-lived [WIP] PRs and closing them without deliverables..."

2. PR [WIP] Document 3 undocumented top-level frontmatter fields and fix broken anchors #36310: [WIP] Document 3 undocumented top-level frontmatter fields and fix broken anchor → Closed

   "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date..."

3. PR [WIP] Remove dead templatable_integer $def from main_workflow_schema.json #36308: [WIP] Remove dead templatable_integer $def from main_workflow_schema.json → Closed

Key Insights

• Conciseness wins: Merged PRs average 443 words vs 603 words for closed PRs — a 36% difference. Shorter, focused prompts consistently outperform lengthy ones (trend confirmed across all 10 historical data points).
• WIP PRs are high-risk: PRs with [WIP] in the title appear frequently in closed PRs. Historical data shows WIP success rates as low as 28–47% vs 82%+ for non-WIP PRs.
• Refactoring prompts outperform expectations: 91% success rate (highest this analysis), suggesting well-scoped refactoring tasks are highly executable by Copilot. Feature and bug-fix prompts are roughly equivalent (~80–82%).

Recommendations

Based on today's analysis:

1. DO: Write concise prompts under 500 words. The data consistently shows merged PRs use fewer words — focus on the specific change, not context padding.
2. DO: Use action verbs with clear scope: "fix X in file Y", "add Z to function W", "refactor A to use B". Merged keywords like explicit, setup, updates reflect this directness.
3. AVOID: Opening PRs with [WIP] titles. These have a ~28–47% merge rate historically. Either start with a complete task or don't open the PR yet.
4. AVOID: Broad engine/schema/proxy changes without clear, atomic scope. These appear disproportionately in closed PRs.

Historical Trends (Last 7 Analysis Runs)

Date	PRs	Merged	Success Rate	Avg Words (Merged)
2026-05-24	1000	810	80.6%	772
2026-05-25	1000	804	80.6%	~608
2026-05-26	1000	802	80.6%	609
2026-05-27	1000	804	80.7%	543
2026-05-28	1000	799	80.5%	511
2026-05-30	1000	802	80.5%	491
2026-06-01	1000	802	80.2%	443

Trend: Success rate is remarkably stable at ~80.2–80.7%. Notably, average prompt word count for merged PRs has declined steadily from 772 words (May 24) to 443 words today — suggesting prompts are becoming more concise and targeted over time.

---

Generated by Copilot PR Prompt Analysis (Run: §26783912095)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · sonnet46 955.8K · ◷

• expires on Jun 2, 2026, 9:50 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #36536.