[prompt-clustering] Copilot Agent Prompt Clustering — 2026-06-05

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-05-17 → 2026-06-05)
Total Tasks Analyzed: 999 copilot-authored PRs
Clusters Identified: 8 (KMeans, silhouette 0.0419)
Overall Success Rate: 75.4% merged
Avg Iterations: 4.38 commits/PR

Eight stable themes emerged. The two largest — shared-helper refactors and safe-output/schema work — account for 59% of all tasks. Success skews high overall, but two clusters drag below the mean: Codex/AWF generated config & defaults (66%) and the Copilot SDK-driver work (68%).

Key Findings

1. Refactor + safe-output work dominates. Clusters C6 and C2 together are 586 of 999 PRs and merge near the mean (~75–76%) — the reliable bread-and-butter of the agent fleet.
2. Generated/Codex config (C7) is the weak spot. At 66% merge with the largest blast radius (avg 78 files changed, ~1054/780 add/del), large auto-generated diffs are the least likely to land.
3. Sous-chef tasks (C4) succeed but grind. Highest merge rate (85%) yet by far the most iterations (9.09 commits/PR) — they get there, but slowly.
4. Small, well-scoped fixes win. "Fix failing Actions job" (C1) merges at 81% in only 3.11 commits — the tightest scope, the cleanest outcome.
5. Trend: success dipped this period. Overall merge rate 75.4% vs 80.3% (2026-06-02) and 78.8% earlier — worth watching whether the SDK/Codex clusters are pulling the average down.

Success Rate by Cluster

Cluster	Theme	Tasks	%	Success	Avg Commits	Avg Files
C6	Shared helpers & error/path refactors	344	34%	75%	4.02	32
C2	Safe-outputs & schema validation	242	24%	76%	4.31	13
C3	Prompts, skills & experiments	131	13%	80%	3.66	11
C7	Codex/AWF generated config & defaults	126	13%	66%	4.19	78
C4	Sous-chef multi-agent (GPT-mini) tasks	54	5%	85%	9.09	35
C5	Model alias/multiplier plumbing	41	4%	76%	4.02	49
C0	Copilot SDK driver/harness mode	34	3%	68%	5.94	33
C1	Fixing failing GitHub Actions jobs	27	3%	81%	3.11	23

Detailed cluster breakdown

C6: Shared helpers & error/path refactors

• Size: 344 tasks (34.4%)
• Outcome: 259 merged / 55 closed / 30 open → 75% success
• Effort: 4.02 commits, 1.9 reviews, 2.91 comments, 32 files (+5008/-261)
• Top terms: shared, helper, behavior, error, path, function
• Example PRs: Target forecast log queries to successful workflow runs #37063, linters: modernize golang.org/x/tools usage — Cursor API, direct assertions, golden fix test #36221, Align copilot harness test with reflect helper API contract #36823, feat: ban python/python3 invocations in action shell scripts #36735

C2: Safe-outputs & schema validation

• Size: 242 tasks (24.2%)
• Outcome: 184 merged / 49 closed / 9 open → 76% success
• Effort: 4.31 commits, 1.8 reviews, 2.94 comments, 13 files (+312/-76)
• Top terms: schema, safe, safe output, issue, output, validation
• Example PRs: fix(chaos-fuzzer): add recreate-ref: true to prevent branch-exists push failure #32769, Prevent chaos create-pull-request fallback when branch already exists #32770, deps: bump github.com/charmbracelet/x/exp/golden to v0.0.0-20260525135217-abeec2b8bf0b #35188, [WIP] Add deprecated: true flag to rate-limit and inline-sub-agents #34776

C3: Prompts, skills & experiments

• Size: 131 tasks (13.1%)
• Outcome: 105 merged / 22 closed / 4 open → 80% success
• Effort: 3.66 commits, 1.66 reviews, 1.48 comments, 11 files (+214/-134)
• Top terms: prompt, guidance, experiment, skill, sub, report
• Example PRs: [WIP] Refactor workflows to adopt github-guard-policy.md #33781, [WIP] Add 5-minute MCP keepalive heartbeat to prevent inactivity timeout #34288, [WIP] Update Go version from 1.25 to 1.26 #34317, [WIP] Refactor to extract progressive disclosure guidelines into shared component #33817

C7: Codex/AWF generated config & defaults

• Size: 126 tasks (12.6%)
• Outcome: 83 merged / 33 closed / 10 open → 66% success
• Effort: 4.19 commits, 2.56 reviews, 7.95 comments, 78 files (+1054/-780)
• Top terms: codex, awf, default, generated, model, step
• Example PRs: Linter Miner: raise execution limits and switch bash tool allowlist to wildcard #32816, Increase default repo-memory MaxFileSize from 10KB to 100KB #32826, Add smoke-antigravity workflow by cloning smoke-gemini onto antigravity engine #34729, [WIP] Fix pin Codex CLI back from 0.133.0 due to HTTP 400 errors #34663

C4: Sous-chef multi-agent (GPT-mini) tasks

• Size: 54 tasks (5.4%)
• Outcome: 46 merged / 5 closed / 3 open → 85% success
• Effort: 9.09 commits, 2.96 reviews, 9.91 comments, 35 files (+515/-139)
• Top terms: sous, sous chef, chef, gpt, mini, model
• Example PRs: feat(pr-sous-chef): run formatters and push to branch #33125, Harden PR Sous Chef and Contribution Check prompts to require explicit add_comment targets #35371, Fix AWF tool-cache mounting so Daily News Copilot can start in chroot #36900, Add opusplan builtin alias to Claude model routing #34263

C5: Model alias/multiplier plumbing

• Size: 41 tasks (4.1%)
• Outcome: 31 merged / 10 closed / 0 open → 76% success
• Effort: 4.02 commits, 1.98 reviews, 3.34 comments, 49 files (+281/-84)
• Top terms: alias, multiplier, model, inventory, aliases, docs
• Example PRs: Add claude-opus-4.7 dotted form to model multipliers #35334, Updating model multiplier updater agentic workflow #37028, Model inventory 2026-05-19: add raptor-mini alias coverage and missing GPT-5 search multipliers #33177, Add /sitemap.xml alias and strengthen README GEO/brand signals #32862

C0: Copilot SDK driver/harness mode

• Size: 34 tasks (3.4%)
• Outcome: 23 merged / 11 closed / 0 open → 68% success
• Effort: 5.94 commits, 2.35 reviews, 6.97 comments, 33 files (+448/-142)
• Top terms: sdk, driver, mode, harness, sdk mode, sdk driver
• Example PRs: docs: update copilot sdk driver spec with token isolation policy (v1.0.2) #37045, Add otlp-env-vars skill for OpenTelemetry SDK env var configuration #32827, Update bundled Copilot SDK to 1.0.0 and recompile lockfiles #36495, chore(deps): update github.com/modelcontextprotocol/go-sdk v1.6.0 → v1.6.1 #35201

C1: Fixing failing GitHub Actions jobs

• Size: 27 tasks (2.7%)
• Outcome: 22 merged / 5 closed / 0 open → 81% success
• Effort: 3.11 commits, 0.89 reviews, 0.19 comments, 23 files (+49/-41)
• Top terms: actions, failing github, actions job, job, github actions, failing
• Example PRs: [WIP] Fix failing GitHub Actions job js-typecheck #32839, [WIP] Fix failing GitHub Actions job agent #34119, [WIP] Fix failing GitHub Actions job lint-go #33548, [WIP] Fix failing GitHub Actions job 'agent' #34639

Representative data table (2 highest-iteration PRs per cluster)

PR #	Title	Cluster	Outcome	Commits	Files
#33033	Add github-app.missing-key ignore mode and guard App token	C6	Merged	15	16
#33129	Fix compound || expressions in prompt markdown never sub	C6	Merged	14	6
#33350	feat(safe-outputs): add required-labels/required-title-prefi	C2	Merged	28	70
#33852	Add create-check-run safe output type for multi-agent PR a	C2	Merged	17	29
#34874	Add inline skill extraction/runtime support mirroring inline	C3	Merged	18	29
#35773	Update gh aw init to create the Agentic Workflows custom a	C3	Merged	17	10
#35286	Centralize compiler enterprise env controls, expand GH_AW_DE	C7	Merged	19	244
#35802	[awf] Fix tool-cache mount handling, smoke-pi runtime config	C7	Merged	17	254
#33273	Add on.pull_request_reviewer: slash_command synthetic trig	C4	Merged	39	257
#36676	Compat-based Copilot CLI install: single remote fetch, jq-on	C4	Open	32	7
#34837	Move model alias/multiplier propagation from step env to act	C5	Merged	18	258
#36634	Tighten ET computation details layout and compact model alia	C5	Merged	13	8
#36358	Fix copilot-sdk harness stdin wiring, SDK installation/resol	C0	Merged	19	247
#36538	Refine Copilot SDK-mode tool permission scoping from engine	C0	Merged	19	68
#32840	[WIP] Fix failing GitHub Actions job JS Tests (shard 4/4)	C1	Merged	7	2
#33165	[WIP] Fix failing GitHub Actions job JS Tests (shard 4/4)	C1	Merged	6	3

Recommendations

1. Tighten Codex/AWF-generated config tasks (C7). The lowest merge rate pairs with the biggest diffs. Split large generated changes into reviewable chunks, or add a pre-merge diff-size guard so reviewers aren't handed 78-file PRs.
2. Investigate Copilot SDK-driver failures (C0). 68% merge over 5.9 commits suggests the harness/SDK mode is still flaky — a good candidate for a focused reliability pass.
3. Cap iteration churn on sous-chef tasks (C4). They land but average 9 commits; clearer up-front task specs or a turn budget could cut the back-and-forth.
4. Keep leaning on tightly-scoped fix prompts (C1). Cheapest and among the most reliable — the pattern to replicate when phrasing new tasks.

Methodology & limitations

• TF-IDF (1–2 grams, domain stop-words removed) over title+body of 999 cleaned PR descriptions; KMeans with k chosen by silhouette across k=4–8.
• Firewall/warning boilerplate, code blocks, and URLs stripped before vectorizing.
• Iterations use commit count as a proxy: the gh-aw workflow-run logs (true turn counts/cost) were not fetched this run, so turn/cost metrics are approximated by commits. Silhouette is low (0.04), expected for short, overlapping technical text — clusters are directional, not hard partitions.
• History persisted to cache for trend tracking across runs.

References: §27010769683

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 151 AIC · ◷

• expires on Jun 6, 2026, 11:18 AM UTC