Copilot Agent Prompt Clustering Analysis - December 2025

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis

Daily NLP-based clustering analysis of Copilot coding agent task prompts using TF-IDF vectorization and K-means clustering.

Summary

Analysis Period: 2025-10-22 to 2025-12-26 (65 days)
Total Tasks Analyzed: 2,381
Clusters Identified: 8
Overall Success Rate: 73.1%
Average Tasks/Day: 36.6

Key Findings

1. 73.1% overall success rate across 2,381 tasks shows Copilot coding agent is highly effective at autonomous software development.
2. Cluster 7 has the highest success rate at 81.7%, suggesting these task types are well-suited for automation.
3. Cluster 1 shows lower success rate at 62.6%, indicating room for improvement in prompt engineering or agent capabilities.
4. Average task involves 20.1 files with +1359/-1400 line changes, indicating substantial code modifications per task.
5. Cluster 4 dominates with 891 tasks (37.4% of all tasks), suggesting this task type is most common.

Cluster Summary

Cluster	Tasks	% of Total	Success Rate	Avg Files	Top Keywords
C4	891	37.4%	73.4%	26.0	github, not, issue
C2	499	21.0%	72.3%	12.6	files, workflow, githubnext
C3	292	12.3%	76.7%	13.8	workflow, agentic, workflows
C5	235	9.9%	68.9%	17.5	cli, issue, command
C8	158	6.6%	73.4%	22.5	agent, copilot, coding
C1	123	5.2%	62.6%	27.5	mcp, server, github
C7	109	4.6%	81.7%	17.5	fix, tests, lint
C6	74	3.1%	77.0%	19.1	code, analysis, duplicate

Full Cluster Analysis

Cluster 4: General Updates & Issues (37.4% of tasks)

• Success Rate: 73.4%
• Avg Code Changes: 26.0 files, +1306/-2398 lines
• Top Keywords: github, not, issue, output, file
• Sample Tasks: Documentation updates, directory creation, JavaScript migrations

Cluster 2: Workflow Investigation (21.0% of tasks)

• Success Rate: 72.3%
• Avg Code Changes: 12.6 files, +662/-581 lines
• Top Keywords: files, workflow, githubnext, all, issue
• Sample Tasks: Investigating failures, fixing heredoc collisions, extracting functions

Cluster 3: Agentic Workflow Management (12.3% of tasks)

• Success Rate: 76.7%
• Avg Code Changes: 13.8 files, +1751/-422 lines
• Top Keywords: workflow, agentic, workflows, create, github
• Sample Tasks: Scheduling workflows, adding smoke tests, semantic refactoring

Cluster 5: CLI & Configuration (9.9% of tasks)

• Success Rate: 68.9%
• Avg Code Changes: 17.5 files, +1959/-576 lines
• Top Keywords: cli, issue, command, version, section
• Sample Tasks: Changing max-turns, raising errors on limits, token updates

Cluster 8: Agent Instructions (6.6% of tasks)

• Success Rate: 73.4%
• Avg Code Changes: 22.5 files, +1356/-503 lines
• Top Keywords: agent, copilot, coding, your, you
• Sample Tasks: Updating variables, debug logging, migrating logger packages

Cluster 1: MCP Server Work (5.2% of tasks)

• Success Rate: 62.6% ⚠️
• Avg Code Changes: 27.5 files, +2887/-3964 lines (largest changes!)
• Top Keywords: mcp, server, github, configuration, safe
• Sample Tasks: Token configuration, MCP server refactoring, transport changes
• Note: Lowest success rate with highest complexity suggests need for task breakdown

Cluster 7: Fixes & Tests (4.6% of tasks)

• Success Rate: 81.7% ⭐ (highest!)
• Avg Code Changes: 17.5 files, +385/-163 lines
• Top Keywords: fix, tests, lint, format, workflow
• Sample Tasks: CI fixes, test expectations, quoting issues

Cluster 6: Code Refactoring (3.1% of tasks)

• Success Rate: 77.0%
• Avg Code Changes: 19.1 files, +2138/-872 lines
• Top Keywords: code, analysis, duplicate, refactoring, output
• Sample Tasks: Duplicate code refactoring, regex optimization, helper extraction

Recommendations

Strategic Recommendations

1. Documentation Tasks: Cluster 7 (fix/tests) shows 81.7% success - these straightforward tasks work well. Prioritize similar well-defined, testable tasks.

2. Complex Refactoring: Cluster 6 (code analysis/refactoring) at 77.0% success suggests autonomous refactoring is viable. Consider expanding to more refactoring workflows.

3. MCP Server Tasks: Cluster 1 (MCP server) at 62.6% success with largest code changes suggests complexity. Break down MCP-related tasks into smaller, focused changes.

4. Workflow Standardization: With 37.4% of tasks in Cluster 4, standardize common patterns. Create templates for frequent task types to improve consistency.

Cluster-Specific Actions

• Cluster 5 (CLI tasks at 68.9%): Review failed PRs to identify common failure patterns. Consider adding more specific instructions or breaking down complex tasks.

• Cluster 1 (MCP server at 62.6%): High comment count (avg 5.1) suggests clarification needs. Improve initial prompt clarity or add more context. Break large tasks into incremental changes.

Methodology

Analysis Pipeline

1. Data Collection: Extracted 2,381 task prompts from Copilot-created pull requests
2. Text Processing: Cleaned and normalized prompts, removing markdown, code blocks, and noise
3. Vectorization: Applied TF-IDF (Term Frequency-Inverse Document Frequency) with 150 features, 1-3 word n-grams
4. Clustering: K-means clustering with k=8 (determined via elbow method and silhouette score)
5. Analysis: Extracted top keywords, success rates, and code change patterns per cluster

Cluster Interpretation

Each cluster represents a distinct type of task based on:

• Semantic similarity of task prompts (TF-IDF vectors)
• Common keywords and phrases
• Code change patterns (files, additions, deletions)
• Success rates (merge vs. close)

---

Next Steps: This analysis will run daily to track trends, identify emerging patterns, and monitor the impact of prompt engineering improvements.

AI generated by Copilot Agent Prompt Clustering Analysis