Copilot/MCP Impact Analysis Is Superficial—Keyword/Priority-Based, Not Functionality-Aware

[dhanveer16]

Description

Currently Copilot/MCP impact analysis for test suite prioritization relies mainly on predefined priority fields and keyword matching, rather than interpreting actual business logic, functional flows, or acceptance criteria—even when detailed documentation is available in ADO.
The result is misaligned prioritization for regression impact validation scenarios, where the relationship between PBIs → test cases → acceptance criteria must be interpreted functionally, not just via labels or keywords.

Current impact analysis relies on keywords and priority fields instead of functional understanding, causing misaligned test prioritization, high manual effort, and missed critical flows.

Expected Behavior

Copilot/MCP should leverage semantic understanding of

1. Acceptance criteria (structured AC, scenario outlines).
2. Functional documents and diagrams (flows, components, data contracts).

Output should:

Propose functionality-aware prioritized test lists with rationale (e.g., “Touches payment auth → prioritize 3DS failover tests”).
Surface risk hotspots (integration points, shared services).
Respect hierarchies and multi-suite context in Test Plans.

Current Behavior

Prioritization appears driven by priority fields and keyword/tag matches from titles/descriptions.
No deep parsing of acceptance criteria or functional dependencies.
Cross-feature integrations and edge conditions are missed.
Explanations (when present) are too generic to be auditable or trusted.

Steps to Reproduce

Provide a Feature with detailed acceptance criteria (e.g., multi-step payment flow with edge cases).
Link multiple PBIs and test cases spanning authentication, payments, and notifications.
Trigger impact analysis/prioritization in Copilot/MCP after a change related to tokenization.
Observe that recommended tests skew to items containing “token,” “auth,” or high-priority fields, omitting critical integrated cases (e.g., 3DS fallback, partial capture, retry after gateway timeout), which are described only in AC/functional docs.

[danhellem]

The MCP Server tools is just a layer that sits on top of the public rest apis. They are simple. If you want to do more, the suggestion is to load all the items you need and use the LLM to analyze and update as needed.

[sruthirangaraj122001]

Thanks for the response @danhellem.

We already maintain multiple functional documents and detailed information within Azure DevOps Test Plans. Downloading all PBIs and test cases and then passing them as context to the LLM is practically not feasible. Is there an alternative approach that avoids bulk data extraction while still enabling functionality-aware analysis?