Skip to content

[Issue] Complete Agent Inspector funnel telemetry and failure diagnostics #9585

Description

@dooriya

Describe the bug
Current BI can observe the azure.ai.inspector command through ext.run, but that only means the command started. It does not prove that the Inspector server started, the browser UI loaded, the UI connected to a running agent, or the user completed an interaction.

As a result, we cannot accurately measure Inspector intent, browser-launch success, agent-connection success, or explain where and why users drop from the funnel. A missing downstream event must be treated as not observed, not automatically as a failure.

Desired funnel:

route selected
-> agent port ready
-> Inspector launch requested
-> Inspector ext.run started
-> Inspector server ready
-> browser open accepted
-> UI ready
-> agent connected
-> first request sent
-> first response received

To Reproduce
Run azd ai agent run for a non-Activity agent and inspect its telemetry. Existing ext.run data cannot distinguish the milestones above or diagnose most launch and connection failures.

Expected behavior
Use the existing extension ReportUsage API; no new azd-core telemetry events or attributes are required.

extension.event = local_client.route.selected
ext.route = inspector | playground | suppressed

extension.event = inspector.funnel.stage
ext.stage = <fixed stage enum>
ext.outcome = succeeded | failed
ext.reason = <optional fixed failure enum>

Metric definitions:

  • Inspector intent proxy: route selected with ext.route=inspector.
  • Successful browser launch: SPA reports setViewReady; browser.OpenURL alone is insufficient.
  • Agent connected: a successful application-level discovery/protocol request.
  • Inspector used: first request forwarded to the agent.
  • Successful interaction: first response or SSE data received.

Acceptance criteria:

  • Emit each successful milestone at most once per invocation.
  • Report known failures with stable, low-cardinality reasons.
  • Never emit URLs, ports, paths, prompts, responses, IDs, or raw errors.
  • Reuse existing Inspector ext.run for command start and final status.
  • Verify Agents-to-Inspector correlation through operation_Id.
  • Update telemetry documentation, tests, and the privacy checklist.

Environment

  • Components: azure.ai.agents, azure.ai.inspector, and azd extension telemetry
  • Platforms: all supported platforms

Additional context
azure.ai.agents owns route selection, launch checks, agent readiness, and launch request signals. azure.ai.inspector owns server, browser, SPA, connection, request, and response signals. The Inspector extension must upgrade to an azd SDK version that supports ReportUsage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agenticAgent-related featuresaiAIarea/telemetryTelemetry, tracing, observabilitybugSomething isn't workingext-agentsazure.ai.agents extensionext-inspectorazure.ai.inspector extension

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions