Skip to content

Commit d85f04c

Browse files
dooriyaCopilot
andcommitted
Instrument local client route selection
Report categorical Inspector, Playground, and suppressed routes through extension telemetry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent eb6c588 commit d85f04c

9 files changed

Lines changed: 147 additions & 5 deletions

File tree

cli/azd/extensions/azure.ai.agents/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,20 @@ effect: activity-protocol agents open the Microsoft 365 Agents Playground rather
3333
than the Agent Inspector, and `--port 8087` on its own collides with the
3434
inspector's own default UI port.
3535

36+
### Local client route telemetry
37+
38+
When installed from the official registry, the extension reports the
39+
`local_client.route.selected` usage event after `azd ai agent run` resolves the
40+
service and protocol profile. Its `ext.route` attribute is exactly one of:
41+
42+
- `inspector` for a non-activity agent;
43+
- `playground` for an activity-protocol agent; or
44+
- `suppressed` when `--no-client` or the deprecated `--no-inspector` is set.
45+
46+
The event is emitted before checking client availability, starting the local
47+
agent, or launching a client. It records route selection, not successful client
48+
launch.
49+
3650
## Migrating Legacy Agent Configuration
3751

3852
New Foundry agent projects keep the agent definition directly on the

cli/azd/extensions/azure.ai.agents/extension.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ description: Ship agents with Microsoft Foundry from your terminal. (Beta)
66
usage: azd ai agent <command> [options]
77
# NOTE: Make sure version.txt is in sync with this version.
88
version: 1.0.0-beta.9
9-
requiredAzdVersion: ">=1.27.1"
9+
requiredAzdVersion: ">=1.31.0"
1010
dependencies:
1111
- id: azure.ai.inspector
1212
version: "~1.0.0-beta.1"

cli/azd/extensions/azure.ai.agents/go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ require (
1212
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerregistry/armcontainerregistry v1.3.0-beta.3
1313
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
1414
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions v1.3.0
15-
github.com/azure/azure-dev/cli/azd v1.28.0
15+
github.com/azure/azure-dev/cli/azd v1.31.0
1616
github.com/braydonk/yaml v0.9.0
1717
github.com/drone/envsubst v1.0.3 // indirect
1818
github.com/fatih/color v1.18.0

cli/azd/extensions/azure.ai.agents/go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWp
6161
github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
6262
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
6363
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
64-
github.com/azure/azure-dev/cli/azd v1.28.0 h1:mqqyV85m7A1XfWJFjV/Ut0QoIEImFeF++1Ruq/cRp0s=
65-
github.com/azure/azure-dev/cli/azd v1.28.0/go.mod h1:Ge7QaU9PoJM7i6J0xArDoQCf2tUn6O7OIKkoItxFTA8=
64+
github.com/azure/azure-dev/cli/azd v1.31.0 h1:p0U4F6w2bPrdzmzavksqfJCnlXoQu9GTQogy+6KXMmM=
65+
github.com/azure/azure-dev/cli/azd v1.31.0/go.mod h1:HFBGeWRWhNsOoYaUcyToqaowibqcbSCfkfJfnIfI4nU=
6666
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
6767
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
6868
github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o=

cli/azd/extensions/azure.ai.agents/internal/cmd/run.go

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ import (
3838
const (
3939
agentInspectorExtensionID = "azure.ai.inspector"
4040
agentInspectorReadyPollPeriod = 250 * time.Millisecond
41+
localClientRouteSelectedEvent = "local_client.route.selected"
42+
localClientRouteAttribute = "route"
43+
localClientRouteInspector = "inspector"
44+
localClientRoutePlayground = "playground"
45+
localClientRouteSuppressed = "suppressed"
4146
// defaultInspectorUIPort mirrors the default UI port of the
4247
// azure.ai.inspector extension. The inspector extension remains the source
4348
// of truth for the actual default: when --inspector-port is unset we do not
@@ -152,6 +157,8 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
152157
// validation can fail without starting a process, and such a failure must
153158
// not clear a session belonging to an already-running agent.
154159
activityProfile := resolveActivityRunProfile(runCtx.Definition)
160+
suppressClient := flags.noInspector || flags.noClient
161+
reportLocalClientRouteSelected(ctx, azdClient.Telemetry(), activityProfile, suppressClient)
155162
if err := validateInspectorPortForProfile(flags, activityProfile.IsActivity); err != nil {
156163
return err
157164
}
@@ -171,7 +178,6 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
171178
// Resolve local-client availability before the agent starts so advisory
172179
// port warnings can account for whether an inspector will actually launch.
173180
// Reuse the result after proc.Start rather than issuing a second RPC.
174-
suppressClient := flags.noInspector || flags.noClient
175181
inspectorInstalled := false
176182
var inspectorInstallErr error
177183
if !activityProfile.IsActivity && !suppressClient {
@@ -379,6 +385,29 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
379385
return nil
380386
}
381387

388+
func reportLocalClientRouteSelected(
389+
ctx context.Context,
390+
telemetry azdext.TelemetryServiceClient,
391+
activityProfile activityRunProfile,
392+
suppressClient bool,
393+
) {
394+
route := localClientRouteInspector
395+
if suppressClient {
396+
route = localClientRouteSuppressed
397+
} else if activityProfile.IsActivity {
398+
route = localClientRoutePlayground
399+
}
400+
401+
if _, err := telemetry.ReportUsage(ctx, &azdext.ReportUsageRequest{
402+
EventName: localClientRouteSelectedEvent,
403+
Attributes: map[string]string{
404+
localClientRouteAttribute: route,
405+
},
406+
}); err != nil {
407+
log.Printf("run: failed to report local client route selection: %v", err)
408+
}
409+
}
410+
382411
func handleInspectorAutoLaunch(
383412
ctx context.Context,
384413
workflow azdext.WorkflowServiceClient,

cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import (
99
"errors"
1010
"fmt"
1111
"io"
12+
"maps"
1213
"net"
1314
"net/http"
1415
"net/http/httptest"
@@ -238,6 +239,73 @@ func TestWaitForLocalPort(t *testing.T) {
238239
})
239240
}
240241

242+
func TestReportLocalClientRouteSelected(t *testing.T) {
243+
t.Parallel()
244+
245+
tests := []struct {
246+
name string
247+
activityProfile activityRunProfile
248+
suppressClient bool
249+
reportErr error
250+
wantRoute string
251+
}{
252+
{
253+
name: "selects Inspector for non-activity agent",
254+
wantRoute: localClientRouteInspector,
255+
},
256+
{
257+
name: "selects Playground for activity agent",
258+
activityProfile: activityRunProfile{IsActivity: true},
259+
wantRoute: localClientRoutePlayground,
260+
},
261+
{
262+
name: "selects suppressed for non-activity agent",
263+
suppressClient: true,
264+
wantRoute: localClientRouteSuppressed,
265+
},
266+
{
267+
name: "suppression overrides activity route",
268+
activityProfile: activityRunProfile{IsActivity: true},
269+
suppressClient: true,
270+
wantRoute: localClientRouteSuppressed,
271+
},
272+
{
273+
name: "reporting failure is best effort",
274+
reportErr: errors.New("telemetry unavailable"),
275+
wantRoute: localClientRouteInspector,
276+
},
277+
}
278+
279+
for _, tt := range tests {
280+
t.Run(tt.name, func(t *testing.T) {
281+
t.Parallel()
282+
283+
telemetry := &recordingTelemetryClient{err: tt.reportErr}
284+
reportLocalClientRouteSelected(
285+
t.Context(),
286+
telemetry,
287+
tt.activityProfile,
288+
tt.suppressClient,
289+
)
290+
291+
if telemetry.request == nil {
292+
t.Fatal("expected telemetry request")
293+
}
294+
if telemetry.request.EventName != localClientRouteSelectedEvent {
295+
t.Fatalf(
296+
"event name = %q, want %q",
297+
telemetry.request.EventName,
298+
localClientRouteSelectedEvent,
299+
)
300+
}
301+
wantAttributes := map[string]string{localClientRouteAttribute: tt.wantRoute}
302+
if !maps.Equal(telemetry.request.Attributes, wantAttributes) {
303+
t.Fatalf("attributes = %v, want %v", telemetry.request.Attributes, wantAttributes)
304+
}
305+
})
306+
}
307+
}
308+
241309
func TestLaunchInspectorUsesWorkflowCommand(t *testing.T) {
242310
t.Parallel()
243311

@@ -743,6 +811,11 @@ type recordingWorkflowClient struct {
743811
called chan struct{}
744812
}
745813

814+
type recordingTelemetryClient struct {
815+
request *azdext.ReportUsageRequest
816+
err error
817+
}
818+
746819
type lockedBuffer struct {
747820
mu sync.Mutex
748821
bytes.Buffer
@@ -772,6 +845,18 @@ func (c *recordingWorkflowClient) Run(
772845
return &azdext.EmptyResponse{}, c.err
773846
}
774847

848+
func (c *recordingTelemetryClient) ReportUsage(
849+
_ context.Context,
850+
request *azdext.ReportUsageRequest,
851+
_ ...grpc.CallOption,
852+
) (*azdext.ReportUsageResponse, error) {
853+
c.request = request
854+
if c.err != nil {
855+
return nil, c.err
856+
}
857+
return &azdext.ReportUsageResponse{Accepted: true}, nil
858+
}
859+
775860
// createVenv sets up a minimal .venv directory structure for testing.
776861
// Returns the path to the .venv directory.
777862
func createVenv(t *testing.T, projectDir string) string {

docs/reference/telemetry-data.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,7 @@ Emitted at provision start by the `microsoft.foundry` provisioning provider (the
460460
| `extension.version` | string | Extension version |
461461
| `extension.event` | string | Extension-chosen event name on an `ext.usage` span |
462462
| `ext.<key>` | string | One extension-supplied attribute on an `ext.usage` span. The key after the `ext.` prefix and the value are chosen by the extension |
463+
| `ext.route` | string | Local-client route selected by `azure.ai.agents`: `inspector`, `playground`, or `suppressed` (`local_client.route.selected`) |
463464
| `extension.installed` | string[] | List of installed extensions (`id@version`) |
464465
| `extension.installed.source.category` | string[] | Installed extension source categories (`id@category`) |
465466
| `extension.version.from` | string | Version before an update or promotion (`ext.update`, `ext.promote`) |
@@ -489,6 +490,12 @@ source succeeds but records nothing, as does any report past the limit of 100
489490
spans per `azd` invocation. This is a configuration-based admission check, not
490491
a cryptographic provenance guarantee.
491492

493+
Reviewed first-party extension usage events currently include:
494+
495+
| Extension | `extension.event` | Trigger | Dynamic attributes |
496+
|-----------|-------------------|---------|--------------------|
497+
| `azure.ai.agents` | `local_client.route.selected` | `azd ai agent run` resolves the service and protocol profile; emitted before client availability, agent startup, and client launch | `ext.route`: `inspector`, `playground`, or `suppressed`; suppression takes precedence |
498+
492499
Source-category fields are classified from the configured source type and location, not the user-defined source name.
493500
Raw source names, URLs, paths, and hosts are not emitted in those fields.
494501
</details>

docs/specs/metrics-audit/feature-telemetry-matrix.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,4 +168,5 @@ reserved field contracts.
168168
| **Up-graph performance** | `up` (graph execution) | (none — enriches the `up` command span) | `perf.provision_duration_ms`, `perf.deploy_duration_ms`, `perf.total_duration_ms` | Emitted from `internal/cmd/up_graph.go` after the graph completes; provision/deploy durations set only when those phases run |
169169
| **VS RPC** | `vs-server` long-running session | `vsrpc.*` (event prefix) | Per-RPC attributes documented in `telemetry-schema.md` | Long-running RPC server for VS integration |
170170
| **Extension telemetry service** | Extension calls `ReportUsage` over the extension gRPC API | `ext.usage` | `extension.id`, `extension.version`, `extension.source`, `extension.event`, plus one `ext.<key>` attribute per entry in the caller's attribute map | Telemetry requires no separate capability or declaration. Only extensions whose configured source matches the verified official `azd` registry name, type, and normalized URL are recorded — a call from any other source succeeds but is dropped, as is any call past 100 recorded events per `azd` invocation. Identity fields are derived from host-signed claims and the installed record, never from the request. Every caller key is prefixed with `ext.` so it cannot overwrite a host field, and the host bounds count and length only — it does not enumerate or pattern-check values |
171+
| **Azure AI Agents local-client routing** | `azd ai agent run` resolves the service and protocol profile | `ext.usage` with `extension.event=local_client.route.selected` | `ext.route` (`inspector`, `playground`, or `suppressed`) | Records one mutually exclusive route before client availability, agent startup, and client launch; suppression takes precedence and the event does not indicate launch success |
171172
| **App detection** | `init`, `up` (fresh projects without `azure.yaml`, via `appdetect.Detect`) | `aspire.apphost.unsupported` | `aspire.apphost.language` (fixed enum — `typescript` / `python` / `go` / `java` / `rust`; not hashed) | Emitted from `internal/appdetect/dotnet_apphost.go` when an Aspire polyglot (non-C#) AppHost is detected; azd surfaces an actionable error referencing [#7138](https://github.com/Azure/azure-dev/issues/7138) instead of falling through to a generic source build |

docs/specs/metrics-audit/telemetry-schema.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,12 @@ guarantees about the whole class:
247247
| Trust | `extension.id` and `extension.version` are derived from host-signed claims; `extension.source` and eligibility are checked against the installed record and verified source config, never from the request |
248248
| Review | Extension telemetry is reviewed when the extension is admitted to the official registry, under the same documentation, classification, and privacy rules as core fields. The eligibility rule above is what ties recording to that review |
249249

250+
Reviewed first-party event contracts:
251+
252+
| Extension | `extension.event` | Trigger | Extension attributes |
253+
|-----------|-------------------|---------|----------------------|
254+
| `azure.ai.agents` | `local_client.route.selected` | `azd ai agent run` resolves the service and protocol profile; this precedes client availability, agent startup, and client launch | `ext.route`: fixed enum `inspector`, `playground`, or `suppressed`; suppression takes precedence |
255+
250256
Because `ext.usage` spans share the command's trace, they join the originating
251257
command in Kusto on `operation_Id`. See
252258
[ADR-001](../../architecture/adr-001-extension-telemetry-events.md) for

0 commit comments

Comments
 (0)