OTEL: Instrument ctx.sample() / sampling loop with spans and metrics

[strawgate]

Summary

The sampling loop (ctx.sample(), ctx.sample_step()) is where most time and cost is spent in MCP servers that use LLM calls, but it has zero OTEL instrumentation. This issue tracks adding spans, metrics, and opt-in content capture to the sampling internals.

Current state

src/fastmcp/server/sampling/run.py has no telemetry imports, no spans, no metrics. The entire sampling loop — LLM calls, tool execution, validation retries, structured output parsing — is invisible in traces.

Proposed spans

sampling/createMessage — wraps sample_impl()

The full sampling loop from start to final response. This is a well-known MCP method name.

Attributes:

• mcp.method.name = "sampling/createMessage"
• gen_ai.request.model (from model_preferences if available)
• gen_ai.request.temperature
• gen_ai.request.max_tokens
• fastmcp.sampling.tool_count — number of tools provided
• fastmcp.sampling.result_type — structured output type name or "str"
• fastmcp.sampling.iterations — total iterations before completion (set on span end)
• fastmcp.sampling.handler_mode — "fallback" or "client"

sampling/createMessage step — wraps each sample_step_impl() call

Each LLM round-trip within the loop.

Attributes:

• fastmcp.sampling.iteration — current iteration number
• fastmcp.sampling.stop_reason — "toolUse", "endTurn", "maxTokens"

sampling.execute_tool {name} — wraps each tool execution within the loop

Follows the google-genai pattern of child spans for tool execution during function-calling.

Attributes:

• gen_ai.tool.name — the tool being executed
• error.type — if the tool errors

Events on the sampling/createMessage span

• sampling.validation_failure — when structured output validation fails (with retry count)
• sampling.text_response_retry — when LLM returns text instead of calling final_response

Content capture (opt-in)

Controlled by the same env var used by google-genai instrumentation: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT (default false).

When enabled, record as span events on the sampling/createMessage span:

• gen_ai.system.message — system prompt content
• gen_ai.user.message — user message content
• gen_ai.choice — LLM response content

This follows the GenAI semantic conventions for content capture.

Metrics

• mcp.server.operation.duration histogram for sampling/createMessage (per the MCP semconv)
• Consider gen_ai.client.token.usage histogram if token counts are available from the handler response

Additional improvements (from instrumentation best practices research)

These apply to all existing spans, not just sampling:

• Add span.is_recording() guards before building attribute dicts (every mature OTEL instrumentation does this — httpx, google-genai, Flask)
• Use type(e).__qualname__ instead of type(e).__name__ for error.type (matches httpx and google-genai convention)

References

• MCP semconv: https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/
• GenAI semconv (content capture): https://opentelemetry.io/docs/specs/semconv/gen-ai/
• google-genai instrumentation patterns: https://github.com/open-telemetry/opentelemetry-python-contrib
• Full audit: https://github.com/strawgate/fastmcp-pr-review/blob/main/otel-audit.md

[marvin-context-protocol]

Claude finished @strawgate's task in 2m 39s —— View job

---

TL;DR

This is a well-scoped instrumentation proposal that fits cleanly on top of the existing fastmcp/server/telemetry.py helper pattern. The only real design decision is metrics: FastMCP does not currently emit any OTEL metrics — no get_meter() anywhere in src/ — so adding mcp.server.operation.duration here would be the first metrics API in the codebase. I'd recommend landing spans + events + content capture now in a focused PR, and deferring the metrics histogram to a follow-up PR that either (a) waits for the broader mcp.server.* metrics work tracked in #3733, or (b) introduces a fastmcp/metrics.py module alongside fastmcp/telemetry.py so it can be reused by #3887's other methods. The "additional improvements" at the bottom (is_recording() guards, __qualname__) should land in their own PR touching all spans, not just sampling — this is exactly what #3886 is for.

Recommendation

Ship this in one focused PR scoped to sampling/createMessage tracing only:

1. Add a sampling_span() helper in src/fastmcp/server/telemetry.py (mirrors server_span/delegate_span), then wrap sample_impl() in sampling/createMessage and sample_step_impl() in sampling/createMessage step, and _execute_single_tool() in sampling.execute_tool {name}.
2. Gate content capture on OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT via a small helper (read once per call, not per event).
3. Defer the metrics histogram — it belongs in a separate PR bundled with Support mcp.server.* OTEL metrics #3733 so FastMCP introduces get_meter() once, consistently.
4. Defer the two cross-cutting improvements (is_recording() guards, __qualname__) to a separate PR that touches every span — those belong in OTEL: Fix attribute compliance with MCP semantic conventions #3886, not here.

Findings

• Sampling file has zero telemetry imports: verified by reading src/fastmcp/server/sampling/run.py — the file imports logging but nothing from opentelemetry or fastmcp.telemetry. sample_impl is the top-level loop (lines 581-743), sample_step_impl is the single-turn helper (lines 464-578), and _execute_single_tool is the inner tool dispatch (lines 282-326).
• Existing span helper pattern: server_span() in src/fastmcp/server/telemetry.py:55-96 is a @contextmanager that creates a SERVER-kind span, sets standard MCP attributes (mcp.method.name, rpc.system, fastmcp.*), auto-records exceptions, and sets error status. delegate_span() (lines 99-123) is the INTERNAL-kind analogue. sampling/createMessage spans fit the INTERNAL kind (they're nested inside an outer tools/call SERVER span during the typical flow).
• No metrics infrastructure exists yet: grep for opentelemetry.metrics|get_meter|Histogram|Counter across src/ returns only one match in an unrelated OpenAPI provider. Neither fastmcp/telemetry.py nor fastmcp/server/telemetry.py define a meter. Adding mcp.server.operation.duration here means introducing the metrics API to the codebase.
• Related tracking issues OTEL: Fix attribute compliance with MCP semantic conventions #3886 (attribute compliance) and OTEL: Instrument all MCP methods, not just tools/call, resources/read, prompts/get #3887 (instrument all 29 methods) explicitly list sampling/createMessage as high-priority. OTEL: Instrument all MCP methods, not just tools/call, resources/read, prompts/get #3887's draft PR [DNM] OTEL: Instrument all MCP methods #3890 touches the same files but does not appear to cover sampling. The "additional improvements" at the bottom of this issue (is_recording(), __qualname__) are cross-cutting and duplicate OTEL: Fix attribute compliance with MCP semantic conventions #3886's scope.
• Content capture env var: The spec-standard var is OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. FastMCP does not yet read this anywhere — it will need a new utility (a plain os.environ.get(...) is fine; this is how google-genai does it).
• Fallback vs client handler: determine_handler_mode() at run.py:139-195 returns a bool indicating whether the fallback handler is used. This is exactly the fastmcp.sampling.handler_mode attribute proposed, but the call happens inside sample_step_impl, not sample_impl. To set the attribute on the outer span the value needs to be surfaced (either re-derive it in sample_impl, or add it as a span attribute from inside sample_step_impl).
• Token usage: CreateMessageResult/CreateMessageResultWithTools from mcp.types does not expose a standard token usage field — handler responses may or may not carry it in model/stopReason or in custom fields. gen_ai.client.token.usage should only be emitted when the result actually exposes counts. This is worth confirming before spec'ing it.

Detailed Action Plan

Files to change

• src/fastmcp/server/telemetry.py — add sampling_span(), sampling_step_span(), sampling_tool_span() helpers (or a single parameterized one). Pattern-match delegate_span (INTERNAL kind).
• src/fastmcp/server/sampling/run.py — wrap sample_impl, sample_step_impl, _execute_single_tool; emit events for validation failures and text-response retries; emit content-capture events when the env var is set.
• tests/server/telemetry/ — add test_sampling_tracing.py mirroring the structure of test_server_tracing.py (use the existing trace_exporter fixture from tests/conftest.py:135).
• docs/patterns/tracing.mdx (or equivalent) — document the new spans and the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env var. Content capture being opt-in is important to call out explicitly.

Implementation sketch

# In src/fastmcp/server/telemetry.py
@contextmanager
def sampling_span(
    handler_mode: str,
    tool_count: int,
    result_type: str,
    model: str | None = None,
    temperature: float | None = None,
    max_tokens: int | None = None,
) -> Generator[Span, None, None]:
    tracer = get_tracer()
    with tracer.start_as_current_span(
        "sampling/createMessage", kind=SpanKind.INTERNAL
    ) as span:
        attrs: dict[str, Any] = {
            "mcp.method.name": "sampling/createMessage",
            "fastmcp.sampling.tool_count": tool_count,
            "fastmcp.sampling.result_type": result_type,
            "fastmcp.sampling.handler_mode": handler_mode,
        }
        if model is not None:
            attrs["gen_ai.request.model"] = model
        if temperature is not None:
            attrs["gen_ai.request.temperature"] = temperature
        if max_tokens is not None:
            attrs["gen_ai.request.max_tokens"] = max_tokens
        span.set_attributes(attrs)
        try:
            yield span
        except Exception as e:
            span.record_exception(e)
            span.set_status(Status(StatusCode.ERROR))
            raise

Then in run.py:

# In sample_impl, just before the loop (after prepare_tools and handler_mode determination)
handler_mode = "fallback" if determine_handler_mode(context, bool(sampling_tools)) else "client"
model = _extract_model_hint(model_preferences)  # best-effort from ModelPreferences

with sampling_span(
    handler_mode=handler_mode,
    tool_count=len(sampling_tools or []),
    result_type=result_type.__name__ if result_type and result_type is not str else "str",
    model=model,
    temperature=temperature,
    max_tokens=max_tokens,
) as span:
    iterations = 0
    for _iteration in range(max_iterations):
        iterations += 1
        with sampling_step_span(iteration=iterations) as step_span:
            step = await sample_step_impl(...)
            step_span.set_attribute(
                "fastmcp.sampling.stop_reason", step.response.stopReason or ""
            )
        # existing validation/retry logic, but emit events:
        # span.add_event("sampling.validation_failure", {"retry_count": consecutive_validation_failures})
        # span.add_event("sampling.text_response_retry", {"retry_count": text_response_retries})
    span.set_attribute("fastmcp.sampling.iterations", iterations)

For per-tool spans, wrap _execute_single_tool in execute_tools() (run.py:282) with sampling_tool_span(name=tool_use.name) that sets gen_ai.tool.name and, on exception, error.type = type(e).__qualname__.

For content capture, read the env var once at the start of sample_impl:

_capture = os.environ.get(
    "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", ""
).lower() in ("true", "1", "yes")
if _capture and span.is_recording():
    if system_prompt:
        span.add_event("gen_ai.system.message", {"content": system_prompt})
    # ...user messages, choice

Testing checklist

• sample_impl creates exactly one sampling/createMessage span with the spec'd attributes
• Nested sampling/createMessage step spans for each iteration, with fastmcp.sampling.iteration and fastmcp.sampling.stop_reason
• Per-tool sampling.execute_tool {name} spans, with gen_ai.tool.name; failures set error.type and status=ERROR
• fastmcp.sampling.iterations set correctly on span end (including exit-on-max-iterations path at run.py:743)
• Validation failure / text-response-retry events recorded
• Content capture OFF by default — no gen_ai.*.message events emitted
• Content capture ON via env var — system/user/choice events present
• Distributed tracing: verify sampling/createMessage correctly parents under an outer tools/call when ctx.sample() is called from inside a tool

What to leave out of this PR

• Metrics histogram: defer to a follow-up that introduces fastmcp.metrics module; coordinate with Support mcp.server.* OTEL metrics #3733.
• is_recording() guards / __qualname__ fix: cross-cutting across every existing span (server_span, delegate_span, client_span, and the new sampling spans). Land as a single sweep PR under OTEL: Fix attribute compliance with MCP semantic conventions #3886.
• Standardizing error.type attribute: covered by OTEL: Fix attribute compliance with MCP semantic conventions #3886.

Related Items

Ref	Relevance
#3887	Parent tracking issue: "Instrument all MCP methods" — this issue is the sampling slice
#3890	[DNM] draft PR for #3887 covering tools/list, resources/list, prompts/list, etc. — does not cover sampling, so this can land independently
#3886	Attribute compliance work — covers the is_recording() / __qualname__ / error.type improvements mentioned at the bottom of this issue
#3889	[DNM] draft PR for #3886
#3733	Broader OTEL metrics semconv work — the right home for the mcp.server.operation.duration histogram
#2869	Original OTel tracing PR that established the server_span/client_span helper pattern
#3221	traceparent/tracestate MCP semconv alignment — shows how distributed context propagation works in _get_parent_trace_context()
#3451	Interop with OTel instrumentation in the low-level MCP SDK — relevant if upstream starts emitting sampling spans

Related Files

File	Relevance
src/fastmcp/server/sampling/run.py	The file to instrument. sample_impl (L581-743), sample_step_impl (L464-578), execute_tools + _execute_single_tool (L251-355)
src/fastmcp/server/telemetry.py	Helper pattern to mirror — server_span (L55), delegate_span (L99)
src/fastmcp/telemetry.py	get_tracer() entry point, record_span_error() helper
tests/server/telemetry/test_server_tracing.py	Test pattern to follow — uses trace_exporter fixture
tests/conftest.py	trace_exporter fixture (L135)
src/fastmcp/server/context.py	Context.sample / Context.sample_step — public entry points that call sample_impl/sample_step_impl

Related Webpages

Name	URL
MCP semantic conventions	https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/
GenAI semantic conventions (content capture)	https://opentelemetry.io/docs/specs/semconv/gen-ai/
opentelemetry-python-contrib (google-genai instrumentation reference)	https://github.com/open-telemetry/opentelemetry-python-contrib
strawgate's OTEL audit	https://github.com/strawgate/fastmcp-pr-review/blob/main/otel-audit.md

• branch: `claude/issue-3891-20260413-0622`

---

Marvin Context Protocol | Type /marvin to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[marvin-context-protocol]

Thanks for the report. This issue goes beyond what our contributor guidelines ask for — we just need a short problem description and an MRE. Please see our contributing guidelines and condense this issue. We'll triage it once it's trimmed down.