Introduce 3rd Party MCP Client/Server testing

[strawgate]

Enhancement

We should introduce a test harness that sets up a FastMCP server with various capabilities and then uses various language SDKs to connect and perform various actions.

[marvin-context-protocol]

Claude finished @strawgate's task in 2m 46s —— View job

---

Triage: Introduce 3rd Party MCP testing

• Gather repository context and understand existing test infrastructure
• Research MCP language SDKs and interoperability testing patterns
• Analyze FastMCP server capabilities to test against
• Formulate recommendation and detailed action plan

Recommendation

Build on the existing tests/conformance/ infrastructure to add cross-SDK interoperability tests. The conformance server (tests/conformance/server.py) already registers tools, resources, and prompts for the official @modelcontextprotocol/conformance suite — extend it to also be exercised by TypeScript SDK and Go SDK clients that connect over Streamable HTTP and verify real protocol-level interop.

Start with TypeScript (Tier 1, already required for conformance tests via Node.js) and Go (Tier 1), then expand to C# and others later. This directly validates that a FastMCP server is compatible with the actual SDK clients that real users will connect with — something the conformance suite alone doesn't fully cover since it tests protocol-level compliance, not SDK-specific client behavior.

Findings

Existing test infrastructure:

• tests/conformance/server.py — A FastMCP server with 13+ tools (text, image, audio, embedded resources, error handling, logging, progress, sampling, elicitation), 4 resources (static text, binary, template, watched), and 4 prompts (simple, with args, embedded resource, image)
• tests/conformance/test_conformance.py — Runs @modelcontextprotocol/conformance@latest against the server via npx
• tests/conformance/expected-failures.yml — Tracks known gaps (completion, SSE polling, subscriptions, DNS rebinding)
• tests/integration_tests/test_github_mcp_remote.py — Tests the FastMCP client against GitHub's remote MCP server
• CI already sets up Node.js 22 for conformance tests (.github/workflows/run-tests.yml:89-92)

Available MCP SDKs (official, from modelcontextprotocol.io/docs/sdk):

SDK	Tier	Client Support	CI Feasibility
TypeScript	Tier 1	StreamableHTTPClientTransport	Already available (Node.js in CI)
Go	Tier 1	mcp.NewStreamableHTTPTransport	Single binary, easy to add
C#	Tier 1	Full client	Requires .NET SDK setup
Python	Tier 1	FastMCP already tests this	N/A (self-test)
Java	Tier 2	Full client	Requires JDK setup
Rust	Tier 2	Full client	Requires Rust toolchain

Key gap the issue addresses: The conformance suite validates protocol-level compliance, but doesn't test that real SDK clients (the ones actual users use) can successfully interact with a FastMCP server end-to-end. For example, a TypeScript Client calling listTools() → callTool() exercises the full SDK stack including serialization, transport negotiation, and response parsing.

Detailed Action Plan

Phase 1: TypeScript SDK client tests (lowest friction)

1. Create tests/interop/ directory with the new test marker interop
2. Create tests/interop/clients/ts/ directory with a small TypeScript client script:
   • Script takes a server URL as an argument
   • Uses @modelcontextprotocol/sdk (the official TypeScript SDK)
   • Performs: initialize → listTools → callTool("test_simple_text") → listResources → readResource("test://static-text") → listPrompts → getPrompt("test_simple_prompt") → callTool("test_error_handling") (expects error)
   • Outputs structured JSON results to stdout (pass/fail per operation)
   • Include a package.json with @modelcontextprotocol/sdk dependency
3. Create tests/interop/test_typescript_client.py:
   • Reuse the conformance_server fixture pattern (start FastMCP server in background thread)
   • Run the TypeScript client via npx tsx or node against the server URL
   • Parse JSON output and assert all operations passed
   • Mark with @pytest.mark.interop
4. Update pyproject.toml: Add interop marker
5. Update CI (.github/workflows/run-tests.yml): Add run_interop_tests job (reuse Node.js setup)

Phase 2: Go SDK client tests

1. Create tests/interop/clients/go/ with a small Go client:
   • main.go that connects via mcp.NewStreamableHTTPTransport
   • Performs the same operations as TypeScript
   • Pre-compile or use go run in CI
2. Create tests/interop/test_go_client.py:
   • Same fixture pattern, run Go client binary against server
3. Update CI: Add Go setup step

Phase 3: Expand coverage

• Add C# client (requires .NET SDK)
• Test additional scenarios: pagination, structured content, progress notifications
• Test both Streamable HTTP and SSE transports
• Test OAuth/auth flows with authenticated endpoints

Implementation notes

• Reuse tests/conformance/server.py — don't create a separate server. The existing conformance server already has the right mix of capabilities
• Keep SDK client scripts minimal — they're test harnesses, not full applications. ~50-100 lines each
• Output structured JSON from each client script for easy parsing in pytest
• Pin SDK versions in the client scripts' dependency files to avoid flaky tests from upstream changes
• Mark as interop not conformance — these are distinct from the official conformance suite
• Skip gracefully if the required runtime (Node.js, Go) isn't available

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
PrefectHQ/fastmcp	#3641 — MCP conformance: resource subscriptions	Related conformance gap — interop tests should also track which operations are expected to fail per SDK
PrefectHQ/fastmcp	#3451 — Interoperating with OpenTelemetry instrumentation	Related interop concern — telemetry propagation across SDK boundaries

Related Files

File	Relevance
tests/conformance/server.py	The conformance server to reuse as the interop test target — already has tools, resources, and prompts
tests/conformance/test_conformance.py	Reference pattern for starting a FastMCP server and running external test processes against it
tests/conformance/expected-failures.yml	Expected failures to be aware of when testing with other SDKs
.github/workflows/run-tests.yml	CI config — already has Node.js setup for conformance, will need Go and potentially .NET
tests/integration_tests/test_github_mcp_remote.py	Example of testing against a real remote MCP server — inverse of what interop tests do

Related Webpages

Name	URL	Relevance
MCP SDKs documentation	https://modelcontextprotocol.io/docs/sdk	Official list of all MCP SDKs with tiers, used to prioritize which SDKs to test against
MCP Conformance repo	https://github.com/modelcontextprotocol/conformance	Official conformance test suite — the existing tests run this; interop tests are complementary
TypeScript SDK	https://github.com/modelcontextprotocol/typescript-sdk	Primary interop target — StreamableHTTPClientTransport client class
Go SDK	https://github.com/modelcontextprotocol/go-sdk	Second interop target — includes conformance client examples at conformance/everything-client/main.go

---

Marvin Context Protocol | Type /marvin to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.