Skip to content

integration_test_analysis.md

Latest commit

 

History

History
143 lines (110 loc) · 4.47 KB

integration_test_analysis.md

File metadata and controls

143 lines (110 loc) · 4.47 KB
Raw

Integration Test Failure Analysis

Critical Finding: Integration Tests Completely Failed to Catch Production Bug

The Problem

  • Users reported: "Claude.ai shows no tools available"
  • Original integration tests: All passed ✅
  • Reality: Server was completely broken 🚨

Why Integration Tests Failed

1. Mock-Only Testing

# What the tests did (WRONG)
server = AuthenticatedMCPServer(...)  # Created object
tools = server.mcp_server.mcp._tool_manager._tools  # Accessed mock
assert len(tools) == 17  # ✅ Passed but meaningless

# What they should have done (RIGHT)  
response = requests.post("http://server/", json=mcp_request)  # Real HTTP call
assert response.status_code == 200  # Would have failed ❌

2. No HTTP Endpoint Testing

  • Never tested actual /tools/list endpoint that Claude.ai calls
  • Never started a real HTTP server
  • Never tested MCP JSON-RPC protocol

3. No End-to-End Validation

  • Tested object creation, not functionality
  • No validation that tools are actually served over HTTP
  • No simulation of Claude.ai's actual workflow

The Bug: Real vs Expected Behavior

Expected (What Claude.ai needs):

curl -X POST http://server/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

# Expected Response:
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {"name": "claude_code", "description": "...", "inputSchema": {...}},
      {"name": "read", "description": "...", "inputSchema": {...}},
      // ... 15 more tools
    ]
  }
}

Actual (What server returns):

# Response:
{"jsonrpc":"2.0","id":"server-error","error":{"code":-32600,"message":"Bad Request: Missing session ID"}}

Root Cause Analysis

1. Native MCP Server Issues

# Native FastMCP server fails:
POST http://127.0.0.1:8360/ → 405 Method Not Allowed
POST http://127.0.0.1:8360/mcp → 404 Not Found
POST http://127.0.0.1:8360/sse → 404 Not Found

2. AuthenticatedMCPServer Issues

# Authenticated server fails:
POST http://server/path → 400 Bad Request: Missing session ID

3. Protocol Mismatch

  • MCP servers expect specific session management
  • Authentication layer breaks MCP protocol
  • Path routing is incorrect

The Comprehensive End-to-End Test

The new E2E test immediately caught the issue:

def test_real_server_startup_and_tools_endpoint(self):
    # Start REAL HTTP server
    server.run_sse_with_auth(host="127.0.0.1", port=port, path="/test-mcp")
    
    # Make REAL HTTP request (like Claude.ai does)
    response = requests.post(f"http://127.0.0.1:{port}/test-mcp/", ...)
    
    # CRITICAL ASSERTION that would have caught the bug
    assert response.status_code == 200  # ❌ FAILED: Got 400 "Missing session ID"
    
    tools_data = response.json()
    assert len(tools_data["result"]["tools"]) > 0  # ❌ Would have failed

Result: Test immediately failed with the exact same error Claude.ai encountered.

Lessons Learned

Integration Test Anti-Patterns (What NOT to do):

❌ Test only object creation
❌ Test only mock interactions
❌ Test only internal methods
❌ Skip HTTP protocol testing
❌ Skip end-user workflows

Integration Test Best Practices (What TO do):

✅ Start real HTTP servers
✅ Make real HTTP requests
✅ Test exact protocol that clients use
✅ Simulate end-user workflows
✅ Test error conditions
✅ Validate actual responses

The Fix Strategy

  1. Immediate: Fix MCP server protocol handling
  2. Short-term: Add E2E tests to CI/CD pipeline
  3. Long-term: Establish "production parity" testing

New Test Requirements

Every integration test must:

  1. Start real HTTP server
  2. Make real HTTP requests
  3. Test actual MCP protocol
  4. Validate tools/list endpoint
  5. Simulate Claude.ai workflow

Conclusion

The integration test failure is a textbook example of why mock-heavy testing is insufficient for integration testing. The tests gave false confidence while the system was completely broken in production.

The comprehensive end-to-end test would have caught this immediately and prevented the user-facing issue.

This demonstrates the critical importance of testing the actual interfaces and protocols that external systems rely on, not just internal object interactions.