Why are all server requests sent to a single process for processing?

[feng19]

Thank you for the author's great efforts.

I am developing an MCP server using this library and have a questions:

At present, through reading the code, I found that all requests sent by the MCP Client will be sent to a single process for processing.

In the design patterns I know, this pattern may have a single-instance bottleneck.

Maybe I have a problem with it, So I'm here to ask why designed like this?

[zoedsoupe]

Thank you for the great question! You're correct that all requests are currently processed through a single Hermes.Server.Base server GenServer per server module. Let me explain the architecture in detail and discuss potential improvements.

Current Architecture Explained

Request Flow Diagram

graph TD
    subgraph "Client Side"
        C1[Client 1]
        C2[Client 2]
        C3[Client N]
    end

    subgraph "Transport Layer"
        T1[HTTP Connection 1]
        T2[HTTP Connection 2]
        T3[HTTP Connection N]
        TS[Transport GenServer<br/>StreamableHTTP/SSE/STDIO]
    end

    subgraph "Server Core"
        BS[Base Server GenServer<br/>Single Process]
        SR[Session Registry]
        S1[Session 1<br/>Agent]
        S2[Session 2<br/>Agent]
        S3[Session N<br/>Agent]
    end

    subgraph "Business Logic"
        H[Handlers Module]
        SM[Server Module<br/>Your Implementation]
    end

    C1 --> T1
    C2 --> T2
    C3 --> T3

    T1 --> TS
    T2 --> TS
    T3 --> TS

    TS -->|GenServer.call| BS

    BS --> SR
    SR --> S1
    SR --> S2
    SR --> S3

    BS --> H
    H --> SM

Loading

Detailed Request Processing

sequenceDiagram
    participant Client
    participant HTTPConn as HTTP Connection
    participant Transport as Transport GenServer
    participant Base as Base Server GenServer
    participant Session as Session Agent
    participant Handler as Handlers Module
    participant Server as Your Server Module

    Client->>HTTPConn: MCP Request
    HTTPConn->>Transport: Plug.call/2
    Transport->>Transport: Decode & Validate
    Transport->>Base: GenServer.call({:request, ...})
    Base->>Session: Validate/Update Session
    Base->>Handler: Route by method
    Handler->>Server: handle_request/2
    Server-->>Handler: Response
    Handler-->>Base: Response
    Base-->>Transport: Response
    Transport-->>HTTPConn: SSE/HTTP Response
    HTTPConn-->>Client: MCP Response

Loading

Why This Architecture?

1. Transport Abstraction: The Transport GenServer provides a unified interface for STDIO, HTTP, and SSE transports. Each
   transport type has different connection models:

• STDIO: Single persistent connection
• StreamableHTTP: Multiple HTTP connections mapped to sessions
• SSE: Server-Sent Events with persistent connections

2. Single Base Server: Acts as the coordination point for:

• Session management and lifecycle
• State consistency (capabilities, initialization)
• Request routing and validation
• Protocol compliance

3. Session Isolation: Each client session is a separate Agent process, providing:

• Isolated state per client
• Automatic cleanup on disconnect
• Concurrent session handling

Addressing the Bottleneck Concerns

As noted in the https://elixirforum.com/t/hermes-mcp-model-context-protocol-implementation-for-elixir/71410/10, there are two
potential bottlenecks:

1. Transport GenServer: For HTTP transports, all connections route through one transport process
2. Base Server GenServer: All requests serialize through a single process

Current Performance Characteristics

In production with hundreds of thousands of users across a multi node Elixir cluster using Horde in the @cloudwalk experience:

• Typical MCP operations (tool calls, resource access) have low request rates
• The architecture handles this scale well because:
  • Most time is spent in I/O operations (LLM API calls, database queries)
  • Request processing is fast (microseconds for routing)
  • Sessions are distributed across cluster nodes
• We also use hermes for MCP client connection for external servers, which kinda follows the same architecture and have even more active users and no bottle neck

Bottleneck Analysis

The serialization through GenServers is rarely the actual bottleneck because:

• Routing overhead: ~0.1ms
• Typical handler execution: 10-5000ms (dominated by I/O)
• GenServer capacity: ~10,000 messages/second

Proposed Improvements

Option 1: Request Worker Pool

Minimal change that preserves the current architecture:

defmodule Hermes.Server.Base do
  def handle_call({:request, decoded, session_id, context}, from, state) do
    # Validate state synchronously
    case validate_session_and_state(session_id, state) do
      {:ok, _session} ->
        # Delegate handler execution to worker
        Task.Supervisor.async_nolink(Hermes.Server.TaskSupervisor, fn ->
          frame = build_frame(decoded, context, state)
          result = Handlers.handle(decoded, state.module, frame)
          GenServer.reply(from, encode_result(result))
        end)
        {:noreply, state}
      error ->
        {:reply, error, state}
    end
end

Benefits:

• Parallel handler execution
• No API changes
• Preserves request ordering per session
• Better CPU utilization

Option 2: Per-Session Request Processing

Move request handling to session processes:

defmodule Hermes.Server.Session do
  def handle_call({:request, decoded, context}, _from, state) do
    frame = build_frame(decoded, context, state)
    result = Handlers.handle(decoded, state.server_module, frame)
    {:reply, result, update_last_active(state)}
  end
end

# Base server just routes
def handle_call({:request, decoded, session_id, context}, _from, state) do
  session_pid = get_session(session_id, state)
  # Async forward to session
  Task.async(fn ->
    GenServer.call(session_pid, {:request, decoded, context})
  end)
  {:noreply, state}
end

Benefits:

• True parallel processing per session
• Natural sharding by session
• Scales with active users

Option 3: Transport Connection Pooling

For high-connection scenarios:

defmodule Hermes.Server.Transport.StreamableHTTP do
  def init(opts) do
    # Create pool of transport workers
    pool_size = opts[:pool_size] || System.schedulers_online()
    workers = create_worker_pool(pool_size, opts)
    {:ok, %{workers: workers, next: 0}}
  end

  def handle_call(request, from, state) do
    # Round-robin to workers
    worker = Enum.at(state.workers, state.next)
    GenServer.cast(worker, {:handle, request, from})
    {:noreply, %{state | next: rem(state.next + 1, length(state.workers))}}
  end
end

Conclusion

1. For most use cases: The current architecture is sufficient. MCP servers typically handle low-frequency, high-value operations
   (tool calls, resource access) rather than high-throughput data processing.
2. For CPU-intensive workloads: Implement Option 1 (worker pool) to parallelize handler execution while maintaining architecture
   simplicity.
3. For high per-session request rates: Consider Option 2 (session-based processing) to eliminate the Base server bottleneck
   entirely.
4. For extreme connection counts: Add transport pooling (Option 3) if you have thousands of concurrent HTTP connections, but i don't think that's so valuable considering MCP spec.

Would you be interested in contributing a PR for any of these approaches? I'd suggest starting with Option 1 as it:

• Provides immediate benefits for CPU-bound workloads
• Requires minimal changes
• Maintains backward compatibility
• Preserves the current handler strategy

What's your specific use case? Understanding your request patterns and scale requirements would help determine the best
optimization strategy.

[feng19]

Thank you very much for your reply. I have two additional points:

1. When the service encounters a DDoS attack.
2. When the execution time of the called tool is too long.

These two situations are common.

Once they occur, they will block both the Transport and Server.Base processes simultaneously.

[zoedsoupe]

transport process now process requests async on #185 and let the base server process answer directly to the SSE handler, avoid 1 of the bottlenecks. does you have suggestion of an improved or simplified architecture that handle horizontal/vertical scalation, distribution/clusterization and parallelism?

[feng19]

Thank you very much for your valuable time in making the modifications.

I've reviewed the latest code. Although a task has been added for asynchronous processing in the transport process, but the transport & server.base process can still be blocked.
Because the ratio of http process : transport process : server process is N:1:1.

I prefer the following changes:

• Put the execution processing of requests into the session process, such as tool.execute.
• Consider transport as the logic layer instead of using process.
• Consider Server as the logic layer instead of using process.

The reason for these changes is that the ratio of http process : session process is 1:1,
Then we can eliminate the singleton pattern.

If you think there are any problems with my suggestions, please point them out.
Additionally, I can also contribute code to this development.

Thanks again. Hope this can be helpful to this project.