ChannelClosedException when server close connection right after releasing the last http2 stream

[huajiang-tubi]

Motivation and Context

sequenceDiagram
    participant aws AS AWS Service
    participant channel AS Channel Event Loop
    participant pool AS HttpOrHttp2ChannelPool.eventLoop

    aws->>+channel: 1. LastHttpContent
    channel->>+channel: 2. ResponseHandler.finalizeResponse
    channel->>+aws: 3. Http2ResetFrame
    channel->>+channel: 4. HttpOrHttp2ChannelPool.release
    channel->>pool: 5. doInEventLoop(() -> release0(channel, promise))
    deactivate channel
    deactivate channel
    deactivate channel

    aws->>+channel: 6. Close The Connection
    channel->>+channel: 7. Http2ConnectionHandler.channelInactive
    channel->>+channel: 8. MultiplexedChannelRecord.closeAndExecuteOnChildChannels
    deactivate channel
    deactivate channel
    deactivate channel


    pool->>+pool: 9. HttpOrHttp2ChannelPool.release0(channel, promise)
    pool->>+channel: 10. doInEventLoop(() -> MultiplexedChannelRecord.closeAndReleaseChild)
    channel->>+channel: 11. childChannels.remove
    deactivate channel
    channel-->>-pool:
    deactivate pool

Loading

1. The AWS service returns the final part of the response.

2. ResponseHandler.finalizeResponse is invoked to complete processing of the response.

3. The client sends an RST_STREAM frame to acknowledge the completion.

4. It then calls release on the channel pool. Since the channel pool consists of multiple layers, this invocation eventually reaches HttpOrHttp2ChannelPool.release.

5. HttpOrHttp2ChannelPool.release needs to access the protocolImpl field, which is only safely accessed from the pool’s event loop. To ensure thread safety, it submits a task to perform the release within that event loop.

6. Meanwhile, the server receives the reset frame and decides to immediately close the connection (No idea why. It's a question for the service developer).

7. This triggers channelInactive on the Http2ConnectionHandler.

8. As a result, MultiplexedChannelRecord.closeAndExecuteOnChildChannels is called. It detects that there are still unreleased child channels because the task submitted in step 5 has not yet been executed. This leads to a ClosedChannelException being thrown and logged as an error.

9. The release task is now running on the pool's event loop, but a little bit too late.

It may also be the root cause of #2914

Modifications

By declaring protocolImpl as volatile, we ensure visibility across threads once it is assigned within the pool's event loop. This guarantees that it can be safely accessed without thread-safety concerns. The underlying BetterFixedChannelPool is thread-safe, as its mutable state is managed exclusively within its dedicated event loop. For release operations, state updates are performed via a future listener after the underlying pool completes the release, effectively avoiding the concurrency issues observed with HttpOrHttp2ChannelPool.

Tests

Reproducing the issue consistently in a test environment is challenging, as it relies on the precise timing and order of event handler invocations.

We’ve been running the fix without the volatile keyword on protocolImpl in one of our production services for some time. It has been working well, as evidenced by a noticeable drop in error logs:

The deployment with the volatile keyword added is now live. I’ll share an update on the results shortly.