S3 access results in improperly terminated TCP sockets.

[tubsandcans]

Describe the bug

I notice that when reading (or writing) objects out of an S3 bucket using the aws-sdk-s3 (1.176.0) gem, a TCP connection is left in the CLOSE_WAIT state. This only goes away when the process running the Ruby program that initiated the connection dies.

This isn't really an issue locally, but in the production server which is a multi-process web-server (Passenger+Nginx), it appears these persistent connections slowly consume system resources to the point of complete exhaustion. I'm genuinely curious if this decision to leave these connections in CLOSE_WAIT was intentional and just a really poor fit for our production environment, or if it was done by mistake. Regardless, it seems at the very least imprudent to keep these connections in CLOSE_WAIT for the lifecycle of the initiating process.

I noticed this after enabling active-storage for our application, and it would slowly consume all RAM until the application crashes. Passenger is unable to re-use these web processes due to them having a TCP socket in CLOSE_WAIT, so it keeps spawning more until all system resources are maxed out.

Ruby version: 3.3.6

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would not expect any evidence of a TCP connection to the S3 service once transmission is completed.

Current Behavior

After reading from an S3 bucket, a TCP socket is left in CLOSE_WAIT state.

Reproduction Steps

1. Run lsof -i -P | grep CLOSE_WAIT and notice there are no TCP connections to an AWS resource in CLOSE_WAIT.
2. Read from an S3 bucket in a Ruby console
3. Run lsof -i -P | grep CLOSE_WAIT and notice a TCP socket for AWS in CLOSE_WAIT.
4. Terminate the Ruby console
5. Run lsof -i -P | grep CLOSE_WAIT and notice this CLOSE_WAIT socket goes away.

Possible Solution

Close the TCP socket upon completion of transmission.

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3 (1.176.0)

Environment details (Version of Ruby, OS environment)

Ruby 3.3.6, macOS 14.6.1

[mullermp]

Thanks for opening an issue. Are you sure this is an issue with the Ruby SDK and not your application/host configuration? If it's not an issue locally when using the SDK, it is likely not the library code. The SDK uses Net::HTTP to make requests and we use a connection pool to re-use connections.

[tubsandcans]

Thanks for the quick reply! When I said it wasn't an issue locally, I just meant that there is very little to no impact locally from having TCP sockets left in CLOSE_WAIT, but on a multi-process production environment, it very much is a problem.

Connection pools only work within-process, for a multi-threaded model. In a multi-process paradigm, connection pools don't help.

[mullermp]

When I attempt your steps locally on my machine, I don't see the same behavior. I see one connection with lsof, then I make a call to s3 get object, and see the same connection again in lsof.

Where in the SDK do you think this is occurring? To be frank, I don't think this is something we can change, it sounds like it's behavior with Net::HTTP or your configuration.

[tubsandcans]

I'm not sure what you're referring to when you say 'your configuration'. Do you mean the S3 configuration? Because that is the only constant between the application in production and my local tests using IRB and my local tests running a brand new Rails8 app with active-storage. There is nothing I'm configuring in my IRB test, just simply loading the aws-sdk-s3 gem and listing the keys in a bucket in our S3.

Keeping TCP connections in this state is not the default behavior of Net::HTTP, so I don't know exactly where in the SDK this would be happening, but I assume the SDK is making these sockets, and would therefore be responsible for closing them properly.

[mullermp]

When I say configuration it could mean anything - I don't know much about your system. It could be your host, how you've configured processes with passenger/nginx, etc.

You can check the connection_pool code here:

aws-sdk-ruby/gems/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb

Line 4 in d31d585

require 'net/http'

. You can try tweaking your idle timeout. Sessions are cleaned regularly based on their idle time and last used.

[tubsandcans]

Right, I'm saying I tested this on multiple systems in multiple ways, so it very likely does not have anything to do with a system configuration.

I'll look into the idle timeout and see if I can influence anything with that, thanks!

[tubsandcans]

Looks like the default value for idle timeout is 5 seconds, but these connections stick around forever as long as the process that initiated them is still running. Something is preventing these from fully closing and it is not due to a configuration setting.

[mullermp]

Are your calls to s3 very spiky? _clean is called when a new session is attempted. You could try calling this method:

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core.rb

Line 148 in d31d585

def empty_connection_pools!

[tubsandcans]

Spiky, as in, mostly low traffic with intermittent periods of very high-volume access? In my simplest case test, I just read the keys from a bucket one time, so no traffic to speak of really.

That link is not pointing to a method, nor does the _clean method appear in the linked file.