Resilient TCP transport #1450
hairyhum
started this conversation in
General & Learning
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Context:
In the current implementation a TCP transport upon receiving a message creates a TCP client connection and a session worker attached to it.
Messages received by this connection would trace a dynamic local address for the connection worker in the return route.
This means that each message to the TCP address creates a new client and to communicate using an existing client the workers have to use the dynamic TCP client worker address.
This may cause TCP connection leaking and servers running out of resources like file descriptors. Also this means that if the TCP connection fails, the worker address can no longer be used to send the messages.
Rust library solves that by having a single client connection per TCP IP:PORT tuple and routing the TCP transport address messages to this connection. This aproach can be used in Elixir, but a single connection might cause some bottlenecks in high throughput scenarios.
Also the TCP handler side of the connection doesn't have the static identity like IP:PORT since the client port is dynamic if connection resets. This means that the handler side of the TCP transport may still break if connection worker is crashing.
In order to address the TCP connection leaking we can start with associating client connection with IP:PORT and maybe adding support for a pool of clients there for scaling.
In order to address resiliency, we need some re-connection mechanism, probably with some state containing client identity.
This mechanism might be a part of the TCP transport, or something more generic which we can use for other session workers.
Beta Was this translation helpful? Give feedback.
All reactions