Is high memory usage expected with large number of websocket connections?

[mplauman]

Hi there!

I'm currently working on a project that involves redirecting websocket traffic from a primary back-end webserver onto a separate service. This new service doesn't do much with them other than send the occasional message.

This setup requires a surprising amount of memory to be allocated to linkerd-proxy. It's currently set to 512MiB and was getting OOM-killed once we hit ~3K established connections/pod.

At that time we were receiving ~200 requests/second on 15 pods (sockets dropping & reconnecting). That feels like a pretty trivial amount of traffic to me and the service had no problem keeping up.

In contrast, we have another service accepting ~200 normal HTTP requests/second on 5 pods. There, linkerd-proxy only uses about 50MiB of memory. That's more in line with what everybody at my org expects.

I suspect it's the open websocket connections that are consuming all the memory in linkerd-proxy. That makes a degree of sense. Unlike normal HTTP requests, websockets stay open indefinitely so resources aren't recycled as much.

Questions:

• Is high memory usage in this situation expected?
• We have a fairly standard linkerd-proxy setup, are there any configuration settings could apply to handle large numbers of websockets?

We're OK throwing more pods at the issue so we aren't blocked or anything. But everybody is surprised by the memory requirements. I'd love to be able to give them a good answer and look smart. :)

Thanks!

[cpretzer]

@mplauman thanks for sharing the metrics and other info in slack.

This report sounds similar to this issue.

With ~2k TCP connections, we expect that there will be more than normal memory usage, especially with the long-lived web sockets connections.

I spent some time testing this and found that the memory isn't leaking, and doesn't continue to grow with time. It grows with the number of connections. So, I'd suggest increasing the proxy memory (and possibly CPU) limits either through the values.yaml file, or by using the
config.linkerd.io/proxy-memory-limit and config.linkerd.io/proxy-memory-request annotations.

The annotations will let you target specific workloads, whereas changing values.yaml will affect all proxies in the cluster.

Let us know how it goes!

[mplauman]

Thanks @cpretzer! That lines up with our experience as well.

For my project I just disabled proxying the incoming requests entirely. Memory usage in linkerd immediately dropped from ~300MiB to about 8MiB in steady state, spiking to 30MiB or so during rolls when we get lots of reconnections.

It isn't super important traffic that we need to rigorously monitor. We were just mostly surprised: after seeing so many articles talking about how lightweight linkerd is, it was surprising to us that we needed to feed it so much memory. Knowing it scales with the number of active connections (vs request rate) is helpful.

[wmorgan]

I think that's a great point @mplauman. At a minimum I think we should update the production runbook to talk about scaling factors for proxy memory consumption.