Stream sourcing throughput capped at ~20K msg/s by a single goroutine

[kaushalaroraharman]

Summary

JetStream's stream-sourcing feature (Sources config on a stream, used directly and via Orbit's pcgroups elastic Partitioned Consumer Group library) funnels all inbound sourced messages for a destination stream through a single goroutine (processAllSourceMsgs in server/stream.go). Across multiple independent load tests on a 3-node R=3 cluster, we consistently measured a hard sourcing throughput ceiling of ~18,300–21,500 msg/s, regardless of:

• Doubling the underlying hardware (8 vCPU → 16 vCPU per node)
• Doubling the number of downstream consumers (50 → 100 elastic PCG members)
• Increasing the sourced (destination) stream's buffer size (2 GiB → 26 GiB)
• Increasing the source stream's retention window (max-age 30m → 2h, max-bytes 20 GiB → 35 GiB)

Meanwhile the cluster nodes themselves were confirmed to have substantial CPU headroom (20-50% utilization observed during these tests, well below saturation), ruling out a hardware/resource constraint. The bottleneck is architectural: the sourcing engine's own code acknowledges this with an explicit TODO.

This single-goroutine ceiling, combined with discard=old retention semantics on the source stream, causes silent, permanent, unrecoverable message loss at any publish rate that exceeds ~18-21K msg/s sustained — with no error, warning, or metric surfaced anywhere in the API to indicate that messages were dropped before ever being read by the sourcing consumer.

My bigger concern is my end-to-end throughput is now restrained at the throughput I can get out of the sourcing single goroutine, which per my performance tests during POC was about 20K msgs/sec. My publish throughput is 70K msgs/sec, which is the expected throughput on production for my product, however, due to slow sourcing when using Orbit library, my subscriber throughput is way less than the expected 70K.

Please let me know if this is a known issue and if there is any plan on improving the concurrency or performance of sourcing mechanism. This is a major blocker for our use case, and could lead us to start looking for some other solution apart from Orbit. We are currently in POC phase for our product.

Details can be found below.

Environment

• NATS Server: v2.12.6
• nats CLI: v0.3.2
• Cluster: 3-node R=3 cluster (bench-cluster), tested on both m8g.2xlarge (8 vCPU/32GiB) and c8g.4xlarge (16 vCPU) Graviton4 instances
• Client library used for sourcing consumption: github.com/synadia-io/orbit.go/pcgroups (Elastic Partitioned Consumer Group), v0.2.2 — though the bottleneck is entirely inside nats-server itself; pcgroups.CreateElastic() is a thin wrapper that just calls js.CreateStream() with a native Sources block. No Orbit-specific logic is involved in the sourcing path.

Probable Root Cause (confirmed from nats-server source)

server/stream.go, lines 4671-4675:

// This will process all inbound source msgs.
// We mux them into one go routine to avoid lock contention and high cpu and thread thrashing.
// TODO(dlc) make this more then one and pin sources to one of a group.
func (mset *stream) processAllSourceMsgs() {

Every message sourced into a destination stream — regardless of how many upstream sources are configured, how many partitions/subject-transforms are in play, or how many downstream consumers exist — is processed through this single goroutine per destination stream.

Test Details

Topology: Single source stream BENCH (R=3, retention=limits, discard=old), sourced into a destination work-queue stream BENCH-cg-bench (R=3) via Orbit's elastic PCG (max-members=100, 100 active members), using a partitioned subject transform (bench.data.* → {{Partition(100,1)}}.bench.data.{{Wildcard(1)}}).

Load: 250,000 concurrent publisher connections publishing at a sustained ~77,000-78,000 msg/s target rate, 3-hour soak, 100-byte payloads.

Observed Data

We measured the destination stream's own Last Sequence growth (the only reliable ground-truth for actual sourcing throughput) across four independent test runs, at two different hardware tiers, with member counts ranging from 50-100:

Test	Hardware	PCG Members	Source stream config	Measured sourcing rate (msg/s)	Measurement method
T054	m8g.2xlarge (8 vCPU)	50	max-age=30m, max-bytes=20GiB	~24,500 (estimated, single-member extrapolation)	Per-member stats extrapolation
T055	c8g.4xlarge (16 vCPU)	50	max-age=30m, max-bytes=20GiB	18,401 – 19,391 (3 windows)	Direct destination-stream Last Sequence delta over time
T056	c8g.4xlarge (16 vCPU)	100	max-age=30m, max-bytes=20GiB	~19,533 (concurrent-publish); ~255 msg/s/member × 100 ≈ 25,500 aggregate consumption capacity (not the bottleneck)	Direct sequence delta; per-member stats
T057	c8g.4xlarge (16 vCPU)	100	max-age=2h, max-bytes=35GiB	18,247 – 19,173 (concurrent-publish, 3 windows); 21,309 – 21,451 (post-publish, consumers still active)	Direct sequence delta across multiple windows

Happy to share full test logs, nats stream info / nats consumer info snapshots, pprof CPU profiles, and the orchestration scripts used to reproduce this at scale, if useful.

[harishva23]

Hi @kaushalaroraharman , I have run into the same issue of capped throughput using this library. Unfortunately, the probable cause you have described is only one of the issues here. There is an open issue when deleting messages in the work-queue stream as well [https://github.com/nats-io/nats-server/issues/7487] . Since this library uses a sourced work-queue stream, it is a double bottleneck that cannot be eliminated using this approach, as each message is also deleted when acknowledged in the sourced stream.

It is best to either rely on pure vertical scaling or design your own specific partitioning logic across consumers using a plain Jetstream consumer, as there are no simple fixes without significant architectural changes.

Also, evaluate whether static consumer groups are compatible with your use case, as the library computes the partition only; this means you will have to over-provision your consumers, requests, limits etc

[kaushalaroraharman]

@harishva23 Static consumer groups won't suffice my use case, as I need to add and remove members per varying load on production, which is the primary reason I decided to go ahead with elastic over static.

The issue I see with custom logic is to manage multiple JetStream consumers, specifically when I add or remove a Kubernetes pod for the cloud application. The support for rebalancing of members when adding or removing application pods is what made me lean towards using Orbit.

Were you using Orbit for strict-ordering, or any other use case, and have you found some alternative?

[harishva23]

I was evaluating elastic consumer groups for strict ordering for a subject and to manage the partitioning of subjects between Kubernetes pods when I hit a ceiling on throughput. A blog post on Synadia's website also states that the advantages of static partitioned consumer groups over elastic are lower latency and faster speeds, and fewer server resources being used.
https://nats.io/blog/orbit-partitioned-consumer-groups

One partial workaround is to create this work-queue stream in memory by forking this library and overriding the stream parameters to include the max stream size in bytes to avoid OOM kills. But this also introduces new problems, such as the stability of the NATS cluster, high cost of memory.

[kaushalaroraharman]

@harishva23 Using static PCG would definitely provide much better throughput. The same communication I received from NATS team on a seperate GitHub ticket. The challenge I see with static is if one of my pod goes down, the members associated with this pod would stop, and the partitions mapped to these members would end up accumulating lag, till my pod recovers, and I also end up with always having to run a definite number of pods for my application. One way I see around it is to have a fixed N number of members which all the M pods registers. At a moment I would have N active members, and (N x M - N) on hot standy. The tradeoff again would be hard-capped parallelism.

[harishva23]

Hi @kaushalaroraharman , I have a fork of the Orbit.go repository in my account where I have overridden the default storage for the work-queue stream to memory storage and replicas to 1.
I achieved a throughput of 28k msgs/sec on a 2 vCPU, 8 GB memory d2as_v5 Azure VM as part of an Azure Kubernetes cluster in these scenarios:

1. If producers and consumers are running together and the messages are well under the max_bytes of the stream (required for streams with memory usage)
2. If max_bytes or max_messages of the work queue stream are set to high values (4 GB and above) and your consumers are capable of processing each message as fast as possible, then you can have the highest throughput till the new messages are consumed from the base stream

However, this workaround does not work for the following scenarios:

1. If you are creating the consumer group for the 1st time with a huge backlog in the base stream. This is limited due to the issue you have described, which requires a major fix in the NATS server itself
2. If consumers are not able to process messages fast enough and the max_bytes or max_messages is being fulfilled every second, by which all consumption of messages from base stream is stopped for 1 second and the cycle continues.

Let me know if you can test my fork in your VMs as I currently do not have the resources for more powerful hardware