Temporary “Stream Not Available” error after restarting a stream or delete replica in RabbitMQ 4.1.0 cluster (previous versions also had this problem)

[yurahaid]

Describe the bug

We are experiencing a temporary unavailability of a stream after performing a restart operation in one of our RabbitMQ clusters. 

Environment:
• RabbitMQ version: 4.1.0
• Cluster configuration: 3 nodes
• Client: Go RabbitMQ Stream client (latest version (v1.5.6)

Issue Details:

In one of our RabbitMQ clusters, when we execute a restart_stream operation on a stream, ours producers and consumers receive a Stream Not Available error for approximately 5 to 20 seconds. This issue does not occur in another cluster with an identical configuration.

Both clusters are running RabbitMQ version 4.1.0 with three nodes each. The consumers are implemented using the latest version of the Go RabbitMQ Stream client.

Reproduction steps

1.	Set up a RabbitMQ cluster with 3 nodes running version 4.1.0.
2.	Create a stream and start a consumer using the Go RabbitMQ Stream client.
3.	Perform a restart_stream or delete_replica operations on the stream - example `rabbitmq-streams restart_stream --vhost "/" "test-rabbitmq-5.stream" --preferred-leader-node="{your_node}"` (it doesn't matter whether the node leader changes or stays the same)
4.	Observe that the consumer or producer receives a Stream Not Available error for 5 to 20 seconds.

{"severity":"ALERT","ts":"2025-05-27T12:03:18.897Z","caller":"/src/cmd/test-rabbitmq/main.go:92","message":"exit with error","context":{"error":"create consumer: Stream Not Available"},"stacktrace":"main.main\n\t/src/cmd/test-rabbitmq/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283"}

Expected behavior

The stream should remain available to consumers and publishers during and after the operations with stream, without any errors.

Additional context

• The issue is reproducible consistently in one cluster but not in another with the same configuration.
• No significant differences have been identified between the two clusters in terms of configuration or client implementation.

This makes it challenging to work with RabbitMQ because we cannot add or remove a node from the cluster without affecting the system's operation.

[acogoluegnes]

Can you provide the server logs for both clusters? Setting the log level to debug would be useful, even it can be verbose.

There must be something different between the 2 clusters, whether it is in the software stack or in the infrastructure, or somewhere else. Could you provide more information about the 2 clusters?

The stream should remain available to consumers and publishers during and after the operations with stream, without any errors.

There will be some unavailability at some point. Whether you notice or not, this is unavoidable. The duration of the unavailability depends on many factors. I suspect there is some unavailability or at least disruption on the cluster without allegedly any problem, as restarting a stream changes its topology, which impacts the connections that use it.

[michaelklishin]

@yurahaid you haven't provided any evidence of a bug.

A stream restart inevitably will result in a short unavailability window. Your clients should be ready to retry the operations.

It's up to you to investigate the difference between the clusters.

[yurahaid]

Yes, I understand that, and we are prepared for it.
However, the issue is that the stream often remains unavailable for an extended period—ranging from a few seconds to several minutes—which poses a problem for us.

This is particularly noticeable when compared to another cluster with an identical configuration, where the stream becomes available almost instantly after a restart.
We expect the stream to be accessible again within 10 seconds post-restart.

Interestingly, the cluster experiencing these issues is more powerful, equipped with 8 cores.
In contrast, I tested on a less powerful cluster (2 cores per node), subjected it to a significant CPU load—making it more overloaded than the problematic one—and still observed that stream restarts were quicker and more stable, with availability restored almost immediately upon the first reconnection attempt.

[yurahaid]

I am adding logs from the clusters.

Both clusters are very similar, hosted on VMs in GCP in different projects.
CPU type n4
Disk: Hyperdisk

The problem cluster has 8 CPUs, and stable has 2 CPUs in each node.

healthy cluster with successful reconnect on the first attempt..log
problem_cluster_3_failed_reconects.log

[acogoluegnes]

Thanks for the extra information, we'll investigate the issue.