You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For some use-cases it is not vital that every message gets through, but it is important to avoid queue overflow in the case that there is not an active channel reader. Some example use-cases:
an app wants to start writing a new DMQ channel, but doesn't want to block on the reader app being implemented
this allows writers to get their work done (and test potential performance issues) without worrying about the reader
can switch to persistent messages if needed only once the reader is ready
we have a very high-throughput channel where if the reader goes down, we will overspill very quickly due to the quantity of data
if we can tolerate the data loss we might want to just drop messages rather than create a big backlog and overspill onto disk
we have a channel where the reader needs to focus on the most current data, and where old messages can be discarded
in this case, processing a DMQ backlog when the reader revives might delay restoration of a well functioning system (even if the size of the backlog is tolerable)
In all of these cases it would be convenient to have either a request type (or a flag for Put requests) that indicates that, in the absence of a channel reader, the DMQ node can just drop the message.
The text was updated successfully, but these errors were encountered:
An extra compromise option might be for messages to be allowed a specified timeout period (in seconds) in which a reader can collect them, before they are dropped.
This might allow us to have a slightly more flexible approach that avoids data loss due to short term network issues but starts dropping messages in the case of longer-term outages. It could be tailored on a use-case-by-use-case basis.
Since the reader needs to focus on the most current data, the least current records need to be dropped. Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.
Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.
That's exactly what I was going to suggest as well 👍
For some use-cases it is not vital that every message gets through, but it is important to avoid queue overflow in the case that there is not an active channel reader. Some example use-cases:
an app wants to start writing a new DMQ channel, but doesn't want to block on the reader app being implemented
we have a very high-throughput channel where if the reader goes down, we will overspill very quickly due to the quantity of data
we have a channel where the reader needs to focus on the most current data, and where old messages can be discarded
In all of these cases it would be convenient to have either a request type (or a flag for
Put
requests) that indicates that, in the absence of a channel reader, the DMQ node can just drop the message.The text was updated successfully, but these errors were encountered: