Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support "fire-and-forget" messages #76

Open
joseph-wakeling-sociomantic opened this issue Nov 26, 2018 · 3 comments
Open

Support "fire-and-forget" messages #76

joseph-wakeling-sociomantic opened this issue Nov 26, 2018 · 3 comments

Comments

@joseph-wakeling-sociomantic

For some use-cases it is not vital that every message gets through, but it is important to avoid queue overflow in the case that there is not an active channel reader. Some example use-cases:

  • an app wants to start writing a new DMQ channel, but doesn't want to block on the reader app being implemented

    • this allows writers to get their work done (and test potential performance issues) without worrying about the reader
    • can switch to persistent messages if needed only once the reader is ready
  • we have a very high-throughput channel where if the reader goes down, we will overspill very quickly due to the quantity of data

    • if we can tolerate the data loss we might want to just drop messages rather than create a big backlog and overspill onto disk
  • we have a channel where the reader needs to focus on the most current data, and where old messages can be discarded

    • in this case, processing a DMQ backlog when the reader revives might delay restoration of a well functioning system (even if the size of the backlog is tolerable)

In all of these cases it would be convenient to have either a request type (or a flag for Put requests) that indicates that, in the absence of a channel reader, the DMQ node can just drop the message.

@joseph-wakeling-sociomantic
Copy link
Author

An extra compromise option might be for messages to be allowed a specified timeout period (in seconds) in which a reader can collect them, before they are dropped.

This might allow us to have a slightly more flexible approach that avoids data loss due to short term network issues but starts dropping messages in the case of longer-term outages. It could be tailored on a use-case-by-use-case basis.

@david-eckardt-sociomantic
Copy link
Contributor

Since the reader needs to focus on the most current data, the least current records need to be dropped. Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.

@gavin-norman-sociomantic

Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.

That's exactly what I was going to suggest as well 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants