Sequence reached end, reader can not provide more buffer.

[engenb]

Hello,

This is more of a question and less of an issue. First, some context:

I'm working through a project that uses a queue to throttle how many stored procedures may be executed on the database concurrently. if too many sprocs run at once (these are very intensive sprocs) the database resources become fully-consumed and things grind to a halt. when a queued stored procedure completes, the results must be stored off for later retrieval by whatever queued the sproc in the first place - claim-check pattern.

I was hoping to use this project to [de]serialize the sproc results into storage as part of that claim-check pattern. however, I've run into the exception mentioned in this issue's title:
An exception of type 'MemoryPack.MemoryPackSerializationException' occurred in MemoryPack.Core.dll but was not handled in user code: 'Sequence reached end, reader can not provide more buffer.'

It looks like a few other people have run into this issue as well, and the suggestion has been to increase the read buffer size due to deserializing some very large strings.

I also have some very long strings. my sproc results include column data that is typed nvarchar(max) so the string size is unbounded. suppose I arbitrarily increase the read buffer size. in that case, I can't guarantee it will be larger than "the maximum length of the utf8 encoding of (my) string" because my "current maximum length" is (virtually) infinite.

my questions:

1. is this read buffer size constraint a deal-breaker to use MemoryPack in this scenario?
2. am I misunderstanding the problem and/or the proposed solution in the other related issues?
3. does this project's predecessor, MessagePack, have similar constraints of needing to know the max size of string data?

would be a shame if I can't use this as the streaming extensions are perfect for my needs!

thanks for taking a look.

[neuecc]

First, as a premise, are you talking about MemoryPackStreamingSerializer?

[engenb]

correct, @neuecc - the streaming is ideal for my use case. I can confirm that the issue does not exist if I use the MemoryPackSerializer.

[engenb]

just wanted to follow up @neuecc - are the streaming apis just not an option for serializing/deserializing strings of unbounded length? is there a possibility of a future update that may add this capability?

[neuecc]

From an implementation perspective, since co-deserializers for each type cannot return success/failure status, they immediately throw exceptions when they fail.
The StreamingDeserializer achieves reasonable success rates by providing a buffer supply margin.
Therefore, to answer the question - unbounded length is not currently supported.
However, since this also involves design considerations,
I'd like to eventually try an approach that can handle this well
while maintaining a balance with performance.
(The current approach was chosen because naive streaming support would cause too much performance degradation)