SpansV2 batcher drops a lot of spans #5236
Description
The span batcher for span first currently has an internal limit of 2000 spans in memory before it force flushes. This default limit seems to make us drop a lot of spans when instrumenting Seer. If the limit is increased to an arbitrarily high number, we don't drop spans. This suggests that Seer creates a lot of spans in a short period of time (the automatic flush happens every 5s).
- Check whether Seer can be considered an outlier in this behavior, or if the batcher performs badly on an average project as well.
- Consider raising the internal flushing limit but make sure to consider the trade-off (keeping more spans in memory).
- Consider adding another heuristic for forcing more frequent flushes, e.g. Flush span buffer based on estimated envelope size #5191.
If we end up determining Seer is an outlier, it's ok to leave the implementation as is, and we can either provide a config option or just directly patch the limit in the batcher in Seer.
Note: We will be moving to telemetry buffers at some point, so the batcher is not the final design.