-
Notifications
You must be signed in to change notification settings - Fork 132
Description
Proposal:
Introduce a new WriteOption config, captureBackpressureData, which can preserve dropped data points on backpressure events. Making it optional so existing clients won't be surprised or affected.
DROP_OLDEST => will contain the oldest points that will be dropped as a result of processing incoming batch
DROP_LATEST => will contain new incoming batch
The additional points in memory will never exceed the batchSize as defined through the client.
Current behavior:
When the backlog grows and backpressure is triggered, points are dropped silently (regardless of backpressure strategy).
Desired behavior:
WriteOptions writeOptions = WriteOptions.builder()
.backpressureStrategy(BackpressureOverflowStrategy.DROP_OLDEST)
.bufferLimit(1000)
.captureBackpressureData(true) // Enable data capture
.build();
writeApi.listenEvents(BackpressureEvent.class, backpressureEvent -> {
List<String> affectedPoints = backpressureEvent.getBufferedLineProtocol();
logger.warn("Backpressure occurred. Affected {} data points:", affectedPoints.size());
affectedPoints.forEach(point -> logger.debug("Affected point: {}", point));
});
Alternatives considered:
Handling it at the application level. Not ideal as client hides batching operations and offers limited visibility/control from the consumer.
Use case:
For many production applications, silently losing points is unacceptable. This change allows the client to report dropped points on backpressure events to ensure at-least once delivery with all incoming requests.
Happy to take this on, focusing on the core Java client.