Skip to content

Add captureBackpressureData WriteOption to preserve dropped points #846

@fredjoonpark

Description

@fredjoonpark

Proposal:
Introduce a new WriteOption config, captureBackpressureData, which can preserve dropped data points on backpressure events. Making it optional so existing clients won't be surprised or affected.

DROP_OLDEST => will contain the oldest points that will be dropped as a result of processing incoming batch
DROP_LATEST => will contain new incoming batch

The additional points in memory will never exceed the batchSize as defined through the client.


Current behavior:
When the backlog grows and backpressure is triggered, points are dropped silently (regardless of backpressure strategy).


Desired behavior:

WriteOptions writeOptions = WriteOptions.builder()
    .backpressureStrategy(BackpressureOverflowStrategy.DROP_OLDEST)
    .bufferLimit(1000)
    .captureBackpressureData(true)  // Enable data capture
    .build();

writeApi.listenEvents(BackpressureEvent.class, backpressureEvent -> {
    List<String> affectedPoints = backpressureEvent.getBufferedLineProtocol();
    logger.warn("Backpressure occurred. Affected {} data points:", affectedPoints.size());

    affectedPoints.forEach(point -> logger.debug("Affected point: {}", point));
});

Alternatives considered:
Handling it at the application level. Not ideal as client hides batching operations and offers limited visibility/control from the consumer.


Use case:
For many production applications, silently losing points is unacceptable. This change allows the client to report dropped points on backpressure events to ensure at-least once delivery with all incoming requests.

Happy to take this on, focusing on the core Java client.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions