Add log_failure_only option to reduce bulk response log verbosity

[rodrigocarvalho-bemobi]

Problem

When log_level trace is enabled, the send_bulk method in out_opensearch.rb logs the full bulk API response:

log.on_trace { log.trace "bulk response: #{response}" }

A bulk request with hundreds of records produces a response object with an equally large items array. Even if only 1 record fails, the entire response is serialized — generating log messages of 10–20 KB per bulk request. In high-throughput environments this floods the log collector and makes triage significantly harder.

This is related to #37, which requests better control over verbosity when bulk requests encounter errors.

Proposed Solution

Add a log_failure_only boolean config parameter (default false, fully backwards-compatible).

When enabled, the bulk response is logged with only the failed items — entries in the items array where the operation returned an error field. Successful items are dropped from the log representation before serialization.

This keeps trace logs actionable for debugging while reducing log volume by orders of magnitude in low-failure-rate scenarios.

Example

With 500 records in a bulk request and 1 failure, instead of logging all 500 items (~15 KB), only the 1 failed item is logged (~200 bytes).

Configuration:

<match logs.**>
  @type opensearch
  log_failure_only true
</match>

Implementation

The change touches only lib/fluent/plugin/out_opensearch.rb:

1. New config_param after log_os_400_reason:

config_param :log_failure_only, :bool, :default => false,
             :desc => 'When true, bulk response trace log includes only failed items instead of the full response.'

2. Modified trace log in send_bulk:

log.on_trace do
  loggable = if @log_failure_only && response.is_a?(Hash) && response['items'].is_a?(Array)
               failed = response['items'].select { |i| i.values.first.key?('error') }
               response.merge('items' => failed)
             else
               response
             end
  log.trace "bulk response: #{loggable}"
end

A PR with this implementation is ready for review.

[Watson1978]

Thanks for sharing your idea.

However, from a design perspective, I believe trace should remain the raw, unfiltered response for deep debugging. Generally, log_level trace is intended for plugin developers rather than general operation.

I'm trying to understand the underlying issue that requires you to use log_level trace. Is there some necessary log information (like specific error details) that can only be retrieved via trace right now?

If that's the case, we should probably consider outputting those specific error details at the debug or warn level instead of adding a filter to trace. Let me know your thoughts.

[rodrigocarvalho-bemobi]

Thank you for the detailed explanation — I fully understand the distinction between trace and warn semantics, and I agree that changing the log level of the bulk response trace would be the wrong approach.

After investigating further with production logs, I realized our actual problem is in a different code path: out_opensearch_data_stream.rb, not out_opensearch.rb. When any document in a batch fails, the plugin logs the entire bulk response at error level — including all the successful items alongside the failed ones:

if response['errors']
  log.error "Could not bulk insert to Data Stream: #{data_stream_name} #{response}"
end

In production, this generates log lines with thousands of characters per flush cycle (hundreds of successful status: 201 items mixed with a handful of status: 400 errors). The @log_level warn setting doesn't help here since this is already error level.

What we'd actually need is to filter the logged response to include only the failed items — something like:

if response['errors']
  failed = response['items'].select { |i| i.values.first['status'] != 201 }
  log.error "Could not bulk insert to Data Stream: #{data_stream_name}, failed items: #{failed}"
end

I'm not sure whether this warrants updating this PR or opening a new one specifically for out_opensearch_data_stream.rb — happy to follow whatever direction you think is best. And I apologize for the initial misdirection; I should have traced the production logs more carefully before opening the issue.

[Watson1978]

I think that filtering the response is the right approach.

However, I have suggestions for the implementation.

1. Refine the Condition
   • != 201 might be a bit too broad or risky in some edge cases. It is generally safer to check for status codes indicating an error (e.g., >= 300).

Thanks.

[rodrigocarvalho-bemobi]

Thank you for the feedback, @Watson1978! Agreed that >= 300 is the safer condition — it correctly handles any 2xx success status OpenSearch may return, not just 201.

I've opened a new PR with the fix applied to the correct file (out_opensearch_data_stream.rb), incorporating your suggestion: #175

I also closed PR #174 since it was targeting the wrong code path (out_opensearch.rb), so it shouldn't cause any noise or confusion on your end.