Skip to content

Commit f02dc07

Browse files
ahmatarokkk
ahma
authored and
tarokkk
committed
Add ES params
1 parent ffac24f commit f02dc07

File tree

8 files changed

+367
-11
lines changed

8 files changed

+367
-11
lines changed

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,9 +106,9 @@ For complete list of supported plugins pleas checkl the [plugins index](/docs/pl
106106
| Name | Type | Description | Status | Version |
107107
|---------------------------------------------------------|:------:|:-------------------------------------------------------------------------:|---------|-------------------------------------------------------------------------------------------|
108108
| [Alibaba](./docs/plugins/outputs/oss.md) | Output | Store logs the Alibaba Cloud Object Storage Service | GA | [0.0.1](https://github.com/aliyun/fluent-plugin-oss) |
109-
| [Amazon S3](./docs/plugins/outputs/s3.md) | Output | Store logs in Amazon S3 | GA | [1.1.10](https://github.com/fluent/fluent-plugin-s3/releases/tag/v1.1.10) |
110-
| [Azure](./docs/plugins/outputs/azurestore.md) | Output | Store logs in Azure Storega | GA | [0.1.1](https://github.com/htgc/fluent-plugin-azurestorage/releases/tag/v0.1.0) |
111-
| [Google Storage](./docs/plugins/outputs/gcs.md) | Output | Store logs in Google Cloud Storage | GA | [0.4.0.beta1](https://github.com/banzaicloud/fluent-plugin-gcs) |
109+
| [Amazon S3](./docs/plugins/outputs/s3.md) | Output | Store logs in Amazon S3 | GA | [1.1.11](https://github.com/fluent/fluent-plugin-s3/releases/tag/v1.1.11) |
110+
| [Azure](./docs/plugins/outputs/azurestore.md) | Output | Store logs in Azure Storega | GA | [0.1.0](https://github.com/htgc/fluent-plugin-azurestorage/releases/tag/v0.1.0) |
111+
| [Google Storage](./docs/plugins/outputs/gcs.md) | Output | Store logs in Google Cloud Storage | GA | [0.4.0](https://github.com/banzaicloud/fluent-plugin-gcs) |
112112
| [Grafana Loki](./docs/plugins/outputs/loki.md) | Output | Transfer logs to Loki | Testing | [0.2](https://github.com/banzaicloud/fluent-plugin-kubernetes-loki/releases/tag/v0.2) |
113113
| [ElasticSearch](./docs/plugins/outputs/elasticsearch.md) | Output | Send your logs to Elasticsearch | GA | [3.5.5](https://github.com/uken/fluent-plugin-elasticsearch/releases/tag/v3.5.5) |
114114
| [Sumologic](./docs/plugins/outputs/sumologic.md) | Output | Send your logs to Sumologic | GA | [1.5.0](https://github.com/SumoLogic/fluentd-output-sumologic/releases/tag/1.5.0) |

‎charts/logging-operator/templates/logging.banzaicloud.io_clusteroutputs.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -410,28 +410,58 @@ spec:
410410
required:
411411
- timekey
412412
type: object
413+
bulk_message_request_threshold:
414+
description: 'Configure bulk_message request splitting threshold
415+
size. Default value is 20MB. (20 * 1024 * 1024) If you specify
416+
this size as negative number, bulk_message request splitting feature
417+
will be disabled. (default: 20MB)'
418+
type: string
413419
content_type:
414420
description: 'With content_type application/x-ndjson, elasticsearch
415421
plugin adds application/x-ndjson as Content-Type in payload. (default:
416422
application/json)'
417423
type: string
424+
custom_headers:
425+
description: 'This parameter adds additional headers to request.
426+
Example: {"token":"secret"} (default: {})'
427+
type: string
418428
customize_template:
419429
description: Specify the string and its value to be replaced in
420430
form of hash. Can contain multiple key value pair that would be
421431
replaced in the specified template_file. This setting only creates
422432
template and to add rollover index please check the rollover_index
423433
configuration.
424434
type: string
435+
default_elasticsearch_version:
436+
description: This parameter changes that ES plugin assumes default
437+
Elasticsearch version. The default value is 5.
438+
type: string
425439
deflector_alias:
426440
description: Specify the deflector alias which would be assigned
427441
to the rollover index created. This is useful in case of using
428442
the Elasticsearch rollover API
429443
type: string
444+
exception_backup:
445+
description: 'Indicates whether to backup chunk when ignore exception
446+
occurs. (default: true)'
447+
type: boolean
430448
fail_on_putting_template_retry_exceed:
431449
description: 'Indicates whether to fail when max_retry_putting_template
432450
is exceeded. If you have multiple output plugin, you could use
433451
this property to do not fail on fluentd statup.(default: true)'
434452
type: boolean
453+
flatten_hashes:
454+
description: 'Elasticsearch will complain if you send object and
455+
concrete values to the same field. For example, you might have
456+
logs that look this, from different places: {"people" => 100}
457+
{"people" => {"some" => "thing"}} The second log line will be
458+
rejected by the Elasticsearch parser because objects and concrete
459+
values can''t live in the same field. To combat this, you can
460+
enable hash flattening.'
461+
type: boolean
462+
flatten_hashes_separator:
463+
description: Flatten separator
464+
type: string
435465
host:
436466
description: You can specify Elasticsearch host by this parameter.
437467
(default:localhost)
@@ -448,6 +478,16 @@ spec:
448478
id_key:
449479
description: https://github.com/uken/fluent-plugin-elasticsearch#id_key
450480
type: string
481+
ignore_exceptions:
482+
description: A list of exception that will be ignored - when the
483+
exception occurs the chunk will be discarded and the buffer retry
484+
mechanism won't be called. It is possible also to specify classes
485+
at higher level in the hierarchy. For example `ignore_exceptions
486+
["Elasticsearch::Transport::Transport::ServerError"]` will match
487+
all subclasses of ServerError - Elasticsearch::Transport::Transport::Errors::BadRequest,
488+
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable,
489+
etc.
490+
type: string
451491
include_index_in_url:
452492
description: With this option set to true, Fluentd manifests the
453493
index name in the request URL (rather than in the request body).
@@ -471,6 +511,14 @@ spec:
471511
description: Specify the index prefix for the rollover index to
472512
be created.
473513
type: string
514+
log_es_400_reason:
515+
description: 'By default, the error logger won''t record the reason
516+
for a 400 error from the Elasticsearch API unless you set log_level
517+
to debug. However, this results in a lot of log spam, which isn''t
518+
desirable if all you want is the 400 error reasons. You can set
519+
this true to capture the 400 error reasons without all the other
520+
debug logs. (default: false)'
521+
type: boolean
474522
logstash_dateformat:
475523
description: 'Set the Logstash date format.(default: %Y.%m.%d)'
476524
type: string
@@ -589,6 +637,12 @@ spec:
589637
description: If you want to configure SSL/TLS version, you can specify
590638
ssl_version parameter. [SSLv23, TLSv1, TLSv1_1, TLSv1_2]
591639
type: string
640+
suppress_doc_wrap:
641+
description: 'By default, record body is wrapped by ''doc''. This
642+
behavior can not handle update script requests. You can set this
643+
to suppress doc wrapping and allow record body to be untouched.
644+
(default: false)'
645+
type: boolean
592646
tag_key:
593647
description: 'This will add the Fluentd tag in the JSON record.(default:
594648
tag)'
@@ -644,6 +698,16 @@ spec:
644698
degree of sub-second time precision to preserve from the time
645699
portion of the routed event.
646700
type: string
701+
unrecoverable_error_types:
702+
description: Default unrecoverable_error_types parameter is set
703+
up strictly. Because es_rejected_execution_exception is caused
704+
by exceeding Elasticsearch's thread pool capacity. Advanced users
705+
can increase its capacity, but normal users should follow default
706+
behavior. If you want to increase it and forcibly retrying bulk
707+
request, please consider to change unrecoverable_error_types parameter
708+
from default value. Change default value of thread_pool.bulk.queue_size
709+
in elasticsearch.yml)
710+
type: string
647711
user:
648712
description: User for HTTP Basic authentication. This plugin will
649713
escape required URL encoded characters within %{} placeholders.
@@ -654,6 +718,20 @@ spec:
654718
with UTC (Coordinated Universal Time). This option allows to use
655719
local time if you describe utc_index to false.(default: true)'
656720
type: boolean
721+
validate_client_version:
722+
description: 'When you use mismatched Elasticsearch server and client
723+
libraries, fluent-plugin-elasticsearch cannot send data into Elasticsearch.
724+
(default: false)'
725+
type: boolean
726+
verify_es_version_at_startup:
727+
description: 'Because Elasticsearch plugin should change behavior
728+
each of Elasticsearch major versions. For example, Elasticsearch
729+
6 starts to prohibit multiple type_names in one index, and Elasticsearch
730+
7 will handle only _doc type_name in index. If you want to disable
731+
to verify Elasticsearch version at start up, set it as false.
732+
When using the following configuration, ES plugin intends to communicate
733+
into Elasticsearch 6. (default: true)'
734+
type: boolean
657735
with_transporter_log:
658736
description: 'This is debugging purpose option to enable to obtain
659737
transporter layer log. (default: false)'

‎charts/logging-operator/templates/logging.banzaicloud.io_outputs.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -410,28 +410,58 @@ spec:
410410
required:
411411
- timekey
412412
type: object
413+
bulk_message_request_threshold:
414+
description: 'Configure bulk_message request splitting threshold
415+
size. Default value is 20MB. (20 * 1024 * 1024) If you specify
416+
this size as negative number, bulk_message request splitting feature
417+
will be disabled. (default: 20MB)'
418+
type: string
413419
content_type:
414420
description: 'With content_type application/x-ndjson, elasticsearch
415421
plugin adds application/x-ndjson as Content-Type in payload. (default:
416422
application/json)'
417423
type: string
424+
custom_headers:
425+
description: 'This parameter adds additional headers to request.
426+
Example: {"token":"secret"} (default: {})'
427+
type: string
418428
customize_template:
419429
description: Specify the string and its value to be replaced in
420430
form of hash. Can contain multiple key value pair that would be
421431
replaced in the specified template_file. This setting only creates
422432
template and to add rollover index please check the rollover_index
423433
configuration.
424434
type: string
435+
default_elasticsearch_version:
436+
description: This parameter changes that ES plugin assumes default
437+
Elasticsearch version. The default value is 5.
438+
type: string
425439
deflector_alias:
426440
description: Specify the deflector alias which would be assigned
427441
to the rollover index created. This is useful in case of using
428442
the Elasticsearch rollover API
429443
type: string
444+
exception_backup:
445+
description: 'Indicates whether to backup chunk when ignore exception
446+
occurs. (default: true)'
447+
type: boolean
430448
fail_on_putting_template_retry_exceed:
431449
description: 'Indicates whether to fail when max_retry_putting_template
432450
is exceeded. If you have multiple output plugin, you could use
433451
this property to do not fail on fluentd statup.(default: true)'
434452
type: boolean
453+
flatten_hashes:
454+
description: 'Elasticsearch will complain if you send object and
455+
concrete values to the same field. For example, you might have
456+
logs that look this, from different places: {"people" => 100}
457+
{"people" => {"some" => "thing"}} The second log line will be
458+
rejected by the Elasticsearch parser because objects and concrete
459+
values can''t live in the same field. To combat this, you can
460+
enable hash flattening.'
461+
type: boolean
462+
flatten_hashes_separator:
463+
description: Flatten separator
464+
type: string
435465
host:
436466
description: You can specify Elasticsearch host by this parameter.
437467
(default:localhost)
@@ -448,6 +478,16 @@ spec:
448478
id_key:
449479
description: https://github.com/uken/fluent-plugin-elasticsearch#id_key
450480
type: string
481+
ignore_exceptions:
482+
description: A list of exception that will be ignored - when the
483+
exception occurs the chunk will be discarded and the buffer retry
484+
mechanism won't be called. It is possible also to specify classes
485+
at higher level in the hierarchy. For example `ignore_exceptions
486+
["Elasticsearch::Transport::Transport::ServerError"]` will match
487+
all subclasses of ServerError - Elasticsearch::Transport::Transport::Errors::BadRequest,
488+
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable,
489+
etc.
490+
type: string
451491
include_index_in_url:
452492
description: With this option set to true, Fluentd manifests the
453493
index name in the request URL (rather than in the request body).
@@ -471,6 +511,14 @@ spec:
471511
description: Specify the index prefix for the rollover index to
472512
be created.
473513
type: string
514+
log_es_400_reason:
515+
description: 'By default, the error logger won''t record the reason
516+
for a 400 error from the Elasticsearch API unless you set log_level
517+
to debug. However, this results in a lot of log spam, which isn''t
518+
desirable if all you want is the 400 error reasons. You can set
519+
this true to capture the 400 error reasons without all the other
520+
debug logs. (default: false)'
521+
type: boolean
474522
logstash_dateformat:
475523
description: 'Set the Logstash date format.(default: %Y.%m.%d)'
476524
type: string
@@ -589,6 +637,12 @@ spec:
589637
description: If you want to configure SSL/TLS version, you can specify
590638
ssl_version parameter. [SSLv23, TLSv1, TLSv1_1, TLSv1_2]
591639
type: string
640+
suppress_doc_wrap:
641+
description: 'By default, record body is wrapped by ''doc''. This
642+
behavior can not handle update script requests. You can set this
643+
to suppress doc wrapping and allow record body to be untouched.
644+
(default: false)'
645+
type: boolean
592646
tag_key:
593647
description: 'This will add the Fluentd tag in the JSON record.(default:
594648
tag)'
@@ -644,6 +698,16 @@ spec:
644698
degree of sub-second time precision to preserve from the time
645699
portion of the routed event.
646700
type: string
701+
unrecoverable_error_types:
702+
description: Default unrecoverable_error_types parameter is set
703+
up strictly. Because es_rejected_execution_exception is caused
704+
by exceeding Elasticsearch's thread pool capacity. Advanced users
705+
can increase its capacity, but normal users should follow default
706+
behavior. If you want to increase it and forcibly retrying bulk
707+
request, please consider to change unrecoverable_error_types parameter
708+
from default value. Change default value of thread_pool.bulk.queue_size
709+
in elasticsearch.yml)
710+
type: string
647711
user:
648712
description: User for HTTP Basic authentication. This plugin will
649713
escape required URL encoded characters within %{} placeholders.
@@ -654,6 +718,20 @@ spec:
654718
with UTC (Coordinated Universal Time). This option allows to use
655719
local time if you describe utc_index to false.(default: true)'
656720
type: boolean
721+
validate_client_version:
722+
description: 'When you use mismatched Elasticsearch server and client
723+
libraries, fluent-plugin-elasticsearch cannot send data into Elasticsearch.
724+
(default: false)'
725+
type: boolean
726+
verify_es_version_at_startup:
727+
description: 'Because Elasticsearch plugin should change behavior
728+
each of Elasticsearch major versions. For example, Elasticsearch
729+
6 starts to prohibit multiple type_names in one index, and Elasticsearch
730+
7 will handle only _doc type_name in index. If you want to disable
731+
to verify Elasticsearch version at start up, set it as false.
732+
When using the following configuration, ES plugin intends to communicate
733+
into Elasticsearch 6. (default: true)'
734+
type: boolean
657735
with_transporter_log:
658736
description: 'This is debugging purpose option to enable to obtain
659737
transporter layer log. (default: false)'

0 commit comments

Comments
 (0)