Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gem::ConflictError when applying ClusterFlow #1251

Closed
sebastiangaiser opened this issue Mar 14, 2023 · 12 comments
Closed

Gem::ConflictError when applying ClusterFlow #1251

sebastiangaiser opened this issue Mar 14, 2023 · 12 comments
Labels
bug Something isn't working

Comments

@sebastiangaiser
Copy link
Contributor

Describe the bug:
FluentD leads to crashloopbackoff and FluentD-configcheck fails

Expected behaviour:
FluentD is not crashing :D

Steps to reproduce the bug:

  1. Install operator via Helm with clusteroutput (in my case opensearch)
  2. Apply ClusterFlow leads to errors as shown

Additional context:

Environment details:

  • Kubernetes version (e.g. v1.15.2): v1.26.0
  • Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): on prem - kubeadm
  • logging-operator version (e.g. 2.1.1): 4.0.0
  • Install method (e.g. helm or static manifests): helm for operator + Kubernetes manifest for clusterflow
  • Logs from the misbehaving component (and any other relevant logs):
    Output of kubectl logs -f -n logging logging-operator-logging-fluentd-configcheck-dcbaaaf3
fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-03-14 17:04:12 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-14 17:04:12 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-03-14 17:04:12 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-03-14 17:04:12 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-14 17:04:12 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-03-14 17:04:12 +0000 [info]: starting fluentd-1.15.3 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
  • Resource definition (possibly in YAML format) that caused the issue, without sensitive data:
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: general
  namespace: logging
spec:
  globalOutputRefs:
    - general
  match:
    - select: {}

FluentD spec (also v1.15 tested) taken from Logging:

disablePvc: true
image: 
  tag: v1.14

/kind bug
@sebastiangaiser sebastiangaiser added the bug Something isn't working label Mar 14, 2023
@sebastiangaiser sebastiangaiser changed the title Gem::ConflictError when applying select: {} flow Gem::ConflictError when applying ClusterFlow Mar 14, 2023
@sebastiangaiser sebastiangaiser mentioned this issue Mar 15, 2023
Closed
@ahma
Copy link
Contributor

ahma commented Mar 16, 2023

@sebastiangaiser, thanks for the report. Could you recheck it with ghcr.io/kube-logging/fluentd:v1.14.6 ?

@sebastiangaiser
Copy link
Contributor Author 

kubectl describe po -n logging logging-fluentd-configcheck-d77653a3 | grep Image
    Image:         ghcr.io/kube-logging/fluentd:v1.14.6

2023-03-17 10:47:48 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @raw
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-03-17 10:47:48 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-03-17 10:47:48 +0000 [info]: starting fluentd-1.15.3 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

Same error for the FluentD pod(s)...

@ahma
Copy link
Contributor

ahma commented Mar 18, 2023

@sebastiangaiser Thanks. Please check with v1.14.6-build.51 and send the logging output definition as well. (without the sensitive data. ;) )

@sebastiangaiser
Copy link
Contributor Author

@ahma

Failed to pull image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": failed to resolve reference "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": ghcr.io/banzaicloud/fluentd:v1.14.6-build.51: not found

@pepov
Copy link
Member

pepov commented Mar 29, 2023

@sebastiangaiser it's ghcr.io/kube-logging/fluentd:v1.14.6-build.51

@sebastiangaiser
Copy link
Contributor Author

Sorry for the late reply:

kubectl describe po -n logging logging-fluentd-configcheck-29ea88f7 | grep Image  
    Image:         ghcr.io/kube-logging/fluentd:v1.14.6-build.51

Same error:

fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-04-11 13:29:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-04-11 13:29:04 +0000 [info]: starting fluentd-1.14.6 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

The ClusterOutput:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: general
  namespace: logging
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      valueFrom:
        secretKeyRef:
          key: password
          name: credentials
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user

@pepov
Copy link
Member

pepov commented Apr 12, 2023 •
edited
Loading

I just ran this both with the latest ghcr.io/kube-logging/fluentd:v1.14 and ghcr.io/kube-logging/fluentd:v1.15 images.

My manifest:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: opensearch
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: default
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: opensearch
spec:
  match:
    - select: {}
  localOutputRefs:
    - "opensearch"
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: opensearch
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      value: asd
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user

The container is running although it is unable to connect to the host, so I guess it passes the error @sebastiangaiser mentioned:

2023-04-12 13:59:47 +0000 [info]: starting fluentd-1.14.6 pid=7 ruby="2.7.8"
...
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).

2023-04-12 14:13:08 +0000 [info]: starting fluentd-1.15.3 pid=7 ruby="2.7.8"
...
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 12. Retry to communicate after 8 second(s).

@sebastiangaiser
Copy link
Contributor Author

sebastiangaiser commented Apr 12, 2023 •
edited
Loading

My error occurs earlier than your Could not communicate to OpenSearch (in the configcheck).

The only difference I can see, you are using normal Flow and Output resources. I did it with ClusterFlow and ClusterOutput but I will try to make a minimal example as you did and paste the result here because I was using https://github.com/kube-logging/helm-charts/tree/main/charts/logging-operator-logging for deploying the CRs.

@sebastiangaiser
Copy link
Contributor Author

Same error with these manifests and operator 4.X (helm upgrade --install -n logging logging-operator kube-logging/logging-operator --version 4.1.0-dev.2):

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
  namespace: logging
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: logging
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: opensearch
  namespace: logging
spec:
  match:
    - select: {}
  globalOutputRefs:
    - opensearch
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: opensearch
  namespace: logging
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      value: asd
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user

@sebastiangaiser
Copy link
Contributor Author

sebastiangaiser commented Apr 17, 2023 •
edited
Loading

@pepov it seams to work now with logging-operator 4.1.0 and logging-operator-logging 4.1.0 and ghcr.io/kube-logging/fluentd:v1.14.

Have you changed something to the images? In case, is there a best practice for pinning the image version for the logging-operator. I remember such a image version pinning for rook-ceph with the recommendation to pin it to a build date (for production environments).

@pepov
Copy link
Member

pepov commented Apr 17, 2023 •
edited
Loading

We don't change the operator image version, so you don't have to worry about that 4.1.0 will stay the same. We have a different build and release setup for the kube-logging/fluentd:v1.14 and kube-logging/fluentd:v1.15 images, but that is going to change once kube-logging/fluentd:v1.15-staging get's merged into v1.15 and all the gem versions will be locked.

And to answer you question I beleive the updates in the elasticsearch/opensearch plugins fixed the conflict issue:

diff -dy --suppress-common-lines <( docker run --rm ghcr.io/kube-logging/fluentd:v1.14-build.51 gem list ) <( docker run --rm ghcr.io/kube-logging/fluentd:v1.15 gem list )
addressable (2.8.1)						|       addressable (2.8.4)
aws-partitions (1.728.0)					|       aws-partitions (1.745.0)
aws-sdk-core (3.170.0)						|       aws-sdk-core (3.171.0)
aws-sdk-s3 (1.119.1)						|       aws-sdk-s3 (1.120.1)
bundler (2.4.8)							|       bundler (2.4.12)
connection_pool (2.3.0)						|       connection_pool (2.4.0)
elasticsearch (7.13.3)						|       elastic-transport (8.2.1)
elasticsearch-api (7.13.3)					|       elasticsearch (8.7.0, 7.13.3)
								>       elasticsearch-api (8.7.0, 7.13.3)
faraday-excon (1.1.0)						|       faraday-excon (2.1.0, 1.1.0)
faraday_middleware-aws-sigv4 (0.6.1, 0.3.0)			|       faraday_middleware-aws-sigv4 (1.0.1, 0.3.0)
fluent-plugin-elasticsearch (5.2.4)				|       fluent-plugin-elasticsearch (5.3.0)
								>       fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-opensearch (1.0.10)				|       fluent-plugin-opensearch (1.1.0)
fluent-plugin-splunk-hec (1.3.1)				|       fluent-plugin-splunk-hec (1.3.2)
								>       fluent-plugin-vmware-log-intelligence (2.0.6)
								>       fluent-plugin-vmware-loginsight (1.4.0)
fluentd (1.14.6)						|       fluentd (1.15.3)
googleauth (1.3.0)						|       googleauth (1.5.1)
msgpack (1.6.1)							|       msgpack (1.7.0)
net-http-persistent (4.0.1, 3.1.0)				|       myslog (0.1.1)
								>       net-http-persistent (4.0.2, 3.1.0)
nio4r (2.5.8)							|       nio4r (2.5.9)
nokogiri (1.14.2)						|       nokogiri (1.14.3)
opensearch-api (2.1.0)						|       opensearch-api (2.2.0)
								>       snappy (0.2.0)
tzinfo (2.0.6, 1.2.7)						|       tzinfo (2.0.6, 1.2.11)
tzinfo-data (1.2022.7)						|       tzinfo-data (1.2023.3)
uri (default: 0.10.0)						|       uri (default: 0.10.0.2)

uken/fluent-plugin-elasticsearch@1e817c9
fluent/fluent-plugin-opensearch@9774a1c

@pepov pepov closed this as completed Apr 17, 2023
@sebastiangaiser
Copy link
Contributor Author

Thank you for the support.
I found another issue that might be related: uken/fluent-plugin-elasticsearch#1014

The question about the tagging was more in the direction for the fluentd images like ghcr.io/kube-logging/fluentd:v1.14. Is the tag v1.14 immutable (I assume not)? How about v1.14.6? Or would it be better to use the build number like v1.14.6-build.67?

@sebastiangaiser sebastiangaiser mentioned this issue Apr 17, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pepov @ahma @sebastiangaiser