Multi line issue within labels

[moeterich]

• fluentd or td-agent version: fluentd 1.4.2
• Environment information: Alpine Linux v3.9
• Kernel version: 4.4.0-1047-aws

Configuration:

    <match **>
      @type relabel
      @label @FINAL
    </match>

    <label @FINAL>
      <filter *.parsed>
         @type record_mapper 
	 map {
  	    "k":"v",
            "k1":10
         }
      </filter>
      <match **>
        @type gelf
        host 1.2.3.4
        port 12201
        protocol tcp
        <buffer>
          @type memory
          flush_mode interval
          flush_interval 1s
          retry_max_interval 60s
        </buffer>
      </match>
    </label>

When using multi line within a label, fluentd will throw an exception:

2019-05-02 19:43:25 +0000 [info]: parsing config file is succeeded path="/tmp/1"
2019-05-02 19:43:25 +0000 [info]: using configuration file: <ROOT>
  <match **>
    @type relabel
    @label @FINAL
  </match>
  <label @FINAL>
    <filter *.parsed>
      @type record_mapper
      map {"k":"v","k1":10}
    </filter>
    <match **>
      @type gelf
      host "1.2.3.4"
      port 12201
      protocol "tcp"
      <buffer>
        @type "memory"
        flush_mode interval
        flush_interval 1s
        retry_max_interval 60s
      </buffer>
    </match>
  </label>
</ROOT>
2019-05-02 19:43:25 +0000 [info]: starting fluentd-1.4.2 pid=26 ruby="2.5.3"
2019-05-02 19:43:25 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-p", "/fluent/plugins/", "-c", "/tmp/1", "--under-supervisor"]
Traceback (most recent call last):
	21: from /usr/bin/fluentd:23:in `<main>'
	20: from /usr/bin/fluentd:23:in `load'
	19: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/bin/fluentd:8:in `<top (required)>'
	18: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	17: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	16: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
	15: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:502:in `run_supervisor'
	14: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:629:in `supervise'
	13: from /usr/lib/ruby/gems/2.5.0/gems/serverengine-2.1.1/lib/serverengine.rb:33:in `create'
	12: from /usr/lib/ruby/gems/2.5.0/gems/serverengine-2.1.1/lib/serverengine.rb:33:in `new'
	11: from /usr/lib/ruby/gems/2.5.0/gems/serverengine-2.1.1/lib/serverengine/daemon.rb:32:in `initialize'
	10: from /usr/lib/ruby/gems/2.5.0/gems/serverengine-2.1.1/lib/serverengine/config_loader.rb:36:in `initialize'
	 9: from /usr/lib/ruby/gems/2.5.0/gems/serverengine-2.1.1/lib/serverengine/config_loader.rb:43:in `reload_config'
	 8: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:630:in `block in supervise'
	 7: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:259:in `load_config'
	 6: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config.rb:39:in `parse'
	 5: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/v1_parser.rb:33:in `parse'
	 4: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/v1_parser.rb:43:in `parse!'
	 3: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/v1_parser.rb:95:in `parse_element'
	 2: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/v1_parser.rb:95:in `parse_element'
	 1: from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/v1_parser.rb:132:in `parse_element'
/usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/config/basic_parser.rb:92:in `parse_error!': expected end of line at 1 line 10,1 (Fluent::ConfigParseError)
  9: 	 map {
 10:   	    "k":"v",

     -^
 11:             "k1":10

Looking into the documentation regarding supported data types, this should work, or do I miss something here ?

[omitrowski]

Hi!
I have exactly the same issue. Map inside a label is not accepted, but I have no other choice in my situation!

[scrwr]

Some finding. This happens only if
gem 'gelf', '>= 3.1.0'
is installed. With 3.0.0 this parser error does not happen.

I don't see something obvious here, which can break the fluentd parser. Anyone?
graylog-labs/gelf-rb@v3.0.0...v3.1.0

[ganmacs]

Map inside a label seems to be accepted.

2019-11-25 15:28:21 -0800 [info]: parsing config file is succeeded path="example/debug/relable.conf"
2019-11-25 15:28:21 -0800 [info]: gem 'fluentd' version '1.8.0.rc2'
2019-11-25 15:28:21 -0800 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2019-11-25 15:28:21 -0800 [info]: using configuration file: <ROOT>
  <match **>
    @type relabel
    @label @FINAL
  </match>
  <label @FINAL>
    <filter *.parsed>
      @type grep
      map {"k":"v","k1":10}
    </filter>
    <match>
      @type stdout
    </match>
  </label>
</ROOT>
2019-11-25 15:28:21 -0800 [info]: starting fluentd-1.8.0.rc2 pid=16369 ruby="2.6.5"
2019-11-25 15:28:22 -0800 [info]: adding filter in @FINAL pattern="*.parsed" type="grep"
2019-11-25 15:28:22 -0800 [info]: adding match in @FINAL pattern="**" type="stdout"
2019-11-25 15:28:22 -0800 [info]: adding match pattern="**" type="relabel"

And graylog-labs/gelf-rb#53 is the root cause of this.

[scrwr]

@ganmacs but only if you write it single-line. Once you use multiple lines like shown in the configuration example (not the log output) it fails, still with version 1.7.4.

[repeatedly]

Yes, the problem is gelf-rb. This is why k8s daemonset doesn't use 3.1.0.

fluent/fluentd-kubernetes-daemonset#219

[scrwr]

I got this part, but sticking with gelf-rb 3.0.0 won't be a permanent solution. Would you like helping to push for a permanent fix at the right place? Or let me know, how we can help here.

[repeatedly]

If you want to fix the problem, sending new patch to gelf-rb is better.
Old patch is stopped: graylog-labs/gelf-rb#64

[scrwr]

With no Ruby Devs in the entire company, that is going to be challenging. ;) But thats the right place. I'll see what I can do.

[ganmacs]

for now, it's gelf-rb's issue, not fluentd issue. i'm closing. Thank you for reporting!