Skip to content

Conversation

@zhaoanliu
Copy link
Owner


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

PettitWesley and others added 30 commits October 14, 2020 21:10
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This patch adds a new plugin to deliver records to Loki (grafana).

It supports the following configuration options:

  host                    Loki hostname or IP address

  port                    Loki TCP port

  tenant_id               Tenant ID used by default to push logs to Loki. If
                          omitted or empty it assumes Loki is running in
                          single-tenant mode and no X-Scope-OrgID header is sent.
                          > type: string

  labels                  labels for API requests.
                          > default: job="fluent-bit", type: multiple comma delimited strings

  auto_kubernetes_labels  If set to true, it will add all Kubernetes labels to
                          Loki labels.
                          > default: false, type: boolean

  label_keys              Comma separated list of keys to use as stream labels.
                          > type: multiple comma delimited strings

One of the interesting features is that 'labels' and 'label_keys' support record
accessor mode, e.g:

    [OUTPUT]
        name        loki
        match       *
        labels      job=fluentbit, rec=$sub['field']
        label_keys  $somekey

Note that 'label_keys' must be prefixed with '$' since it's a record accessor pattern.

This plugin is still in development, community feedback is welcome.

Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
…uent#2698)

The following patch fix the Stream labels management and in addition
it perform other improvements:

 - if a stream label value is empty, skip it (don't pack an empty value)

 - set the default label 'job=fluent-bit' only when no other label has
   been set.

Signed-off-by: Eduardo Silva <[email protected]>
A user (jsubirat) requested that in_winlog should be able to output
StringInserts in log records.

This patch implements that feature, which enables you to write the
plugin configuration like:

    [INPUT]
    Name winlog
    String_Inserts true

... and here is an example record after this patch:

    {
        "RecordNumber"=>333,
        "TimeGenerated"=>"2020-08-11 10:56:41 -0000",
        "TimeWritten"=>"2020-08-11 10:56:41 -0000",
        "EventType"=>"Information",
        "Message"=>"Starting session 0 - 2020-08-11T10:56:41.595894000Z.",
        "StringInserts"=>["0", "2020-08-11T10:56:41.595894000Z"]
    }

Signed-off-by: Fujimoto Seiji <[email protected]>
Fix the problem that the task cannot be executed due to the wrong status in non-multiplex mode. When it traverses the task queue, it resets the flag used to determine the task status

Signed-off-by: zhanghjster <[email protected]>
Signed-off-by: zhanghaijun3 <[email protected]>

Co-authored-by: zhanghaijun3 <[email protected]>
… connections after configured uses (fluent#2704)

Allow users to configure the max number of uses of a connection before it gets
thrown away. In cases where fluentbit is used with an upstream behind a
load-balancer, we can get better distribution amongst upstreams by tuning this
value.

Signed-off-by: William Orr <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
edsiper and others added 30 commits December 3, 2020 15:29
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
`val.via.str.ptr` here is not a regular C string - it is not `\0`
terminated as it comes from `msgpack`. Thus, using the regular `strstr`
here is bad as we might "jump over" to other content in memory.
Introduce a new `flb_strnchr` which operates on possibly NULL-terminated
strings.  Use it in the `kafka` output plugin.

Using `strncmp` is fine because the char arrays need not be
NULL-terminated.

Fixes fluent#2836. Cannot reproduce it locally after this fix.

Huge, huge kudos to @deimantastumas for a lot of investigation and a
very good test case. <3

Signed-off-by: Giedrius Statkevičius <[email protected]>
Co-authored-by: Deimantas Tumas <[email protected]>
Windows does not allow directory names to contain a colon. For this
reason, S3 plugin was unable to create the backlog directory.

This patch fixes it by changing the directory name from

    /tmp/fluent-bit/s3/bucket/2020-12-01T12:31:21

into:

    /tmp/fluent-bit/s3/bucket/2020-12-01T12-31-21

Signed-off-by: Fujimoto Seiji <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Prevent a segfault in the error path when the JSON format is used:

```
[2020/12/24 00:05:00] [ warn] [engine] failed to flush chunk '651791-1608761099.630731466.flb', retry in 6 seconds: task_id=4, input=dummy.0 > output=kafka.0
[2020/12/24 00:05:00] [ warn] [engine] failed to flush chunk '651791-1608761100.174611647.flb', retry in 7 seconds: task_id=16, input=dummy.0 > output=kafka.0
munmap_chunk(): invalid pointer

Thread 2 "flb-pipeline" received signal SIGABRT, Aborted.
--Type <RET> for more, q to quit, c to continue without paging--
[Switching to Thread 0x7ffff7885700 (LWP 651795)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
    at /home/gstatkevicius/dev/fluent-bit/plugins/out_kafka/kafka.c:409
```

All of the paths besides `FLB_KAFKA_FMT_MSGP` operate on `flb_sds_t` so
the appropriate function to use here is `flb_sds_destroy`.

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.