output: add multiplex option for output plugin

[Binyang2014]

Add keep order option for output plugin.

This option will help user send logs to the storage server which can not sort log lines according to timestamp.
Such as AzureBlob/S3.

Issue: #1746

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[Binyang2014]

config: the http port is inaccessible, so will always retry to limit.

[OUTPUT]
    name                  http
    match                 *
    multiplex             on
    Host                  127.0.0.1
    Port                  8080
    Retry_Limit           1

logs: all chunks are keep in order

[2021/03/19 16:15:32] [error] [src/flb_http_client.c:1156 errno=32] Broken pipe
[2021/03/19 16:15:32] [error] [output:http:http.0] could not flush records to 127.0.0.1:8080 (http_do=-1)
[2021/03/19 16:15:32] [ warn] [engine] failed to flush chunk '3681-1616141727.980231600.flb', retry in 6 seconds: task_id=0, input=dummy.0 > output=http.0 (out_id=0)

[2021/03/19 16:15:37] [error] [src/flb_http_client.c:1156 errno=32] Broken pipe
[2021/03/19 16:15:37] [error] [output:http:http.0] could not flush records to 127.0.0.1:8080 (http_do=-1)
[2021/03/19 16:15:37] [ warn] [engine] chunk '3681-1616141727.980231600.flb' cannot be retried: task_id=0, input=dummy.0 > output=http.0

[2021/03/19 16:15:41] [error] [src/flb_http_client.c:1156 errno=32] Broken pipe
[2021/03/19 16:15:41] [error] [output:http:http.0] could not flush records to 127.0.0.1:8080 (http_do=-1)
[2021/03/19 16:15:41] [ warn] [engine] failed to flush chunk '3681-1616141731.996901200.flb', retry in 10 seconds: task_id=1, input=dummy.0 > output=http.0 (out_id=0)

[2021/03/19 16:15:51] [error] [src/flb_http_client.c:1156 errno=32] Broken pipe
[2021/03/19 16:15:51] [error] [output:http:http.0] could not flush records to 127.0.0.1:8080 (http_do=-1)
[2021/03/19 16:15:51] [ warn] [engine] chunk '3681-1616141731.996901200.flb' cannot be retried: task_id=1, input=dummy.0 > output=http.0

[Binyang2014]

@edsiper @koleini @fujimotos Please help to review.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[Binyang2014]

pr is still relevant

[shinji62]

@edsiper any chance to get this merge ?

[patrick-stephens]

I'm interested in this if it solves issues with out-of-order failures for Loki as well

[MikaelElkiaer]

@Binyang2014 Does this PR alone fix the tail -> loki "message out of order" issues?
I mean, at least for those caused by non-multiplexed input->output - I guess there's still an issue related to re-sending failed messages.

[Binyang2014]

Hi @MikaelElkiaer, this doesn't fix the tail issue, it just for output plugin. For Loki, multiplex should ready be disabled by default

fluent-bit/plugins/out_loki/loki.c

Line 1373 in bcce842

.flags = FLB_OUTPUT_NET | FLB_IO_OPT_TLS | FLB_OUTPUT_NO_MULTIPLEX,

So it just solve output retry (due to network issues), and doesn't solve the input retry. I didn't notice there is out of order issue for tail plugin, but if there is such issue, we may need another solution.

[MikaelElkiaer]

I didn't notice there is out of order issue for tail plugin, but if there is such issue, we may need another solution.

Not an issue with tail as such, but AFAIK there's no method of handling logs that already are out of order, in the case of services that do async logging.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.