Fluent-bit skip logs very frequently

[nitishkumar71]

Hello Team

We are trying to capture nginx ingress logs and are pushing it to database. The whole pipeline is working but we are losing logs very frequently. We have looked at all points for any error but nothing is being reported either in fluent-bit or database. We enabled metrics in fluent-bit but none of the filter is recording any dropped records. There is no error being recorded for output plugin too. We enabled debug mode there is no error or warning being recorded there.

Based on suggestions from issues and discussion, we have increased Buffer_Chunk_Size, Buffer_Max_Size and storage.total_limit_size to very high but still logs are missing very frequently for some span as shown in the attached image.

Here is the config we have used for some of the attributes value may look very high but we have been testing them from default values and gradually increasing them to test. We are expecting more than 80G of logs in a day and all of that logs is being captured by single fluent-bit daemon.

[SERVICE]
    Log_Level debug
    Daemon Off
    Flush 1
    Log_Level debug
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On
    storage.path /var/log/flb-storage/
    storage.checksum On
    storage.backlog.flush_on_shutdown On

[INPUT]
    Name tail
    Path /var/log/containers/ingress-nginx*.log
    multiline.parser docker, cri
    Tag ingress.*
    Buffer_Chunk_Size 25M
    Buffer_Max_Size 500M
    Skip_Long_Lines Off
    DB ingress.db
    Read_from_Head On
    storage.type filesystem
    threaded true

[FILTER]
    Name parser
    Match ingress.*
    Key_Name log
    Parser nginx-ingress
    Reserve_Data On
    Preserve_Key On

[FILTER]
    Name record_modifier
    Match ingress.*
    Remove_key _p
    Remove_key stream
    Remove_key reg_id
    Remove_key upstream*
    Remove_key proxy_upstream_name

[OUTPUT]
    name http
    tls on
    match ingress.*
    host ${url}
    port ${port}
    URI ${uri}
    format json_stream
    json_date_key time
    json_date_format iso8601
    http_user ${user}
    http_passwd ${password}
    storage.total_limit_size 500M

[nitishkumar71]

Apolgoies, there is no issue with fluent-bit. In our case, we are generating logs very frequently, so we should have reduced Refresh_Interval in tail plugin.