Output ES fails to flush chunk with status 0 #3301

@konradasb

Description

Bug Report

Describe the bug
Constantly (but not always) fails to flush chunks with HTTP status=0 from ES in Kubernetes. Debug/Trace logs don't show anything specific, even with Trace_Error On

[fluent-bit-5gqwz] [2021/03/29 21:53:42] [ warn] [engine] failed to flush chunk '1-1617054818.967282961.flb', retry in 9 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-mw7sw] [2021/03/29 21:54:03] [ warn] [engine] failed to flush chunk '1-1617054839.820871907.flb', retry in 7 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-5gqwz] [2021/03/29 21:55:32] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-5gqwz] [2021/03/29 21:55:32] [ warn] [engine] failed to flush chunk '1-1617054927.935218441.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-2z84q] [2021/03/29 21:55:47] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-2z84q] [2021/03/29 21:55:47] [ warn] [engine] failed to flush chunk '1-1617054942.558870892.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-mw7sw] [2021/03/29 21:56:23] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-mw7sw] [2021/03/29 21:56:23] [ warn] [engine] failed to flush chunk '1-1617054978.505299097.flb', retry in 11 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 

Your Environment

  • Version used: 1.7.0 and 1.7.1
  • Configuration:
    Tried to remove everything and leave only in_tail and out_es. Results are the same.
  •   config:
        service: |
          [SERVICE]
              Flush         5
              Grace         120
              Log_Level     info
              Daemon        off
              Parsers_File  parsers.conf
              Parsers_File  custom_parsers.conf
              HTTP_Server   On
              HTTP_Listen   0.0.0.0
              HTTP_PORT     2020
        customParsers: |
            [PARSER]
                Name        CRI
                Format      regex
                Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
                Time_Key    time
                Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        filters: |
            # -- Applied to All --
            [FILTER]
                Name                kubernetes
                Match               kube.*
                Merge_Log           Off
                Keep_Log            Off
                K8S-Logging.Parser  On
                K8S-Logging.Exclude On
            [FILTER]
                Name                nest
                Match               kube.*
                Operation           lift
                Nested_Under        kubernetes
            # -- Ingress Nginx --
            [FILTER]
                Name                rewrite_tag
                Match               kube*ingress-nginx*
                Rule                $stream ^(stderr)$  ingress-nginx.error false
                Rule                $stream ^(stdout)$  ingress-nginx.access false
            [FILTER]
                Name                parser
                Match               ingress-nginx.access
                Parser              json
                Key_Name            log
        inputs: |
            [INPUT]
                Name                tail
                Tag                 kube.*
                Path                /var/log/containers/*.log
                DB                  /var/log/flb_kube_containers.db
                Parser              CRI
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
                Refresh_Interval    10
                Skip_Long_Lines     On
            # -- SystemD --
            [INPUT]
                Name                systemd
                Alias               systemd-kubelet
                Tag                 systemd-kubelet
                Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
                Path                /var/log/journal
                DB                  /var/log/kubelet.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-kube-node-installation
                Tag                 systemd-kube-node-installation
                Systemd_Filter      _SYSTEMD_UNIT=kube-node-installation.service
                Path                /var/log/journal
                DB                  /var/log/kube-node-installation.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-kube-node-configuration
                Tag                 systemd-kube-node-configuration
                Systemd_Filter      _SYSTEMD_UNIT=kube-node-configuration.service
                Path                /var/log/journal
                DB                  /var/log/kube-node-configuration.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-kube-logrotate
                Tag                 systemd-kube-logrotate
                Systemd_Filter      _SYSTEMD_UNIT=kube-logrotate.service
                Path                /var/log/journal
                DB                  /var/log/kube-logrotate.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-node-problem-detector
                Tag                 systemd-node-problem-detector
                Systemd_Filter      _SYSTEMD_UNIT=node-problem-detector.service
                Path                /var/log/journal
                DB                  /var/log/node-problem-detector.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-kube-container-runtime-monitor
                Tag                 systemd-kube-container-runtime-monitor
                Systemd_Filter      _SYSTEMD_UNIT=kube-container-runtime-monitor.service
                Path                /var/log/journal
                DB                  /var/log/kube-container-runtime-monitor.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
            [INPUT]
                Name                systemd
                Alias               systemd-kubelet-monitor
                Tag                 systemd-kubelet-monitor
                Systemd_Filter      _SYSTEMD_UNIT=kubelet-monitor.service
                Path                /var/log/journal
                DB                  /var/log/kubelet-monitor.db
                Mem_Buf_Limit       5MB
                Buffer_Max_Size     1MB
        outputs: |
            [OUTPUT]
                Name                   gelf
                Match                  systemd*
                Gelf_Short_Message_Key MESSAGE
                Gelf_Host_Key          _HOSTNAME
                Host                   ${GRAYLOG_HOST}
                PORT                   ${GRAYLOG_PORT}
                Mode                   udp
            [OUTPUT]
                Name                   gelf
                Match                  kube*
                Gelf_Short_Message_Key log
                Gelf_Host_Key          host
                Host                   ${GRAYLOG_HOST}
                PORT                   ${GRAYLOG_PORT}
                Mode                   udp
            # -- Ingress Nginx --
            [OUTPUT]
                Name                   gelf
                Match                  ingress-nginx.error
                Gelf_Short_Message_Key log
                Gelf_Host_Key          host
                Host                   ${GRAYLOG_HOST}
                PORT                   ${GRAYLOG_PORT}
                Mode                   udp
            [OUTPUT]
                Name                   es
                Match                  ingress-nginx.access
                Index                  nginx
                Logstash_Format        On
                Host                   ${ES_HOST}
                Http_User              ${ES_USERNAME}
                Http_Passwd            ${ES_PASSWORD}
                Logstash_Prefix        nginx
                Logstash_DateFormat    %Y%m%d
                Retry_Limit            5
                Type                   _doc
                Port                   443
                Tls                    On
                Buffer_Size            50M