相关文章推荐
时尚的萝卜  ·  淘宝开放平台 - 文档中心·  11 月前    · 
威武的钥匙扣  ·  Python编程之子进程管理(subproc ...·  2 年前    · 
灰常酷的楼房  ·  Python 网络编程 | 菜鸟教程·  2 年前    · 
销魂的风衣  ·  如何将二维列表转换成二维数值数组？ - ...·  2 年前    · 
玩篮球的韭菜  ·  七年之痒 ...·  2 年前    · 
Code  ›  Output ES fails to flush chunk with status 0 · Issue #3301 · fluent/fluent-bit · GitHub
var systemd input log
https://github.com/fluent/fluent-bit/issues/3301
烦恼的毛豆
7 月前

Output ES fails to flush chunk with status 0 #3301

New issue
New issue
Closed
Closed
Output ES fails to flush chunk with status 0 #3301
Labels
Stale
@konradasb

Description

@konradasb
konradasb
opened on Mar 29, 2021

Bug Report

Describe the bug
Constantly (but not always) fails to flush chunks with HTTP status=0 from ES in Kubernetes. Debug/Trace logs don't show anything specific, even with Trace_Error On 

[fluent-bit-5gqwz] [2021/03/29 21:53:42] [ warn] [engine] failed to flush chunk '1-1617054818.967282961.flb', retry in 9 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-mw7sw] [2021/03/29 21:54:03] [ warn] [engine] failed to flush chunk '1-1617054839.820871907.flb', retry in 7 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-5gqwz] [2021/03/29 21:55:32] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-5gqwz] [2021/03/29 21:55:32] [ warn] [engine] failed to flush chunk '1-1617054927.935218441.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-2z84q] [2021/03/29 21:55:47] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-2z84q] [2021/03/29 21:55:47] [ warn] [engine] failed to flush chunk '1-1617054942.558870892.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 
[fluent-bit-mw7sw] [2021/03/29 21:56:23] [error] [output:es:es.0] HTTP status=0 URI=/_bulk 
[fluent-bit-mw7sw] [2021/03/29 21:56:23] [ warn] [engine] failed to flush chunk '1-1617054978.505299097.flb', retry in 11 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) 

Your Environment




    



  • Version used: 1.7.0 and 1.7.1
    • 

  • Configuration:
    
Tried to remove everything and leave only in_tail and out_es. Results are the same.
    • 

      config:
    service: |
      [SERVICE]
          Flush         5
          Grace         120
          Log_Level     info
          Daemon        off
          Parsers_File  parsers.conf
          Parsers_File  custom_parsers.conf
          HTTP_Server   On
          HTTP_Listen   0.0.0.0
          HTTP_PORT     2020
    customParsers: |
        [PARSER]
            Name        CRI
            Format      regex
            Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    filters: |
        # -- Applied to All --
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Merge_Log           Off
            Keep_Log            Off
            K8S-Logging.Parser  On
            K8S-Logging.Exclude On
        [FILTER]
            Name                nest
            Match               kube.*
            Operation           lift
            Nested_Under        kubernetes
        # -- Ingress Nginx --
        [FILTER]
            Name                rewrite_tag
            Match               kube*ingress-nginx*
            Rule                $stream ^(stderr)$  ingress-nginx.error false
            Rule                $stream ^(stdout)$  ingress-nginx.access false
        [FILTER]
            Name                parser
            Match               ingress-nginx.access
            Parser              json
            Key_Name            log
    inputs: |
        [INPUT]
            Name                tail
            Tag                 kube.*
            Path                /var/log/containers/*.log
            DB                  /var/log/flb_kube_containers.db
            Parser              CRI
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
            Refresh_Interval    10
            Skip_Long_Lines     On
        # -- SystemD --
        [INPUT]
            Name                systemd
            Alias               systemd-kubelet
            Tag                 systemd-kubelet
            Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
            Path                /var/log/journal
            DB                  /var/log/kubelet.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-kube-node-installation
            Tag                 systemd-kube-node-installation
            Systemd_Filter      _SYSTEMD_UNIT=kube-node-installation.service
            Path                /var/log/journal
            DB                  /var/log/kube-node-installation.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-kube-node-configuration
            Tag                 systemd-kube-node-configuration
            Systemd_Filter      _SYSTEMD_UNIT=kube-node-configuration.service
            Path                /var/log/journal
            DB                  /var/log/kube-node-configuration.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-kube-logrotate
            Tag                 systemd-kube-logrotate
            Systemd_Filter      _SYSTEMD_UNIT=kube-logrotate.service
            Path                /var/log/journal
            DB                  /var/log/kube-logrotate.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-node-problem-detector
            Tag                 systemd-node-problem-detector
            Systemd_Filter      _SYSTEMD_UNIT=node-problem-detector.service
            Path                /var/log/journal
            DB                  /var/log/node-problem-detector.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-kube-container-runtime-monitor
            Tag                 systemd-kube-container-runtime-monitor
            Systemd_Filter      _SYSTEMD_UNIT=kube-container-runtime-monitor.service
            Path                /var/log/journal
            DB                  /var/log/kube-container-runtime-monitor.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
        [INPUT]
            Name                systemd
            Alias               systemd-kubelet-monitor
            Tag                 systemd-kubelet-monitor
            Systemd_Filter      _SYSTEMD_UNIT=kubelet-monitor.service
            Path                /var/log/journal
            DB                  /var/log/kubelet-monitor.db
            Mem_Buf_Limit       5MB
            Buffer_Max_Size     1MB
    outputs: |
        [OUTPUT]
            Name                   gelf
            Match                  systemd*
            Gelf_Short_Message_Key MESSAGE
            Gelf_Host_Key          _HOSTNAME
            Host                   ${GRAYLOG_HOST}
            PORT                   ${GRAYLOG_PORT}
            Mode                   udp
        [OUTPUT]
            Name                   gelf
            Match                  kube*
            Gelf_Short_Message_Key log
            Gelf_Host_Key          host
            Host                   ${GRAYLOG_HOST}
            PORT                   ${GRAYLOG_PORT}
            Mode                   udp
        # -- Ingress Nginx --
        [OUTPUT]
            Name                   gelf
            Match                  ingress-nginx.error
            Gelf_Short_Message_Key log
            Gelf_Host_Key          host
            Host                   ${GRAYLOG_HOST}
            PORT                   ${GRAYLOG_PORT}
            Mode                   udp
        [OUTPUT]
            Name                   es
            Match                  ingress-nginx.access
            Index                  nginx
            Logstash_Format        On
            Host                   ${ES_HOST}
            Http_User              ${ES_USERNAME}
            Http_Passwd            ${ES_PASSWORD}
            Logstash_Prefix        nginx
            Logstash_DateFormat    %Y%m%d
            Retry_Limit            5
            Type                   _doc
            Port                   443
            Tls                    On
            Buffer_Size            50M
     
    推荐文章
    时尚的萝卜  ·  淘宝开放平台 - 文档中心
    11 月前
    威武的钥匙扣  ·  Python编程之子进程管理(subprocess)详解 - LoveFishO - 博客园
    2 年前
    灰常酷的楼房  ·  Python 网络编程 | 菜鸟教程
    2 年前
    销魂的风衣  ·  如何将二维列表转换成二维数值数组？ - 问答 - 腾讯云开发者社区-腾讯云
    2 年前
    玩篮球的韭菜  ·  七年之痒 只因七年后你已改变？|七年之痒|婚姻|人体_新浪时尚_新浪网
    2 年前
    今天看啥   ·   Py中国   ·   codingpro   ·   小百科   ·   link之家   ·   卧龙AI搜索
    删除内容请联系邮箱 2879853325@qq.com
    Code - 代码工具平台
    © 2024 ~ 沪ICP备11025650号