同事在一次 spark 程序中，通过 spark 在 kafka 中的 topic 中写入数据发生报错，错误日志如下:

2021-01-25 19:58:11,794 INFO utils.AppInfoParser: Kafka version : 2.0.0-cdh6.1.1
2021-01-25 19:58:11,794 INFO utils.AppInfoParser: Kafka commitId : null
2021-01-25 19:58:11,984 INFO codegen.CodeGenerator: Code generated in 80.049669 ms
2021-01-25 19:58:12,020 WARN clients.NetworkClient: [Producer clientId=producer-1] Connection to node -3 could not be established. Broker may not be available.
2021-01-25 19:58:12,034 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,034 INFO clients.Metadata: Cluster ID: aKKlHlDqQtalfjbLYRW1GQ
2021-01-25 19:58:12,136 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 9 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,241 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,346 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 11 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,451 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 12 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,554 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 13 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,658 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 14 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,763 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 15 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,867 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 16 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:12,971 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 17 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,076 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 18 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,181 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 19 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,285 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 20 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,390 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 21 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,495 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 22 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,600 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 23 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,704 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 24 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,808 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 25 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:13,913 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 26 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:14,044 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 27 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:14,149 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 28 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
2021-01-25 19:58:14,254 WARN clients.NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 29 : {tagging-api-1611575841019=INVALID_REPLICATION_FACTOR}
探索步骤:
 
根据日志中的信息：INVALID_REPLICATION_FACTOR 初步判断，出现了不合法的副本数。
检查kafka的配置文件发现server.properties中默认的配置是三副本配置。
检查被写入kafka的topic是否存在，发现目标topic并不存在，初步怀疑是建立topic失败
得知同事以前可以在这个kafka的集群中成功执行同样的任务，遂后开始检查kafka集群的状态。
检查出有一台的kafka的broker结点挂掉，重启失败。发现是磁盘空间被占空导致重启失败，遂清理无用日志空出空间，启动kakfa发现kafka成功执行没有失败
同事再次启动程序，这个时候已经可以成功运行了。问题解决√
 
在水下这篇文章之前，看了下报错日志。其中日志里提到了这样的一句话:Connection to node -3 could not be established. Broker may not be available.，所以，emmmm，如果仔细一下日志的话，可以更加快速的定位到问题。所以，我再次记录一下这个文章的目的是为了警醒自己，一定要仔细看日志，仔细看日志。
                    Spark 连接kafka报错: Error while fetching metadata with correlation id xx [topicName=INVALID_REPLICATIO]
                    问题来源同事在一次spark程序中，通过spark在kafka中的topic中写入数据发生报错，错误日志如下:2021-01-25 19:58:11,794 INFO utils.AppInfoParser: Kafka version : 2.0.0-cdh6.1.12021-01-25 19:58:11,794 INFO utils.AppInfoParser: Kafka commitId : null2021-01-25 19:58:11,984 INFO codegen.CodeGener
错误信息如下
Error while fetching metadata with correlation id 1 : 
{topic_lcc=LEADER_NOT_AVAILABLE}
对方配置如下
集群启动后如下
3. 解决方法1
3.1 原因
获取相关ID为xx的元数据时出错
3.2 问题解决
修改...
InvalidReplicationFactorException: Replication factor:
1 larger than available brokers
原因是kafka的zk中的配置与创建参数对不上
打开server.properties
vim /opt/module/kafka/config/server.properties
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka
注意版本，不同的版本情况不一样。我以 2 .7 为例
首先配置：config/server.properties
advertised.listeners: 把注释去掉，在PLAINTEXT://后面放入你的公网ip
如：PLAINTEXT://192.168.0.1：9092
如下异常：
[2021-01-21 10:59:28,658] WARN [Producer clientId=console-producer] Connection to 
报错信息
在终端执行kafka发送、消费消息遇到如下错误
Error while fetching metadata with correlation id 2 : {abc123=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient
2  ps -ef | grep kafka
				
1.创建topic中出现出现错误
kafka出现:
Error while fetching metadata with correlation id   : {LEADER_NOT_AVAILABLE}
表示无法识别kafka hostname
正确处理姿势
1、修改server.properties如下，目的是让kafka监听的到
vi   /opt/kafka/kafka/config/se...
				
kafka创建topic报错ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException
在kafka创建topic中输入以下命令
 bin/kafka-topics.sh --zookeeper node01:2181 --create --replication-factor 3 --partitions 1 --topic first
ERROR org.apache.kafka.common.errors
				
golang源码分析：sarama kafka client(part I:生产者) - 墨天轮
https://github.com/Shopify/sarama 是一个纯go实现的kafka客户端，是gopher学习kafka一个很好的资料。说实话sarama的代码组织很烂，密密麻麻一堆源码文件都在一个目录，让人无从下手，下面列出了一部分：
examples
mocks
tools //基于客户端，实现的kafka客户端工具
    tools/kafka-producer-performance
					
 Error while fetching metadata with correlation id 37 : {flink_kafka9=LEADER_NOT_AVAILABLE}
                gitlab reconfigure 卡住 ruby_block[wait for redis service socket] action run
                    StarDream-Online: 
                    大佬，成功了，只是首次安装的时候会卡住，第二次修改就没有了
                Python:设置 Pandas的dataFrame的index索引起始值为1
                    山茶花开时。: 
                    你这个方法舍去了index索引列值为0的那一行数据，不符合要求
                Hive 3.0.0 的安装（图文教程）
                    芹菜学长: 
                    说的是，我改一下。谢谢提醒
                Hive 3.0.0 的安装（图文教程）
                    surpass-_-: 
                    你那个path配置的就是坑，把系统的path路径全覆盖了，