Kafka consumer list - Stack Overflow

相关文章推荐

有腹肌的萝卜 · 事例敏感性 - Azure Repos | ...· 1 月前 ·

飘逸的肉夹馍 · 请介绍图形用户界面（GUI）平台 - ...· 9 月前 ·

高大的苹果 · StackView实现卡片堆叠如此简单-腾讯 ...· 1 年前 ·

叛逆的橡皮擦 · 1.15 ...· 1 年前 ·

博学的鸵鸟 · Sql超时时间已到,在操作完成之前超时时间已 ...· 1 年前 ·

I need to find out a way to ask Kafka for a list of topics. I know I can do that using the kafka-topics.sh script included in the bin\ directory. Once I have this list, I need all the consumers per topic. I could not find a script in that directory, nor a class in the kafka-consumer-api library that allows me to do it.

The reason behind this is that I need to figure out the difference between the topic's offset and the consumers' offsets.

Is there a way to achieve this? Or do I need to implement this functionality in each of my consumers?

Please consider changing the accepted answer given that things have changed and zookeeper is not used in newer versions of Kafka. – Gray Mar 1 at 15:15 bin/kafka-consumer-groups.sh --list --zookeeper localhost:2181 Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API). bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092 Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). – Raja Krishnan Jan 25 '18 at 0:12 Consumer group list command lists all the consumers groups in the cluster, but is there a way to show consumer groups for specific topic? – user3366706 Jun 1 '18 at 22:32

you can use this for 0.9.0.0. version kafka

./kafka-consumer-groups.sh --list --zookeeper hostname:potnumber
to view the groups you have created. This will display all the consumer group names.
 ./kafka-consumer-groups.sh --describe --zookeeper hostname:potnumber  --describe  --group consumer_group_name
To view the details 
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
Kafka stores all the information in zookeeper. You can see all the topic related information under brokers->topics. If you wish to get all the topics programmatically you can do that using Zookeeper API.
It is explained in detail in below links
Tutorialspoint, Zookeeper Programmer guide
                This is actually incorrect in the newer versions of Kafka. Zookeeper is no longer used to store offsets. The offsets are stored in Kafka itself in specially named topics
                    – Basanth Roy
                Jun 17 '18 at 22:00
High level consumers are registered into Zookeeper, so you can fetch a list from ZK, similarly to the way kafka-topics.sh fetches the list of topics. I don't think there's a way to collect all consumers; any application sending in a few consume requests is actually a "consumer", and you cannot tell whether they are done already. 
On the consumer side, there's a JMX metric exposed to monitor the lag. Also, there is Burrow for lag monitoring.
I realize that this question is nearly 4 years old now. Much has changed in Kafka since then. This is mentioned above, but only in small print, so I write this for users who stumble over this question as late as I did.
Offsets by default are now stored in a Kafka Topic (not in Zookeeper any more), see Offsets stored in Zookeeper or Kafka?
There's a kafka-consumer-groups utility which returns all the information, including the offset of the topic and partition, of the consumer, and even the lag (Remark: When you ask for the topic's offset, I assume that you mean the offsets of the partitions of the topic). In my Kafka 2.0 test cluster: 
kafka-consumer-groups --bootstrap-server kafka:9092 --describe
    --group console-consumer-69763 Consumer group 'console-consumer-69763' has no active members.
TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
pytest          0          5               6               1               -               -               -
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
                        Related
                            0In Kafka 0.9 is there a way I can list out the offset for all the consumers in a consumer group?
1Tools to read consumer offset from kafka 0.9
6How to delete kafka consumer (created via new consumer api)?
3kafka-node start consume from last offset
16Kafka consumer for multiple topic
0Kafka partitions and consumer groups for at-least-once message delivery
0Getting error while consuming kafka message
0If Kafka Consumer fails (Spark Job), how to fetch the last offset committed by Kafka Consumer. (Scala)
0Kafka track consumer progress
1Reading last Kafka partition's record that's transactionally written
site design / logo © 2019 Stack Exchange Inc; user contributions licensed under cc by-sa 4.0
                            with attribution required.
                    rev 2019.9.10.34858