Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

kafka: Commit offsets failed with retriable exception. You should retry committing offsets

[o.a.k.c.c.i.ConsumerCoordinator] [Auto offset commit failed for group consumer-group: Commit offsets failed with retriable exception. You should retry committing offsets.] []

Why does this error come in kafka consumer? what does this mean?

The consumer properties I am using are:

fetch.min.bytes:1
enable.auto.commit:true
auto.offset.reset:latest
auto.commit.interval.ms:5000
request.timeout.ms:300000
session.timeout.ms:20000
max.poll.interval.ms:600000
max.poll.records:500
max.partition.fetch.bytes:10485760
What is the reason for that error to come? I am guessing the consumer is doing duplicated work right now (polling same message again) because of this error.
I am neither using consumer.commitAsync() or consumer.commitSync()
                Hey @ViacheslavShalamov, I am sure this is a late reply. I was not able to fix the issue. I had to restart the broker! Still hoping to be able to find the cause whenever it appears again :p
– enator
                Mar 19, 2019 at 16:54
                enable.auto.commit:true calls commitAsync under the hood, so it's the reason why you are getting this error
– eparvan
                May 22, 2019 at 8:49
                of course. but isn't that the point of enable.auto.commit:true? if SDK is managing the commit then why the developer has to face Commit offsets failed with retriable exception. You should retry committing offsets? I was hoping that using auto-commit we shouldn't worry about offset management.
– enator
                Jul 2, 2019 at 8:00
Consumer gives this error in case if it catches an instance of RetriableException.
The reasons for it might be various:
if coordinator is still loading the group metadata 
if the group metadata topic has not been created yet
if network or disk corruption happens, or miscellaneous disk-related or network-related IOException occurred when handling a request
if server disconnected before a request could be completed
if the client's metadata is out of date
if there is no currently available leader for the given partition
if no brokers were available to complete a request
As you can see from the list above, all these errors could be temporary issues, that is why it is suggested to retry the request.
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.