Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am trying to implement a loop where we will process the a batch of messages in loop, post successful processing we will commit the offset of the message, however for below scenario I am not able to find any tangible input.

Read: 1,2,3,4,5....10 I will process them in loop sequentially, if the message processing succeed then only the offset should be committed.

so imagine 1,2 succeeded but 3 failed, 4,5 success, 7,8 fail... if I only commit the offsets 1,2,4,5 what will happen to uncommitted offsets? 3,7,8...

Any idea will be read back again?

If your largest committed offset is 5 and you stop your consumer (or it crashes), the next time your consumer runs will start reading from offset 6 onward. Kafka does not provide a mechanism to deal with uncommitted individual offsets between committed offsets. Put differently, Kafka has no knowledge about any commit gaps. When the offset 5 has been committed, Kafka interprets this that your consumer has consumed all messages until offset 5.

To solve the problem you are describing, your application should be able to catch the failed messages and send them somewhere else (e.g. another Kafka topic). That way you can analyze the problematic messages and in case re-ingest them into your application.

Thanks for contributing an answer to Stack Overflow!

• Please be sure to answer the question . Provide details and share your research!

But avoid …

• Asking for help, clarification, or responding to other answers.
• Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers .