Kafka Consumer and Consumer Groups Explained

Learn how the data is read in Kafka! Understand how Consumers work and how to use them!
Get the Learn Apache Kafka for Beginners v3 course at a special price!

Don’t forget to subscribe to get more content about Apache Kafka and AWS!

I'm Stephane Maarek, a consultant and software developer, and I have a particular interest in everything related to Big Data, Cloud and API. I sat on the 2019 Program Committee organizing the Kafka Summit. I'm also an AWS Certified Solutions Architect, Developer, SysOps Administrator, and DevOps Engineer.

Follow me on social media:

[sathyanarayanano]

I really got that diagram into my head... Thanks

[ravikirandesai]

Nice explanation by Stephane, remember, visual memories are strongest. So he ensured to explain with diagrams.

[rafapiotrowicz]

Really great video. Finally I saw it. I have this question, if I can. If I have listener applications with a given group id and 4 brokers, will each instance read from a different broker (partition) in parallel after autoscaling to 4 instances of my application?

[kms]

great thought of storing consumer offset👍

[tyapka]

I think it would be more accurate to say that Kafka stores offsets until which a partition has been read by any consumer in a consumer group. It does not really care about an identity of particular consumer.

[ananthakrishnankrishnan]

If I just write one consumer, then can I get multiple instances of that consumer to attach to different partitions? Also if I attach a consumer to just a topic is it by default bound to a partition or is it a slave to all partitions (since one consumer can read from multiple partitions)?

[madhusudhann]

Thank you for this nice video Stephane! I understand that each consumer is a separate independent process that reads from one or more partitions. No two consumers from the same group will read the same partition. So to achieve a parallelism of 5, should I create and run 5 consumers, all grouped under same consumer-group?

[rrsruji]

4:04 who will take care of committing the consumer_offset? Looks like point 3 days consumer does the commit. But, if in case there is a bug in the code that doesn't let consumer commit, would it be reading the same message again and again or it'll just keep on waiting for new message past the current offset?

[GAURAVGUPTA-zubu]

How can I create multiple consumer instances within a consumer group?

[pradeepgr]

Hi Stephane, Need your expertise here. I'm stuck in my last step of my assignment.
1) Create a kafka consumer to consume messages from topic 'topic-1'
2) store them in '/tmp/kafka-messages' .
Can you please assist in the step 2. Appreciate your help on this. Thanks

[vishalmishra]

i have a doubt can two consumer grp read data from same partition

[sarathbaiju]

hi, i tried to set up a consumer. it's worked but after sometimes i got error like "consumer instance not found.". how i can troubleshoot this type of senerios?

[rachanahm]

How to find the group Id of a kafka topic if no separate consumer groups are created.
what is number in consumer-offsets-(Number)

[annalisetrite]

The offsets are nice but what if some consumer app has just read a bunch of data and fails while it's processing it...how can we guarantee no data loss? Is there a mechanism to control read acknowledgement similar to write acknowledgement?

[rag]

Zookeeper never comes into picture while taking care of these consumer offsets?

[Hepward]

how kafka identifies single consumer? for example consumer goes down, 5mintues later hes up so we suppose to continue where he finished last time but how kafka knows which consumer it is and if he even been connected previously? by ip? what if there are two consumers on single machine - so ip+port?

[GenerativeAI-Guru]

"Messages are read in order within partition, but they read parallel across partition"

[markusjackson]

Bro you have to speak more cleary....open your mouth.