Apache Kafka 101: Consumers (2023)

In this video you learn about Apache Kafka consumers and consumer rebalancing. Youl also learn how to use the consumer API and the KafkaConsumer class to connect to the cluster to subscribe to one or more topics.

- - -

ABOUT CONFLUENT

#kafka #kafkastreams #streamprocessing #apachekafka #confluent

[ZFlyingVLover]

Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it.

The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.

[rtvdbcz]

I like the way Tim explains, straight to the brains

[tomaszvinys]

Wow - a totally perfect video, clearing most of my doubts in less than 7 minutes!
Thank You!

[quentinautomation]

Best Kafka educational videos out there. thank you

[sumitt]

Tim has explained kafka `simplified` exceptionally well.

[robinhopeixoto]

Great video! Any tips of:
1) how to define and monitor a good number of partitions?
2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?

[aleman]

Hello, great video. I'm trying to find a strategy to resolve transactions of microservices communicated to each other by Kafka. Do you think that is possible? Thanks for your help.

[yogeshmore]

How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?

[rk]

We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?

[gab]

where i can bought that t-shirt! it's so cool!

[chauvoluuhuong]

I have a question, because of two instances in the same consuming apply (group) will be assigned to differences of partition right ?
so could i it miss messages from partition it not assigned ?

[maxmnnit]

Number of consumers is restricted to number of partitions?

[shankil]

Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer