Understanding Kafka partition assignment strategies with in-depth intuition & Practical using Python

This video explains the Secrets of Kafka Partition Assignment Strategy across multiple consumers within same Consumer Group.

Prerequisite:
------------------------------
Consumer & Consumer Group in Kafka
Apache Kafka Rebalance Listener & Implementing a database sink for Kafka
Kafka Consumer Groups CLI Demo | Kafka-Python
Manual Offset Commits & Exactly-Once Once Processing in Kafka Consumer using Python

Code:
------------

Producer Code:
---------------------
from time import sleep
from json import dumps
from kafka import KafkaProducer

def custom_partitioner(key, all_partitions, available):
"""
Customer Kafka partitioner to get the partition corresponding to key
:param key: partitioning key
:param all_partitions: list of all partitions sorted by partition ID
:param available: list of available partitions in no particular order
:return: one of the values from all_partitions or available
"""
print("The key is : {}".format(key))
print("All partitions : {}".format(all_partitions))

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'),partitioner=custom_partitioner)
topic_name='hello_world'

for e in range(0,100):
data={"number":e}
sleep(1)

Consumer Code:
----------------------

from kafka import KafkaConsumer
from kafka import TopicPartition , OffsetAndMetadata
import kafka

import json

class MyConsumerRebalanceListener(kafka.ConsumerRebalanceListener):

def on_partitions_revoked(self, revoked):
print("Partitions %s revoked" % revoked)
print('*' * 50)

def on_partitions_assigned(self, assigned):
print("Partitions %s assigned" % assigned)
print('*' * 50)

consumer = KafkaConsumer(bootstrap_servers=['localhost:9092'],
group_id='demo112215sgtrjwrykvjh', auto_offset_reset='earliest',
enable_auto_commit=False,partition_assignment_strategy=[RoundRobinPartitionAssignor])

listener = MyConsumerRebalanceListener()

for message in consumer:
print(message)



Learn Apache Kafka form scratch

Check this playlist for more Data Engineering related videos:

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

[bharathreddy]

Thanks for your great efforts and nice explanation, Please make a video on Kafka monitoring tools Prometheus & Grafana

[chenguo]

Thanks for the lecture! In your explanation, either for the range or round robin assignor, the order of consumers plays an important role here. Can I assume that consumers are always ordered by the time it is created? For example, consumer 1 is always created earlier than consumer 2 and consumer 2 is always created earlier than consumer 3.

[lifeitnow]

Hi, @Knowledge Amplifier, Thanks for all the amazing content!!! My request for the content is to on-premises data migration from oracle to snowflake using AWS. Can you please create any such content, or if you already created plz share the link for the same.

[chamathjayasekara]

Hi KnowlegeAmplifier1, I have issue with consumer group member where it has only 2 members out of 3 brokers I have, sometimes only one member, so when the cordinator broker goes down that consumer group is useless, how do I fix this?