Apache Kafka 101: Partitioning (2023)

Learn how partitioning works in Apache Kafka. With partitioning, the effort behind storing, processing, and messaging can be split among many nodes in the cluster.

- - -

ABOUT CONFLUENT

#kafka #kafkastreams #streamprocessing #apachekafka #confluent

[vinod]

finally i found the partition logic in this video. Thanks a lot for the crisp video tim.

[Neosam]

man your style of explanation is just awesome..I mean how can you explain things so easily that too without much animation or something...one of the best instructor I must say

[philipackerley]

Yet again Tim, rock solid short snappy overview.

[azharmobeen]

It's simply a great explanation. Thanks Man

[Sulerhy]

incredible visualized video. Thank you so much

[unny]

Thanks Tim.. This is best video over internet for those who just jumped in to Kafka....

[yang]

When consumer read the topic, how does it know which partition to read the message out?

[matifibrahim]

Thanks to animation in this video now I better understand partition

[aditya-desai]

What is the relation between events and messages?
Messages have key-value pairs? or Events have key-value pairs?
what exactly ... Events or messages are stored in partitions with key-value pairs?

this was a helpful video
Take key HASH Output mod (Total no. of partitions) Resulting no. is Partitions number where message going to store.
I was unaware of this concept.
Thank You Tim.

[facusana]

what if we want to store information for each day in the NASDAQ stock market with ~3000 symbols and 1 billon of trades per day. Should we use one topic for each symbol or just one topic with handred of partitions?
I want to understand a real case with a hugh amount of data!

[ireallylikeshoes]

Excellent explanation, but one thing that I see just about all videos lack is explaining WHY partitioning is useful and when it is not.

[cuongtranchi]

hi, thank you so much for this nice course.
Might I have one further question. Does the ordering and key mechanism also work for difference topic.
Suppose that I have 2 topics A and B. There are 2 partitions with 2 consumers for these 2 partitions. Each consumer consumes both topic A and B but in difference partition.
When I produce message 1 to A and 111 to B with the same key AAA. Does it can be stored in the same partition [0] or [1]?
When I produce message 2 to A and 222 to B with the same key BBB. Does it can be stored in the same partition [0] or [1]?

If the messages 1 and 111 are not consumed by the same consumer then it is a problem since one of them can be consume by other consume which can be consumed earlier it should be.
In this case we expected the message 1 should be consume before and then 111. But the 1 and 111 can be consumed at the same time by 2 consumers if the ordering does not work for the difference topic.

Thanks a lot!

[glennbullock]

When I subscribe to a partitioned Topic, I still get all the messages eventually, just not necessarily in the correct order, right?

[alen]

so it is generally correct, if not always, to say that messages in different partitions within same topic are mutually exclusive?

[lianglu]

If the number of partitions changes, does Kafka re-hash and redistribute events? If not then events with the same ID could end up in different partitions?

[Betty_]

Pessimistically, because fund transfers always go without a hitch, right?

[padmanathanramasamy]

Do we need to mentioned the number of partition while creating the topics?
For example if I create the key with CustomerID and create the Kafka topic with 5 Partition then when there is a 6th Customer comes with different Key then which Partition will stored?

[kamal-xdid]

What if one partition node goes down, then the order can be messed up?

[victorsterling]

Why don't use consistent hashing but use hash/ num_partition?

[nimasarayan]

What's the name of this song? ps: Kafka is amazing