Apache Kafka: a Distributed Messaging System for Log Processing

Apache Kafka is a very popular distributed event streaming system. It's most popular use cases are message sending and event processing.

In this video, we dive into what Apache Kafka is, how it helps engineers scale their messaging systems, and how it works internally. We will be referencing the 2011 paper by Jay Kreps for this video.

Common design patterns with Kafka include publisher-subscriber and event sourcing. If you are new to Kafka, or looking to understand its internals, this video will be useful to you!

00:00 What is Apache Kafka?
01:47 High-Level Design
04:36 Scaling Kafka
07:45 Message Batching
08:40 Atleast Once delivery
09:40 Atmost Once delivery
10:07 Exactly Once Delivery
12:45 Zero Copy Messaging
15:12 Thank you!

#Kafka #ResearchPaper #SytemDesign

[NiksChavan-rq]

I am liking this channel, watching this for fun... Mainly because there is no fancy stuff . Its so natural that it feels like I am learning from my friend.

[ch]

One of the best explanation i have come across

[kidkool]

Wow I was just out of college when I first found your channel.

I’m a fresh senior now preparing for interviews, great video!

[ashutoshsamantaray]

zookeeper is no longer used in the latest version of kafka. The newer version is Kraft using consensus protocol among brokers. Makes it more scalable

[tuhindey]

Great Explanation for beginners also.
Thanks for sharing and keep this up🙏🙏

[tayyabachowdary]

Have been following you since long and i am so impressed with the consistency and humility. Yoir explanations are simple and you seem like a college mate we are getting to learn from.thanks for all the free content

[sunnykumarsingh]

Such a simple yet elegant video Gaurav!

[sarwajeetbharti]

Great Explanation with lovely animation !Thanks Gaurav Bhai !

[TheCodingChamelion]

Tomorrow is exam and this video was saviour for the kafka topic

[rajkumar-insk]

6:58 If I remember correctly, ZooKeeper uses the Zab (ZooKeeper Atomic Broadcast) protocol instead of Paxos for consensus. Starting with version 2.8, Kafka no longer relies on ZooKeeper for leader election. Instead, it uses KRaft (Kafka Raft), Kafka’s implementation of the Raft consensus algorithm, for managing metadata and coordination internally.

[Engineerhoon]

I haven’t received my Amazon delivery yet. What might be the possible reason?

[SaiBharatBatchu]

It’s a great video. It covered the features of kafka well. I feel the way you explain a problem and explain how to overcome it might not be obvious to the listeners that kafka automatically takes care of these.

For eg, you said multiple consumers might consume a message from the same influencer topic leading to notification duplication, and you explained consumer grouping as the solution to it. But you might have missed to explain that it is not actually possible, any extra consumers more than one active on a partition sits idle and kafka takes care of it automatically. Viewers might think that as some inherent problem with kafka and they would be required to handle it. Apart from these minor things the video is very good. Thanks for making it!

[priyankavaidya]

Damnn this was such a great explanation dude am so proud that we are also from the same undergrad school Fr CRCE. Thanks dude you're a savior! 😎

[varunv]

Awesome! The best kafka internal architecture video

[WinnersDontWin]

Been trying to study this for the best to weeks, and this is by far the best youtube on the topic

[NikhilKumar-oymx]

I have worked on sqs n sns but this helped me understand that concept
Thanks a lot 🎉🎉

[bharathraj]

Intresting bro looking forward to work in Kafka... Soon i am going to complete my HTML class 🎉

[uioehqsakjlc]

10:39 Consumer Group has nothing to do with Exactly Once Delivery. Am I missing something?

[sojanmathew]

Hi @gkcs, it looks Kafka partition leader handles both read & write.
Write Acknowledgments: The level of acknowledgment (acks) from the leader to the producer depends on the producer configuration:
acks=0: The producer does not wait for acknowledgment.
acks=1: The producer waits for acknowledgment only from the leader.
acks=all: The producer waits for acknowledgment from all in-sync replicas (ISR).
[ Read ] Consistency: For strict consistency, consumers can be configured to only read from committed offsets (data replicated to all in-sync replicas).

Also newer version of Kafka (2.8 onwards) introduced KRaft (Kafka Raft) as replacement for Zookeeper to simplify operations & system design.

[satyamjha]

Amazing video as always! Really informative!