Apache Kafka Crash Course

Apache Kafka is a distributed stream processing software developed by LinkedIn and written in Scala and Java.

Chapters
0:00 Intro
2:32 Kafka Broker
5:39 Kafka Producer
7:11 Kafka Consumer
8:34 Kafka Partitions
12:50 Queue vs Pub-Sub
16:47 Consumer Group
23:06 ZooKeeper
29:45 Coding Example
1:02:48 Kafka Pros-Cons
1:17:00 Summary

Download slides here
(Members get all slides for all my video courses for free)

In this video I want to focus on the basics of Kafka, talk about how it works give finally spin up a Kafka cluster and write a producer and a consumer.

Kafka Components
_ producer
_consumer
_topic
_partition
How Kafka works
Example ( Ride sharing )
Spin Kafka cluster
Write a producer
Write a consumer
Kafka Applications
Pros and Cons

Source Code

Docker commands spin up kafka
//if you ever mess up or you want to start fresh just stop and remove the containers and rerun the command

docker stop zookeeper kafka
docker rm zookeeper kafka

docker run --name zookeeper -p 2181:2181 -d zookeeper

docker run -p 9092:9092 --name kafka -e KAFKA_ZOOKEEPER_CONNECT=husseinmac:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://husseinmac:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -d confluentinc/cp-kafka

Resources
KIP 500 (Remove Zookeeper)

Become a Member on YouTube

🧑‍🏫 Courses I Teach

🏭 Backend Engineering Videos in Order

💾 Database Engineering Videos

🎙️Listen to the Backend Engineering Podcast

Gears and tools used on the Channel (affiliates)

🖼️ Slides and Thumbnail Design
Canva

🎙️ Mic Gear
Shure SM7B Cardioid Dynamic Microphone

Cloudlifter

XLR cables

Focusrite Audio Interface

📷 Camera Gear
Canon M50 Mark II

Micro HDMI to HDMI 

Video capture card

AC Wall for constant power

Stay Awesome,
Hussein

[_sr]

27:28 answer:
Every broker on a Kafka cluster is also named a Bootstrap Server. All brokers have the metadata required for the clients (producer or consumer) to discover brokers. When a client connects to one of the brokers (which are already configured as bootstrap servers in the Kafka configuration) it makes a "metadata request". The response includes information about topics, partitions, leader brokers, etc. Once the client gets this info, then - in the case of a producer- it makes the write request directly to the leader broker for that specific partition.

[jithin_zac]

I always got demotivated by the complexity I experienced when I tried to learn this on my own but this one is a gem. I know it might have taken a lot of learning and experience for you to explains this in a simple manner Nevertheless thank you for sharing your knowledge!

[bokilsbk]

Great tutorial as always. Literally a crash course fit into about an hour. Has everything to get started with kafka, especially because of the hands on.

[satalkar]

I rarely comment on videos but this was genuinely one of the best computer science tutorials I've ever seen!

[ershad]

did not feel like a tutorial, but like a friendly chat between deeply connected friends, every word makes clear sense. Kept me engaged every second till the end, yes I made to the end. Well done Sire... 👏👏👏

[manishbolbanda]

i had to deal with Kafka in one of my project, so searched for some tutorials, As soon as I saw that you have made a video on this, i had a sense of relief that i dont have to worry about the resource atleast. Thanks for all your efforts man.

[mubashirimtiaz]

I just wanted to express my heartfelt appreciation for the amazing Kafka crash course, Your dedication to simplifying complex topics shines through, and I can't thank you enough for making Kafka so much easier to understand.

[zmm]

how do you only have 23k subs?? you're so good at explaining difficult concepts, tysm!

[talkohavy]

I didn't know you before this video, but after this video? I'm gonna watch everything you upload man. Every tutorial you're selling? I'm buying! Obviously I liked & subscribed.

[HirschDaniel]

Thanks, I feel like I learned everything I need to know about Apache Kafka from this video. Sure, there are more advanced concepts. But this is the essence.

[aatifbangash]

Another perfect tutorial. Everything is explained very nicely. Thanks for making Kafka easy to understand in this tutorial.

[luqmansen]

I just watched your RabbitMQ crash course and at the end of the video you mention how Kafka better at some point, then I bumped into this great video. Very clear explanation, love your video!

[asht]

Dude, I made it to the End. You kept us engaged throughout. I have soo much to learn about Comp Sci!

[alexsiuwh]

this is the only tutorial i found that go in the depth and details of a pro should know, and i can compare Kafka with other messaging system like JMS and MQ. Thanks so much for sharing👍👏🙏

[mikejeffery]

This was an awesome tutorial. You're incredibly good at explaining things and keeping people engaged when consuming so much information.

[stmasanti]

One of the best tutorials i’ve seen. Great class and very funny. Subscribed and waiting for more content!

[DestinyAmbakederemo]

This is well detailed even for a beginner, I completely understand whats going on. Brilliant content.

[hnasr]

Course Table of Content time codes: (EDIT you should also see youtube chapters)

Download slides here

Kafka Broker: 2:32
Kafka Producer: 5:39
Kafka Consumer : 7:11
Kafka Partitions : 8:34
Queue vs Pub/Sub 12:50
Consumer Group 16:47
ZooKeeper 23:06
Example 29:45
Kafka Pros & Cons 1:02:48
Summary: 1:17:00

Example time codes
Spin up Zookeeper 31:37
SPin up Kafka 34:00
Connect to Kafka (40:30)
Create Kafka Topic (45:30)
Create a Producer (49:44)
Create a Consumer (55:24)
Final test (1:00:00)

[rubensingh]

bro your tone when explaining these videos are so good. Didn't even feel like someone was teacahing me. It was as if a friend was talking to me lol. Good job man. Your excitement is intoxicating. :)

[MOHDSALMAN-sjzu]

I made it to the end. :P Thank you so much for the amazing content Hussein.