Apache Kafka Key Terminologies and Architecture Internal Working | Features | In Depth Explanation

In this video of code decode we have kafka architecture which is very important in terms of kafka interview questions and answers

Udemy Course of Code Decode on Microservice k8s AWS CICD link:

Course Description Video :

Kafka implementation with producer and consumer :

Topic — A bucket of messages (or events in case of kafka) where services may place or read messages from. Messages are organized and durably stored in topics. A topic is similar to a folder, and these messages are the files in that folder. Topics are multi-producer and multi-subscriber. Messages can be read as many times as needed, unlike traditional messaging systems, msgs / events in kafka are not deleted after consumption. Instead, you can define for how long Kafka should retain those events/msgs.

Kafka topics are divided into a number of partitions, which contain msgs/events in an unchangeable sequence. Each msg in a partition is assigned and identified by its unique offset. A topic can also have multiple partition. This allows multiple consumers to read from a topic in parallel.

In Kafka, replication is implemented at the partition level. The redundant unit of a partition is called a replica. Each partition usually has one or more replicas over a few Kafka brokers in the cluster.

Every partition (replica) has one server acting as a leader and the rest of them as followers. The leader replica handles all read-write requests for the specific partition and the followers replicate the leader. If the lead server fails, one of the follower servers becomes the leader by default.

When a producer publishes a msg to a topic, it is published to its leader. The leader appends the msg to its offset and increments its msg offset.

Summary
Kafka has a sized component called a cluster.

Inside a cluster, we’ve got several servers, also known as brokers (usually at least three brokers to provide enough redundancy.)

when a message gets sent to a broker, it gets sent to a particular topic.Topics allow us to categorize data. The data can eventually get broken into several partitions.

By splitting the topic into multiple partitions, we’re facilitating scalability, as each partition can be read by a separate consumer.

Each broker is responsible for receiving messages from producers and committing those messages to disk. The broker is also responsible for answering consumers’ fetch requests and serving them.

While designing a Kafka system, it’s always a wise decision to factor in topic replication. As a result, its topics’ replicas from another broker can solve the crisis, if a broker goes down. Broker1 has Topic 1 and 10 partitions , its replica is in Broker2, so on and so forth. It has got a replication factor of 2; it means it will have one additional copy other than the primary one. It is not possible to have the number of replication factor more than the number of available brokers.

Moreover, exactly one consumer instance reads the data from one partition in one consumer group, at the time of reading.

Since, there is more than one consumer group, in that case, one instance from each of these groups can read from one single partition.

However, there will be some inactive consumers, if the number of consumers exceeds the number of partitions. Let’s understand it with an example if there are 8 consumers and 6 partitions in a single consumer group, that means there will be 2 inactive consumers.

Kafka Cluster: A Kafka cluster is a system that comprises of different brokers, topics, and their respective partitions.

Subscriber and Follow Code Decode

#kafka #codedecode #kakfainterviewquestions

[sudhiradlakha]

You are doing a Spectacular job by providing these video tutorials. I wish you may achieve greater heights. May God bless you always!!
Looking forward to more practical sessions on Kafka series, Kafka Stream APIs and so on.

[abcdefgful]

I just love the way you teach, very well explained, a series of great tutorials. Thanks a lot.

[saigirindra]

I really like the way you teach keeping it simple but effective. Keep up the good work and continue doing the same. I would like to ask you to add a playlist with interview questions for Microservices. It will greatly benefit large number of people

[johnypremkumar]

Please cover micro services design patterns completely.. humble request from loyal fans😀

[AshishYadav-sedb]

Its a great learning again with code Decode, Thank you so much for your effort🙏

[rameshnonia]

Speechless... Superb Explaination for this tough part

[RAJANKUMAR]

thank you so much for this wonderful video

[johnnybreathe]

Yes interview questions 🧑‍💻 and some water lol just kidding. This a big topic to go through wow. Thanks for the big effort 🤗 and the videos.

[gbcyywp]

I have an interview for Kafka, Springboot based role.
After seeing this video I got the confidence to crack the interview.

[purnendushekharpatra]

Hi.... Its was a great learning with code Docode, Thank you so much for your effort

[dipaliyadav-koli]

Best explanation as always .. 👌👌👌 Thanks a lot for creating such a great video. It’s very informative… 🙌

[novagoyal]

Thanks very well explained. You are doing a great job 👍

[johnypremkumar]

way of explanation is short and simple...voice is unique and cool... I want to thank you so so much.... But please cover full cover java, Springboot, spring core, full length videos like 2 hour or 3 hour deeply, one of the underrated channel

[sandeepradhakrishna]

A very good tutorial, thanks a lot for the clear explanation

[soulfulGirl_]

Your teaching is too good please cover more on Kafka, spring boot, microservices interview related senario based questions for experienced it's a humble request 😊, keep up the good job I am totally enjoying your video's 😊

[aditisingh]

Thanks for all your efforts.
Please create video on kafka just for interview purpose, it's a much needed one .

[tejeswarpala]

thanks for the detailed explanation.
but i have few doubts coming like what if i have 2 brokers but replication factor is 3.
then 1 partition will be on 1 broker and other 2 partitions will be on another broker ?
Please cover these combinations in next video on kafka from interview perspective.

[tanveersyed]

Nice explanation ....thanks for uploading

[rakeshpramanik]

Nice one .. Can you please make next video of Kafka current Challenges like if Cosumer throws exception how it can restart from particular offset where it lefts or skip it ..configuration changes reqd

[KaranSingh-rudl]

Hi Ma'am, please make video on the implementation of Active MQ and Rabbit MQ also. Thanks for these videos and please continue adding videos in this microservices series. :)