Producer Retries & Idempotent Producer in Kafka | in-depth intuition

In this video Kafka Producer Retries & Idempotent Producer concepts are explained in-detail.

Architecture:
---------------------
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
Multiple Producer & Multiple Consumer in a Kafka Topic
Intuition on Log files in kafka & Kafka Brokers
Broker Cluster and Zookeeper in Kafka
Topics, partitions, and offsets in Kafka
Kafka Cluster with Multiple Brokers
Topic with Replication in Multiple Broker Kafka Cluster
Setting up a single node Kafka Cluster on EC2
Kafka Log Segments in-depth intuition
Kafka Producer Key & Message Acknowledgements
Kafka Producer Internals
Kafka Topic Partitions & Producers using Python

Check this playlist for more Data Engineering related videos:

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

[KLNP-otrq]

Great job in explaining important components

[kamalakantabehera]

You have covered many in depth concepts of Kafka. I couldn't find how transaction is processed . Could you please create a video how to handle transactions in kafka

[shreyas-sg]

it is impossible that someone might have started you kafka series and by this point he/she would not have subscribed.

[mhaesh]

@KnowledgeAmplifier1 is there any chance in future will see apache flink tutorial . I am looking for this tutorial .
Can you please let me know any good documents or videos are available for this course.
Thanks advance

[shameekagarwal]

i can be wrong, but i feel like the two different concepts are being mixed?

i felt like you suggested that idempotence is needed for avoiding duplicate, and max in flight is needed to be set to 1 to ensure ordering

but i think with idempotence, the max in flight property is not needed to be set to 1 even for ordering

in your example when you said kafka writes first batch, third batch and then the second batch, with idempotence, kafka would know the third batch is out of order so throw an exception, so max in flight can still be a high value. but i can be wrong, not sure

[gayathri.mahendran]

Hi sir, can you please post a video for matlab program for blood pressure Estimation from ppg signal using cnn

[Rishabh]

could u please start a series on Snowflake🙏🙏

[Nexatek]

You dint explain how duplicacy check happens? That’s the most important question