Apache Kafka Producer - Python | Confluent Kafka Python | SASL_SSL | Apache Kafka Series - Part 08

For serverless Kafka solutions, please check below:

Thanks in advance :)

-------------------- Connect with me --------------------
---------------------------------------------------------------------

𝐃𝐄𝐒𝐂𝐑𝐈𝐏𝐓𝐈𝐎𝐍:
This is the 8th part of my Apache Kafka series. In this video, you will learn the following topics about a Kafka Producer practically:
1. How to write a simple Kafka producer application that connects to Kafka via SASL_SSL security mode using Python.
2. Batching the incoming messages
3. Queuing the message batches
4. Partitioning the incoming messages
5. Message Acknowledgements (ACKS)
6. Message Retry

Please check the below videos in my Apache Kafka series if you want to know more about kafka producer protocols and kafka security :

𝐔𝐬𝐞𝐟𝐮𝐥 𝐥𝐢𝐧𝐤𝐬:

Thanks for watching !!
நன்றி _/\_
#kafkaproducer #apachekafka #sasl_ssl

0:00 Agenda
1:49 What is confluent kafka API and librdkafka
3:27 Producer credentials and ACLs
6:44 Installations
9:29 Confluent API documentation
11:23 Simple Kafka producer
46:38 Batching
51:40 Partitioning
56:24 Queuing
1:02:13 ACKs and message retries
1:17:27 Summary and next video

[DataEngineeringMinds]

In part-5 where I implemented the SASL security, I created the topics via the insecure zookeeper port 2181. However, moving forward, we need to create the kafka topics by using the parameter --bootstrap-server rather than --zookeeper for two reasons - 1) Security and 2) Removal of Zookeeper in the future from the Kafka ecosystem.

[rossivan]

Your passion to teach is commendable. I love the way you speak in easy to understand words. Great going!

[cdsiema]

man, I can't express how much I love u right now, thank you for the video.

[jagadeeshit]

1:01:22
Hi, can you please explain what is poll method in producer. In which phase of this producer protocol it will be part of? why this should be used.

If callback is available in produce method, do we need to use poll again in the producer. Bit confused and not clear with information on documentation

[esakkisundar]

Regarding message queue, in your video at 1:01:39, how come the queue got full during the first two iteration itself and went into the BufferError block and the queue size is 2.

[SaddamBinSyed]

Hi @
Data Engineering Minds, Thanks for your amazing work, as stated, Did you uploaded the next video in this series (Kafka-consumer API)?

just curious ...

thank you

[kinggamar-pymx]

Good jod!!! Excellent explain!!
Could you please do video on java kafka??

[vipinsharma-jtqt]

where to find ca.cert for MSK in AWS and how to pass those configuration to producer and consumer? Thanks

[lexh]

excelent content !, very clear, im pretty new in kafka, so im trying to understand some best practices, so if i want to pass messages to the topic when only new data arrived from my storage or database, do i need to use something like kafka connector for watching changes in real time and send the new data to the topic or maybe create some lambda function or something like that, ? whats the best approach for sending streaming data to topics?

[surendramullapudi]

hi Thanks for the video. we are using confluent kafka on GCP. what should be the location

[sudhakararuchamy]

Watched all videos. Thanks for the effort to get the detailed and practical session. Hope it continuous further. One clarification. I'm trying to get SASL_SCRAM with delegate token using python. Unfortunately, librdkafka not yet support for the same. Not sure they have plan to support as well. Any suggestion or tweak you see!!!

[lokeshkumar]

your videos are really informative.I have one question, Is it possible to use zookeeper address to write on topic instead of bootstrap server? If yes then can you share the code snippet for this? thanks

[damodargarugu]

How to create a avaro producer in python bro and is it possible to install librdkafka local?? In windows

[junsuibantsu]

Very good explanations Vinod. Well done! I learnt a lot from it. Do you mind doing one on using Kafka JDBC sink connector to send messages from Kafka to Postgresql database? If not, do you have recommendations?

[lokeshkumar]

Please post some videos on kafka stream using python...

[souravchaurasia]

Couldn't find the part 9 of the playlist

[gudapati]

Could you please explain schema registry setup in Kafka