Confluent Kafka - A Python Library for Distributed Stream Processing

Welcome to my video series on Apache Kafka and in this video we're going to see how we can use apache kafka in python using confluent kafka library.

Confluent kafka is a python library from confluent and is one of the widely used library for apache kafka. There are other libraries also available like kafka python, but this video is not about the comparison of the libraries, this video is all about how to use confluent kafka to interact with kafka and create kafka topics, kafka producers and Kafka consumers.

Thanks for watching the video, here are the contents of the video

Timecodes:
0:00 : Confluent Kafka Python Library for Distributed Stream Processing
0:36 : Kafka topic using python confluent kafka library
1:15 : Confluent Kafka Admin Client and Newtopic
2:05 : Creating a Kafka topic using Admin Client
3:22 : Confluent kafka producer and callback function
4:43 : Confluent kafka consumer within a group and offset reset
5:25 : Polling for data in kafka topic
5:46 : Kafka topic consumer with latest offset
6:33 : kafka consumer with earliest offset
7:25 : kafka consumer groups in confluent kafka
8:27 : kafka consumer group statistics
9:35 : Multiple consumer groups in kafka

#kafka #apachekafka #eventdriven #softwaredevelopment #programming #streaming

About Me i.e. An Insightful Techie

Greetings and Thanks a lot for checking out my YouTube Channel. I’m Deepak K Gupta (Daksh). This channel is a medium for me to share my learnings and journey as a Techie so that you not only learn from my experience but also from my mistakes

As a techie I am closely associated with software development, so most of my content will revolve around Programming Languages, AI, Machine learning and Databases. I share things which will help you learn, remember and use these things in an effective and efficient way.

As a Techie, I also know that there is a life beyond software development and it has a profound impact on our career as well as on our well being. I also share those learnings which I feel are worth sharing with the intention that it MAY help you in taking better decisions in your life.

Last but not the least, I'm a traveler and explorer by nature. I would like to take you with me on some of the interesting journeys around the world. Believe me there is more to learn outdoors than indoors

Hope you’ll like my contents and will be part of my journey 🙏

[peterboxberg]

Very well presented! I hope you continue, maybe with the topic "schema validation on consumer and producer side"?
Consideration of the different protocols (avro, json, protobuf)....

[renatogustavocarneiro]

Great videos, thank you for sharing this knowledge.

[beto.]

What an amazing explanation!!!
THANK YOU!!!

[vardhamanjain]

Hi great tutorial, please create more tutorial on what more can be done using confluent

[bradydyson]

I'm encoding with utf-8 when producing, but the consumer is not able to decode with utf-8 because of some invalid bytes ("'utf-8' codec can't decode byte 0xff in position 10: invalid start byte"). It seems like something is happening to the data between the producer and consumer. How is that?