Install and run Apache Kafka & integration with Python using Kafka-Python

Показать описание

Set-up kafka in Local:
------------------------------------
Check the pinned comment for details

Check this playlist for more Data Engineering related videos:

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

Knowledge Amplifier

Рекомендации по теме

Комментарии

Set up Kafka --

Create 2 folders in F drive--
kafka_logs-- zookeeper
kafka_logs-- server_logs

change the zookeeper.properties:

maxClientCnxns=1

This property limits the number of active connections from a host, specified by IP address, to a single ZooKeeper server.

change the server.properties:

uncomment listeners

Start Zookeeper:

Start Kafka-server:

Create topic:

--create --topic hello_world --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

Start Producer:

--topic hello_world --bootstrap-server localhost:9092

Start Consumer:

--topic hello_world --from-beginning --bootstrap-server localhost:9092

kafka-python installation:

pip install kafka-python
(To know more about this client, you can refer this link :

Python Code:

from time import sleep
from json import dumps
from kafka import KafkaProducer

topic_name='hello_world'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'], value_serializer=lambda x: dumps(x).encode('utf-8'))

for e in range(1000):
data = {'number' : e}
print(data)
producer.send(topic_name, value=data)
sleep(5)

KnowledgeAmplifier

Thanks a lot. As usual THE Best. However please complete the entire Kafka series and create a playlist..Learners like me will be heavily benefited.

animeshsen

Sir
Great Knowledge and skills video channel.
So Kafka also explained with high clarity. Excellent Teaching.
Thanks So Much

vasutke

just first time getting to know ksfka. Thanks for your tutorial. Best one out there

tuonghoangtri

Thanks for covering Kafa in all the possible permutations...very informative videos as usual

moizdewaswala

Simple and straight to the point. Thank you !

AymenALITALEB

finally succeeded sir....Thankyou so much but, i still dont know how this can be practically applied on projects

shreyas-sg

Thanks for the nice courses. Just a question. I am beginners in Apache Kafka. Should I follow the playlist courses or the courses are randomly?

aminmohebbi

Great tutorials! Thanks for all your efforts!

ackshaysuruli

Thanks for this valuable contents and will you cover ssl and tls also

shahezadalam

why you are using virtual env for only producer and consumer ?

BOSS-AI-

When I run KafkaProducer in main.py, I have an issue
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'
Please give me a solution!!

HoangGiang-sryh

Let me ask a question. I have setup kafka on EC2 as you have explained in one of your video. I can send and receive message from the producer and the consumer terminal. However when I am sending message from the python producer I am not able to see the messages on the terminal. I have tried producer.flush() as well. But there is no effect. Have you faced any such issues ?

ritabratasaha

i have got an error
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'
while running the python program in main.py. Please help!!

bhaumikraj