buffer.memory , max.block.ms & Producer IO Thread in kafka

buffer_memory is the total bytes of memory the producer should use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block up to max_block_ms.

Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
Multiple Producer & Multiple Consumer in a Kafka Topic
Intuition on Log files in kafka & Kafka Brokers
Broker Cluster and Zookeeper in Kafka
Topics, partitions, and offsets in Kafka
Kafka Cluster with Multiple Brokers
Topic with Replication in Multiple Broker Kafka Cluster
Setting up a single node Kafka Cluster on EC2
Kafka Log Segments in-depth intuition
Kafka Producer Key & Message Acknowledgements
Kafka Producer Internals
Kafka Topic Partitions & Producers using Python

Code:
-----------
from time import sleep
from json import dumps
from kafka import KafkaProducer

topic_name='hello_world'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'))

for e in range(1000):
data = {'number' : e}
print(data)

Check this playlist for more Data Engineering related videos:

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

[AyushMandloi]

Thanks Man. From where u have learned all this ?

[shreyas-sg]

sir, i would want to do a little correction in code, that producer.flush() should come inside for loop, other wise its of no use. Thankyou !!!

[naikmshweta]

I am getting BufferExhaustedException, we are sending 60k request, what is best way to solve this

[olayoridickson]

Can you disable batching ? Is it required? Also Does is the buffer memory is on the client or server side?

[namangarg]

How can we check if the buffer is getting full or not?