Snowflake Snowpipe streaming, Kafka setup with python why Kafka, snowflake real time data ingestion

This video is the part 1 of Snowpipe streaming implementation demonstration. In this vide we will complete the setup of kafka and python and will see how kafka works in real project scenario. In the part 2 we will use this setup to ingest the kafka topic events into snowflake in real time.

Adding below is the kafka setup file i used.
-------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------

Set up Kafka --

Download setup and extract

Create logs directory
------------------------------------------------
kafka_logs-- zookeeper
kafka_logs-- server_logs

------------------------------------------------------
dataDir=E:/kafka_logs/zookeeper
maxClientCnxns=1

----------------------------------------------------
uncomment listeners

Start Zookeeper: : Ensure Java installation
---------------------------------------

--ERROR : The system cannot find the path specified.
solution : $env:JAVA_HOME=
set JAVA_HOME = ""

Start Kafka-server:
-----------------------------------------

Create topic:
------------------------------------

Start Producer:
--------------------------------------

Start Consumer:
-------------------------------------

kafka-python installation:
--------------------------------------------------
pip install kafka-python

Python Code:
----------------------------------
from time import sleep
from json import dumps
from kafka import KafkaProducer

topic_name='hello_world'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'))

for e in range(500):
data = {'message counter=' : e}
print(data)
sleep(2)
-------------------------------------------------------------------------------------------------------------------------------------

#snowflake #snowflaketutorial #snowflakedeveloper #snowpipe #snowflakeStreaming #snowpipeStreaming #python #kafka #database #datascience #dataengineering #tutorial #dataintegration #datawarehouse #datawarehousing #realtime #dbt #datamodeling #datamodelling #streamlit #plsq #oracle #postgresql

[ashdbms]

Wonderful !!! Nice and simple explanation.

[abhilashmathew]

Good Explanation.. Thanks for the video.. where can i see the part 2 video