Set up Streaming Data Pipeline to ingest Kafka Data To Snowflake with a Python Produce from Scratch

The aim of this short tutorial is to get the viewer up and running with Snowflake Connector for Kafka.
The Snowflake documentation does a good job of explaining the concept/working of this connector. This video can be used in conjunction with the documentation.

Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
Building Serverless Data Stream pipeline using Kinesis data streams and Firehose for Snowflake
Key Pair Authentication in Snowflake for Linux machine from scratch
Introduction to Kafka Connect

Note:
---------
As this is POC , so the kafka connect is running in same cluster where our broker is running , but in prod env , the kafka connect cluster runs in a different environment , not in same cluster where our kafka brokers are running.

Kafka Snowflake Integration:
--------------------------------------------------------

Check this playlist for more Data Engineering related videos:
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

[roshankumargupta]

I'm still not getting the reason to include ec2 instance here? Would be great if you can help on this. Thank you

[chamaljayasinghe]

I got no any error but data not load into DB ? why is that ?

[Kalpana-flr]

Here you have used Snowpipe internally to ingest Kafka messages.
How to do the same with Snowpipe streaming? Can we get rid of the Snowflake connector jar file by using this method?
Many sensitive data systems will not allow to place jars in their lib folder, so is there a way where in we just configure the properties file and achieve Snowpipe streaming?

[chrisbrad]

Please provide notepad file used in this . I am not able to find it.

[gopinath]

Can we do the SSL connection through WSL(Windows Sub system for Linux)? instead of launching EC2 instance

[roberbonox]

Hi if i'm getting avro messages in my kafka topic and loading this avro messages in a snowflake table (into two columns RECORD_CONTENT and RECORD_METADATA), how can i translate this avro files into a columnar table? using flatten function? thanks

[KaziNayeem]

Hey Ramu, great playlist. I was able to get started on Kafka locally.
I am still trying to figure out how I can read files from a folder into a Kafka cluster . Are you able to shed some light on that please ?

[itzharshu]

Just one questions i have
Looks like in this case kafka connect is running on broker node (in this case your local windows machine) and is this possible to run kafka connect away from broker ?

[lamaguilar]

i have a question ? can I insert in multiples snowflake. schema with one connection or how can i create multiple connections to the same database but differents snowflake.schemas? thanks

[pankajkumarbhoi]

can you make a video for the reverse pattern ? I mean to say, from snowflake to kafka. CDC from snowflake to kafka queue.
Thanks

[rameshwarpawale]

Thank you for the video. It looks like Snowflake is using Snowpipe to move the underlying data. Is it possible to use Snowflake Streaming Ingest SDK for Python to achieve the same?

[shasmeenhameez]

when connecting with putty i get fatal error .. can you pls help me

[padmalankipalli]

Can u share for mssql to kakfa connectoe demo pls

[__gangstr__]

Thank you for the video. it was very informative. 🙏

[almaraya]

Thanks for great tutorial. Question: How do I save my messages to a postgresql database? Is it possible to create the messages via a POST method using POSTMAN?

[linuxier]

instead of AWS instance, we can try with windows DSL

[bacchemankesacche]

Your playlist of KAFKA is really awesome, can you make the same for streaming data pipeline for other db such as mysql, cassandra, Thanks again😃

[KishanKumar-fdon]

Hi
I am getting errors while running the connector.
ERROR [SF_KAFKA_CONNECTOR] Validate Error msg:[SF_KAFKA_CONNECTOR] Exception: Failed to prepare SQL statement
Error Code: 2001
Detail: SQL Exception, reported by Snowflake JDBC
Message: SQL compilation error:
Object does not exist, or operation cannot be performed.
Caused by: Connector configuration is invalid and contains the following 1 error(s):
Snowflake.database.name database does not exist

But the database is there.