Kafka Connect Tutorial For Beginners | Zero to Hero

#KafkaConnect #Kafka

Transcript:
If you are a beginner in Kafka, things become very challenging to learn sometimes, but this video we will explain everything in very simple Manner.
At the end of this video, You will have a clear picture of how to get started with your project.

So, we will start with what is kafka connect?
How kafka connect works?
How can you scale up or down kafka connect?
We will also learn how to use kafka connect transformations.
And finally we will learn about how the architecture is designed for Kafka connect.
We will also make a sample project on how to get started with kafka connect from scratch, but thats the scope of next video.
First lets see why Kafka came to picture. Kafka was originally developed at LinkidIn to solve the Data Integration problem.
Here is a diagram that shows a the structure where data is flowing from many source systems and many target system. The point is e very straightforward. You cannot design a single application that does everything in an enterprise.
Because that is practically impossible and not economical.

Some of them are custom designed and developed running in-house. And other maybe third party application.
Applications might be running outside of the organisation boundaries and maintained by partners and service provider. you don't have any direct access to their system data and these data may be available on demand or as feed.

For Example,
Finance Service may be requiring data from invoicing system.
Inventory service may required data from WareHouse or Shipments Service Systems.
Your analytics service will require data from all of your enterprise applications.
And at some point of time you to create so many data pipelines that it becomes impossible to visualizen and maintain.

This is the problem we are trying to solve and this is where Kafka comes into picture.

have a Look at the example here,
Let's assume that we have and invoicing application here, and it is connecting with backend database.
Now we have requirement to move data from this database to a snowflake datawarehouse . How do we do that?
We look for many options and we find that Kafka ecosystem is very helpful for this purpose because it keeps our data integration very simple.
Bringing the data from invoicing application to kafka cluster is a one-time activity. once you have data is in kafka cluster you can easily send it to Snowflake datawarehouse.
If you want to move the same data to other applications, you can very well consume from kafka cluster. All these pipelines are going to be one to one link. Right?
Now let's assume what will happen if we remove the kafka cluster.
Now you will have to build one to many pipelines. And I am sure that your invoicing application will not be able to handle the load due to each connection. So till now we know that we are going to use Kafka Broker.
But how do you bring your data from Invoicing application to Kafka cluster?
Do you want to create a producer?
Let's try and evaluate that option.

Slide 8:
There are two ways in which you can create producer depending on the source code availability of the source system.
If you own and have the source code of your invoicing system, you can directly e write the producer code in the invoicing system and send the data directly to kafka cluster. Its Practically possible and very straight forward.
Lets have a look at 2nd Scenario, Where Invoicing application is a third party application and you do not have access to the source code. You can not write a producer in someone else code right.
In that case,
You can write your own producer, which reads the data from invoicing application, then send it to kafka cluster.
Both Options are good, you can evaluate and go with the best option that suits your business.
but, If You are going to settle for the second option then you are going to solve a problem which has already been solved for you.
Slide 9:
Yes, you are right
That's where kafka connect comes to picure.
So Kafka connect is a system which is placed in between your cluster and source application, then all you do is to configure it to consume data from the source system and send it to the kafka cluster.

Support Us by Connecting

kafka connect
source connector
kafka connectors
snowflake kafka connector
kafka connect jdbc
kafka connect docker
kafka connect elasticsearch
confluent kafka mongodb connector
confluent cloud managed connectors
confluent bigquery connector
confluent jdbc source connector
confluent http sink connector
confluent kafka connect docker
kafka jdbc sink connector
confluent cloud connectors
kafka snowflake connector

[pcl]

One of the best and quickest explanations of Kafka Connect that I've found. Good stuff.

[kavitakumawat]

You explained this very well.🔥👌

I think everyone who saw your first vedio completely are still wsiting for second one😶

[mahidharkadiyala]

Hi bro, good explanation
But we all are looking for the video for how to set up the Kafka connect practically with step by step, whether it is in Apache Kafka or confluent.

Could you please help us, hope will get the video asap

[hariramk]

Hi sir, the video good and helpful why can't make a Realtime video on Kafka connect. I'm eagerly waiting for it.

[nishaahuja]

whats in it ? Just the theory...there is no video to tell the example of kafka... it is said there is next video but tere is no one.