Real-Time ETL (ELT) with Kafka connect; change data capture from mysql to sql server.

This video demonstrates the power of kafka connect; using built-in connectors to perform incremental load (CDC). It shows how to extract and load data with Kafka Connect,Confluent Platform.
It showcases how to pull data (source) from a mysql into Kafka, and to push data (sink) from a Kafka topic to a sql server.

GitHub link for connector and mysql code:

Confluent kafka download:

Confluent kafka installation:

Kafka connect mysql source example:

Learning Journal kafka overview:

Kafka connect deep dive - JDBC source connector:

[jonnysanty]

*dances* Thank you! This video convinced me to attempt kafka setup.

[masttubefun]

Hi Sir, it's a very usefull and informative video. It's very detailed explained in the vidoe. One question could you let help with same with SQL server ?

[neutralcommentor]

What is the difference between Debezium and Kafka Connect? If Kafka needs Debezium for CDC, why is Debezium not a part of Kafka itself, yet?

[chasedoe]

From what I understand, Kafka connect does not provide a delete action.
With that said, I never know that it can handle schema change.

[mumair]

Hi Haq Nawaz,
I am very new to Kafka. I have an external database (Third party) in oracle, and on my premises there is a Warehouse database in SQL Server. Where do i need to start setting up kafka so that the data is read from the Oracle Database into the Azure SQL Server database.
Also i would like some aggregates from some tables instead of reading records directly, i would like to have aggregated values. Is it possible ? Please Help.
Thanks

[ytw_tv]

How about multiple table (source) to single table (sink). Can Kafka connector handle it ? Thanks

[ML_Enthusiast]

Thanks for your amazing tutorials.

I have followed all your instructions, other than the Docker installation step, which I had to run the consumer console in the connect container. All the parameters are double checked but still right after any record incremental, no messages appear in the console, besides no record in the sink database. PLZ HELP!

[kamranahmed]

after the successful sink connection, the bulk data is kept inserting and loading the duplicate data. how to stop duplicate data load?

[Smile-elyy]

Bro could u please let me know which one is paid one which is open source freeware

[mohdfaraz]

Sir I am using Azure function kafkaTrigger to read data from Kafka topic and load it into SQL server

Its taking alot of time
Can you plz suggest the best approach

[pandemicplayseverything]

Why it is your kafka always insert data to target database?, even when you do update in source db, in target db it looks like always insert new data

[DanishHussain-mz]

How did you configure??? Can somone help me out on this

[tunglethanh]

Very clear and visual. Thanks!. Question: For the updating actions, can we update existing rows in the destination instead of inserting more?

[benpracht]

Hi Haq,
Thank you for this concise video. I'd like to use Kafka in my work project against a source table that has no primary key. How does the connector distinguish a new record from a modified record in this case.
Many Thanks,
Ben

[JP-dtem]

Could you please make a tutorial on how to stream data from azure data lake to neo4j

[abusaidsarkar]

Is it possible to change data capture from sql server to sql server.

[Rajnish-lk]

Hello Nawaz,


Can you please share source code for this ? Have you written it in Java or Python ?

[Rajnish-lk]

is Confluent platform free for doing ETL ?

[SpiritOfIndiaaa]

#BIInsightsInc thanks a lot, i tried but unable to connect the PostgresSQL from connector, where as i am able to connect from pgAgent ...error -X POST -H "Accept:application/json" -H localhost:8083/connectors/ -d @cdc_deb.json
{"error_code":400, "message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.\nYou can also find the above list of errors at the endpoint



Reply

[SpiritOfIndiaaa]

#BIInsightsInc thanks a lot, i tried but unable to connect the PostgresSQL from connector, where as i am able to connect from pgAgent ...error -X POST -H "Accept:application/json" -H localhost:8083/connectors/ -d @cdc_deb.json
{"error_code":400, "message":"Connector configuration is invalid and contains the following 1 error(s):\nError while validating connector config: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.\nYou can also find the above list of errors at the endpoint