How to stream data from MySQL to Apache Kafka® | Kafka Tutorial

From this video you will learn how to stream changes from #mysql database to #apachekafka using #debezium connector!

💥 Subscribe for more videos about #Kafka, #KafkaStreams and #streamprocessing

== Prerequisites

docker version
docker-compose -v

===
Viktor Gamov is a Developer Advocate at Confluent, the company that makes a streaming platform based on #ApacheKafka.
===

== timecodes / chapters

0:00 - Intro
1:44 - explanation of the use case
2:39 - installing MySQL and enable Change Data Capture (CDC)
6:59 - adding Apache Kafka, Kafka Connect, and configuring Debezium connector
12:53 - validating that data flows through from MySQL to Kafka
13:44 - materialize data from Kafka using ksqlDB
19:32 - it's a wrap!

[ViktorGamov]

*timecodes* / *chapters*

0:00​ - Intro
1:44​ - explanation of the use case
2:39​ - installing MySQL and enable Change Data Capture (CDC)
6:59​ - adding Apache Kafka, Kafka Connect, and configuring Debezium connector
12:53​ - validating that data flows through from MySQL to Kafka
13:44​ - materialize data from Kafka using ksqlDB
19:32​ - it's a wrap!

[julioribeiro]

Dude, you saved me hours of work with these 2 lines:

"transforms": "unwrap",


Thank you very much.
CDC has been a pain with all those metadata, ... but now I figured out that we can rely on SMTs for some dirty jobs.

[AndreyRebrov]

Good stuff! Would be great to see Postgres, Redshift and Snowflake integrations in future.

[maham]

How to fetch flat file from Kafka and load into sql server

[toenytv]

Want to clone your brain Viktor. Lol great edit. Miss all the jazz from the livestream. Thanks and keep up the sharing we all benefit from a Db superhero.

[sujanyt]

need help
the debezium source connector disconnects from postgres DB after 15-16 minutes

[icangicung]

i have difficulties in converting timestamp and datetime from mysql

[rydmerlin]

Does the connector let you present both before and after image. Ie is it tied into the log of the db or is it just polling?

That soundtrack could get you demonetized :-)

[KaifKhan-pxov]

Where do we create the kafka topic where debezium connector will dump the data?

[JAIRTA]

thanks for sharing ... very educational and easy to understand. 😃👍

[cedriclamalle]

This one was quick but great!
Maybe one cool think would be to make a KTable from an actor topic coming from debezium too and join it with the movie stream...

[kerolesmonsef]

hello i need to show the changed columns and values only as before and after
@ViktorGamov

[gabemc]

Nice presentation - is it possible to use a connector to hit a kafka rest endpoint?

[aliakseydruzik]

Victor, what do I have to install on my Mac in iTerm2 in order to run command in style like you - 'dcupd'?

[DhananjayaKarandana]

Can anybody instruct me how to install kafka-avro-console-consumer on mac os please