The Ultimate Guide to CDC Setup: Kafka, Debezium, PostgreSQL, and MySQL | Enterprise level setup

In this comprehensive, step-by-step guide, I walk you through setting up an enterprise-level Change Data Capture (CDC) system. We use Amazon MSK for Kafka, PostgreSQL as the source database, and MySQL as the target database, leveraging Debezium for CDC. This 1-hour detailed tutorial covers everything you need to know, from configuring MSK and RDS to setting up Kafka Connect UI and Kafdrop. Whether you're a seasoned architect or a developer looking to scale your data systems, this tutorial is packed with insights and practical knowledge.

🔑 What You'll Learn:

- Setting up Amazon MSK (Managed Streaming for Apache Kafka)
- Configuring PostgreSQL as the source database
- Using MySQL as the target database
- Deploying Debezium for CDC
- Managing Kafka Connect UI and monitoring with Kafdrop
- Best practices for enterprise-level CDC setups
- Automate management and monitoring of CDC health using APIs

By the end of this video, you'll have a fully functional CDC setup that can handle real-time data replication in an enterprise environment. Don't miss out on this in-depth tutorial designed to elevate your data infrastructure skills!

[kasunjalitha]

Thanks a lot Krish. Please do more like this. waiting for more.... ❤🤞

[Osandawedamulla]

Quiet the explanation Thank you for your effort !!

[isurusrimal]

Hi Krish great video again. I have a request. Can you please do a SEO related video ?

[jj_pog_champ]

Thank you so much Krish, learnt a lot from you regarding architectures. Kafka is too challenging, steep learning curve . Just the thing needed CDC that too enterprise level, this is gem💎.

[essakinatarajan]

Your explanation is very helpful. However, I need assistance on how to set up filtering of messages based on a specific value.
Config params sample.
"transforms": "filter",
"transforms.filter.type": "io.debezium.transforms.Filter",
"transforms.filter.language": "jsr223.groovy",

[isurucumaranathunga]

Was waiting for this, thanks a lot Krish ❤

[razmyasdf]

@Krish Dinesh, Thank you so much for providing such a valuable and detailed insight into the CDC setup. Your explanation was incredibly helpful, and it has given me a much clearer understanding of the process.

It would be even more beneficial if you could create a simple tutorial or step-by-step guide on how to use this CDC setup specifically for implementing audit logs. A practical walkthrough would greatly help those of us looking to apply this setup in our own projects.