Stream your PostgreSQL changes into Kafka with Debezium

Watch your Postgres changes stream into Kafka in realtime using Debezium! End to end example of CDC from Postgres all the way into Kafka in realtime.

In this video we go over a tutorial where we stream PostgreSQL changes into Kafka using the Debezium Connector. We go over the docker containers necessary and we demonstrate end to end how the whole thing works.

This is a real life example of how CDC works with Postgres & Kafka.

#kafka #postgres #streaming #realtime #debezium #cdc #systemDesign #tutorial #programming

[manojjoshi]

This is an extremely rich and high quality video. I stumbled upon this by chance when I was searching for some info about Debezium for some project we might be interested in. Interestingly enough, the files provided in the video description did not quite work for MAC and it took about 2 days to make the required adjustments including getting the correct images and port configuration, network configuration etc. but in the end things worked. My experience was that I learned a lot as I went through the process, not just about Debezium but a great deal of other things on what docker-compose structure, network dependency, port issues and all that. So it was worth. Kudos to Irtiza for taking all efforts to put together such as amazing tutorial....!!

[intruderstube]

Thanks, it is helpful. "Weird" thing with psql shell is just because of missing semicolon at the end.

[bnssoftware]

Really like your teaching style. Thank you.

[vjntstar]

Very good presentation short and to the point. I got the information I needed. Thank you

[chrisschuck]

Awesome concise explanation. Appreciate your work!

[morph-]

Thanks for the comprehensive explanation.

[martingonzalez]

I have a question, when you did the update the event on the kafka topic had a '{"before": null, ' why is that null if the row already had information before ?

[SpiritOfIndiaaa]

Thanks a lot, simple and straight forward and clear

[r-rtz]

Your 'Trevor' update is showing up in Kafka not as an update, but as a new record ("before": null)

[sushantsinha]

Thanks ! Clear Explanation in this video . Do you have PostgreSQL Kafka Sink connector tutorial similar to this ?

[udaynj]

Nice video. Like your teaching style. Quick question - How stable is Debezium if you want to run this at scale in production?

[CuriouslyContent]

Any idea why the last update had a 'before' value of 'null'? Since the last thing was an update, shouldn't the 'before' have the values as they were before the update (id:2, name:mike)?

[UmarFarooq-ugh]

can i used this design for RAG system in production for real time update postgres to vector database?

[mmfStudent]

Why after the sql update command, event has `before: null` in Kafka?

[SoumilShah]

Hello i need some help i was able to implement everything you showed in video
i cannot consume messages from my local kafka-python can you please help ?

[sanjaymadhavan]

How to deal with the concept of foreign key in debezium?

Great video BTW! Life saviour

[vijeandran]

Hi, Thanks for the video... Whenever I insert or update the data in postgres, those changes are not showing in the docker run command

[KshitijKohli-hk]

Awesome tutorial! I wanted to understand 1 piece more. There are 2 patterns which I am now aware of of streaming CDC to Kafka. 1 via debezium connector that you talked about, other is via the Outbox pattern where the application service commits to an Outbox table in the same commit it writes to other application tables, post which tools like Ceres can stream the data to Kafka. What are the core differences in these 2 approaches and is there a recommendation of one over the other?

[pyramidofmerlinii]

If we have airflow in docker-compose, we don't need Debezium, right?

[edisonngizwenayo]

Great explanations, just have few questions. The first one is how the configuration should be made in debezium.json file if the the source is an API other than Database. The last one, how to allow debezium to listen to change from multiple database tables. Thanks