Realtime Streaming with Apache Flink | End to End Data Engineering Project

In this video, you will be building an end-to-end data engineering project using some of the most powerful technologies in the industry: Apache Flink, Kafka, Elasticsearch, and Docker. In this video, we dive deep into the world of real-time data processing and analytics, guiding you through every step of creating a robust, scalable data pipeline.

Timestamp
0:00 Introduction
0:55 The system architecture
08:00 Sales Analytics Data Generation
19:10 Producing Data into Kafka Broker
25:00 Setting up Apache Flink project
32:28 Consuming data from Kafka with Apache Flink
43:30 Starting Apache Flink on Mac
54:25 Writing Kafka Streams to Postgres Database
1:20:00 Aggregating Transactions per Category into Postgres
1:36:00 Aggregating Transactions Per Day into Postgres
1:39:46 Aggregating Transactions Per Month into Postgres
1:51:52 Writing Kafka Streams Data into Elasticsearch
2:05:00 Reindexing Data on Elasticsearch with Timestamp
2:10:52 Creating Streaming Dashboard on Elasticsearch
2:22:46 Realtime Dashboard Results
2:24:14 Recap
2:25:34 Outro

🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟

🔗 Useful Links and Resources:

✨ Tags ✨
Big Data Engineering, Apache Flink, Kafka, Elasticsearch, Docker, Data Engineering, Realtime Data Processing, Big Data, Data Pipeline, Streaming Data, Data Analytics, Tech Tutorial, Data Science, Flink Streaming, Kafka Streaming, Elasticsearch Tutorial, Docker Containers, Data Engineering Project, Realtime Analytics, Big Data Technologies, Data Engineering Tutorial, Data Engineering Projects, Data Engineer

✨Hashtags✨
#ApacheFlink, #Kafka, #Elasticsearch, #Docker, #DataEngineering, #RealtimeData, #BigData, #DataPipeline, #TechTutorial, #DataScience, #StreamingData, #Flink, #KafkaStreams, #ElasticsearchTips, #DockerContainers, #DataEngineeringProjects, #RealtimeAnalytics, #BigDataTech, #LearnDataEngineering, #dataengineers

[idokofrancis]

Your content are truly amazing, clear, and precise.

[levimatheri]

Awesome content. I'm a .NET developer but I really enjoyed this tutorial! Subscribed!

[sagarHimanshu]

Another great video, Will surely implement. Thanks for the good work. Appreciations from India...

[ericlaw]

ur content getting better and better !! keep it up. i have been following your end to end projects

[hritikapal]

Absolutely terrific content you're delivering, your content reached INDIA too thanks for helping!

[lpfojuz]

You are Amazing! Very detailed and made everything simple 👏

[romilpatel]

Amazing. I love what you do. Keep growing brother.

[harshityadav]

Thank you so much for this wonderful video I have been looking for full end to end project for Apache flunk

[Flumxz]

Love the energy bro, keep up the good tutorials 🙌

[khawlaallak]

Thank you so much for your good explanation 👍 great work

[henriquevalentim]

What a great video. Very helpfull. Thanks for this content

[nyupxrr]

Your content is truly amazing, making data engineering interesting. I have been a devoted follower of yours, keeping up with your projects. Currently, I am working on a project that involves extracting data from a blockchain. However, I am unsure about the appropriate processing method to use. Should I opt for stream processing using Apache Flink, or would real-time processing with Apache Spark be more suitable? I am still in the learning phase and would appreciate any guidance you can provide.

[mohammadshamsher]

Thankyou buddy, Some amazing stuff you just gived to us. I have a query, How the paid projects from your website are different from this one of on your channel.

[sreesanjeev]

you have both the create table and insert in the Flink datastream, so does it check for create table command for each message in Kafka? Is that not slow? Better we create a separate DDL statement for tables and just use the insert command in the stream?

[kevinding]

Great content, wish I could see it earlier! I'm curious why we're using Flink here rather than a traditional Kafka consumer, what if we only want to sink data in Elastic, (no Postgres) here, would Flink still be a good option?

[wiss]

i have been following your end to end data engineering projects, I am really proud of you...waiting for snowflake data warehousing projects

[imikhan]

hi bro really great Tutorial i have request if you can make the flink part as in Scala or SQL API that would be a really great learning

[jacobdev]

Thanks for the detailed explanation and you are amazing !!

I have doubt here. in this example you were mentioned about sales by day or sales by category. If i restart my kafka server then how it will work ? it will start with latest data and store latest result by wiping old data ? and If want to persist data then how can i do that ?

Also if kafka is running with earliest event and if my flink job restart how i will work with old data ? is it ingested again ?

[nmmnxdt]

Does this project include cloud computing service?

[a.kabaki]

@CodeWithYou Thank you for this detailed project tutorial. Question: Why do you need Kafka in the middle? Can't you stream directly to Flink?