Scalable & Event Driven Food Ordering App with Kafka & Python | System Design

In this video, I design a highly scalable backend system for a food ordering app. The system is event driven. Every important thing is an event, such as an order being placed and an order being confirmed.

These events are writtent to Apache Kafka which is a central bus for all moving data. Other systems, such as the transaction system or the email system, will be built on top of Apache Kafka. They will subscribe to different Kafka topics that they are concerned with, and process in real time as these messages are written to Kafka.

We use a library called kafka-python to hook up a locally running Kafka broker and our Python code.

The system should be highly scalable. Given the decoupled nature, you can scale up or down individual components as required. These individual components can either be microservices or some stream processing jobs or just plain Python files like I do here.

The system is also very extensible because of the nature of Apache Kafka. The messages for a given topic can be retained within Kafka. That means, if you were to build a new component, you could replay all the messages of a topic from the beginning and make the component process all messages from the beginning of the topic.

Similarly, given the persistence of Kafka, if there is an issue with any downstream or upstream system, you won't suffer from any data loss.

Other helpful videos:

Further Resources

#kafka #systemDesign #python

[bjugdbjk]

Fantastic buddy !! YouTube took so long to recommend me this channel !!

[DuyTran-sslu]

This is the coolest demostration. Thanks

[voidisperfect]

thank you, amazing explanation! always struggled with understanding how kafka would work in real scenarios (used to overcomplicate it)

[shamsularefinsajib]

Fantastic little explanation, I cannt be more grateful for the explanation and easy demonstration

[chanduiit]

Brilliant.. I am new to back-end engineering and most people just talk it..while you demonstrate it..awesome work..👏

[imdadood]

Wow. This is a solid introduction for a person who has no idea what Kafka is. Thanks for this.

[amanchandra]

One of the best demonstrative video for kafka using python. Thank you so much.

[PavanKumar-pomskipper]

In one video, you have given fundamental working of kafka .. thank you

[aurumlabs]

This was a great tutorial! I was wondering how I would scale a microservice like this! Would be a great tutorial.

[shihabullah]

Very clear and awesome demonstration! Thank you for clearing my concept more as I have just started learning it.

Can you please make a step by step kafka data pipeline setup tutorial using docker and any cloud infrastructure?

You are doing great and should not stop helping people like us!

[abessesmahi]

Clear as crystal
Thank you so much for efforts
Please be aware that music during tutorial are disturbing.

[tejaslotlikar]

awesome tutorial. please make a video oh how to "wake up the consumer" once the producer pushes in the topic. my terminology might not be correct

[ashwanikapur]

Great explanation. Thanks for uploading it.

[pemmanalakshmi]

Hello Irtiza,
This is the cleanest and best kafka hands-on video I have seen! 🙂
I have an use case required to be done, can you please suggest how to do it. We have an Flask application running on Kubernetes, I need to write a KafkaConsumer which reads the messages from a specific topic ID and perform some operation with the message fetched.
Can we make the consuming part asynchronous(Background) so that it won't block the Flask app. Please feel free to suggest alternatives as well

[mukeshnayal]

Thank you brother for the beautiful explanation ! After watching this content my expectation going to increase in many fold.
May I expect a project with fast api where celery, asynchronous, zookeeper, kafka or rabbitmq, micro Services will be use along with the deep explanation of each one.

[cybernetinc]

This tutorial is awesome. Thanks for it!
I just have one question, In production, I don't think we would be running a while True loop. is there any other way to configure the listener? maybe using something like signals etc..

[zendr]

This is awesome. Can we have more of these? Please

[sribastavrajguru]

Simple yet elegant. Amazing tutorial. One question I have here is, if one of the consumers(let's say email) was down for few minutes, will it loose the details it received during that time??Ideally it should not, right?? If yes, how do we achieve that?

[wriiko]

in the github repository you have a consumer.py its not being used. Why? are u using the order_backend.py instead?
Btw, excellent explanation! thanks

[nice-jemp]

Thank you. If there's a lot of traffic on the real-time exchange,
What do you think about writing kafka + Celery?