Redis pub-sub vs Kafka? What to use for a chat application?

This video is about a specific question around System design of a chat application, which generally remains un-addressed unless you actually have a design discussion. The question is : In the publisher subscriber section, why did we use Redis? Why not Kafka?
The short answer is : Redis is the right choice for the use case of real time chat application.
There are 4 reasons
1. SPEED
Redis is in-memory , so it is faster. Where was Kafka has to make disk operations.

2. STORAGE REQUIREMENT
Redis does not store messages , they just pass through to all the consumers 
So, you only need to increase the processing and not the storage. Where-as in Kafka you need both compute and storage.

3. COST OF CREATING A TOPIC
Kafka: Has extra cost because the partition needs to be created in all the Redis:  no cost of creating a new channel.  In fact no channel is created, you just send a message to a channel and you are good. 

4. COST OF DELETING TOPIC 
Deleting a topic in Kafka is not straight forward. Sometimes, delete topic operation takes long time and then you have to manually delete the topics from server.

COMMANDS

Redis:

Start redis server: redis-server
connect to redis server: redis-cli
subscribe channel: SUBSCRIBE channel
publish command: PUBLISH channel1 "hey"
Observation : fast message sending and receive

Kafka

00:00 Intro
01:30 Redis & Kafka Basics
05:45 problem statement
07:08 Traditional solution
10:00 Demo and explanation
25:55 Conclusion

You can also DM me if you need guidance as a software engineer.

[ashenguththilawickremasing]

Valuable information. Thanks
Keep it up 👍

[dhavald]

kafka stores data in disc cache page which is in memory and later flushed to actual disk.

[satyajit]

Excellent Video. For Redis, How both the web socket services differentiate messages from various sessions of users. its the same one topic right?

[aditya]

Thanks for this really insightful video and very well presented and explained.

I have one question. Does redis pub sub scale horizontally, if we think about scaling this even further for millions of users using this system at the same time.

[shirsh]

Good work Anubhav
So i am curious when should I use Kafka or RabbbitMq then?

[vikrantkaushal]

Hi Anubhav
Thanks for creating this video. I have a question around persistence in Redis. What if Redis host crashes? Won't we loose all in-flight messages? Would you still prefer Redis?

Please correct me if I am wrong Redis can also store data on disk which should be helpful in recovery if the Redis host crashes but not sure about latency in that case. Is there any other significant difference between Redis and Kafka that we should consider before making the decision?

[shivamsingh]

What will be the flow to rotate redis data to database mongodb means what will the time to do that should we do for every message as it produce or batches it and store after an interval?

[abdullah]

*there is an option in redis as well for disk persistance!!!

[harshwardhanshetty]

one question big brother can i store my data in mongodb by using redis?

[jitendrasamal]

Are you available for custom development?

[hemanthaugust]

Consider Whatsapp scale and persistence to the messages. Does this scale?
Redis becomes.a bottleneck. U need to shard it, if pub-sub can scale like Kafka then it's good.

[meditationdanny]

Scaling websockets horizontally is an expensive task thats the fundamental problem redis is solving here

[TikTokTrendsCompilation]

imagine millions of chat messages every second or minute. you think that's healthy for keeping in memory without causing memory overflow?