Designing Notifications Service for Instagram

This video is a snippet from my System Design Masterclass and in it, we are discussing How Instagram Scales its notification systems. The primary challenge of such a system is doing a real fast fanout and we discuss how to do this very efficiently.

# Arpit's System Design Masterclass

I teach an 8-week cohort-based course on System Design - a masterclass that helps you become great at designing systems that are scalable, fault-tolerant, and highly available.

The program will have a blend of Live Classes happening on Weekends, 1:1 Mentorship sessions happening on weekdays, assignments and, group projects. The program is designed to be intense and crisp so as to accelerate the overall learning.

### Subscribe to my free newsletter

🔥 Once a week, in your inbox, an essay about programming languages internals, or a deep dive on some super-clever algorithm, or just a few tips on building highly scalable distributed systems.

### Follow me on other social platforms

Stay Awesome,
Arpit

[harekrishna]

Some important points to highlight during this discussion:

1) Kafak vs SQS and where in system to use what and Why ?
2) Push vs Pull - Fan out problem: (Solutions like push to bell icon subscribers only, let other users pull instead of push, or a subscribe for notifications)
3) Brain to filter notifications
4) Priority of sending notification
5) Does order of notification delivery matters. (storing notifications for a user in a specific queue hashed on user id consistent hashing ensuring message ordering)

[ramannanda]

For the in app stuff, you can store this is Cassandra with composite primary key (user_id, message_id) and message data with cluster ordering on message id. To ensure ordering on message id(have a timestamp component in message_id).you can store info like last_read_message_id on client device and then efficiently query for the message_id>last_seen for a user partition per device.

[B-Billy]

This is the best of best system design video I have ever seen... You don't talk just about theories but it shows that these are really practical.

[ShreeharshaV]

Hi Arpit, thanks for the video. I have watched quite a lot of your videos on this channel. Thank you so much for all the content. I really wish I had a teacher like you growing up. :)

[anaysarkar]

I have atleast seen this thrice this year & each time i have a 'aha' moment

[nytlyf]

this is good. keep the read from publish topic fast and then heavy lifting can be done by other service.

[mrinaalarora]

Bro I don't even know what system design is yet found this video interesting.

[MdJunaidSiddique]

Hi Arpit!
I follow you on LinkedIn and have been watching your videos. They're amazing and so very insightful! I have been a Backend Dev for about 2 years now. I wish I'd met/or worked with you since the beginning of my career.

[shishirchaurasiya]

It was an amazing experience as well as learning, finally completed this series dayum.❤️ can't believe

[swappybird]

Super nice to hear some good engineering stuff again! Keep posting!

[kartech]

Additionally, if you need to track user response on the notifications such as it was dismissed or clicked then it will be interesting to see how it is handles across multiple users and devices?

[rohanarora]

Kafka partition parallelism to SQS is smart.

[munjal]

"kafka consumers are limited by their partition count" : correct, but then you should take care of your throughput and set this number correct well in advance.

[gmmkeshav]

bhai ap great ho ap ko charo aur se pram hai

[nytlyf]

Aggregation would be on client side if I am not wrong. Basically app decision to show notif or not but again on-prem presence of app is a PITA for any updates. so, maybe fanout should consider this as a filteration step. existing design might change. WDYT...

[aashishgoyal]

Great content Arpit. Expecting more such knowledgeable content from you

[manchukondamanoj]

This is amazing, Notification service adds a lot of value to the app to improve it's user base and have loyal customers & designing this at scale is awesome.

I have one doubt,

There is an e-commerce website which sends push notifications(Email & SMS) to all its users to shop and get 20% discount in specific categories within an hour.
In this case how will the fan-out service( or it's Publisher) sends or prioritise the notifications assuming the plat form has some 50M users. looping and sending notifications takes time and the one hour window will be reduced to the last user (or batch if we send in batch) by the time user gets a notification.

My assumption is we autoscale the workers to a higher limit in this case and increase the publishers & make async calls?

or do we have a different approach?

Thank you

[aashishgoyal]

Subscribed and clicked on bell icon as well😀

[JyotiSingh-pxfh]

Thank you for the video! Does the SQS vs Kafka decision change if we need to support 1M+ notifications per second? Is the inflight message limit for SQS a bottleneck?

Would we use SQS for all queues if we didn't need multiple consumers for first event stream?

[vyshnavramesh]

- How do we take care of sending notification to offline users (store notifications), failed notifications (retry mechanism), order of notifications?