Notification Service Design Deep Dive with Google SWE! | Systems Design Interview Question 22

Time to get a women @ google shirt next, need to add to the collection

00:00 Introduction
01:18 Functional Requirements
02:08 Capacity Estimations
02:51 API Design
03:44 Database Schema
04:45 Architectural Overview

[cambriandot]

5:30 Stream processing
6:14 Message brokers
7:30 Duplicate messages - 2 Phase commit, idempotency - bloom filters
9:55 Queues - partitioning, zookeper
10:43 Fan-out to users
13:23 Notifications database - Hbase
14:23 Topics database - MySQL
15:02 Architecture

[raghavkaushik]

Awsome video dude! Just one thing missed CDC from TOPICS table to Flink using message queues partitioned on userID.

[AnkitGaurav-qths]

What changes would you do in your design if a requirement comes in where notifications have different priority, like P0 notifs need to be sent before any P1 and P2 notifictaion?

[vanderer_prashant]

I see only Flink in between topics message queue and user message queue in the diagram. Who puts message from topic MQ put into user MQ?

[vetiarvind]

what about phone notification - apple has some Apple Pushnotification service...i think Android has Firebase cloud messaging as a free option

[fangao]

Questions here: I'm assuming that the publishMsg API will finally trigger the service to push SSE msg to the Frontend Users, so the consumer servers as the ones who pushes the messages, but how does it know that the client is connecting to itself, what if the notification consumer picks up a message and find that the client is not connected to it ? Is it being controlled by the UserId Sharding queue ? so that the queue always know which consumer to send to ?? (and what if a consumer is down, does the queue know it and take care of the reconnected consumer ?)

[povdata]

Thank you for video. I still have one question. How to to avoid sending duplication messages? You say that unique id is attached to each notification message. How to use it then? It which step?

[dibll]

Why do we need to use Zookeeper to store metadata like partitions in a topic, IP Addresses of those partitions etc. Why can't we use a DB for that purpose? What are the advantages of using Zookeeper over a normal key value DB

[Glashutte]

The problem is that we have a chat app and when the app is in kill state in android or ios the push notification does not work so we go for FCM and we dont want to use firebase as google data issiues so do you think it is possible to build your own push notification in kill state like fcm. i will be looking forward for your answer

[Gerald-izmv]

what about celebrity problem? ... would you do pull vs push in this case? you also mention user queues - how do you batch the messages for a user? ... imagine the user doesnt want to receive 20 messages per day - instead 3 or 4 message might be a better user experience...

[ariali]

Curious why we use SSE/websocket here instead of long polling? I would assume notification is more like uni-directional communication instead of bi-directional here.

[VivekMahadevanv]

Followup question.. Why do you think a userId sharded 2nd level of MQs' are necessary? In most practical applications such as say chat for example, the notifications still go through multiple mediums based on what the user subscribes to (say push notification, email but not SMS). This information could be made part of the payload as tags that the broker can aggregate on for easier consumption. For any medium, it would then make sense to have a storage mechanism to hold messages that were not delivered.

I don't think the current design saves any resources on the server(threads?) on the server either since you probably need to maintain a connection with each device the user has in order to send the message. Thoughts?

[TheImplemented]

Did you intentionally skipped Subscriptions table (user_id, topic_id) ?

[jayshah]

we should probably add a notification acking mechanism from client back to service?

[prateekaggarwal]

Can you desribe a bit more about how Configuration service is helpul here, like what information it holds and how that information is used.

[sanampreetsingh]

Great design ! Of restricting user to just one web socket connection with server by queuing messages !

Couple of questions !
1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock, so our general topic can be product in stock but user is only interested for product X in stock notification ! ?



Thanks

[SlashTagged]

On using bloom filters for idempotence: Wouldn't we start getting a lot of false positives at some point (due to hash collisions)? That would lead to erroneously rejecting non-duplicated notifications, which seems counterintuitive to idempotency. We're essentially being so "aggressive" that we're actually rejecting "good" (read: non-duplicated) notifications.

[sanampreet]

Hi Mate, can u please elaborate more on local state management using flink here ? Like consumers are sharded on hash of user id and topic has list of users to notify, how state management is happening here !

[sanampreetsingh]

Great design ! Of restricting user to just one web socket connection with server by queuing messages !

Couple of questions !
1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock, so our general topic can be product in stock but user is only interested for product X in stock notification ! ?
2- we are keeping Queue per user design which might hog resources as user base increases?

Thanks

[bet_more_pls]

How would you deal with spiky traffic - e.g. load increases by 10-100x because of some special event (black friday sales or something)?