Design a Distributed Message Queue - System Design Mock Interview

A senior engineering manager, designs a distributed message queue.

When designing a distributed message queue, consider both functional and non-functional requirements. Use a topic-based system with a pull model for scalability and control. Use a key factor for sharding. Store data with SQL, NoSQL, or a write-ahead log system, and use metadata storage for important info.

To ensure fault tolerance, use a leader-follower approach with a coordination service and acknowledgment for write success assessment. Follow a structured approach and incorporate multiple producers and consumers for different topics to design an effective message queue.

Chapters (Powered by ChapterMe) -
00:00 - Intro
00:13 - Functional and distributed queue requirements
01:42 - Queue types topic base, fan out, order creation
02:36 - Direct message queues in ecommerce
06:39 - High-level design for messages with producers
11:10 - Scaling consumer for faster consumption
12:20 - Different options for queue design
13:29 - Key and sharding for message storage
13:56 - Different sharders for different buyers
14:21 - Storage options SQL, no SQL, write ahead
15:00 - SQL-based log management solution achieves high performance
18:01 - Partitioning 300TB files using buyer ID
18:52 - Partitioning, segmentation, metadata storage for Q
21:09 - Data storage, consumption, and fault tolerance
25:47 - Replicating messages in Kafka
28:57 - Faster interview questions highlight advantages of depth analysis
29:18 - System design interviews short summary, follow pattern
31:20 - Check-in with interviewer helps prepare for interview

Watch more system design videos here:

ABOUT US:
Did you enjoy this interview question and answer? Want to land your dream career? Exponent is an online community, course, and coaching platform to help you ace your upcoming interview. Exponent has helped people land their dream careers at companies like Google, Microsoft, Amazon, and high-growth startups. Exponent is currently licensed by Stanford, Yale, UW, and others.

Our courses include interview lessons, questions, and complete answers with video walkthroughs. Access hours of real interview videos, where we analyze what went right or wrong, and our 1000+ community of expert coaches and industry professionals, to help you get your dream job and more!

[samirvasani]

this video explains more on how Kafka works then a pure distributed system..Content can be improved by mentioning a use case so that viewers can corelate the problem and the solution.Thanks

[soumikdc]

Although it was good explanation of distributed queue, it hardly seemed like an interview. Usually the interviewer asks multiple counter question and takes the discussion into different direction as per his/her choice but here there was no such thing. This could have been more interactive.

[afraz-khan]

less than 15 mints to design and explain this huge and complex system? chill brah :) but anyway, a greate video and very fluid approach to design. was helpful. thanks guys.

[DebasisUntouchable]

the system design mock interview was good, but there are things were not fine,
1) It started with distributed queue, but you ended doing pub-sub
2) 100 bytes and initial calculation of topics and messages was not convincing,
3) You said there is a tradeoff between latency and throughput, that is not so, you are not thinking in distributed way.
4) You said, WAL, Consistent Hashing based on buyer_id and then Leader-Follower architecture, I guess you are bit confused.
5) And a segment of 10KB? that means for a message of 100 bytes, you will have 100 messages in a segment? It does not make sense right to have so less segment size.
6) And the consumer only reads from Leader not from replica?
7) Leader syncs to replica, you didn't say whether its async or sync replication.
On the positive side, I liked your handling of meta-data and cluster service inclusions.

[demogyani]

From Distributed Queue he went to messaging brokers. I prefer low level system designs as compared to these things.

[pioneerx]

The interviewer didn't define what he meant by distributed. This just turned into a pub-sub topic system.

[Deepakkumar-dyen]

Great content❤
About leaders and followers, i think there should be leaders for each topic plus partition instead of just topic.

[raw_tech_with_tom]

A distributed message queue. I think that is different from a message stream. I see more of message stream like kafka design than an actual queue like rabbitmq.

[ofvc]

Which application/website is being used to draw diagrams in the video?

[jmitesh]

segment size could be of 1 GB as in Kafka(vs here taken as 10KB which is quite small).

[rajeshkishore]

Overall it was good but implementation has lot of complexity, could not find any thoughts on implementation for any of feature thoroughly

[amylovescoffee]

Is this a typical PM interview or more for engineers?

[ranjanable]

which editor he is using to explain system design?

[qutaibabs]

what whiteboarding softare he is using ?