Distributed Consensus and Data Replication strategies on the server

We talk about the Master Slave replication strategy for reliability and data backups. This database concept is often asked in system design interviews with discussions on consistency and availability tradeoffs.

Very closely tied to the master slave architecture is the concept of distributed consensus. When designing a system, we must make sure that the individual components can agree on a particular value. (Leader election, distributed transactions, etc...) Some popular techniques are 2 phase commit, Multi Version Concurrency Control, SAGAs and Quorum.

Chapters
0:00 Problem Statement
0:53 Replication
1:24 Synchronous replication vs. Asynchronous replication
3:36 Peer to Peer data transfer
4:44 Split brain problem

Also useful when used with Master Slave replication is sharding. You can check out the video mentioned below.

Looking to ace your next interview? Try this System Design video course! 🔥

Course chapters:
1) Design an email service like Gmail
2) Design a rate limiter
3) Design an audio search engine
4) Design a calling app like WhatsApp
5) Design and code a payment tracking app like Splitwise
6) Machine coding a cache
7) Low level design of an event bus

The chapters have architectural diagrams and capacity estimates, along with subtitled videos. Use the coupon code of 'earlybird' to get a 20% discount.

References:

You can follow me on:

#SystemDesign #Databases #Replication

[gkcs]

Guys, sorry for the last 4 minutes of the video where the camera focus went haywire. I keep running into these technical issues every video. Maybe I should do a course on videography...any suggestions?
All the best!

[mukundsridhar]

Excellent video as always gaurav.
A few points from my side.
1. Taking snapshots at regular intervals of the db is one way to avoid the single point of failure.
2. Log reconstruction is one more way. Before writing to the db first flush to a log file. So if the db crashes the log file can be used to restore the db.
3. I think when you talk about the databases communicating with each other, you mean there is an API that sits in front of them and these API's communicate with each other.
Or is it that there is an API in front of the db which writes to a queue and another process that polls the queue and talks to the second db.
4. One way to deal with the split brain problem is through the paxos algorithm.

Great job and thanks !!! :)

[vyshnavramesh]

Key terms for Data Replication during interview: Atomicity (A of ACID), Master- Slave, Master-Master, 2PC, 3PC, MVCC, SAGA, Split Brain, Distributed Consensus, Regular DB Snapshots, Log reconstruction, Paxos Algorithm

[TheHpb]

Great content (upvoted and subbed). Don't see any other channel with short and precise summaries of fairly complicated concepts.

One more benefit of Master-Slave - Move replicated data closer (geographically) to the end user. For better user experience - lower latency.
Also sharding and replication scheme are tangential and often work in tandem. Generally you will have sharded databases with each shard replicated. Replicated shard could be a slave or a master (consensus based).

I vote for distributed consensus. In fact this would cover the meat of distributed systems and could be broken down into multiple smaller introductory videos
Video 1> Inherent problems with distributed systems (failure detection, n/w partitions, byzantine(non) faults, clock skews).
Video 2> Why do we need consensus ? Protocols to tackle #1 and what they tradeoff to offer consensus (CAP axis). Protocols - 2pc, 3pc, paxos (intro).
Video 3> Real applications and tradeoff they make for better performances. (Why distributed acid applications are hard and to be avoided).

I see lot of interviewees(i'm interviewer in one of the FANG) have surface knowledge of these concepts and often incorrectly adjust to a constraint (been guilty myself). Content like yours can help someone find gaps in their understanding.

[manishprajapati]

The point where the video gets blurry, I felt like I got hit by some sort of vision problem 😂😂 Video was very interesting and fun to learn.. Thanks Gaurav Sen

[shubhamqweasd]

Hey Gaurav, great videos :), I believe 2PC and MVCC are completely different things, they cannot be compared to each other, MVCC(weak isolation) is a way of handling transaction Isolation on a single node, whereas 2PC/3PC are ways of handling distributed consensus (as you have explained already). Transaction Isolation levels should have no relation with distributed consensus.
Please correct me if I am mistaken here, will help a lot.

[nandanrathod]

Hi Gaurav, good video. Can you make a video over "how large scale Indian Unified Payment Interface works"? Focus on technical/Networking and Database aspects. Thanks !!!

[jatinmotiani]

Great explanation Gaurav, You just made my interest in system design also😊

[alokuttamshukla]

Great work brother!!! Awesome as always

[phd]

Dude thanks very much. Your work is highly appreciated!

[computernetworking]

Awesome explanation Gaurav Sir 🤩 I learn a lot from your videos keep the good work up.

[DivyanshuBansal]

Nice work Gaurav! This is the only channel for which I have enabled notifications. You are helping the whole community. Thanks a lot.
PS: start an instagram page and post all these memes there also.😀

[nirupam]

MVCC is used by MySQL InnoDB storage engine as well.

[ghanishsingla]

Where is Distributed Consensus actually explained ? Did you make a video for that already ? Please share if you did. Thanks for the videos. Keep up the good work.

[sdfg]

Hey Gaurav,
Great timing for the upload btw,
Thanks for a very exhaustive discussion regarding master slave architecture. Much new stuff I came to know. Distributed consensus or "Quorom", as mentioned in your earlier videos, will be highly dependent on many factors as mentioned, but I believe that these concepts dealing with "system vote" in case of potential failure will require a branched study of its own, as it slightly outside the topics in discussion.
Hey, your video was very complete regarding master-slave, can't focus more on that, but it looks like the "quorum" part also contains some of the architectural meat, and can be discussed in dedicated manner at once, for all the possible use cases.

[jayantsogani]

Hey ! It was really helpful video for beginners ! Keep rocking

[MarketesCM]

Thanks for the video!! I think it would also be interesting to explain how a consensus algorithm like paxos works and what problems it prevents.

[umeshmalhotra]

Hi Gaurav, polling for next video - Reporting engine. In many systems, heavy reportings/analytics is involved. There are various solutions like using sql views/ late update using cron jobs. Using separate database or even servers. Please enlighten us on this.

[rakibjahankhan]

If there is single database, my application is scaling up in terms of user then the load balancer will not help, all the load will be in single database, for that how to serve fast in terms of user experience(read/write), how to scale or design the database??? i m seeking help for sql server

[calvinsugianto]

Hi, thanks for the great work! I just have a question regarding the syncronization lag between master and slave DB. What is the best practice to solve this issue for a read heavy system ?