Dropbox system design | Google drive system design | System design file share and upload

Let's design a file hosting service like Dropbox or Google Drive. Cloud file storage enables users to store their data on remote servers. Usually, these servers are maintained by cloud storage providers and made available to users over a network


#systemdesing #dropbox

[sumonmal]

idea scope 1:38
scale 2:10
HLD 2:41
problem to solve 4:55 6:57
solution 10:41
metadata file 15:26
HLD 17:38
messaging service detail 25:01 device sync feature
metadata handling 28:40
metadata schema 31:48
edge store usage to serve metadata 36:16
search feature 40:01

[roopaschannel]

Thanks for your channel Naren! Brings back my love for computer science. We need more such teachers that can break things down and explain it as simply as you have done here.

[chepaiytrath]

Clients described at 18:26, taking an example of Google Drive, refer to the various "Backup and Sync" desktop clients which you might have active on multiple devices. All these clients keep listening to a messaging queue. In case one device makes changes to a file, the change is propagated to S3 and all clients are notified of this by publishing the change to the messaging queue which they are listening to. The client which is the originator of the change doesn't care but other clients do and when they know of a change they update their local copies (download the whole file if not present).

Update:
It's not just one Q2. Each client will have its own queue on which the change is broadcasted. This is to have an asynchronous behaviour wherein the client can be offline for a period and then when it is online it starts listening to the queue for any changes

This is my understanding. Correct me if I'm wrong

[vaibhavsinghx]

Another reason to use async queues: one cannot assume that only a single file will be uploaded. There could be a case in which multiple files could be uploaded and a queue ensures that chunks do not get mixed with each other. I guess one can also talk about failover (what happens when a chunk gets lost during transmission/gets corrupted) but that might not be required.
Edit: NVM he covers this case as well LOL. Love the depth he goes into when covering different components.

[simpleurbanliving]

Enjoyed this video more than others because of the cute doggo interruptions. :) Thank you!

[samahome]

Your explanations and approaches in explaining these System Design Problems is absolutely phenomenal.

[ragingpahadi]

Give this man Bharat Anmol Ratna : ]. Thanks for SD series it helps us broaden our thinking and not just defect fixing and small CR.

[akinkanju]

Hello Naren! your channel is a goldmine. I've learned quite a lot. Please consider creating content that dives deep into data models/schemas/datasets. Thanks 🙏

[eugenekim]

Great system design. I really wish he explained why file change sets need to be ordered and consistent, in which led him to use a relational database for the metadata.

If you look at his design for google docs, it doesn't even use a relational database for massively concurrently updated files.

[RakeshGajjar]

Give this man the credit he deserves 👏🏼👏🏼👏🏼

[RandomShowerThoughts]

15:39 LMFAO! great video man, you are my go to for system design prep

[rohittiwarirvt]

A Great Video on Understanding file storage service design like dropbox, Preparing for an interview and this content is helpfull

[helloworld]

honestly as a swe working at dropbox, i don't feel like this is an answer i am looking for. It misses a lot of important stuff like how do you design your database schema for storing the metadata and how would your sync protocols looks like? what if there are write conflicts during sync how do you deal with that? and the search engine part i guess is the least likely bonus question i'll ask in an interview(probably makes more sense in design twitter)

no offense to Narendra, i think you put in a lot of effort/research into this and even referenced dropbox's blog post on network edge infra.
but i think this's a problem to almost all of these youtube system design videos, like, yes you will learn a little bit here and there, but it's not the same as a real interview and don't expect to memorize some sys design solution and pass the interview.

better ways to learn system design:
read DDIA, web scalability for startup engineers, take a distributed system class
listen to real mock interviews if you somehow can(or some faang engineer does these mock interviews and post them somewhere i guess)
design and implement projects at your job if you have the opportunity

[deepakzworld]

The best part I like about your videos is you do a lot of research to put the information from various sources about a topic into one place. You are our Edgestore ;)

[Yan-rvmi]

33:56 Here you threw the problem that we need to rebalance/re-shard as we get more and more data in one shard, but the subsequent mentioned approach "edgestore" does not seem to solve this, does it? It seems like the edge wrapper simply provides a better interface for developers to read/write data. How does the "edgestore" help in regards to the data sharding parts?

[stevemew]

Great work Narendra. This is the best video I have found so far on YouTube on the DropBox architecture.

[viktorartemov]

Needs an explanation of how exactly does one detect which chunk was changed. Because your applications, video editor, for example, doesn't know anything about chunks, it doesn't change a chunk, it changes your file. It's up to your Dropbox client to figure out which chunk the change corresponds to. And that is not immediately obvious especially for huge binary files.

[pavankumaruppuluri]

While explaining why we need queue instead of http call to sync service you mentioned we need it as client may not always be connected. My question is if client dont connect to internet for example, even that message also cant be transmitted to queue right ?

[saiprajeeth]

WTF. only 633 likes out of 38, 663 views for this gold? Come on viewers, you are beholden for this guy who is putting enormous effort to share knowledge beyond his boundaries.

[yog]

Amazing no nonsense serious designs which are really good hatsoff bro 👍 keep doing good work