Google SWE teaches systems design | EP5: Database sharding/partitioning

My ex would've been really good at picking a fixed number of partitions because she told me I wasn't too big or too small, just right (pain).

Also, slight error on my part: dynamic partitioning doesn't necessarily mean the database rebalances on its own - sometimes it will make the suggestion and an operator has to do so manually.

Recommended Reading:
Designing Data Intensive Applications, Chapter 6

0:00 Introduction
1:45 Objectives of partitioning
2:25 Methodologies for partitioning
4:01 Hash Range Partitioning
4:35 Hash Range Partitioning Tradeoffs
5:44 Indexes in a partitioned database
7:01 Local Index Tradeoffs
7:29 Global Indexes
8:53 Rebalancing Partitions
10:28 Fixed Number of Partitions
12:20 Fixed number of partitions per node
13:03 Sharding Summary
14:05 Sharding Summary Continued

[ajayreddy]

me and my friends as binging on these videos rn and jordan's subtle humour and on point content here is what driving us to finish playlists after playlists.

[marcosoliveira]

This channel will be my Netflix for the next few days. Learning a lot from this sexy cute mid guy hahahaha

[nishankdas]

I wish I had started watching your videos when I started my system design journey.

[xavierperemiquel]

I love your videos buddy, simplicity and straight to the point

[zodman]

Best video of sharding/partition data thanks pal!

[alexanderbalasky]

Binging these rn but also facts on those “day in the life videos”. Filmed a bunch for one that I was going to post for like you said “more views” but realized i wasnt about it. Niche swe content it is.

[rodrigoalvesvieira]

Excellent video, very concise and full of information. Thank you

[shuaisan]

There is a small typo on the page for the local indexes. Next to the table on the top, position: small forward should be [2] and the shooting guard should be [1, 3].

[kamalsmusic]

At 10:05 you mention that data is copied over to a new node while the old node still handles reads and writes for a given partition, and the new node handles reads/writes once the copy is finished.

What if reads and writes never stop coming into the old node for partition X (or more realistically, lets say the reads/writes for partition X keep coming in for a very long time), while partition X is still being copied to the new node? If reads and writes are always coming in for partition X, then the copying process would take a long time to finish because those new writes coming into the old node also have to be copied over, right?

[tarunnurat]

What's the point of local secondary indexes, if they're going to slow down reads? I thought the point of an index was to speed up reads? Is it just so that for a given partition, doing a read based on some value will be faster due to the local secondary index?

[zodman]

at the global index and secondary index locally, if you can draw the lines from the data to the indexes list will be helpful i have to read that part 3 times to understand both

[tarunnurat]

For the slide with the local index example with the basketball players, is the secondary index for partition 1 supposed to have shooting guard as [1, 3] and small forward as [2]? I think those two got mixed up. Just checking.

[scuderia]

Unrelated qn - For the Meta Product Architecture Interview ( not system design ), would you suggest going into so much detail? ( since that one is more focused on API design and an E2E product )

[LeoGoldenLabrador]

what books did you read to learn these things?

[collinmeredith]

I’m working on evaluating this systems security and if it’s up to my companies policies but I can’t get over the fucking name😂