Database Sharding and Partitioning

Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.

In the video, I discussed the importance of sharding and partitioning in scaling systems. Sharding distributes data across multiple machines for improved throughput and availability. We explored how databases evolve through stages, the differences between sharding and partitioning, and when to introduce these concepts. I also highlighted the benefits of a collaborative system design course I offer. Scaling databases vertically involves increasing resources, while horizontal scaling adds more servers for higher throughput. Sharding splits data across shards, while partitioning divides data within a shard. Strategic partitioning is crucial for efficient data management.

# Recommended videos and playlists

If you liked this video, you will find the following videos and playlists helpful

# Things you will find amusing

# Other socials

I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.

Thank you for watching and supporting! it means a ton.

I am on a mission to bring out the best engineering stories from around the world and make you all fall in
love with engineering. If you resonate with this then follow along, I always keep it no-fluff.

[shishirchaurasiya]

I was literally consfused in gaining the clarity untill you came to the point where you transposed this theory into understanding through tables and the reference with SQL queries, thanks a lot to your efforts for this loving beautiful explaination Arpit sir

[AlokMehta]

Excellent video Arpit . Coming from no software and system engineering background, this was the best video to explain data sharding and partioning . I am a Tech PM for AWS Supply Chain and data partitioning and sharding is real deal for us. Thank for making this extremely easy to understand video

[virendersingh]

You have a brilliant way to explain, only one who has gone through the journey would be able to teach it this way.

[codecspy]

2 Important points which i felt could be discussed more are 1) When you said the choice of partitioning depends on the load, usecase and access patterns, can you please give an example of each case ?? 2) When you were talking about the advantages and disadvantages of sharding, have you written these points considering only sharding and no partitioning or have you written considering both sharding and partitioning ??

[visheshjindal]

I have just started learning System Design and not from a backend background. Still able to understand the concepts. Thanks for creating such content.

[ranjithpals]

Thanks a lot ! That was well explained with clear and concise explanation. Looking forward to enrolling in your complete system design course.

[movieswatchify]

liked the way concepts are explained like a story.

[chaitanyawaikar]

One of the best videos explaining the nuances between partitioning and sharding. Thank you @ArpitBhayani

[nikhilrajput]

Wow...really nice. Nowadays a lot of people are selling and talking about system design and always try to build some optimistic solution straight forward without going into the internals and in fact they have not even worked on a lot of systems. I strongly feel the way of your explanation is very very nice and I am going to buy your system design plan to improve mine.

[jaskiratwalia]

Wonderfully explained! Cleared all my doubts. Please keep making such videos. These are also well timed, not too short nor too long.

[yuvrajchauhan]

Amazing explanation! The diagrams were really helpful in helping me solidify my understanding of the difference between sharding and partitioning.

[garpaygavs]

I am just stunned how you describe every concept that easily and even if the concept is so complicated. I love you and your works man please continue doing it. I promise I will join as a member to this channel as soon as I become an earning developer. :D

[vipulsharma]

This is such a great and simple explanation of partitioning and sharding of a database. Would love to watch the video on partitioning strategies when it is uploaded.

[ishwarhingorani]

Great clarity and in depth explanation. Thank you

[mystiqkc]

I looove this kind of explanation. i.e stepping back and discussing the scenarios behind why something came up to be. Thanks a lot for these videos man!

[aksharcharagi]

When i finished watching this video, concepts looked super easy. Thanks

[GaganJain]

Does it mean Sharding and replication are the same? 22:16

[iHariPatel]

As my view Partition is more complex because you have to work with partition key! With wrong query accidentally query scan all partition’s.

[nuclearniraj]

One video and all the clutter on Sharding and Partitioning is clear. Thank you so much Arpit.

[anandahs]

Very good explanation with right examples. Hats off to you. Thanks for great content. I always thought shard and partitions are same but you clarified it very well.