MongoDB Internal Architecture

I’m a big believer that database systems share similar core fundamentals at their storage layer and understanding them allows one to compare different DBMS objectively. For example, How documents are stored in MongoDB is no different from how MySQL or PostgreSQL store rows.
Everything goes to disk, the trick is to fetch what you need from disk efficiently with as fewer I/Os as possible, the rest is API.

In this video I discuss the evolution of MongoDB internal architecture on how documents are stored and retrieved focusing on the index storage representation. I assume the reader is well versed with fundamentals of database engineering such as indexes, B+Trees, data files, WAL etc, you may pick up my database course to learn the skills.
Let us get started.

0:00 Intro
2:00 SQL vs NOSQL
18:00 MongoDB first version MMAPV1
26:30 MongoDB Wired Tiger
38:00 Clustered Collections

Follow me on Medium

Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon)

Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)

Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)

Introduction to NGINX (link redirects to udemy with coupon)

Python on the Backend (link redirects to udemy with coupon)

Follow me on Medium

Become a Member on YouTube

Buy me a coffee if you liked this

Arabic Software Engineering Channel

🔥 Members Only Content


🏭 Backend Engineering Videos in Order

💾 Database Engineering Videos

🎙️Listen to the Backend Engineering Podcast

Gears and tools used on the Channel (affiliates)

🖼️ Slides and Thumbnail Design
Canva


Stay Awesome,
Hussein

[hamdaankhalid]

Yoooo have to do Postgres internals all the way from storage to query planning please!

[alirezarohami]

Designing Data-Intensive Applications by Martin Klepmann is the go to reference for anyone who wants to understand the differences between different needs and pros and cons of each.

[Xavierpng]

Thanks a ton for these videos. Please try making an architecture video for cassandra database as well.

[adarshpatel]

Waiting for this since long time.
Thank you for providing this information.

[arnabchatterjee]

Thanks for creating these videos. Definitely worth the time 🚀

[siya.abc]

Waking up to a long form Hussein video on a bright Saturday morning 🌞 is one of the best ways to get going

[tylersustare]

The main difference in RDBMS and “NoSQL” is denormalization and querying efficiently instead of joins. It can be done in PG of course.

[burakhansen]

I am addicted to your videos, thank you sir

[taman]

Pretty neat, Thanks a lot Hussein.
Looking forward your video about NewSQL DBs like TiDB

[JohnnysaidWhat]

In the 60’s and 70’s storage was incredibly expensive. So it was clever to use tables. You could store more for less. You pay in write/read performance.

Now storage is dirt cheap. So you pay a small amount in storage but you get horizontal scaling and more performant write/read.

There are other differences like ACID compliance but I think it’s good to know the history and motivations.

[realWorldDevStudio]

I love the your channel and the content you post, but I am trying to find more channels like yours with in depth explanation on topics to fill my yt feed with such videos but couldn't find many. Feel free to reply to this commend with recommendations .

[darpanmalhotra]

Question around 14:00 ---> How is writing to WAL different from writing to data file? Same risks apply to both... Flushing pages of data file must be same as flushing pages of WAL. So, what does DBMS really achieve by offloading writes to WAL files from data files?

[iftekharuddin]

Hussein is the gift that keeps on giving. 🙏

[sonamphuntsog]

I'm confused. In WAL, If writing to the disk is expensive so instead we write in memory and to the log file. Isn't writing to the log file as expensive as writing to the data file directly ? Where is the performance gain ?
Unless writing to the log file is faster than writing to different place in data files. Is that it ?

[Gns]

The so called 'NoSQL' movement purpose was to create a new breed of storage engines that could leverage distributed computing and separate storage from compute. That required a huge investment from companies and community in terms of time so the frontent part for those new breed database engines wasn't a priority at the beginning. So many people mistakenly saw this as a war against the old standard interaction protocols (SQL etc) missing the bigger picture. Now after a decade or so we see all those new breed databases embrace all the concepts of the past (sql, tables, columns etc). Just a clarification for those they think structure was the motive behind 'NoSQL'

[shahariarhriday]

Hussein, Why don't you create a course for mongodb in depth? That will be very helpful to us.

[susmitvengurlekar]

Column level or field level locking can cause problems if value of other column/field is being set based on the value of another column/field which suddenly someone has changed. Though if the read operation was performed in the same transaction in which update is being performed, we can prevent this issue.

[nitishbhatia]

Hi @hussein
Want to understand 1 concept.
Since file system allows complete page to be read and flushed
Then how is sequential io fast? The one we use for WAL.
Wouldn’t that sequential io have to write the entire page back to disk?
Please help me clarify this doubt.
Thanks.

[Shwed]

How could people dont like sql? Its so cool!

[brymstoner]

A full-sized, long form Hussein video... that's an easy click!