Design a Basic Search Engine (Google or Bing) | System Design Interview Prep

Problem Statement:

Provide a design overview of a basic search engine. Your search engine system must support the following:

- **Retrieval:** The search engine should display a list of relevant web pages in response to a user query. The results should include the page title, URL, and a brief summary.
- **Indexing:** The system should be able to crawl and index web pages from the Internet. The indexing process should store metadata about the web pages, such as their URL, title, and a brief summary.
- **Scalability:** The system should be designed to handle a large number of queries and indexed web pages, ensuring that response times remain low as the search engine scales.

Finer concerns such as query processing & page ranking can be briefly addressed, but are not mandatory.

Table of Contents:
0:00 - Requirements
0:20 - How Search Works
1:57 - API: Accepting Search Queries
2:16 - Database: Storing Site Metadata
4:19 - Database Demands
4:51 - Page BLOB Store
5:17 - Database Sharding
6:10 - Global Index
6:33 - Text Index
7:09 - The System Thus Far
7:52 - Crawling
9:24 - Crawler Demands
10:31 - The System So Far
11:04 - URL Frontier: Priority
11:39 - URL Frontier: Politeness
12:01 - Naive URL Frontier
12:31 - Multiple Queues
13:35 - Solving for Politeness
15:51 - URL Frontier: Recap
16:16 - URL Frontier Demands
17:24 - Full Design Review
17:49 - Extensions

Socials:

[prathamshenoy]

needless to say.... your channel will grow superbly. this video was RECOMMENDED by youtube

[artemvolsh]

Channel currently hugely underrated, material is just delicious, especially for those who seek examples of complex system schemes.
Love it.

[carlboneri]

One of the best walkthroughs I've ever seen, regardless of the topic or technical depth. Superb work man.

[marwanezzat]

Dude, your content quality is superior, Keep going.
Yesterday you had 145 subscribers and now you have 245 i am so happy for you.

[lucasoliveira-xsyh]

Awesome content! I liked to see some data structure (such as queue and heap) used in practice, because the simple examples are good in the beginning, but it is not that good with the time. Continue with this, really a hidden gem this channel

[dmitrydmitriev]

Greetings, I just came to YouTube to watch video about SQL optimization and your channel was offered. And I started to watch this video. It is amazing, the way you explain is brilliant and outstanding. Very clear, full of information, not boring because too obvious, not difficult because too sophisticated and convoluted - a golden middle.

Thank you!

[bandr-dev]

lmao if this was the interview question I'd just not do it.. but I'm not there yet

[raghuboyapati]

This channel is gonna explode. The content is just too good. Thank you.

[vladd]

Clean, clear, efficient. ❤
I’d love to see more videos like this from you!

[williefr]

I really enjoyed the video! Thank you guys for taking your time and posting it, it was very entertaining and educational. Best regards

[andydataguy]

Your course looks great. I love that you have a teaching assistant and the explaining styles are awesome. Will try it out! Only thing is I really wish you supported Rust 🦀🙏🏾

[SlaHu.]

woow loved it
by 1:48 I was in love because you ruled out everything every small detail required + planning this makes understanding alot easier rather than directly jumping into code and saying on the go

[juanitoMint]

Really appreciate the back-of-the-envelope calculations in between!
Great work!

[rembautimes]

One application of this solution is for horizon risk scanning. The use case is that a large multinational corporation wants to have an idea of new risks which are emerging and adopting this approach allows them to have a traceability back to the web source. Of course they won’t be crawling 100M pages but maybe 100k pages.

[MrRetroboyish]

Only a 1/3 of the way through and already one of the best I've seen. Focused, logical leaps from topic to topic, minimal digressions. Keep it up

[linonator]

Wooooah!!!! This is what I needed in my life 😢. I’m now complete

[JM_utube]

I really appreciate this video! Information was clear and concise. Levels of depth are perfect for the viewer to be able to continue educating themselves about any of the topics mentioned here. Thank you so much

[Roshen_Nair]

Loved the video! A video I'd love to see in the future is system design for video streaming applications e.g. YouTube, Netflix.

[dave]

Dang, I never thought I could understand this whole process. I typically wrote off most of the implementation details as a black box, but this seems halfway approachable.

Has me thinking a lot about single page applications, and how the crawlers handle them. A similar type of video would be awesome if you had it.

[BraisonsCrece]

keep it going!
High quality content and a very solid platform! Without a doubt, I will buy the subscription soon and start learning!
a hug from a new Spanish subscriber