How Roblox Went Down For 73 Hours

A look into what happened behind the scenes during the longest outage in Roblox history.

Sources:

Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution

Corrections:
- At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a **buffered channel of size 1** however the overall point still stands.

Music Credits:
- Impact Prelude by Kevin MacLeod
- We're Finally Landing by Home

[sirgamsay]

Imagine just doing a Hobby project to understand a piece of Software and suddenly the complete Roblox Infrastrucure is build on it.

[justliliandev]

every time I hear that someones hobby project caused a major outage somewhere I get the feeling that maybe big corporations should maybe check what software they are built on and support it's development/maintenance

[klhmn]

Turns out, this video could be a great introduction to modern backend architecture and development.

[cphr]

the negative 900 million dollars hits hard 😭

[ratm]

"A massive company with ... -$924 million net income" 💀

[yeetyeet]

github repo: "it was a toy project never meant for production"

multibillion dollar company: "YAYEET"

[ccccy-ox]

Hi Kevin, amazing content as always! One minor correction @9:54 tho: Go unbuffered channel's length is 0, instead of 1, and it means the sender will get blocked until a receiver receives the value. What the video @9:54 showcase is actually a buffered channel with length 1 (e.g. result of make(chan string, 1)).

[useruser-tiog]

This is like the XKCD of all of the world depending on a toy project someone abandoned 10 years ago

[theprantadutta]

This is one of the biggest challenges of modern programming, depending on various 3rd party packages, not knowing what that package is, what it does, or whether it's even reliable, and moreover knowing what are the dependency of that 3rd party package and whether they are safe or not.

[mrdabup]

I still remember the day that it went down, people were blaming Chipotle (american fast casual chain) because they had an event that same day where you could claim a free burrito. People suspected that it was due to a mass influx of people, I knew (and a bunch of people too) that this wasn't an issue with influx of people. At the end of the day, it was a fun journey (more or less with the conspiracies, guessing correctly that it went down for 3 days months before this outage, and youtubers just milking on the outage). Thank you for making a video about this.

[TheeSirRandom]

Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.

[nebufabu]

Whatever it took to make a video about a Roblox server crash and not use the "oof" SFX even once... I salute it.

[skirn]

the kids enter angry
the kids leave confused

[yaakovwaxman]

This is by far my favorite documentary channel on yt

[BananasAintCheap]

It’s crazy how much of the internet as a whole is in the hands of solo developers who made a thing in their spare time for fun

[MaximumADHD]

Oh shit I was gonna suggest this as an idea, awesome to see that you did it!

[superbobsaget]

Thank you for all the work you put into making this!!

[i-am-linja]

I'd imagine programmer Hell is just a bug like this which takes all of Eternity to fix, also it takes down the company's internal issue tracker and communication system.

[sergelorenzvillasica]

Can't wait for the CrowdStrike episode 😀

[davidslevs]

Roblox is actually a bigger company than most think. Thanks for doing a video on it.