Why Did Facebook Go Down? - Computerphile

Just what was it that took Facebook, Instagram & WhatsApp offline on 4th October 2021? - Dr Steve Bagley investigates!

previously titled "Facebook's Day Off"

The visualisation software Steve was using was BGP play, downloaded from the RIPE website.


This video was filmed and edited by Sean Riley.

[HerrHeisenheim]

What's a phone book you ask? Well, it's like DNS, but for people and their telephone numbers.

[jonathanjacobson]

Ironically, Facebook's mission statement is best fulfilled when Facebook is disconnected from the Internet.

[danielcarlossmd]

it's like Stackoverflow going down and their devs not having where to search for the fix

[robertoricardoruben]

The phone book names are the same that the teacher calls in "Ferris Bueller's day off". Brilliant

[Muthwill]

I love how you're talking about DNS, IPs, ISP and several technical terms but need to explain what a phonebook is <3

[thedrusus]

The world briefly became a better place

[allesarfint]

It was beautiful, but as everything beautiful it didn't last long.

[ferdievanschalkwyk]

Love the Apple magic mouse in its natural position, upside down, asking you to recharge it.

[AttilaSVK]

This reminds when I worked on tech support at an antivirus company. The IT admin of a mining company called us that he had accidentally pushed the wrong configuration out all of their computers, including the mission critical ones down in the mine. I cannot recall what his config was meant to look like, but the wrong one was set up to run a total scan of all hard drives every 10 minutes. Since a single scan took significantly longer than 10 minutes to complete, many parallel scans were running, slowing the system down even more and every 10 minutes a new scan job would start.
We had to use a special, last resort tool, which would kill the antivirus kernel, but getting hold of it was hard (it was given out only with a special permission).
My advice for that IT admin was to try new antivirus configurations on his own PC first (or get a sacrificial one for tests), then push out the config to 10 machines (preferably of tech savvy people). If there are still no problems, push out the config to 20-50 more computers, and so on, but never ever do it to the entire network.

[lecsg]

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."
Douglas Adams

[grahameida]

One thing I learned the hard way on this, whenever working on a remote router always set a timed reboot on the router before you make the configuration changes. So if the worst happens you just wait for the reboot into the saved configuration and not the running one. instead of jumping in your car to the remote location.

[marklonergan]

It may have been a throwaway question, but i was talking to my 11 year old nephew a couple of weeks ago and i mentioned a phonebook and he didnt have a clue. His best guess was a diary that you write your names and numbers into but he couldn't get his head around the idea that we used to get a catalogue of everyones phone number posted out to us every year.

[AlanTheBeast]

I like that FB designed things to lock themselves out of FB.

[fmdj]

We need to start explaining things the other way around: a phone book is like a DNS for phone numbers.

[LoPhatKao]

imagine a world where it took days rather than hours to fix

personally, i think this timeline lost

[locust]

I don’t understand how Facebook didn’t have an out-of-band back door in case their ASN completely vanished from routing tables everywhere. I mean, these guys helped write the book on data center infrastructure Best Practices

[FlorianEagox]

it's kinda wild to me how people LOST THEIR MINDs over this.
Back only a few years ago, stuff would go down and you'd just say, "idk, shit's down right now, I'll try again later"

[MrRyanroberson]

What's a phone book, grandpa?
"Well, an old philosopher put it like this: it was the great book of doxing. It told you the names, numbers, and sometimes locations of everyone in town."

[flamencoprof]

The national network I have worked on since 1975 used to have a thing called "the Shout-down". It consisted of analogue audio circuits over copper only, between major control rooms or what-have-you. Everyone had to call in every morning to check its integrity. It was intended to circumvent network failures and keep communication alive robustly between these places. (As a young man then it reminded me of those old marine speaking tubes from the early 20th century; Phweet! "Half astern!" "Aye-aye, Captain! Half astern it is!")
I knew the rot had set in when about a decade ago some genius decided that it was old-fashioned copper tech and they moved it to a VOIP connection.

[Novacification]

When I was studying to be a software engineer we had a semester on networks. We were assigned network equipment and tasked with setting up a network with security, services and so on. It was common for students to update the router configuration resulting in cutting off the connection that they were using to configure the router and having to get help to reset things. Almost everyone did this (including our group) and it was like a rite of passage.