What Caused the Largest IT Outage Ever?

This week, Marques, Andrew, and David talk about the Tesla Roadster finally (maybe) possibly being ready to begin shipping in 2025. Then they discuss the CrowdStrike IT disaster before switching to Samsung devices at the Olympics. After that, they go over some rumors that Apple might actually make screens with folding displays in the near future before wrapping it up with a new keyboard that will help you cheat. It's a fun one, enjoy!

Chapters
00:00 Intro
01:05 Welcome to the Fediverse
02:32 Tesla Roadster Coming Soon?
12:33 CrowdStrike Fiasco
30:18 Trivia
31:24 Meta AI (Sponsored)
32:26 Samsung at the Olympics
38:19 Apple Rumors and Renders
49:20 Alphabet Commits $5B to Waymo
53:01 Trivia
54:16 Meta AI (Sponsored)
55:19 A Keyboard for Cheating?
01:08:24 Trivia Answers
01:17:41 Outro

Links:

Shop the merch:

Socials:

TikTok:

Join the Discord:

Music by 20syl:

Waveform is part of the Vox Media Podcast Network.

[vaGin]

Ellis should get the trivia point every time he sweeps the crew and they all get the question wrong

[iMikeTech]

One insane thing about the Crowdstrike issue that you don't hear a ton about is Correctional facilities. I work in a federal jail and when this issue hit, my facility lost all computers AND all cameras. we went like 18 hours without camera access. Scary situation to be in.

[bojome]

Big fan but this meta ad gives me chills every time. Such a contrast to any other ad.

[Pottapatri]

David saying I don't know" at 45:31 should totally be made into a soundbite.

[thanos]

I still remember Y2K even though I was only a few years old. We coincidentally had a power outage and I heard some random dude outside screaming It's forever etched in my brain 😂

[xayushx]

I didn't know *Majin Buu* was into tech. Now Goku is literally dumber than Buu

[jaganathannatchimuthu]

because of all the actually potentially apparently allegedly stuffs in this podcast, this could potentially be the biggest and the greatest tech conspiracy theory podcast. Welcome!

[soundninja]

If anyone wants a less technical explanation to the crowd-strike issue than Dave's, I recommend the computerphile video on the topic. It's two CS professors explaining it to a non-technical layman. Theo - t3 also has a great video called "diving into the embarrasing engineering behind CrowdStrike" that breaks down crowdstrike's statement. It's more geared towards people with dev backgrounds that aren't versed in security, but I'm pretty sure it should be perfectly understandable by any layman. But it's very ranty, so you've been warned.Also to comment on a couple of the things you guys were wondering about:

You're completely right that they tried to find a way to circumvent Microsoft's validation methods by pushing code to kernel level software without pushing code to kernel level software. In their legalese, they're trying to justify this by saying it's just binaries labeled as content and not code (which is true, but they're not adressing the fact that it's just as bad). Yes it's allowed by microsoft, but it's the same way loopholes in the law are perfectly legal. It's against the spirit of the rules.

About not testing on production. Software validation is never perfect, as these tests are written by engineers, often the same engineers that write the code that the tests validate, and you can't always catch everything. That's why staggered release would be better. Essentially, you send the update out to a very limited number of machines, preferably your own machines. Let's say 100 test units somewhere that the devs can check on. Then 1000 client machines. Then 5000 more... etc so that you have much more time to make sure nothing breaks as disastrously as this.

[dit]

The bigger issue was that they didn't target the deployment. It just went everywhere all at once. Even after testing, you normally push things out in waves so as to not have issues like this. Also the reboot 15 times was because of when the BSOD occurred. the computer was booted for a really short amount of time.After 15 reboots it was possible the CrowdStrike fix would reach the machine while it was online before it died and allow it be fixed.

Another big issue people had were that their bit locker keys were stored on machines that couldn't boot so IT had to fix infrastructure in order to get the keys to even try to fix other computers.

[BreakawayB]

That “Steeplechase” joke made me LOL so hard 😂

[Pjty]

As a keyboard enthusiast I’m glad that you mentioned Wooting as the pioneer in hall effect switches and analog input for keyboards. I think Wooting as a company is still way too underrated, even tough (as the underdog) they single handedly changed the entire gaming keyboard game and and were the innovative spearhead for years (and still are)

[cocutou]

George Kurtz, the CEO of Crowdstrike was a previous CTO of McAfee in 2009. Back then, McAfee also BSOD a bunch of Windows XP computers by pushing out a software update that deleted Windows system files.

Let's just say, this guy has a history of causing worldwide computer outages and didn't learn a single thing.

[sean_vikoren]

Nice nerd out on the 'hall effect', otherwise known as;
magnetism and electricity are two side of a very strange coin.
It's also how to cheat at the 'i'm not touching you' game.

[Eesemoore]

Ellis casually growing a moustache. And nobody is talking about it.

[godbeforeme]

I wish I had my own personal Ellis to answer my own random tech questions in such a concise and instantaneous manner.

[sachinraj]

Drawing QR codes out? 😂 I see what you did there Marques. Subtle and effective 😂

[upstarter]

Re: Razer keyboard.
This conversation is as old as time in technology. For example, I DJ for over 24 years. You can imagine going from vinyl to digital media how much the equipment has changed. “Back in the day“ DJs had to manually beat match songs so they mixed without sounding like a train wreck. Now, or I should say quite a while ago a feature was introduced called “SYNC“ to wear instead of using the pitch feeder and manually having to listen to the track to judge how fast or slow it is, you press the button and it locked every track loaded to the correct beat per minute. TJ is all around. The world had strong opinions about it, and now a decade or so later it’s as ubiquitous as a technic turntable used to be. Move with innovation or get left behind by it.

[SellyS.]

Dave also mentioned there was apparently some regulation that prevented them from implementing an API that wouldn't prevented this sort of thing from happening...

[meta_NX]

It was 10AM in the morning and I was in a Teams call when suddenly I got the BSOD. I spent the next 3 hours trying to reach out to the tech support and finally when I got connected I was informed that it was a global outage. Over 60% of the systems from my company were down. And to solve this you either had to take your machine to the Support centre or connect with Tech support... Which took another 2 hours on Sunday to fix.

[nikeadidas]

It would be honestly cool to have David Plummer be a guest on this podcast. Guy has a lot of insight since he’s an OG dev for Windows