The CrowdStrike Problem Isn’t A Simple Fix…

The CrowdStrike Falcon Blue Screen of Death (BSOD) crisis is insane. I still can't fathom just how bad this is. Hopefully this helps y'all understand what happened and why it's such a big deal.

TY AGAIN FOR THE STUDIO @learnwithjason

SOURCES

S/O Ph4seon3 for the awesome edit 🙏

[tdotgg]

Sorry about the frame rate issues, CrowdStrike took down my main recording rig and I had to do this on my Mac :(

[samcalder]

This is the best named company in history. This is the exact same outcome as if the entire crowd went on strike.

[mrtnsnp]

To everyone cleaning up this mess: my condolences, may your weekend rest in peace.

[kameronbrooks]

When I got multiple calls at 2AM I knew this was going to go down as one of the worst days in recent IT history.

[PatNeedhamUSA]

The largest disruption in human history caused by a missing try/catch block

[Super-idbq]

Imagine having "I broke the planet" as a hold my beer anecdote whenever you and your colleagues start trying to one-up each other on times you screwed up at work.

They failed to do a smoketest of their agent after build but before deploying it worldwide. Sounds like their software and update development process is just really not up to professional software engineering standards. At Meta, we had to have other engineers, sometimes multiple, review diffs before they would be accepted. And then there were multiple layers of CI/CD testing before exponential deployment with canary testing. You don't just push new code to all the machines all at once, because it's way too dangerous.

[gorak]

Crowdstrike is ransomware, they just have a different payment plan. You pay up front for the privilege of being ransomwared at some unknown point in the future. Turns out the unknown point in the future was today! Surprise!

[AndrewEddie]

Definitely a "zero" day problem. 

The only things saving CrowdStrike from a class action is most law firms are Windows users too :)

[einargs]

I mean, all the malware also targets windows because that's the big user facing desktop OS.

[michaelgebauer]

Sending out a sys file filled with nulls looks to me like sabotage

[kwilt]

I literally spent my entire friday manually fixing computers and explaining to people at remote locations how to fix their computers. Our entire IT department became helpdesk because of this update. You don't know the pain of explaining to a non-tech person over the phone - how to make a bootable USB, boot to it, and then enter their bitlocker recovery key so they can delete a file via command prompt until you've done it personally. I got to do that dozens of times on Friday and theres going to be lots more of this for the foreseeable future... I cannot express how much this sucks to fix even though its a relatively simple fix. It just can't be automated and it's horrible for that reason.

[MasterOfMisc]

The problem with the bootable USB thing is that a lot of corporate devices block booting from USB by default, which means the IT Team would have to tell the end user the BIOS password to get into the BIOS to change the boot order to enable booting from USB. Its a total nightmare!

[erroneum]

I mean, Windows might be the least secure how most people use it, but there's another huge facet to why it's the target of randsomware: it's absolutely dominates the end-user/workstation market, especially when you are wagering the victim can't just restore from a backup and ignore you.

[koyotecow]

They don't want to apologize cuz they don't want to admit fault and open them up to lawsuits.

[AvanaVana]

I literally just turned down an offer from Crowdstrike two weeks ago in favor of another job offer…it was a tough decision to make at the time but now it’s definitely looking like I made the right decision! 😬

[thezoidmaster]

the fact that one company can take everything down like this is scary, one bad actor and this could've been a mass malware attack instead of a simple driver error

[jasonfreeman]

Where in the hell is the testing cluster they should have deployed to first? CrowdStrike should deploy their Falcon updates to all their own machines and if they don’t BSOD after a week THEN release to the entire galactic empire.

[ValZarGaming]

"Windows is the only OS that is insecure enough to have problems like this"
Let me tell you why that's bullshit - Crowdstrike did this to our production Linux fleet back on April 19th.

[chbrules]

Yeah, I'm one of those tech guys. I'm in charge of our enterprise's cloud infrastructure (which is all our servers). I was up till 2am restoring a couple servers affected on our European side, thinking it was some weird Win update that took things down. I went to bed and was woken up 3 hours later by my boss freaking out. I spent all morning force shutting down systems, detaching and attaching drives to working systems to remove this .sys file and all. What a HUGE pain. I finally got everything working after like 5 hours of doing this crap nonstop. The poor helpdesk was stuck doing bitlocker based safe mode fixes for end users. I don't envy them...