Diving into the embarrassing engineering behind CrowdStrike

The constant failures of CrowdStrike are truly unbelievable. A $10 gift card??? Ugh.

SOURCES

S/O Ph4se0n3 for the awesome edit 🙏

[LadyEmilyNyx]

This is one of those times when if you wrote it in a movie, everyone would say "That's so unrealistic. No company could be THAT incompetent without going out of business."

[grimhavoc_]

Theo: "..and my conclusion is that either Crowdstrike is incredibly stupid, or they are lying."
Me: "Why can't it be both?"
Crowdstrike: "Would you like a giftcard? :D"

[kevinletterer]

The 10 dollar gift card is a trap, if you accept it then they can claim that they have already compensated you for any damages and you agreed to it.

[arcanernz]

It’s not crowdstrike’s fault the update worked on Dan’s computer; the millions of other computers were just edge cases.

[JoyceMuller-xvkh]

This global internet outage is insane! All airlines grounded and i was stock the airport and even banks, media, and offices from the U.S. to Australia. How can CrowdStrike have such a monopoly that could help restore such a massive amount of tech?

[autohmae]

If they haven't signed the 'content', this is probably the worst failure of all, for a _security_ company, because they are creating a HUGE security issue.

[PatNeedhamUSA]

I find it kind of hilarious that a third or half of their PIR document is marketing material

[desertdude]

The craziest thing is that the CEO of Clownstrike was CTO at McAfee when they pushed the update that thought svchost.exe was malware and nuked millions of Windows XP installations.

[stagefan]

This company is named 'Clownstrike' for the rest of time.

[hugazo]

So tldr: so many avoidable errors, that exploded in the most spectacular way

[kyle]

"Disasters don't just happen, they're a chain of critical events"

[cabanford]

So this update bricked every machine it was pushed to... Screw the Rolling Updates - didn't they test this on even a "single" internal machine before their Friday push??? WTF. Managers heads (CEO)

[halbeik]

A smoke test would've caught this at any point in the pipeline

[luketurner]

Aviation has the Swiss cheese model
ClowdStrike had the NO cheese model

[OhhCrapGuy]

The ABCs of running a company like Crowdstrike:

Airline: We forgot to check if we put any fuel in the plane, or if the engines worked, or if they were attached, but at least there was a pilot in the plane before takeoff. And it's a good thing that Southwest pilot was on our flight.

Bar tending: We have an excellent bar, high quality glassware, exquisite lighting, a phenomenal sound system, and only the best point-of-sale system available. Let's open the doors, it's time for the grand opening. ... wait, we have to buy the booze? I thought whenever a bartender came in, they'd bring the booze with them! What do you mean "hire a bartender"?

Car manufacturing: Look, we made sure that all the wheels were firmly attached to the vehicle. Sure, one was bolted to the hood, another was attached to the middle of axle, which caused the axle to snap in half, and the last one was put in place of the steering wheel, but we made DAMN sure that it had all 3 wheels attached this time. ...what do you mean "4 wheels"?

[conceptrat]

A little bird told me that the CI/CD process was taking too long and the maxing out the processors. So they removed some parts of the process (tests?) and then pushed start again. And here we are. Partially.

[DuraanAli]

I have a small software business, it's not even my full time job and I don't do updates unless it goes through unit tests, staging and behavior monitoring with applications like Sentry and Hotjar! It's so weird to see a huge company like that just pushing updates even if they're urgent.

[toastrecon]

I don’t think that the PR team is necessarily inept, I’d bet more than a few $10 Uber Eats cards that the statement was wordsmithed for hours by engineers, lawyers, executives and whoever else felt like their neck was on the line. Endless revisions until it said everything and yet nothing.

[wrfsh]

3:54 Hey Theo, you're wrong here about graphics drivers. WHQL is not there just to put you into the next windows update. It allows the driver to be signed, and the windows kernel won't allow you to load unsigned drivers (normally, unless you're in debug mode and don't have secure boot enabled). So even if you distribute your driver through your own channel, you still need to get it signed, and for some drivers that means going through WHQL. CS sidestepped it with the channel files because they don't actually load those files as drivers, as i think you correctly point out.

[leexgx]

20:20 rolled out the update on a Friday at 10pm pt (-7 or 8 gmt, 5am ish uk/eu) the best time to brick 9 million systems