CrowdStrike IT Outage Explained by a Windows Developer

Follow me for updates!
Twitter: @davepl1968 davepl1968

Opinions are mine only, not a spokesperson!

I should add that I don't know if the channel definition update files were privately signed or not; what I meant is that I presume they do not go through the WHQL signing process. But even if they do, we've learned that trusting their content (which was all zeros in this case) didn't go well.

[Yandarval]

"Agile, ambitious and aggressive" the sarcasm with which this phase was uttered, wonderful.

[NealB]

3 days ago no one outside of IT had ever heard of Crowdstrike. Now the entire world knows the name. Reputation destroyed in an instant.

[rezephuafrogs]

When I was in high school I had a teacher that had a way of explaining things to you that temporarily elevated you to a fraction of his level of understanding. Today I got to experience that again. Thank you Dave! 🤯

[ShawnWrona]

What a great explanation!
No bull crap. No conspiracy theories. No badmouthing. Just plain facts. Even me… who rarely uses a computer anymore understands, and follows Dave’s explanation and walks away a little more knowledgeable. Thanks Dave😊

[StarLightDotPhotos]

As a former CrowdStrike employee this is the best explanation I have heard and is 100% accurate.

[zug-zug]

While this is technically what crashed machines it isn't the worst part.

CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of business have a N-1 or greater staging policy and only test systems get the latest updates immediately. My work for example has a test group at N staging, a small group of noncritical systems at N-1, and the rest of our computers at N-2.

This broken update IGNORED our staging policies and went to ALL machine at the same time. CS informed us after our business was brought down that this is by design and some updates bypass policies.

So in the end, CS caused untold millions of dollars in damages not just because they pushed a bad update, but because they pushed an update that ignored their customers' staging policies which would have prevented this type of widespread damage. Unbelievable.

[mikezimmermann]

Dear God! I’ve been out of the IT world for 15 years now, and I still understood his explanations. I’m VERY IMPRESSED by Dave’s clear and concise presentation and astounded by the fact that I remembered enough of this “stuff” to finish some of his sentences! Until today, I was convinced that a benevolent universe had purged all that out of my head to make room for important stuff (like cocktail recipes).

[VeritasAlienari]

You're obviously a skilled and experienced technical powerhouse, but the writing style (sarcasm, wit, technical aptitude combination) and delivery make this more than just a "system dump" of data the viewer has to try and digest. Instead, we're treated to a bit of entertainment as we debug.

Thank you for the package deal.

[MrKvasi]

The company I work at got bought by a bigger one. They required us to install Crowdstrike on all servers. We found a memory leak, that Crowdstrike still hasn't fixed after 6 months so I have refused to install it until then. I was on vacation when I saw all URGENT emails from other divisions.
Thank you Crowdstrike for not fixing your memory leaks, it saved my vacation. =P

[Vladimir_Kv]

The most funny thing is that CEO of Crowdstrike was a CTO at McAfee... during their worldwide faceplant.

[fbmowner]

What I've learned so far is that every OS has a big boss and that big boss ensures everyone follows the rules and as soon as someone gets out of line the big boss shuts the party down before the looting begins. In all seriousness this is a great video. Subbed!

[garysturdivant]

Extremely well and clearly described, Dave. As a former kernel developer (at Tandem Computers), we didn't allow such back doors, but then we were being deployed as a 24x7 hardware/software fault-tolerant server system and did not have millions using our systems, developing third-party drivers (or attacking them).

Yes. Multiple failures at Crowdstrike. Someone wrote that driver code without the requisite error checking, no one caught it in reviews/inspections (if they do that, and if they don't...don't even want to go there), no one in QA thought to test for it or ran the test, someone in the release chain submitted that file (or failed to substitute the correct one if the default is an all zeroes file), etc. I don't expect today's developers/QA to think like we did (what could be corrupted if the processor/driver/adapter/etc fails between this instruction and the next and how can I prevent that corruption). Too time consuming and non-agile. But...apparently no one considers the consequences of not doing so and the damage to customers and the company it causes, or the bean-counters dismiss it as too unlikely and worth the risk.

[mikeyoung]

Love that while stuck at the airport Dave opened his MacBook. A fair amount of dry humor in this vid.

[alleneng]

for some reason dave's explanation was waaay easier to understand than every other video about this

[spyderwalker]

being in IT for 30 years, your video is precise, easy to follow and on point. Well done.

[rokombolo]

Our engineer dodged this one by not signing up for CS and keeping Sophos. CS charges about $30k extra for content filtering, which Sophos includes. We have computers all over the world so this would have hit us hard not being able to get to all those remote users and sites.

[mhewett]

I am a network systems engineer that had to deal with this for 14 hours that day. This was one of the most informative videos I have ever seen. You helped simplify Windows OS in 15 minutes in a way that hours of reading hasn't. Something about real world scenarios to tag the concept with in my memory really helps. Thanks!

[CHmLgN]

I just learned more about system functions in 5 minutes then I would’ve imagined. What a clear breakdown on things.

[locutus]

I haven't heard talk like this in almost 40 years! Thanks for the memories! 😁

[BoStern]

Thanks Dave, found myself to be on the spectrum just a few years ago, at 53. Changed everything! Thanks for your extremely lucid, helpful and complete lessons on this channel! 🙏🏻