What Happened With the CrowdStrike Update and Azure Outage

CrowdStrike said a problem with an update the company pushed to Falcon sensors on Windows hosts on July 18 caused a blue screen of death, an issue that coincided with a Microsoft Azure outage and widespread outages across airlines, banks, hospitals, and other services.

[frederichardy]

I'm also missing a lot of things:
How the risk assessment team of these big companies didn't think that a kernel level automatic update application can brick a station/server???
For the encryption keys: no offline backup for something so important? really? It's the basic for backup (search for "3-2-1 backup rule") and for that kind of information you should have more than "basic".
Deploy an update to all your clients in a very small period? Was it really so urgent? critical vulnerability? because otherwise this is really bad practice...
I don't know what really happened but as usual there is not just one guilty party but a succession of bad decisions...

[Cantor]

I heard that the update ignored the N-1, N-2 configuration setting (where N is the latest release).

Also, if this is update was just to push "content" (e.g. virus definition signatures...) then why was the fix to delete which is presumably a file that contain device drivers or hardware configurations for the system. You would think a data only update would not modify .sys files. CrowdStrike might not tell us the details but hopefully someone whose machine bug checked will analyze the memory dump and maybe provide some insight as to what the driver was doing that prevented the rest of windows from functioning. Was it consuming IRP's and not passing them down the driver stack? It would be nice to know the lowest level of detail as possible.

[edasher]

Shouldn't have pushed a change during the middle of the Asian/European business day. Lazy American centric view of risk assessment. Software will always let you down. In the old days software changes were designed to facilitate rollback.

[therealjohnshelburne]

I am missing something. Who uses windows as an operating system for a VM? Ubuntu Linux is default system. At least that is what I was taught. I thought Windows operating system is really expensive in a vm.