IT admins around the world are scrambling to fix a major issue with Windows computers today after a faulty update from cybersecurity provider CrowdStrike knocked thousands of PCs and servers offline with a Blue Screen of Death (BSOD) error. While CrowdStrike has fixed the update that originally caused the problems, many systems are still offline, with banks, airlines, supermarkets, and TV broadcasters struggling to cope without their machines.
The fix, for many, won’t be easy. IT admins are still trying to use an initial workaround provided by CrowdStrike, which involves booting Windows systems into Safe Mode and deleting a system file:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:WindowsSystem32driversCrowdStrike directory
- Locate the file matching “C-00000291*.sys” and delete it
- Boot the host
These steps force Windows to boot into a Safe Mode environment where third-party drivers like CrowdStrike’s kernel-level driver aren’t able to load. IT admins then have to locate the faulty driver on the disk and delete it. This workaround requires, in most cases, physical access to a machine. And in some environments, it could be complicated by disk encryption like BitLocker or even a lack of admin rights to be able to delete the faulty driver.
The other option is to wait for CrowdStrike’s fix to come through — but getting it has been a problem. Some IT admins are simply rebooting machines over and over, hoping that the CrowdStrike update will get pushed through the network stack before CrowdStrike’s protection engine initializes and then BSODs the machine. Turning machines off and on again (yes, really) seems to be working for some, with reports of machines coming back online after being rebooted multiple times.
CrowdStrike’s update server and content delivery networks are likely being hammered by the millions of machines reaching its servers for an update, so it may take some time for the reboot method to work.
Businesses running virtual desktops may be able to recover quicker than others by simply restoring affected hosts back to a point before CrowdStrike’s faulty update wreaked havoc. In environments where rebooting isn’t working, the workaround of booting into Safe Mode looks like the best option right now.
Either way, this issue isn’t going to be resolved in a matter of hours like the typical internet outages we see from cloud providers. “It could be some time for some systems that won’t automatically recover, but it is our mission to make sure every customer is fully recovered,” says CrowdStrike CEO George Kurtz in an interview with NBC News.
In that same interview, Kurtz apologized for the damage caused by CrowdStrike’s update, but there will undoubtedly be questions around how a faulty update like this ever managed to hit thousands or millions of machines around the world.