Widespread Windows Crashes Due to Crowdstrike Updates

CrowdStrike

American cybersecurity technology company

What is CrowdStrike?

CrowdStrike delivers a unified platform approach to modern security so you can protect and build your future.

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas. It provides cloud workload protection and endpoint security, threat intelligence, and cyberattack response services  The company has been involved in investigations of several high-profile cyberattacks, including the 2014 Sony Pictures hack, the 2015–16 cyberattacks on the Democratic National Committee (DNC), and the 2016 email leak involving the DNC.  In July 2024, a faulty update to its security software caused global computer outages, impacting air travel, broadcasters, and other services.

  • July 19, 2024

CrowdStrike update crashes Windows systems, causes outages worldwide

A faulty component in the latest CrowdStrike Falcon update is crashing Windows systems, impacting various organizations and services across the world, including airports, TV stations, and hospitals.

The glitch is affecting Windows workstations and servers, with users reporting massive outages that took offline entire companies and fleets of hundreds of thousands of computers.

According to some reports, emergency services in the U.S. and Canada have also been impacted.

“We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers” – George Kurtz

CrowdStrike’s CEO says that a fix is available and advises customers to access the support portal for the latest updates.

The company also provides two options to address the issue in cloud and virtual environments, one variant being to roll back to a snapshot before 04:09 UTC. The second option is the following seven-step procedure:

  • Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Detach the volume from the new virtual server
  • Reattach the fixed volume to the impacted virtual server

Outage hits airlines and hospitals worldwide

By the time of the correction, though, many large organizations across multiple verticals had already been affected.

Some reports say that CrowdStrike’s update impacted some 911 emergency service agencies in the state of New York (EMS, police, fire department), Alaska, and Arizona, as well as 911 services in parts of Canada.

A 911 telecommunicator in Illinois said that they were “working off of paper until things come back.”

There also reports that the health hotline in Catalonia, Spain, is impacted and authorities are asking citizens not to call 061 unless there is an emergency.

Dutch broadcasting organization NOS said that the glitch created disruptions at Schiphol Airport and “forced several flights to be grounded” (operated by KLM and Transavia).

Melbourne Airport said that it was experiencing “a global technology issue which is impacting check-in procedures for some airlines.” The most affected are passengers departing internationally via Jetstar and Scoot airlines.

A few hours ago, in the latest update, the Zurich Airport says that “flights with destination Zurich that are already in the air are still allowed to land,” no aircrafts “are currently taking off for Zurich Airport,” and there are no departures to the U.S.

Furthermore, there are delays and cancellations and passengers of individual airlines must be checked in manually.

Other airports affected are in Berlin, Barcelona, Brisbane, Edinburgh, Amsterdam, and London.

In the U.S., the Federal Aviation Administration received requests to assist multiple airlines (American Airlines, United, Delta) with ground stops until “a technical issue impacting IT systems” is resolved.

On JFK and LaGuardia airports in the U.S., flights have been grounded due to outages from the CrowdStrike update, leaving passengers stranded.

Some hospitals in the Netherlands – Scheper in Emmen, Slingeland Hospital in Achterhoek, and emergency posts in Hoogeveen and Stadskanaal were also impacted.

In Barcelona, the Terrassa University Hospital and the Catalan Oncology Institute experienced issues earlier today due to the CrowdStrike issue but have started to return to normal activity.

In the U.S., Bellevue hospital in New York and NYU Langone Hospital are also impacted.

On Friday morning, multiple television stations and news outlets, such as Sky News and ABC suffered disruptions as computers crashed.

A large number of users started to spill their frustration in Reddit comments about tens and even hundred of thousands of computers crashing after CrowdStrike’s update and the impact on their companies:

Malaysia here, 70% of our laptops are down and stuck in boot, HQ from Japan ordered a company wide shutdown

210K BSODS all at 10:57 PST….and it keeps going up…this is bad….

Workstations and servers here in Aus… fleet of 50k+ – someone is going to have fun.

Failing here is Australia too. Our entire company is offline

Same here in OZ. Entire company is down.

Half the company down. Somehow it has hit our AWS servers also. Major service downtime for our customers

Entire org and trading entities down here. Half of IT are locked out.

Seeing major issues here in NZ at the moment, company wide outage impacting servers and workstations.

Supporting Philippines and China Locations. All experiencing the same as well

Despite a fix being deployed and CrowdStrike providing a workaround for Windows hosts already crashing, companies will feel the effects from the issue for a while.

Admins are going to have a long weekend, especially with computer fleets of tens or hundreds of thousands of computers, employees working remotely, off-premise data centers, or cloud environments where booting in safe mode is not an option.

CrowdStrike’s CEO says that a fix is available and advises customers to access the support portal for the latest updates.

Update [July 19, 09:59 ET]: Article edited to include mitigation details for cloud and virtual environments.

READ FULL ARTICLE HERE


SOURCE: BLEEPING COMPUTERS