![shareicon](https://www.globallogic.com/wp-content/themes/GLTheme/gl-assets/images/shareicon.png)
-
-
-
-
URL copied!
Incident Summary
On July 19, 2024, CrowdStrike released an update to its Falcon platform that resulted in widespread system instability, specifically causing Windows systems to experience the "Blue Screen of Death" (BSOD). The issue was traced to a channel file named “C-00000291*.sys” included in the update, which caused system crashes upon deployment. “Channel files” are part of the behavioral protection mechanisms used by the Falcon sensor, and are updated several times a day in response to new tactics, techniques, and procedures (TTPs) discovered by CrowdStrike.
This flaw had a significant impact on global operations, affecting critical infrastructures such as airports, hospitals, and news outlets. Although CrowdStrike acted quickly to identify and revert the problematic update, the disruption required extensive manual remediation across numerous systems.
Key Takeaways
This incident first and foremost underscores the importance of rigorous pre-release testing and deployment protocols. Moreover, the critical role of disaster recovery plans and efficient communication protocols are evident in managing and mitigating such widespread disruptions.
This incident also highlights significant security risks, including the potential for vulnerability exploitation during periods of system instability, as well as increased risks associated with the operational disruption of critical infrastructures like airports and hospitals.
Best Practices
We will not discuss here the technical remediation plan for the incident, but rather use GlobalLogic’s extensive experience and expertise in software development and deployment to suggest some best practices companies should consider to mitigate unnecessary risks and avoid such incidents in the future:
Comprehensive Testing: Ensuring updates are thoroughly tested in varied environments before deployment can prevent issues. Testing should include not just functionality, but also performance and stress testing on different Windows configurations and versions.
Staged Rollouts: Gradually deploying updates in stages allows for monitoring and addressing issues before they affect a wider user base. This approach can help identify and rectify problems early in smaller, controlled groups.
Rollback Mechanisms: Implementing automated and efficient rollback mechanisms can quickly revert changes if issues are detected. This helps minimize downtime and disruption.
Monitoring and Analytics: Continuously monitoring the performance and behavior of updates through analytics can provide early warnings of potential issues, allowing for quicker intervention.
Communication and Transparency: Prompt and transparent communication with customers about potential issues and ongoing fixes helps manage the impact and maintains trust. CrowdStrike’s provision of remediation steps and communication about the issue were essential, but earlier and more proactive communication might have lessened the impact.
Disaster Recovery Plans: Having robust and well-practiced disaster recovery plans ensures that there are clear, effective procedures to follow in the event of a widespread issue. This includes having backups, failover systems, and clear communication channels.
Conclusion
Implementing these best practices, backed by GlobalLogic’s proven track record, can help organizations mitigate risks associated with software updates and ensure smoother, safer deployments. By focusing on comprehensive testing, staged rollouts, robust rollback mechanisms, continuous monitoring, effective communication, and well-developed disaster recovery plans, organizations can significantly reduce the likelihood of disruptions and enhance their overall risk management strategy.
Trending Insights
![If You Build Products, You Should Be Using Digital Twins](https://www.globallogic.com/wp-content/uploads/2023/03/Digital-Twins.jpg)
If You Build Products, You Should Be Using...
Digital TransformationTesting and QAManufacturing and Industrial![Empowering Teams with Agile Product-Oriented Delivery, Step By Step](https://www.globallogic.com/wp-content/uploads/2020/01/automotive.jpg)
Empowering Teams with Agile Product-Oriented Delivery, Step By...
AgileProject ManagementAutomotiveCommunicationsConsumer and RetailMedia![ajax-loader](/wp-content/themes/GLTheme/images/ajax-loader.gif)
Let’s Work Together
Related Content
IIoT: The Future of Manufacturing
Evolution of Industrial Innovation: How IIoT Will Impact Manufacturing in the Future? The Manufacturing Industry is entering a new era thanks to the Industrial Internet of Things, or IIoT. This revolutionary technology is dramatically reinventing manufacturing with the integration of digital technology into processes that enhance output quality, reduce costs, and increase productivity. IIoT … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Generations and GenAI
This is probably a well-known fact in sociology or some other such discipline, but it struck me the other day that only the generation that knows how to do something can be the one to make that thing obsolete. Take driving a car, for example. My generation and the ones preceding me in the U.S. … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
We’re getting the chance to live in the future
Early 20th Century motivational speaker and author Dale Carnegie once wrote “Today is the tomorrow you worried about yesterday.” I believe that Mr. Carnegie’s point was that unless today is the literally the worst day of your life (and my sincere sympathies if it is), then the energy you spent worrying about it yesterday was largely wasted. I haven’t read much … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Intelligence is Intelligence, even if it’s Artificial
I had a stimulating conversation with the head of our GenAI practice, Suhail Khaki, a few weeks ago. Suhail made the remark that the more he works with GenAI, the more it strikes him that it’s less like conventional computer software, and more like a person in the way it interacts. He made the remark: … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Retail as a Conspiracy
I was one of the early buyers of the first release of Apple Vision Pro AR headset early this year. I got up at 5am my time to place an order on-line at the first moment when the device became available for pre-order. I then made an appointment at my local brick-and-mortar Apple Store to … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Share this page:
-
-
-
-
URL copied!