-
-
-
-
URL copied!
Incident Summary
On July 19, 2024, CrowdStrike released a content configuration update for the Windows sensor that resulted in widespread system instability, causing Windows systems to experience the "Blue Screen of Death" (BSOD). The issue was traced to a channel file named “C-00000291*.sys” included in the update, which caused system crashes upon deployment. “Channel files” are part of the behavioral protection mechanisms used by the Falcon sensor, and are updated several times a day in response to new tactics, techniques, and procedures (TTPs) discovered by CrowdStrike.
This flaw had a significant impact on global operations, affecting critical infrastructures. CrowdStrike acted quickly to identify and revert the problematic update, but the disruption required extensive manual remediation across numerous systems.
Key Takeaways
This incident underscores the importance of rigorous pre-release testing and deployment protocols. The critical role of disaster recovery plans and efficient communication protocols are evident in managing and mitigating potential widespread disruptions.
Significant security risks, including the potential for vulnerability exploitation during periods of system instability, as well as increased risks associated with the operational disruption of critical infrastructures require an extensive and proactive mitigation approach.
Best Practices
Best practices to mitigate unnecessary risks and avoid potential outages to critical infrastructure in the future include:
Comprehensive Testing: Ensuring updates are thoroughly tested in varied environments before deployment can prevent issues. Testing should include not just functionality, but also performance and stress testing on different Windows configurations and versions.
Staged Rollouts: Gradually deploying updates in stages allows for monitoring and addressing issues before they affect a wider user base. This approach can help identify and rectify problems early in smaller, controlled groups.
Rollback Mechanisms: Implementing automated and efficient rollback mechanisms can quickly revert changes if issues are detected. This helps minimize downtime and disruption.
Monitoring and Analytics: Continuously monitoring the performance and behavior of updates through analytics can provide early warnings of potential issues, allowing for quicker intervention.
Communication and Transparency: Prompt and transparent communication with customers about potential issues and ongoing fixes helps manage the impact and maintains trust. CrowdStrike’s provision of remediation steps and communication about the issue were essential, but earlier and more proactive communication might have lessened the impact.
Disaster Recovery Plans: Having robust and well-practiced disaster recovery plans ensures that there are clear, effective procedures to follow in the event of a widespread issue. This includes having backups, failover systems, and clear communication channels.
Conclusion
Implementing these best practices, backed by GlobalLogic’s proven track record, can help organizations mitigate risks associated with software updates and ensure smoother, safer deployments. By focusing on comprehensive testing, staged rollouts, robust rollback mechanisms, continuous monitoring, effective communication, and well-developed disaster recovery plans, organizations can significantly reduce the likelihood of disruptions and enhance their overall risk management strategy.
Author
Top Authors
Senior Manager
Consultant
Chief Technology Officer, AI
VP of Healthcare Technology
CTO
Let’s Work Together
Related Content
Accelerating Digital Transformation with Structured AI Outputs
Enterprises increasingly rely on large language models (LLMs) to derive insights, automate processes, and improve decision-making. However, there are two significant challenges to the use of LLMs: transforming structured and semi-structured data into suitable formats for LLM prompts and converting LLM outputs back into forms that integrate with enterprise systems. OpenAI's recent introduction of structured … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Connected Vehicle Cybersecurity Considerations That Vehicle Manufacturers Need to Know
According to Deloitte, there will be 470 million connected vehicles on highways worldwide by 2025. These connected vehicles provide opportunities and have a higher cybersecurity risk than any other connected devices; even the FBI had to make a statement about it. A typical new model car runs over 100 million lines of code and has … Continue reading Crowd-Striked: Lessons Learned and Best Practices for Future Prevention →
Learn More
Accelerating Enterprise Value with AI
As many organizations are continuing to navigate the chasm between AI/GenAI pilots and enterprise deployment, Hitachi is already making significant strides. In this article, GlobaLogic discusses the importance of grounding any AI/GenAI initiative in three core principles: 1) thoughtful consideration of enterprise objectives and desired outcomes; 2) the selection and/or development of AI systems that are purpose-built for an organization’s industry, its existing technology, and its data; and 3) an intrinsic commitment to responsible AI. The article will explain how Hitachi is addressing those principles with the Hitachi GenAI Platform-of-Platforms. GlobalLogic has architected this enterprise-grade solution to enable responsible, reliable, and reusable AI that unlocks a high level of operational and technical agility. It's a modular solution that GlobalLogic can use to rapidly build solutions for enterprises across industries as they use AI/GenAI to pursue new revenue streams, greater operational efficiency, and higher workforce productivity.
Learn More
Share this page:
-
-
-
-
URL copied!