While we slept here in Kansas City, the other side of the world started noticing an issue that would soon be felt globally—a software update from CrowdStrike that prevented Windows PCs from properly booting after a restart. Users became stuck in a blue screen of death loop, and the day has been dubbed by some as “Blue Screen of Death” day.
You might be thinking, CrowdStrike is a huge company, surely they have multiple layers of Quality Assurance checks along the way for these software updates. Yes, and no. Essentially, there are two types of updates in the cybersecurity industry. The first is the update most people think of: a program update. These are normally pushed extremely slowly and methodically in batches. So if there happens to be a problem, it won’t affect every single customer at the same time.
However, in the cybersecurity realm, there are also security and definition updates. These updates allow security software, like CrowdStrike, to respond to new threats. Because new threats arise daily, definition updates are pushed as soon as possible to everyone. Similar to how a vaccine teaches your body how to fight against viruses, definitions act the same way for security software. Unfortunately, in this case, the definition update brought down millions of computers.
CrowdStrike's Apology: Acknowledging the Effort of Support Teams
The fix for this issue, while simple, is time-consuming and cannot be automated. It also requires some technical skill. This means that technical support teams will need to manually assist users in resolving the problem, which can be labor-intensive, especially for large organizations. That’s why many techs have been working overtime. Imagine if you were the only technician at KU Med and had to visit all their health system buildings and hundreds of medical offices to do a 5-minute fix. Do the math. CrowdStrike even sent out $10 Uber Eats gift cards to support staff and partners as an apology.
How Business Data Services Helped Kansas City Businesses During the CrowdStrike Outage
While only a minority of clients were affected, the Business Data Services Sidekicks were able to help Kansas City businesses in two ways:
- Augment IT Departments: It’s an understatement to say IT departments had their hands full. Our team helped alleviate some of the pressure from IT teams by taking some of the workload off the internal IT team. In some cases, we even acted as boots on the ground for satellite offices.
- Blocking the CrowdStrike definition update: Our security team found a solution for clients with our Cybersecurity Service. With our BDS Storage Control solution, we were able to block read and writes to the CrowdStrike C-00000291*.sys files that CrowdStrike has named as problematic. By denying reads and writes to that block, our clients’ systems were able to reboot successfully and bypass the Blue Screen of Death loop. This served as a temporary solution to restore operations, and we recommend following CrowdStrike’s instructions for a permanent fix.
Essential Preparedness Steps for Kansas City Businesses
Whether you were directly affected or not, this is a key lesson on being prepared for unexpected technological disruptions. Here are some steps every Kansas City business owner needs to implement or re-evaluate to safeguard against similar events:
- Establish or Re-evaluate Your Incident Response Plan
- Document Your Plan: Ensure you have a written incident response plan. This plan should outline steps to take if critical systems go down.
- Regular Updates and Reviews: Periodically review and update your incident response plan to account for new threats and changes in your IT environment.
- Test Your Incident Response Plan
- Conduct Table-Top Exercises: Simulate scenarios where critical systems, such as power, internet, or essential software, are unavailable. Practice how your business would continue operations under these conditions. Involve multiple departments and different levels of each department.
- Identify Alternatives: Have alternative methods and tools in place to keep your business running smoothly during disruptions.
- Technical Solutions
- Implement Preventative Software: While a storage control system like ours was originally designed for security purposes, it served us well to temporarily restore operations for many businesses. Having this software in place can serve as a solution to restore operations temporarily while your IT team continues to implement the fixed from CrowdStrike
- Communicate with Your IT Team
- Regular Check-ins: Maintain regular communication with your IT company or IT team to stay informed about potential threats and updates.
- Leverage Expertise: Utilize your IT team’s expertise to ensure that your business is protected with the latest cybersecurity measures and that any incidents are handled swiftly and efficiently.
The CrowdStrike incident cost businesses millions of dollars. What would happen if your business was down for 1 day, 5 days, or a week? In fact, there are 5 SIGNIFICANT financial costs you will incur when a ransomware attack or incident like this happens. These risks are being routinely underestimated and have far-reaching consequences that most business owners and CFOs aren’t accounting for or insured against. To learn more about these 5 Significant Financial Risks, download our free report here or call us at 913-239-0368 to request a copy!
