Gabe Gambill As Texas dries out from the devastation left by Hurricane Harvey, Florida is in the midst of cleaning up from Hurricane Irma. Historians are calling them the biggest storms to hit the U.S. in modern times. Estimates of the overall damage caused by Harvey alone vary — Moody’s estates overall U.S. economic losses between $6 billion and $10 billion. Goldman Sachs predicts a reduction in third quarter GDP of as much as 0.2 percentage points dragged. But this does not include the business costs incurred beyond physical damage. Manufacturers in particular have losses that impact customers, suppliers, employees and partners worldwide caused by disruption in services and data being vulnerable.
Factory floor leaders need to keep their equipment and processes going no matter what, yet maintaining productivity is more complex today as manufacturers often use sophisticated software in tandem with their machines. Whether an equipment failure, staff injury, security incident or natural disaster like a hurricane, blizzard or earthquake, some kind of downtime will happen to a manufacturer — mandating a re-evaluation of backup disaster and recovery (BDR) plans. Here are a few ways to prepare your manufacturing IT team for a fast recovery when the unexpected happens.
Make sure the leadership understands the value of BDR – before it’s needed.
When the C-suite is focused on hitting production numbers, they may not think server backups are valuable. To sell them on the need for better BDR, calculate your downtime costs. A halt to production can be expensive. To anticipate problems, identify the failure points that would stop production in the event of downtime. Which of your plants experience the most downtime? What have been the costs of past outages? Understanding how past downtime has delayed orders and deliveries can help you identify the top risks and the measures needed for swift recovery.
Map out the factory or plant ecosystem.
A typical manufacturing environment today tends to be an interconnected web of equipment and software. Complicating it further, some infrastructure may be local and some supported by remote corporate IT systems. Take a look at your critical data, software, and programs, as well as local infrastructure like switches, routers and servers. Identify how they work together, and what kind of failure could set off a chain reaction that leads to a plant or organizational shutdown. If your data is in silos and you don’t have a firm sequence planned for resurrecting each app and server, you’ll want to map that out now before disaster hits.
Include accidents and test your disaster recovery plan.
Make sure your disaster recovery plan includes the right medical response when a worker becomes injured on the line. Determine how an accident or injury could take a piece of equipment offline, and how you might compensate for it being out of service. Also, it’s not always human error that causes accidents. Backups can go wrong, too. Maybe your BDR system hasn't worked in months or the staffer in charge of manually creating backups stopped doing it. Testing is the only way to know for sure your backups will be there when you need them. To keep it simple, look for a solution with automated testing.
Protect and duplicate critical files now.
Your organization should already be digitizing files, but if you still rely on old filing cabinets, make a plan for preserving that data now and store them offsite. Also, make more than one set of backups. Having only one copy of your data could set yourself up for massive data loss. Fires and storms can hit datacenters, tape backups can be corrupted or ruined, or a system may malfunction. Play it safe with multiples sets and partner one local set of backups with cloud copies.
Embrace the cloud.
If your manufacturing IT team is still relying on a tape drive system, your future recovery is already sabotaged. Tape won’t offer a fast or guaranteed recovery. Transition to a BDR system that keeps your data safe in the cloud and lets you recover in minutes with only a click. You could end up saving the day when disaster hits — and saving a fortune by averting lost production revenue. Whether in the cloud or on-premises, validate your backup frequency. Your backups won’t be much help if they’ve been snapshotting critical servers far too infrequently. To avoid being surprised by a significant data loss, check your snapshot timing and see if it’s equal to the importance of the assets they’re guarding.
Prepare for ransomware attacks.
While natural disasters may be top of mind, a growing reason for downtime is ransomware attacks. Invest in security training for employees and spread awareness of common attack methods like phishing emails. Check the speed of your BDR solution: can you hit a recovery time objective (RTO) fast enough to avoid paying the ransom? Consider using a tool specifically designed for ransomware recovery.
Create a formal disaster recovery plan.
Draft a plan that tells everyone what they need to do, which data and processes should be prioritized for critical operations, and the different actions to take in the event of a natural disaster, error, hardware failure or attack. Once you know how various equipment components and software systems interact on a local and enterprise level, identify who’s responsible for getting IT systems and equipment back online. If plant managers rarely interact with IT managers, make sure they understand the priorities and processes at both levels in the event of a disaster. This should include a way of communicating between offices and factory floors even when the internet is down or email or mobile devices are impacted. By documenting and distributing the plan before an outage, your team will be able to jump right into action instead of asking for orders.
When your systems go dark, you don’t have time to read complicated instructions or track down the three people in the entire organization who understand how to operate the 12-step failover process of your solution. And you definitely don’t have the room to experiment, make a mistake and possibly make things worse. For this reason, use a simple and intuitive BDR solution with a one-step failover process that literally anyone can operate and get you up and running fast. A disaster recovery plan is the ultimate insurance policy that can position your team for confident action instead of chaos, and deliver peace of mind during a storm.
Gabe Gambill is Vice President of Product and Technical Operations at Quorum.
