Do you have an adequate business continuity/disaster recovery (BC/DR) plan and system in place? Most operations think they do, but it’s often not the case.
Natural or man-made, disruptions and disasters that could pose threats to the well-being of our employees, physical plants and business viability are more common than we think — and it’s human nature to be overconfident about our preparedness.
As operations consultants, we’ve found that most companies are inadequately prepared — particularly for scenarios in which there is a loss of the mainframe or server farm. Moreover, we have seen more than a few facilities severely damaged or even destroyed by natural disasters during the past 25 years.
Concerns about costs, as well as the “it can’t happen to us” mindset, prevent many companies from addressing these crucial issues. A 2010 survey by Forrester found that companies are allocating on average a relatively modest 6% of IT operating and capital budgets to BC/DR, although 32% reported plans to increase this spending by at least 5%.
But sufficient investment in well-researched, prudent BC/DR plans and systems is a must for all businesses. These are some of the key steps in developing and ensuring the effectiveness of such a plan — and keeping it current.
Establish a formal BC/DR committee. This planning is far too critical to approach on an informal, piecemeal basis. Appoint senior managers who understand the business and can objectively assess its areas of vulnerability, as well as provide leadership in the event of a disaster or disruption. Make sure that all key business operations areas/departments are represented on the committee.
Focus on employees. Employee safety is, of course, the first priority in any emergency. Further, preparing personnel with a clear emergency plan is critical to business recovery. Carefully think through how you will respond to your organization’s personnel needs during a disaster.
Make regular checks and tests of the emergency warning and evacuation systems in all of your facilities. Merely adhering to minimum code standards may not necessarily be adequate.
Be sure you have a strong emergency leadership team in place, with clear, written instructions spelling out individual responsibilities. As part of this, consider whether the responsibilities of those who have been designated as facility and floor leaders in the event of a fire might be extended to include other forms of disaster.
The plan should start with a contact list (phone and email) for notifying key personnel, who in turn are given a designated group of personnel to call.
Your plan should include designated meeting places both for employee evacuation purposes and for business recovery purposes. Provide clear instructions for the tasks that each leader needs to perform prior to meeting at the business recovery site, such as bringing off-site back-up.
You’ll also want to designate back-up personnel to cover key responsibilities. These folks should have adequate training in these alternate duties.
Assess all potential operations threats, objectively and methodically. In thinking through disaster scenarios and business systems/services replacement or back-up options, focus first on critical activities, including call centers and customer service, fulfillment facilities, information systems and headquarters.
For example, disruptions in physical fulfillment are much harder to defend against if you have a single warehouse — one of the biggest vulnerabilities for many companies.
Another major point: Do you currently use third-party overflow call centers that can be deployed in the event of disruption? If not, what is your contingency plan for call center/customer service recoverability?
IT issues potentially affect a wide range of critical services and business information, so you must think through, realistically assess and address all potential circumstances and vulnerabilities in systems.
Once you’ve completed a thorough assessment of risks and needs in various circumstances, create a clear, step-by-step plan to enable rebooting existing operations as quickly as possible and, ideally, running the business from one or more alternate locations while main facilities are compromised. Here are two areas to look at closely.
- INFORMATION SERVICES RECOVERY
-
In the online environment, do you have mirrored disc storage in place so that if one site goes down, there is no business interruption? If so, is it located far enough away to minimize the chance of it also being affected?
-
Do you have off-site hard storage and electronic back-up of all key headquarters documents that must be retained for legal and business continuity reasons? Employee laptops and desktops contain much of a company’s most important information. Are adequate, routine back-up systems in place for this data, as well as for centralized data?
Information services recovery is an area where you may need the assistance of service providers, and possibly your software vendors and a consultant.
Hosted, cloud-based or SaaS services for websites, order management systems, third-party call-center overflow, payroll and human resource services can critically enhance recovery capabilities.
Another major issue: What type of physical system-recovery site will you use for your in-house centralized systems, if needed?
A hot site — one that’s fully hardware-equipped and ready to roll as soon as the servers are loaded with the current operational and data recovery back-ups (assuming back-ups are not already residing there) — will allow you to resume operations most rapidly. Maintaining a hot site service is costly, but may well be a justifiable “insurance policy,” particularly for companies that stand to sustain large losses with even a short period of downtime.
Cold-site facilities (which do not contain hardware or back-up data, and are often used for storage purposes until needed for an emergency) are the least expensive option. But recovery takes much longer — days to weeks, in our experience — due to the need to acquire hardware and restore the data environment from the back-up media.
A warm site falls in the middle in terms of investment and recovery seed. These have some replication of a company’s hardware environment, and can be used to store back-up recovery data — although it may not be the most current data. This type of environment allows for quicker recovery than a cold site, but the lack of a full duplicate hardware configuration is likely to limit through-put.
Selecting the right type of recovery site for your company requires weighing the business-loss costs involved in various threat scenarios (potential losses per hour of downtime in sales and downtime-incurred expenses) against the investment/expenses associated with the various back-up or continuity options.
And remember, you will struggle to get back in business unless you have been doing regular back-ups — not only of your data, but of your whole operating environment. This includes operating software, all executables, databases, all ancillary data and all other job-control software needed to run the operation.
-
- CALL CENTER RECOVERY
-
Have you considered how you would take calls if your trunk line was temporarily disconnected — for instance, as the result of local utility work near your facility? Is it worth investing in a plan that includes redundant trunk lines, with some coming into the facility from opposite directions, or even from different central offices?
-
If you use an overflow or off-hours third-party call center, does the provider have the capacity to take on all of your calls on short notice? Is it possible to use a facility or facilities close enough that your contact center representatives could go there to take the calls? Should you hedge your bets by having one back-up call center nearby and one far enough away that it is unlikely to be affected by a natural disaster affecting your main call center?
-
If you don’t have a third-party call center arrangement, should you create a reciprocal agreement with another division (in a large company) or with a local business with sufficient capacity to enable your reps to go there and answer your calls?
Note that if you use a browser-based application for taking orders and responding to customer service calls, you could do so from any environment that enables Internet access and has phone-switch capabilities that can split in-bound calls to your reps.
-
As solid as you may believe your research and planning to be, only regular testing will confirm a plan’s viability, uncover its weaknesses, and enable you to update it to reflect your changing business needs, as well as changing technology and other factors. To remain effective, a business continuity/disaster recovery plan must be viewed as a critical and continuously evolving project.
Curt Barry ([email protected]) is president of F. Curtis Barry & Co., a multichannel operations and fulfillment consultancy.