Backup Plans: How to set up a disaster-recovery plan

Oct 02, 2012 2:26 AM  By

As we become ever more dependent on technology and its ability to sustain our businesses, isn’t it time to consider a business-continuity (BC)/disaster-recovery (DR) plan?

We’ve seen on the news or experienced ourselves the natural disasters that can occur: hurricanes, earthquakes, tornados, blizzards, floods, etc. But do we think enough about their implications when it comes to our businesses and how prepared we are?

One of the best precautions any company (large or small) can take is to create a BC/DR plan. In this article, we’ll discuss what should be in a BC/DR plan, how to know if the plan will work, and what you need to do to keep it current.

Establish BC/DR committee
Appoint senior leaders who understand your business, who can design and organize the company in case of disaster. Many companies already have organized facility and floor leaders in case of fire, for example. Can these people’s responsibilities be extended to include other disasters? What are the threats you are going to design the plan to defend against?

As you think through the potential for disaster and the replacement of the services you’ll need to remain in business, it can be helpful to focus on major entities or activities—your contact center and customer service, fulfillment facilities, information systems, and headquarters’ operations.

For example, contact center/customer service is far more recoverable if you use a third-party overflow contact center for some of your work today. Physical fulfillment is much harder to defend against if you have only a single warehouse and it’s destroyed. For most companies, this is one of the biggest vulnerabilities.

Since IT is at the hub of how we take orders and serve customers, focus on the issues of backup and recovery for a wide variety of circumstances. Additionally, much of your company’s best information may be on personal desktop or laptop systems. How are these being backed up?

In the headquarters’ operation, what key documents must be retained, legally or from a business continuity sense, in off-site storage?

Develop the plan
The plan should outline a step-by-step process and include enough detail to get the business up and running as quickly as possible. Information about who is assigned to take specific action when there is an interruption to business continuity or a natural disaster must be easily understood. All aspects crucial to enabling the business to function elsewhere need to be identified.

From a financial perspective, there are several important considerations. For example, based on the various threats to your business, what is the potential loss per hour of downtime—both sales per hour lost as well as expenses incurred from downtime? Additionally, various forms of data redundancy, hot or cold recovery sites, etc., have an investment or expense that can be considerable. Develop your plan with an eye to balancing business continuity with the expense or investment required.

Focus on your people
The plan should start with a contact list of key personnel, with their phone numbers and email addresses. Each should have a list of key personnel to notify—who in turn should have a designated group of personnel to call. Your people are, obviously, the key to recovery.

The plan should contain a designated meeting place, and each person could have designated tasks to perform prior to meeting—for example, bring the off-site backup to the hot site. You also need to have backup personnel in place should “a branch on the tree not be available,” and they should be adequately trained to perform the tasks of the people they are replacing.

One of the highest priorities is your personnel and their families. In a catastrophic event such as hurricane Katrina, people’s lives become the top priority. Personnel will be able to focus on the company’s sustainability needs when they know that their families are safe and taken care of. Think this through: How do you respond to the needs of your organization’s personnel during the disaster so that they can help you recover?

Test the plan
As with any plan of this type, it needs to be tested. By testing we mean that, at some point, the BC/DR committee should declare or stage a simulated disaster and execute the plan. Not until you actually test the plan will you know for sure if it is viable and that you have all aspects in place to be able to run your business from another location.

IS recovery sites: Hot, cold or warm?
This is an area where you may need the help of a consultant, service providers and possibly your software vendors in order to develop your plan. Obviously, hosted, cloud-based or SaaS services for websites, order management systems, third-party contact center overflow, and payroll and HR services play a big role in the variability of the recovery plan.

For your in-house centralized systems, what type of system recovery site will you use, if needed?

A hot site will allow you to resume your operations because it is fully equipped with all the computer hardware you’ll need to run your operation as soon as the servers are loaded with the current operational and data recovery backups. This facility may also be the location of your backed up or redundant data. Clearly, there is a cost to having a hot-site service. But a hot site is like an insurance policy—something you pay for but may never use.

What data does IT need to back up daily, whether it is a full backup or incremental? And where is the backup being taken to or residing? In addition to data backups, a backup of your operating environment should be done after any updates to the same so that it is in sync if you have to utilize the backup.

If the disaster affects your website, do you have a mirrored disc storage environment in place so that if one site goes down there is no business interruption? If you have a mirrored environment, is it located far enough away—in another state, for example—to minimize the chances of it also being affected?

A cold site, on the other hand, provides you with a facility but no hardware infrastructure to run your business. This is the least expensive type of recovery site, but because hardware will need to be acquired and then restored to your environment from your backup media, it will take much longer to get your operation back up and running. In our experience, this can take days to weeks to establish.

A warm site is the third type option. It would have some replication of your hardware environment, but not an exact duplicate. This kind of facility may also be where you are currently storing your backup recovery data, but it may not be the most current. This environment will allow you to be back up in a shorter timeframe, but with limited through-put because of the lesser hardware configuration.

The type of recovery site that is right for you depends on how fast your business must be back up and responding to your customer’s needs and maintaining operations. There are several companies that offer availability services with several options—some with sites worldwide, such as EMC and SunGard.

However, before you can be effectively up and running at an offsite facility, whether hot, warm or cold, you need to be taking backups on a regular basis of not only your data, but your whole environment—including operating software, all executables, databases, all ancillary data, and all other job control software needed to run the operation. Without this information and the ability to load it at the recovery site, you will have a difficult time getting back in business quickly.

Contact center recovery
If there is utility work being done near your facility and they cut your communication lines, do you have a plan to be able to answer the phones? Some of our clients have redundant trunk lines coming into the facility from opposite directions—and some even from different central offices.

If you utilize an overflow or off-hours third-party contact center, does it have the capacity to take on all of your calls on short notice? Is it possible to use a facility or facilities close enough so that your contact center reps could go there and take the calls? If not a third-party contact center, do you have a reciprocal agreement with another division in a large company or a local business with a capacity that would allow your contact center reps to go there and answer your calls? If you utilize a browser-based application for taking orders and responding to customer service calls, you could do so from any environment that allows you access to the Internet and has phone switch capabilities that could split your inbound calls to your reps.

Curt Barry (cbarry@fcbco.com) is president of F. Curtis Barry & Co., a multichannel operations and fulfillment consulting firm.

  • Sean Carter

    Having a plan set up like this is key! We never know when we ill need the disaster recovery plan we set up but the safety and protection it has already offered has proved invaluable. There is so much that can go wrong out there

    • Charlie

      What is your biggest fear? My biggest fear is my auto may quit on me. I am thing of a Mini They seem to have plenty of quality cars for sale
      Do you agree?

      • Al

        I do

        • Cyrusmiley

          My fear is a crash