Installing new systems is like watching your worst enemy go over a cliff in your mint-condition Cadillac: “Oh yes! Oh no!” Every business looks forward to the significant improvements that come from updated technology, including enhanced functionality, increased productivity, and better throughput. But implementing new systems often proves to be extremely painful and can create major setbacks to daily operations. As a result, many executives face a Catch-22 situation – they recognize the need to make a change, but are reluctant to take on the risk associated with a systems project.
Success in systems implementation requires a well-coordinated effort on numerous fronts. One of the more difficult tasks can be correctly identifying scenarios that test systems functionality, especially if you have never undertaken such an effort before. Sophisticated tools can create scenarios and test scripts from requirements documents and/or technical specifications, but since most companies are forced to start with a manual approach to systems implementation, I will focus on that here.
You can remove much of the mystery from systems tests by taking a structured approach to your scenario development. Your agenda must include: 1) analyzing your daily business operations; 2) creating a matrix of business variables; and 3) selecting components of the matrix to create your scenarios.
Zap the bugs Testing scenarios provide a structure for walking through all business operations in a controlled environment prior to installing the system in the real world of daily operations. The object is to discover any software bugs that might cause disruption in the production environment. Bugs can result from poor software coding, improper configuration of the system, or miscommunication in the interface layer between existing systems and the new one.
Although most operations don’t want to admit it, bugs can also be the result of poorly defined or non-existent requirements discovered during the testing process. By creating scenarios that exercise all of the intended functionality of the new system, most of these flaws can be detected in time to fix them before going live. Failure to allow appropriate time for the testing process prior to installation only means that the system will actually be tested on real customers.
A new system usually implies new operating procedures throughout the business. Yet, it is a rare company that fully considers all the ramifications of these changes. If you define and execute the test scenarios properly, there should be no ugly surprises when the system goes live.
Winners take all To be successful, an implementation team must have the full support of the executive leadership, as well as a strong leader who has a wide range of experience and is capable of structuring the activities of many different people and organizations. Many companies choose to hire outside consultants for this task. Although it is certainly beneficial to capitalize on the experience of industry professionals, it is advisable to pair an outside consultant with an internal resource. If the team leader is an outside consultant, the project structure should emphasize transferring his or her knowledge to internal resources throughout the implementation.
The team should comprise individuals from many distinct business units within the company. In a traditional consumer-direct business, this means that the team should field members from the customer contact center, the warehouse, and the accounting, marketing, and IT departments.
Finally, the team must be composed of qualified, capable, and available people who can be dedicated nearly full-time to the project and who are empowered to communicate the needs of their respective departments. In the course of visits to many companies that have experienced difficulty in systems implementation, my team has discovered one theme that stands out as a reason for less-than-immediate success: failure to dedicate sufficient internal resources to the project early enough in the process. As a rule of thumb, if it doesn’t hurt to give up a person from the daily operation, then it’s probably the wrong person.
The first step in creating effective testing scenarios is to have each of the functional experts document the areas of their business that are affected by the new software. To do this, the team members should consider the daily functions they perform and assess whether the new system will affect the way they do business.
It is helpful for the team members to walk the floor of the operation and visit the site of each function that is a part of daily activity. The team can then create operational flowcharts and determine which of the functions the new software affects. Creating these flowcharts is important because it requires the team to go through the operation in sequential order and brings the level of detail down to floor level. Many software selection projects take place at a strategic level, and it is only in the implementation phase that the team analyzes the impact at every point of contact in the business. If the system implementation requires revised operating procedures, then the flowcharts must reflect those changes. By walking the floor first, the team can be reasonably sure that the new operational process accounts for all existing functions.
The product of this phase of the project is a set of flowcharts that accurately represent the operation as it is intended to perform after the implementation of the new system. The team will highlight each functional box on the flowchart that is affected by the new system.
Including digital photographs of particular operational functions in this or an accompanying document communicates ideas to the rest of the business and can be an integral part of new training materials.
Create a matrix The next step is to use the operational process flow information to develop a matrix that will identify all the possible combinations of events that can occur in a transaction. The testing process would be easy if it needed to account only for the straightforward order and could ignore all the pesky exceptions that customers insist on. It is very easy to create a scenario for an order that requires three in-stock items to be sent to Florida for a customer who is paying with a credit card. It is far more difficult to create a scenario for the customer who orders two items for herself, one for a friend in San Francisco, wants gift wrap on the friend’s merchandise, and is paying part of the order with a check and part with a credit from a returned package received last week.
To manage this process, it is helpful to create a matrix that categorizes the variables of an order or operational transaction and then populates it with all the possible conditions. An example of one such matrix for a catalog management system is shown on page 68.
Content for both the column headings and the fields below them will vary depending upon the type of business and the software package being implemented. The flowcharts created above should be a guide for writing column headings. For example, tender information is unnecessary for installing a warehouse management system. A WMS matrix might include headings such as receiving, quality control, prep, putaway, wave planning/inventory allocation, picking, replenishment, value-added processing, packing, and truck loading.
It is a good idea for team members to check their list of variables with their departments to ensure that it is exhaustive. The implementation team should then meet to compile and review the entire list. Each team member should understand the details of each variable well enough to judge its impact on his or her own area. During these sessions, people who have worked in one area of the business will be often amazed to discover the effect of their actions on the rest of the process. This understanding helps to create better scenarios in the next step and facilitates troubleshooting.
You now have a complete list of all the factors that might influence system functions. You then create scenarios by combining different variables into a series of single transactions that will be entered into the test environment. It is helpful to start a new chart with the same headings as your business conditions matrix. The team selects meaningful elements from the matrix and assembles the types of transactions necessary to create an individual scenario. The example on page 72 creates a scenario for a standard order. The goal is to determine all of the complicated transactions and exceptions needed to fully test the system. The team should list the reason for creating the scenario as part of this document. Examples would be “tests multiple methods of payment for the same order” or “tests replenishment triggers for unslotted merchandise.” An example of a complicated scenario is shown below.
It will be possible to use the same order to test multiple functional elements of a system. The drawback of combining scenarios is that it can make results analysis and problem resolution more complicated. It is a good idea to retain a file with the individual scenarios so that if a combined scenario develops a bug, you can re-run the test using the individual components to isolate the problem.
To use the matrix effectively, each element must be represented in at least one of the scenarios. In addition, the team must understand where combinations of multiple elements in the matrix are a) likely to occur and b) relevant to systems functions. This is when a carefully selected team will pay off for the entire organization.
Now is an excellent time to enlist the help of the people responsible for creating the business requirements documents and technical specifications for modifications. They should be able to help the team understand all of the complex functions specifically created for your operation. That understanding will help the team members design scenarios that could cause the system to have problems if it has not been coded and configured properly. In other words, they should know how to “break” the system better than anyone else. For instance, it might take a system designer to understand that an order with a bill-to address in New York and a ship-to address in New Jersey needs to be created to properly test the tax function of an order management system.
When the list is near completion, review the contents with each department to seek further input. It is likely that additional scenarios will be created. The number of scenarios on the final list depends on the scope of the system implementation. A simple upgrade to one function might require only a handful of scenarios, whereas a new WMS will likely require hundreds.
The scenarios should give the team a comprehensive road map of the intended functions resident in the new system. To use the scenarios, the team will have to add three elements:
(1) Data. Each of the variables from the matrix will require its own unique set of data to test. Item characteristics, for example, will need a different SKU for products in stock, backorders, gifts with purchase, value-added services, substitutes, and complimentary merchandise, at a minimum. It is possible that other item characteristics, such as the components of a kit, can be created by combinations of data.
Once information has been created or copied from existing systems, it should be populated for each scenario. This can be done in Microsoft Word, Excel, or Access, whichever your organization prefers. Keep in mind that this information will need to be accessed by the entire team and will require frequent manipulation. Be certain that all the team members are comfortable with the chosen tool.
(2) Schedule. Create a manageable daily workload is created and ensure that the program follows a logical sequence. As one astute young man observed during a recent project, “We need to ship packages before we can return them.” Many truths like this will inform your scheduling decisions, such as, “You must receive product to put it away,” “You must put it away to pick it,” and so on. For complicated sequencing issues, the team can refer to the flowcharts created in the first step of this process. Testing the straightforward scenarios first and working up to the more complex ones will make problem isolation and resolution easier.
(3) Expected results. To be effective, the scenarios must contain a list of expected results that you can use to validate outcomes. This task requires discipline to perform correctly, but is vital to the project. This is especially true if the testing team includes people who have not participated in the scenario development.
Checkered flag The final product of this exercise should be a book of scenarios that will be used by the testing team to validate system functions. An sample sheet is shown below.
Although there are many ways to get to this result, flowcharts and a variables matrix are excellent tools for starting the implementation team down the right path. By accurately identifying as many different business conditions as possible and turning them into viable testing scenarios, it is possible to uncover most of the software problems that can lead to a failed implementation. It’s the best way to turn a potentially hair-raising trip into a pleasant scenic drive.