Lists and Data: It’s Time to Test Again

Some direct marketers put the brakes on testing during the past two years, largely due to budget and staffing constraints, as well as poor results. While this is understandable in the short term, it’s time to return to testing — in a focused, strategic manner.

Merchants who haven’t tested in two or more years are working from outdated assumptions about their customers and prospects. Given the shifts in consumers’ priorities and lifestyles, and purchasing-decision dynamics wrought by the economy-along with the influence of the social media explosion — your customers, and how they react to your message and offerings, have likely changed.

The ability to test and accurately measure the ultimate effectiveness of channel, brand, item, price, offer and creative — and then apply the insights — is a core strength of direct marketing. While not much has worked in the past two years, there are signs of at least the beginnings of recovery. So you need to test your old assumptions within the new reality.

Should you just start testing everything? No. Constrained resources remain a reality for most of us; plus, unfocused testing will cloud the critical issues and complicate test results without driving profitability. Instead, zero in on three key variables that drive your business, and test existing assumptions regarding those variables.

Don’t select these critical variables on intuition, though. Identify them through exploratory research and pretest qualifying — the first two stages of the five stages of testing. (See sidebar on page 25 for the testing stages.)


Acquiring customers has always been one of the most difficult and expensive business challenges, regardless of channel. And in today’s ultra competitive and fast-changing climate, any direct marketer that can’t acquire profitable new customers will face collapse even sooner than it might have in years past. Testing is the only truly effective way to ensure that you are maximizing acquisition performance.

As an example, one apparel merchant used to require that 10% of its total acquisition advertising spending go to testing offers and creative. Then the recession hit, and the testing budget was eliminated.

As a result, the cataloger went for two years without retesting its free-shipping promotion control offer. Response rates had declined quite a bit during that period, but blame was laid entirely on the economy and stale lists.

But when the company recently retested the offer, the results showed that a $20-off promotion generated both a higher response rate and a higher AOV than free shipping. The merchant reacted intelligently, doing more retesting over a period of time before rolling out the new offer. In this case, the initial test results ultimately were confirmed.

What are the critical tests for acquisition efforts? For most merchants, it’s channel, offer, catalog cover/envelope creative, and a separate acquisition printed piece. Today, you must also determine how to measure the impact of your social media and search campaigns.

How are you defining profitable customer acquisition in those channels? Are you measuring the impact of one channel on another and accurately crediting the order-generating channel with the sale?


When it comes to your house file, consider these three questions: 1) How do you maintain or increase the profitability of your best customer segments while offsetting increased advertising expenses? 2) How do you move one-time and two-time buyers into your most profitable, three-time-plus segments? 3) Are you measuring channel preferences effectively and incorporating those findings into your overall contact strategy?

If done right, testing your current customer list can provide answers to these questions and more. For instance, on the promotions front, you might want to determine whether your free-shipping offer has lost its effectiveness. Is free shipping simply expected now — especially during holidays? Are you correctly testing this offer, including within the context of other offer ideas also being tested at this time?

You need patience when testing your house file. Don’t rush into a full execution plan based on results from one test. Tested segments must be retested and measured over time, with the same test-segment buckets maintained for each retest, to interpret results accurately. This typically requires a minimum of six months.

One cataloger recently measured channel preference over a full year. Early results showed a strong preference by a majority of the tested segments for online-only contacts. But at six months, the dominant preference had shifted to a combination of print and online contacts.

The company ultimately discovered that 15% of its file could be contacted purely through the online channel, while maintaining response rates and average order value. Another 24% needed to receive the print catalog on a monthly basis. The majority — 61% — needed a combination of the printed catalog (but every two months, as opposed to monthly), complemented by weekly email campaigns.

The cost savings realized from the online-only and reduced-print contact groups more than offset the print and postage increases for the other segments. Had the merchant acted a decision based solely on the initial test result, its plan would have resulted in reduced response rates from both of the print-contact groups that were ultimately identified through further testing.

Philippe Graner ( is director of marketing strategy for J. Schmid & Associates.


Deriving maximum benefit from house file or prospect testing requires an understanding of five distinct testing stages and the purposes and benefits of each.

Exploratory research is used to probe strategic questions. It should take place any time you start a new business division, develop a new category of products or services, enter a new market, encounter viable new competitors, or detect signs of change or new behaviors within your target audience.

You should also conduct exploratory research on segments that used to meet profitability targets, but are now performing poorly. What strategic areas do you need to measure in order to determine what factors could push customers up the loyalty ladder?

Pretest qualifying is critical to identifying what you want to test, on a more specific, tactical level. What are those three critical levers that will most affect your business?

These could include product or service attributes, a cross-channel brand or creative plan, offers that can be rolled out throughout your organization, or a pricing strategy that increases market share while preserving profitable margins.

The main testing stage requires a clearly defined process — in particular, methods to ensure that equal buckets of similar customers, with similar transactional, item-purchasing habits and demographics, are selected for each testing segment.

Primary testing also requires a clearly defined timeframe and a plan of action based on the results. It should take into account seasonality, channel, offer, product or service in question and competitive landscape.

Post-test analysis is used to answer two critical questions: What happened? and why did it happen? It is also used to confirm that the initial test was properly executed — that those equal segments were selected, that the messaging was on target, and that the creative presentation accurately reflected your brand.

The retest stage is key to maintaining overall business profitability. You should retest immediately to confirm the accuracy of the initial testing process and validate your analysis or diagnosis. You should also retest over time to ensure that you identify anything that has changed the business environment since the last test. — PG

Partner Content

Hincapie Sportswear Finds Omnichannel Success in the Cloud - Netsuite
For more and more companies, a cloud-based unified data solution is the way to make this happen. Custom cycling apparel maker Hincapie Sportswear has leveraged this capability to gain greater visibility into revenue streams, turning opportunities into sales more quickly while gaining overall operating efficiency. Download this ecommerce special report from Multichannel Merchant to more.
The Gift of Wow: Preparing your store for the holiday season - Netsuite
Being prepared for the holiday rush used to mean stocking shelves and making sure your associates were ready for the long hours. But the digital revolution has changed everything, most importantly, customer expectations. Retailers with a physical store presence should be asking themselves—what am I doing to wow the customer?
3 Critical Components to Achieving the Perfect Order - NetSuite
Explore the 3 critical components to delivering the perfect order.
Streamlining Unified Commerce Complexity - NetSuite
Explore how consolidating multiple systems through a cloud-based commerce platform provides a seamless experience for both you, and your customer.