Can You Trust Your Test Results?

Direct marketing is all about the numbers. Whether it is response rate, lifetime value, square-inch analysis, or any other metric you can dream of calculating, we all rely on measurability to give our businesses the advantage they need to succeed.

This measurability also affords us the opportunity to test ideas in the marketplace with a high degree of accuracy and a low degree of risk. That’s a powerful combination. But lately we’ve been seeing some data at Lenser that caused us to ask; How high is that degree of accuracy? Are we taking anything for granted?

Executing a direct marketing test mirrors the venerable scientific method, brilliant in its simplicity: We develop a hypothesis, set up a control group, change one variable in a test group that is exactly like the control in all respects, and measure the impact of that change. But an interesting thing happens to the scientist in us during this process. That is, we tend to bias our analytical efforts toward the test group, and understandably so. Implicit in this method is the assumption that we know everything there is to know about the control group; otherwise you couldn’t confidently ascribe any measured changes to whatever you are testing.

When one mailer recently started seeing repeated (albeit isolated) counterintuitive test results, we thought it was time to take a hard look at that “control” group. We tested the predictability of the control by setting up two identical controls in a recent test instead of just one. None of the customers selected into these control files were mailed a catalog in a chosen drop. Our goal was to determine the incremental value of that specific catalog contact. When it came time to analyze the results of the test, we looked to see if the two “identical” groups of customers were behaving identically, as one would expect.

We were lucky to be able to test across more than one title in this case, ending up with richer data, and we also had the ability to isolate Internet-acquired buyers from catalog-acquired buyers. The results are shown below, and what we discovered was surprising – a 32% to 34% difference in sales being generated by the two holdout control groups. This seemed inexplicable to us, with the alarming notion that if we can’t trust the control, we can’t trust the results of the test. But let’s take a closer look. That variance was being driven largely by the average order value (AOV) of Internet-acquired buyers in both titles. By contrast, catalog-acquired buyers and requestors were much less volatile in their buying behavior, and this is significant in understanding what’s happening here.

Can we calculate these same results by e-mail, by search engine optimization, by search engine marketing, by affiliates? Our guess at this time is that just one or two of those factors are behind this apparently quixotic behavior.

And so, as is often the case in this discipline, there is always more data to mine. We are reminded that with the proliferation of multiple marketing channels, changes in our customers’ behavior are accelerating. We would be wise to occasionally challenge assumptions that pre-date seismic changes in our marketing arsenal, be they tactical or strategic. And finally, multichannel marketers must be prepared for multidimensional analyses. I believe our business metrics are telling us to do it. The corollary to that is, of course, that our customers are telling us to do it.

Jude Hoffner is director of circulation, business-to-consumer markets, for San Raphael, CA-based catalog consultancy Lenser.

Partner Content

Hincapie Sportswear Finds Omnichannel Success in the Cloud - Netsuite
For more and more companies, a cloud-based unified data solution is the way to make this happen. Custom cycling apparel maker Hincapie Sportswear has leveraged this capability to gain greater visibility into revenue streams, turning opportunities into sales more quickly while gaining overall operating efficiency. Download this ecommerce special report from Multichannel Merchant to more.
The Gift of Wow: Preparing your store for the holiday season - Netsuite
Being prepared for the holiday rush used to mean stocking shelves and making sure your associates were ready for the long hours. But the digital revolution has changed everything, most importantly, customer expectations. Retailers with a physical store presence should be asking themselves—what am I doing to wow the customer?
3 Critical Components to Achieving the Perfect Order - NetSuite
Explore the 3 critical components to delivering the perfect order.
Streamlining Unified Commerce Complexity - NetSuite
Explore how consolidating multiple systems through a cloud-based commerce platform provides a seamless experience for both you, and your customer.