When Testing, More Is More

Apr 01, 2007 9:30 PM  By

Do you know how big your “buy” or “submit” button should be? How about what color? Should your online registration process involve more than one page?

Right about now the old-school direct marketer in you is thinking something like, “I’m not sure, but I know how to find out: Test. Test. Test.”

Thing is, if you’re like most of your colleagues when it comes to the Web, you’ve been paying lip service to direct marketing principles online, but that’s about it. The reason: Online marketing is cheap and, as a result, hasn’t forced the type of discipline onto marketers that print does.

Moreover, testing various sales and marketing approaches on the Web up until recently required getting the IT department involved, something most marketers would rather avoid. So ironically, many direct marketers threw a lot of their own rules out the window as they worked to establish a presence on the most efficient direct marketing channel of all time.

“There is still a relatively small percentage of people who are really embracing testing as part of their culture and methodology,” says Eric Hansen, president of SiteSpect, a Boston-based company whose technology allows marketers to perform multivariable testing on their Websites. He estimates the percentage of marketers testing to be in the low double-digit range.

But more marketers are likely to embrace testing moving forward, as the cost of buying media online continues to rise and the amount of inventory continues to decline. “At the beginning, with the advent of Google, you could buy very targeted clicks very cheaply, and everybody was making reams of money,” says Mark Wachen, CEO of Optimost, a New York-based provider of multivariable testing services. “But because everybody was making reams of money, those bid prices just keep going up, so all of a sudden where you used to pay $0.12 for a click, you’re now paying $1.20 per click. When you pay $0.12 a click you can be pretty sloppy in how you convert the people, but when you’re paying $1.20 or $5.00 or $10.00 a click it’s a different ballgame.”

So testing is coming into vogue online, but with a twist. Instead of doing single-variable, A/B-split tests, a few marketers are embracing multivariable testing. Matt Roche, chief executive of San Francisco-based Offermatica, a provider of testing and optimization services, estimates that no more than 300 companies have tried it so far. But many of those that have are getting some bang-up results. Roche says that multivariable testing can result in a triple-digit increase in performance.

“If you’re going to do testing, multivariable testing is just orders of magnitude more efficient with the same amount of traffic than A/B-split testing,” says Wachen. “For example, 10 variables and five values for each give us 10 million possible permutations. Now, we’re not going to run all 10 million versions, but we might run between 100 and 200. In the offline world, creating 100 or 200 versions of a catalog would be a big endeavor.”

“The reason multivariable testing is possible here, when it wasn’t all that realistic in [print] direct marketing, is that the cost of production of variants is almost zero,” says Roche. “The multivariate approach has been around for 50 years, but the cost of producing multivariants relative to how you could exploit it didn’t make any sense. You’d have to do 15 print runs for each piece, and by the time you were done you’d have a winner, but you wouldn’t have a population left you could send it to.”

Even online, Roche continues, multivariable testing wasn’t practical until a couple of years ago, when the programming language JavaScript became powerful enough to allow Website operators to change the look of pages without getting IT overly involved. “That was really around 2003,” he says. “To some degree it is a Web 2.0 phenomenon.”

Beyond the obvious

Multivariable testing allows marketers to test what Wachen refers to as nonobvious variables. “If you’re doing an A/B test and you can test only one thing, you’re going to test something big, like the offer,” he says. “But what we’re seeing is that the little things can mean a lot, and with the multivariable approach there’s no additional overhead to go from five variables to 10 variables, so you don’t have to stop at the offer and the headline.”

Speed is another advantage of online multivariable testing. When electronic postage marketer Stamps.com recently tested seven elements of a landing page — including a testimonial vs. no testimonial, three vs. six bullet points, and differently worded calls to action — it was able to determine the winner in less than two weeks.

“It really doesn’t take more than a couple of weeks in most cases,” says Sebastian Buerba, marketing director, small business for Los Angeles-based Stamps.com. “In direct mail that would take you forever, and the cost would be huge.”

In contrast, a midsize or larger business can expect to pay $8,000-$10,000 a month for any of the multivariable testing services available, says Hansen. SiteSpect’s services start at $2,000 a month.

It’s not surprising that as head of a testing firm, Offermatica’s Roche would declare, “You should not be doing anything without testing, because the odds that you’re going to be doing something wrong are almost 100%.” But Stamps.com’s Buerba would likely agree with that statement, based on an A/B/C/D test the company performed recently to determine the most effective design for a landing page. The winner ended up being the version no one expected it to be. “Everybody thought version D was the worst one, and it ended up being 13% higher [in registrations] than the control,” Buerba says.

Multivariable testing has also helped Buerba nail down which bullet points to include in the company’s sales copy, another test that would be impractical offline.

“Any time I start writing down features and benefits — and I think this is the case with most companies — I end up with a list of 20, and I don’t want to eliminate any of them,” he says. “You want to tell [prospects] everything, but you have only a limited amount of time, and if you include everything you confuse people. So with multivariable testing you can test all 20 bullet points and find out exactly which of the 20 are the top five. Testing that in direct mail would take you forever.”

Buerba adds that multivariable testing also is great for limiting those maddening meetings every company has where each person in the room has an opinion on what will work, but the opinions are mostly based on anecdotal evidence. “Online, you can test any crazy idea you may have,” he says. “If someone has an idea I can put it up there, and if it performs really badly, I can shut it down right away.”

The whats and wheres

So what portion of the Website should a marketer using multivariable testing attack first?

“That’s always a good question, but to be frank, I don’t think it matters,” says Roche. “There’s a strong argument for testing the shopping cart or form first because that’s where your highly qualified customers are. But there’s also a strong argument for beginning by testing a landing page off a campaign because that’s where you can have the most impact.”

As far as Roche is concerned, you don’t even have to limit yourself to one area of the site at a time. “You can do something on the cart or the form, you can do something on the product page, the landing page, and the home page. You can plan for all four because it’s not that hard to do,” he says.

If you want to start more slowly, Optimost’s Wachen suggests looking at the campaign landing page first. “In a lot of cases landing pages are not as technically complex as pages downstream, such as a billing page, a product page, or a credit-card page with dynamic content,” he says.

And beginning with landing pages will also shed light on how other pages are performing. “While you’re improving the landing page, you’re also going to be learning about what’s going on downstream and where you’re losing people,” Wachen says. “We’ll optimize a landing page based on whatever the key metric is, be it sales or revenue per visit. But at the same time, we’ll keep an eye on those intermediate steps because if you can figure out ways to get twice as many people to the shopping cart and then you’re having some drop-off there, you can attack the shopping cart to see what’s going on there.”

As for the sorts of things you should test within each area, “clearly price, promotion, and product are going to have much more impact than button color,” says Roche. “The problem is if you’re a reasonably smart person, you start thinking, ‘I can test everything,’ and then you get into dangerous territory where you say, ‘Let’s test everything,’ whereas I say, ‘Let’s be disciplined about this.’”

Online tests, offline results?

Can the results of multivariable testing online help you refine your efforts in other channels?

“Absolutely,” says Eric Hansen, president of testing technology provider SiteSpect. “You can put something out and get real-time response data like no other medium, and yes, you can take what you learn online and use that to influence your offline marketing efforts.” For example, a travel client learned that men in Texas responded to different offers than men in California, he says.

But while results from online testing can help guide offline marketing, they shouldn’t dictate it. “Don’t take something from online and expect there to be a one-to-one translation offline,” Hansen warns. “You are in a different medium, and people are in a different head space when they shop online compared to when they shop offline.”

Electronic postage merchant Stamps.com takes a common-sense approach. Says Sebastian Buerba, marketing director, small business, “The stuff that seems to make sense on the Website, we apply offline.”
KM