In science-fiction movies, all the best computers talk intelligently to their users, listing their options and the relative chance of success of each. In real life, of course, we must diligently analyze everything in our systems to figure out what is working and what is not, and draw our own conclusions as to what to do next.
But what if your e-commerce system could talk? What if it could just tell you which promotion would convert best or which page design would keep visitors shopping longer?
A/B testing comes close. It will unambiguously tell you which of two online elements is performing better, making it easy to decide where to put your efforts to generate the best result.
Online A/B testing helps you obtain optimal performance out of key selling elements such as promotions, page designs, messaging, and images. It gives you solid information about how your customers react to specific site elements and helps you focus marketing and merchandising campaigns, form your development roadmap, and improve your site design. With some forethought and the application of e-commerce tools you may already have, you can incorporate A/B testing into your marketing program.
Online A/B testing is similar to the A/B testing that direct marketers have conducted for decades. You create two items (A and B) and run them side by side to see which is more effective, as measured by the sales each generates. You retain and continue refining whichever of the two performs better.
Typically an A/B test is set up to improve an existing item or feature, such as a promotion or a page design. Only a small portion of your customer base is allowed to see the new item until you can gauge how your audience as a whole is likely to respond. You are in effect using a portion of shoppers on your live e-commerce site as a focus group and gathering shopper response data on new site elements before publishing them to your entire customer base.
The variety of things you can test is virtually limitless. The following examples represent some of the most commonly tested Website elements.
Landing pages
Before you expose your entire customer base to a new landing page, determine how effective the page is by publishing it to a small sample group of your customers. For this discussion, “landing page” refers to any page where your customer enters your site. Landing pages most often serve as entry points from links in promotional e-mails, online ads, or print ads. They can be home pages, specially created entry pages, or even product pages.
First, design your new landing page and test it to make sure it functions properly. Second, set up your e-commerce application so that it presents the new page (option B) to a small percentage of your visitors (25%, for example) and loads the older page (option A) for all the rest. Third, use your reporting application to track the results and determine which of the two performs better statistically. Finally, publish the successful page to all visitors. You should then design a new test to refine this page further, since there is always room for improvement.
You can use the landing page test to assess
-
page template design — Which layout is easier to navigate? Do your customers relate best to a busy design with lots of messaging and product slots, or do they prefer a clean design with one large image and a few clearly defined links to individual products or product categories?
-
ratio of content to promotion — Does your page generate more clicks when it contains a lot of promotions, or do your shoppers respond better when it contains more links to informational or product content?
-
large product image vs. small “kickers” — Do your shoppers want a large image to give them context for the site, or do they want a collection of smaller product ads?
-
page themes — Are shoppers drawn in by drastic seasonal color changes and image changes, or do they prefer more subtle seasonal changes? What color schemes are best?
You can also perform similar tests on product category pages, product group pages, and other types of Web pages.
Promotions
How do you know which promotion is going to be most effective with your customer base? Assume that you want to find out if a free-shipping promotion works as well as or better than the percentage-off promotions you have relied on in the past. Set up two promotions for the same spot on the page; one should offer a percentage off, and the other should offer free shipping. Use the same design elements for both (images, fonts, placement) so that you know your customers are responding to the different offers and not to different design elements.
Set up your site so that it exposes a small percentage of your customer base to the new free-shipping promotion, with the remainder seeing the standard percentage-off promotion. Measure the results, and the statistics will tell the story.
You can also set up promotions testing for
-
promotional imaging — What promotional images grab shoppers’ attention best?
-
promotional messaging — Do visitors respond better to flashy callouts and aggressive messaging, or do they prefer more subtle messaging?
-
promotional pricing structure — You could also try fixed dollar amount off, buy one get one free, and other promotional structures.
Promotion paths
The promotion might be just fine, but what about the page the shopper clicks to? Find out by testing the same promotion with different link results. It is critical that you know which linked page yields the greatest success.
In this example, assume that you are using a “percentage off all women’s summer apparel” promotion. You now need to determine whether your client base responds best to a page with a leisure-time “fun in the sun” approach or one with a “summer office wear” theme. Following standard testing procedure, you create an appropriate page for each approach and send a certain percentage to one page and the remainder to the other. Measure and analyze the results.
The promotional path test allows you to test
-
product groupings — Are groupings by product type (say, casual business suits) more effective than grouping by theme (such as “fun in the sun” shorts, sandals, and swimsuits)?
-
persistence of promotional messaging — Research is showing that carrying a promotional message throughout the shopping path encourages order placement. Test the effectiveness of persistent promotional messaging from the home page to the product category page to the product detail page.
Shopping cart
Many merchants are adding low-key promotions to their shopping carts these days, but others are concerned that these ads will hurt sales. To find out how well they work for you, design a shopping-cart page that contains some small cross-sells and upsells. Again, present that page to a small percentage of your customers and the regular page to all the rest. Key metrics to watch are number of completed sales, dollar amount of completed sales, and cart abandonment rate.
More elements
As we said, the range of variables you can test is broad. Among other Website elements and features you can fine-tune via A/B testing:
- Offers and messaging
- Directory pages
- Kickers or on-site ads
- Off-site banner ads
- Product page layouts
- Product thumbnail size and quantity
- Navigation scheme, button placement, and button design
- ‘More info,’ ‘tell a friend,’ ‘view larger,’ zoom
- Product category names and types
- Colors and designs
Specific segments
If your e-commerce platform features customer-base segmentation and personalization tools, you can apply A/B testing to specific customer segments. Testing in this way can be very important because a change that elicits a positive result among one customer group could easily have a negative result with another.
Suppose you are running a campaign to increase customer order sizes, and you want to test pitches for your customer loyalty program. You may not want your first-time visitors to see these pitches. You can first set up your site to present one targeted home page to new visitors. Returning customers, on the other hand, would be split between Home Page Loyalty Pitch A and Home Page Loyalty Pitch B.
You could further broaden your testing using detailed segmentation information. Returning male customers could be split between page A and page B, for instance, and returning female customers could be split between page C and page D. All first-time visitors would still see your default home page.
You can also narrow your testing with laser-like accuracy, targeting such specific groups as returning male customers who purchased product X between Oct. 1 and Dec. 20. If this is your targeted segment, split them in a test between two new pages or promotions and see which one fares better. Meanwhile, the rest of your customer base remains isolated from these very targeted pitches and sees whatever default page you have assigned to their group.
Tips and hints
-
Consult with your e-commerce platform vendor to determine what capabilities it provides. Find out exactly what tools are at your disposal, and learn to use them.
-
Change only one thing per test so that you know exactly what led to the results you get.
-
Perform quality assurance testing on all new pages and promotions before you start your test. Unforeseen technical or design flaws will skew your test results.
-
The bulk of your A/B testing should be done where changes will yield the greatest return: your most profitable pages, points of high cart abandonment, poorly used features, or seldom-visited pages.
-
Test again. Use further A/B testing to refine the better performer and to yield incremental increases in sales or conversion.
-
Start with simple tests and increase their sophistication as your expertise grows.
-
Personalize your tests. If you can, target specific customer groups with tests designed just for them.
-
Don’t let testing replace real product/market research. Make sure you are testing sound ideas. A test between two poorly conceived marketing ideas will show only which one is less bad.
One more thing…
Take note that differences in performance between two tested items might be subtle. If both convert nearly the same number of sales, dig deeper into your data to see if one generated more page views, kept visitors on your site longer, or led to more repeat visits afterward.
Once a shopper makes it to your site, you have to be sure that everything he encounters does its utmost to convert him into a buyer. Steady testing and incremental changes can have an enormous impact on how relevant your site is to visitors and consequently how effectively it will convert them. You may be surprised at how quickly A/B testing reveals the truth about prior assumptions regarding customer preferences, improves your conversion rates, and increases your sales.
Ken Burke is founder/CEO of MarketLive, an e-commerce technology and services provider based in Petaluma, CA, and is the author of Intelligent Selling: The Art and Science of Selling Online.
MULTIVARIANT TESTING
More complex than A/B testing, multivariant testing can string one test after another. This lets you simultaneously run individual tests without muddying the results, since the results of each string of A/B tests can be measured and tracked independently.
Let’s expand on the women’s summer apparel promotion test cited on page 24, in which we’re trying to determine whether customers respond best to a page with a “fun in the sun” or a “summer office wear” theme. You randomly direct 50% of the “fun in the sun” clickers to a page that has colorful lifestyle imagery of a family enjoying their swimming pool. The other 50% get a page with imagery of a tropical resort. Those who click on “summer office wear” are split between a page that focuses on separates and one that showcases apparel ensembles. In this case you have four offer combinations:
Fun in the sun + family pool party
Fun in the sun + resort wear
Summer office wear + separates
Summer office wear + ensembles
Your analytics application should make short work of showing which of these generated the best result. But be wary of adding too many variants to your testing. Each variant you add increases the number of possible results exponentially, and you could quickly end up with such a complex array of data that it becomes impossible to make an appropriate business decision. Start small, and add new layers of testing only after you feel comfortable and confident in your testing system.
— KB