Has your e-commerce site been load-tested lately? If not, get it done. After all, the IT folks are always writing new code, which can have unforeseen side effects. Also, your site is probably showing double-digit traffic growth. And your e-mail file has almost certainly grown as well.
All of which means that when you need your site to perform its best — when handling the massive response to that killer e-mail campaign, for instance — it could crash, so that customers and prospects trying to log on get error messages. Or just as bad, your Web pages … load … very … slowly.
According to a report published this past summer by JupiterResearch, 33% of dissatisfied online shoppers attributed their dissatisfaction to a Website’s being too slow. Another 28% attributed it to having received error messages. More damning still, three-quarters of online shoppers said that when they experience a site that freezes, is slow, or has a convoluted sales process, they will never shop from it again, and 27% said they’ll tell their family and friends about their negative experience.
Load-testing can help you avoid this sort of calamity.
Making the case
Also known as stress testing, load-testing monitors a Website’s performance under a full load of visitors or transactions. In a white paper, Borland Software Corp. defines it as “the systematic exposure of an application to real-world, expected-usage conditions in order to predict system behavior and to pinpoint/diagnose errors in an application and its infrastructure.”
In the same white paper, Borland reminds us that in eBay’s early days, the online auction site lost $3 million-$5 million in revenue and suffered a 26% decline in its stock price as a result of a single, 22-hour site outage: “EBay understood very quickly the very real impact of poor performance on their bottom line and today utilizes a very robust internal performance center to thoroughly test software prior to deployment.”
While load-testing would seem to be common sense, a surprisingly large number of merchants don’t do it, says Peter Kirwan, chief strategy officer at San Diego-based Webmetrics, a provider of Website performance services and applications. Most of the big brands conduct regular load-testing, says Kirwan, but load-testing is even more crucial for small and midsize businesses.
“The bigger companies that have the financial resources can throw more servers at [a problem],” Kirwan says. “A medium- to small-size company needs to know what’s actually going on so they can solve their problem, because they can’t afford to throw extra stuff at it.”
Load-testing can pay for itself by spotting software issues that may be causing site-performance troubles, Kirwan continues. “If you can find out that it’s a software problem, why throw servers at it?”
Testing can also help prevent unexpected logjams that can happen with just a small spike in Website traffic. “It’s like traffic. Just a couple fewer cars, and the whole thing would run smoothly. But give it a 5% increase, and you’ve got a huge traffic jam,” Kirwan says. “If you get a huge backlog with just a small increase in traffic, then you could lose lots of customers in just a few hours.”
Kirwin recommends testing a site any time there’s been a major change: “If you don’t load-test it, it’s like going out there naked and you have no idea what’s going to happen.”
In addition, Ken Burke, CEO of Petaluma, CA-based Website services provider MarketLive, advises his business-to-consumer clients to load-test their sites before the holiday shopping season begins. “Code gets out of date over the year,” he says. “One line of code can create so much inefficiency that it can blow everything up. And a simple load test can solve all of that.”
Once you load-test your site for the holidays, Burke says, you should go into what he calls “code freeze — no code gets uploaded after a load test until after your peak season. You don’t want to be messing with your site every day, because suddenly the load test becomes invalid.”
Getting it done
There are multiple ways to load-test, including manual testing, using inhouse-developed applications, with licensed software, with open-source — or free — tools, and via outsourced services.
Borland, for one, strongly recommends against manual testing, saying that it is akin to not testing at all. Manual testing is pretty much what it sounds like: If you wanted to test the effect of 50 people ordering on your site at once, you would need to have 50 people on 50 computers conducting the same sort of transactions at the same time, all while somehow recording any errors that might occur.
“Even with five users, this is not practical,” according to Borland. “And load-tests involving hundreds or thousands of virtual users would be impossible.”
Buying load-testing software is impractical for most online marketers as well, Burke says. “It’s hard for companies to go out and buy the software and run their own load tests because it’s so darned expensive. I literally got a quote for $400,000 from IBM to license its software and just about died.”
And open-source software is not for the faint of heart, Burke adds. It takes serious equipment and inhouse expertise. MarketLive, for example, bought $100,000 worth of hardware and has two technicians on staff to use open-source software to test its clients’ Websites.
When using an outsourcer, load-testing can cost anywhere from several thousand dollars to $60,000. Burke says that most midrange merchants can expect to pay about $20,000 for a load-test of 2,000-3,500 concurrent users. He adds that to accommodate the fixing of problems discovered in the initial test, service providers often sell load-tests in packages of three. “Typically in three load-tests you’re good,” says Burke.
Kirwan says that companies will often load-test a new application on so-called staging or development servers before puttng the application into place on their Website. Then again, load-tests sometimes have to be performed on the site itself.
“This is dangerous, and it’s why monitoring is so important,” Kirwan says. Not surprisingly, load-tests on live sites are usually scheduled for off-peak times. “Often the IT department will schedule it for one in the morning or some other time where their traffic is low and they’re all available.”
The first thing you must do to perform a useful load-test is to forecast the site’s peak traffic numbers by looking at year-over-year growth, Burke says. Moreover, “you have to understand your traffic by the hour.”
Almost every Web analytics package offers hour-by-hour reporting, Burke says. “So you have to look at your busiest day, but you also have to look at your busiest hour, because that’s really what makes the difference.”
Along with accounting for site traffic patterns and year-over-year growth, you need to take into account how much your business has grown. If your e-mail list has expanded appreciably during the past year, for instance, you need to increase your Website traffic projections.
Even the types of offers you plan to make will affect site traffic. One MarketLive client recently sent an e-mail campaign with an offer on jewelry that resulted in four times the site’s normal peak traffic throughout the day.
“The offer was so exclusive, the traffic was off the charts,” Burke says. “The problem was, the site was down for most of the day.”
Testing as a marketing issue
Burke tells the story to offer evidence that load-testing is a marketing issue. “Don’t think of it as a technical issue,” he says. “Think of it in terms of what’s going on in the business. That’s what I like to understand before I do a load-test.”
In fact, Burke contends that the number-one contributor to Website traffic spikes is e-mail: “I guarantee that if you ask 100 online merchants if they’ve sent an e-mail campaign that took their site down, 95 will say yes.”
But in addition to e-mail promotions, there are other situations to take into account, such as what Burke calls “the Oprah effect”: A merchant’s product appears on her show, and people flock online looking for it. “Oprah can spike traffic and take down a site like you wouldn’t believe,” he says. A major snowstorm on the East Coast will also send merchants’ Web traffic through the roof, he adds.
If e-mail is the leading cause of traffic spikes, Burke says the leading point of failure on e-commerce sites is their databases. “Everything else is typically somewhat redundant, but if the database server goes down, everything goes down. I suspect that’s what happened at Wal-Mart,” he says, referring to the retail giant’s site crashes during the past holiday season.
Marketing departments often plan major promotions without telling the IT staff — and that’s asking for trouble, Kirwan says. It’s important to let Web developers know not only that a promotion is planned but what type of promotion it is so that they can perform the right load test.
“If you’ve got a promotion planned that will drive people to your home page, that’s one thing. If they’re going to go look at buying stuff, that’s another thing. If they’re going to go to a part of your site that has information fed from a third party, that creates still another type of load,” Kirwan explains. “Imagine if you’re the IT department or software development. They can’t possibly know this. It has to come from marketing.”
Quick tip: Don’t be cheap
Peter Kirwan, chief strategy officer at Website performance services provider Webmetrics, says that load-testing may enable some companies to avoid buying unnecessary hardware by identifying problems that can be resolved by simple software fixes.
But that doesn’t mean you should pinch pennies when it comes to servers, warns Ken Burke, CEO of Website services provider MarketLive. “Don’t chintz out on hardware,” says Burke. “It’s the stupidest, most narrow-minded thing a merchant can do.”
Merchants tend to be frugal, Burke says. But failing to buy enough hardware to accommodate growth ends up costing more money than it saves. “The time you want to be a rock star is the time when you’ve got all these new people coming to your site,” he says, and if you don’t have sufficient server capacity, “that’s the time that you fail.”
— KM