If you are like most merchants, you’ve followed the advice of your natural-search optimization (NSO) firm and completed some basic site optimization projects. You routinely spot-check your Google indexation and your rankings on 100 or so “trophy” keywords to show your executive team. And a look at your Web analytics shows that sales driven by your natural-search efforts are growing. So what’s wrong with this picture?
The problem is that you are looking only at the surface of the situation, oblivious to something gigantic that lies beneath. To explore deeper, you need more-sophisticated tools and methods to understand and diagnose the opportunity so that you can take appropriate action. In fact, chances are high that your natural-search channel is seriously underperforming.
As NSO best practices become commonplace among your competition, you need to look deeper to gain an advantage. Fortunately, by approaching natural search with an understanding of long-tail dynamics, new key performance indicators (KPIs), and yield management techniques, you can move beyond the traditional project-based mentality of NSO to a more-sophisticated approach to capture the enormous potential that lies beneath the surface.
The long tail of natural search
To understand the KPIs and management techniques, let’s examine what lies beneath the surface: the long tail — a common statistical distribution featuring a tail-shape curve, as in the chart below.
In the long-tail graph, the Y axis is Website traffic as measured in hits, and the X axis represents unique search terms. Traffic from branded search term accounts for the spike at the far left side of the graph. Traffic from unbranded search terms is visible as you follow the graph farther to the right.
According to a research report, “Chasing the Long Tail of Natural Search,” we at Netconcepts recently published, for every search that occurs for the average merchant’s brand name, nearly 40 relevant searches occur for more-generic, brand-neutral keywords.
For instance, if there are 100,000 searches for “L.L. Bean” this month, there are an estimated 4 million searches for hundreds of thousands of “unbranded” keyword markets that L.L. Bean pages could compete within: “furniture slipcovers,” “women’s flannel pajamas,” “men’s reindeer sweater”… The matching category or product pages on www.llbean.com ideally could position the L.L. Bean brand high enough in the search results to win these millions of click decisions — and sales — on the cheap. And even if these pages do not convert 100% of these searches into clicks, if they are well ranked the pages could nonetheless build brand by association with these millions of corresponding “unbranded” searches among the searchers who used the terms.
This is the long tail of natural search — the universe of diverse, unbranded keyword markets that, while perhaps not as frequently searched on an individual basis, cumulatively, add up to hugely greater search potential than the branded search traffic that most merchants naturally receive. This long tail is the prize every savvy merchant seeks.
Because brand popularity is relative, a more objective way to think of this long-tail potential is as a function of your Website size. Our research suggests that for every unique page on a merchant’s site, nearly 100 unbranded searches are conducted in an average month. If you have a 20,000-page Website, your long-tail potential would be calculated as 2 million unbranded searches. Tracking your unique pages and their yield is therefore a critical measurement to understanding the dynamics of your natural-search channel performance.
Capturing the elusive tail
The long tail of unbranded keywords is a new concept for most marketers to grapple with. What drives it? How do you quantify it? What is good performance? What is bad? How do you capture it?
To answer these questions, we developed what we refer to as page yield theory. It aims to break down, quantify, and model the components of a merchant’s long tail into manageable units and was developed using natural-search data gathered from a few dozen top online merchants using our GravityStream natural-search optimization proxy technology. In the process, we developed a set of metrics to understand the dynamics of the long tail. We then used the results to develop seven KPIs for the natural-search channel.
KPI #1: BRAND-TO-NONBRAND MIX
What percentage of your natural search comes from brand keywords vs. nonbrand keywords? What does it mean if your keyword curve is dominated by brand terms? If most of your traffic is coming from searches for your brand, this is symptomatic of a larger problem lying just beneath the surface: Very few of your pages are actually yielding traffic.
Many retailers find that 95% of their search traffic comes for brand terms, with — not coincidentally — a small percentage of their Website powering that brand traffic. Once you’ve done some search optimization, however, you should find a very different distribution curve. For instance, you may discover that 40% of your pages are yielding traffic, and that 60% of that traffic is unbranded keyword traffic. This in fact is the core hypothesis of page yield theory: Unbranded-keyword traffic volume grows as the number of pages yielding natural-search traffic grows.
Click to see larger image
 |
KPI #2: UNIQUE PAGES | |
|
|
|||
|
|||
|
|||
Just how many pages does your Website have? This is a critical metric for establishing a yield management foundation.
You may be tempted to approximate the number of pages using a product count. This will most likely significantly understate your actual pages. Or you might want to use a search engine’s reported index. This will return mixed results, as each engine indexes dramatically different numbers of pages for each Website.
But search engines are in the business of discovering pages. So we recommend using the number of pages that the bots (such as Googlebot) are able to crawl over a 30- to 60-day period as the most useful proxy for how many unique pages are available on your site. (This assumes that your URLs are in a condition that permits bots to crawl deeply. If they are not, you may need to resort to an approximation.)
As a comparison, the average merchant in our research had roughly 73,000 uniquely crawled pages.
|
KPI #3: PAGES YIELDING TRAFFIC | |
|
|
|||
|
|||
|
|||
What percentage of those pages yield search traffic? Is it 10%? 90%? This ratio essentially dictates the length (the X axis) of your unbranded-keyword long tail and suggests remaining potential.
If you do not know your number of unique pages or yielding pages, you can use the matrix at right for an estimate. It was developed using data from the average merchant in our research and illustrates the inverse relationship between page yield and brand/nonbrand traffic. Simply seek your current level of brand and nonbrand traffic to estimate how many pages are accounting for that traffic.
The average merchant in our study had 14% of pages yielding traffic.
|
KPI #4: KEYWORDS-PER-PAGE YIELD | |
|
|
|||
|
|||
|
|||
Now that you have an idea of your page-yield rate, how many keywords does each of those producing pages yield over the course of a month? Two keywords? Ten?
The keyword-yield KPI is responsible for creating scale. That is, the more keywords each yielding page attracts or targets, the longer your tail.
The average merchant in our study found 2.4 keywords produced per yielding page.
|
KPI #5: VISITOR-PER-KEYWORD YIELD | |
|
|
|||
|
|||
|
|||
This KPI is basically determined by how highly a page ranks for a keyword; it tells you how much traffic each keyword drives. This metric determines the height or the thickness of your long tail.
The average merchant in our study experienced 1.9 visitors per keyword.
|
KPI #6: INDEX-TO-CRAWL RATIOS | |
|
|
|||
|
|||
|
|||
The first step in the search-conversion funnel is converting a crawled page to an indexed page.
The average merchant in our study saw a 3:1 index-to-crawl ratio in Google. That is, for every page that was crawled, there were three in the index, as strange as it sounds. If a merchant finds the index shrinking to, say, a 1:2 ration, that may be a signal of crawl pattern changes, or perhaps pages are being shifted into the supplemental index and are therefore much less likely to appear on results pages.
|
KPI #7: ENGINE YIELD | |
|
|
|||
|
|||
|
|||
Each search engine has a different audience size. How do you fairly compare the referral traffic you get from each? Simply calculate how much referral traffic each sends for every page it crawls or consumes. Compare engine by engine. What we have found is that MSN and Yahoo! tend to crawl a lot more pages, but the yield per crawled page from Google is typically significantly higher.
|
Reconstructing the tail | |
|
|
|||
|
|||
|
|||
Once you have the metrics for these KPIs, you can create a vivid picture of your natural-search performance and make smarter decisions. For instance, brand and nonbrand traffic composition (KPI #1) provides a baseline on the effectiveness of your channel yield. You can contrast this against your total available pages (KPI #2) and actual yielding pages (KPI #3) to understand the gap between current and potential performance. The rate at which these pages yield keywords (KPI #4) multiplied by the number of visitors driven by each keyword (KPI #5) forms the length and height of your tail. Meanwhile, your index-to-crawl ratios (KPI #6) and engine yield (KPI #7) help quantify your site’s effectiveness at converting Web pages into units of “search produce.”
Let’s say the number of your pages that yield traffic is low (less than 10%). It may then be time for some basic, scalable URL rewriting to maximize the flow of “link juice” through your internal link text.
On the other hand, if 80% of your pages are yielding traffic, but only at a rate of one keyword per page and one visitor per keyword, it may be time to conduct some auto-generated title-tag tests to scalably lift rankings of thousands of pages by incorporating more relevant keywords per page. Each situation is unique and requires a custom approach to optimization.
The secret to capturing the long tail of natural search is this: Fully maximize the page yield of your Website. That is, develop a metrics-driven optimization process that empowers each page to be viewed as the authority on the subject, whether the subject is “furniture slipcovers” or “men’s reindeer sweater.” Empowering thousands of pages to yield natural-search traffic may seem daunting, but it can be achieved by focusing on these new KPIs, managing your NSO effort against these yield metrics, and committing to an iterative NSO testing program.
Increasingly, the natural search game revolves around getting your brand associated with unbranded terms. While these new KPIs do not change the basics of NSO, they can provide the context and management discipline that enable you to make more-informed decisions how to best spend time and budget optimizing your long tail.
Brian Klais is vice president, search for Netconcepts, a Madison, WI-based Web marketing agency that offers search optimization services.