Do you remember the 10 commandments of marketing database content management from previous columns? In case you don’t, here’s a quick recap:
- The data must be maintained at the atomic level
- The data must not be archived or deleted
- The data must be time-stamped
- The semantics of the data must be consistent and accurate
- The data must not be overwritten
- Post-demand transaction activity must be kept
- Ship-to/bill-to linkages must be maintained
- All promotional history must be kept
- Proper linkages across multiple database levels must be maintained
- Overlay data must be included, as appropriate
These 10 commandments lead to what we at Wheaton Group refer to as best-practices marketing database content. Everything within reason must be kept, even when its value is not immediately apparent. Best-practices marketing database content enhances data mining, often dramatically. This, in turn, allows the deep insight into behavior that is required for effective data-driven decision-making.
Today’s failures Many of today’s databases don’t come close to best-practices content. A key reason is that most database developers are not deep-dive data miners. Rather, they are pure technologists, so they have no first-hand experience in the nuances of what is required to support best-practices data mining. Couple that with client companies that have experienced neither best-practices marketing database content nor best-practices data mining and you have the blind leading the blind. This is a sure path to subpar content and the inevitable subpar data mining. Remember, data mining is only as good as the underlying data.
We discussed in February why you must not archive or delete the data, using the example of a database marketing firm. This company tried to build a model to predict which customers would respond to a holiday promotion. But all data content older than 36 months was rolled off the database on a regular basis; it wasn’t even archived. So the database reflected only three years of history for a customer who had been purchasing for 10 years.
To build the model, the company had to go back to the previous holiday promotion. This reduced to 24 months the historical data available to drive the model. Worse yet, the company needed to validate the model off another holiday promotion, the most recent of which had taken place two years earlier. This reduced to 12 months the amount of available data. So the resulting model was far from optimal in its effectiveness.
What I did not mention in the February article is that the database was built by one of our industry’s largest data management firms. The development team included the full complement of project managers, database architects, database engineers, and subject matter experts from the client’s marketing and IT departments. Unfortunately, no one knew that limiting the database to three years of customer history would severely hamper most data mining efforts. Sadly, many of the data management company’s other marketing databases contain only two years of data.
Today’s business intelligence and campaign management software, with its GUI interfaces and eye-catching output, have revolutionized the ways that marketers experience their data.
But all too often this impressive technology is constructed upon the rotten foundation of inferior database content. With inferior content, the most advanced business intelligence and campaign management tools in the world will not get your company where it needs to go. This is because a marketing database is only as valuable as its underlying content.
Creating and maintaining best-practices marketing database content is hard, ugly work. There is nothing glamorous about sifting through every data source you can get your hands on in order to organize, fix, and enhance it, or, implementing and religiously adhering to quality-assurance procedures during all subsequent database update cycles.
Don’t get me wrong: I am a huge proponent of the wonderful tools available to today’s database marketers. But such tools must be coupled with best-practices marketing database content. Cutting-edge software with subpar content are a dangerous combination. It is analogous to a builder’s taking a pile of lumber, turning on some power tools, and throwing the tools onto the pile. It is certain that he will not end up with a house!
Waxing nostalgic Some of our firm’s work is pure data mining and therefore does not involve data management. All too often in such cases, I am frustrated because of the limitations of the databases that we are called in to leverage. Incredibly, it can be impossible to do what I was able to accomplish nearly 20 years ago with mainframe databases that employed non-table-based, proprietary technology. This is inexcusable!
During the late 1980s and 1990s, I worked for a now-defunct data management firm, Wiland Services, which back then was building systems with better content than many in operation today. I took over Wiland’s data mining and consulting group in January 1991, and a major retailer was one of my group’s clients. Wiland maintained a customer database for the retailer that comprised 11 million active customers.
The database offered no real access except for green-bar reports and extracts to statistical software packages such as SAS. But, the system met the modern standards for best-practices marketing database content. Therefore, the past-point-in-time customer views (“states”) that are essential for virtually all meaningful data mining could be easily re-created. This allowed my group to develop a large number of specialized regression models to drive the retailer’s sophisticated target marketing programs, and to fine-tune or rebuild the models at will.
During every database update, more than 12,000 lines of SAS code cost-effectively evaluated all of the billions of dollars of atomic-level order, item, and promotion history information across all of the 11 million customers. In addition, the code could be altered significantly and then put into production with less than one day of lead time.
The marketing database also automatically maintained up to 36 historical point scores and segment codes for each model. This allowed the creation of longitudinal velocity measurements, such as an indication that a group of customers was declining significantly in month-over-month scores and therefore was ripe for palliative measures. As a result of the target marketing programs supported by the database’s best-practices marketing database content, the retailer was able to decrease general advertising expense by 25% while increasing revenue significantly.
Is it too much to ask that, in terms of content, today’s marketing databases match the capabilities of a system built nearly 20 years ago? As a data-mining professional, I have no use for some of the modern databases our firm runs in to, in which powerful, GUI-based business intelligence and campaign management tools are driven by subpar content. And neither should you!
Jim Wheaton is a cofounder/principal at Wheaton Group, a Chicago-based data management, data mining, and decision sciences practice. The firm also offers full list processing capabilities through its Daystar Wheaton Group affiliate.