BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

« Privacy and Syndicated Data | Main | Will Web Services Degrade the Relational Idea? »

January 30, 2007

DIdn't Anyone Notice Incorrect Reports Before MDM?

The current issue of TDWI's What Works (Volume 22): includes a summary of some market research conducted by Phil Russom on Master Data Management (MDM) that really caught my attention. To the extent that it demonstrates a significant demand for something like Master Data Management, it is very interesting, but insufficient. But it also raises some questions about how we create and publish market research in this industry (and I am including myself in this assessment).

First of all, I want to say that Phil is a serious professional, I like him personally and have a lot of respect for his work, and he is an excellent analyst. But, like all analysts, myself included, some or all of our research is funded by vendors. In Philís case, this research was funded by Actuate, ASG, BusinessObjects, DataFlux, Informatica, SAP, Sunopsis and Teradata. At a certain level, the analyst is attempting to uncover the sentiments in the market toward certain ideas and trends, but the sponsors are expecting a document that validates their current marketing of some concept and/or product. This is because buyers mostly operate in herd mentality, and a favorable report from a respected analyst is a useful tool. It's a difficult position for an analyst. Because of this balancing act, it's important to learn how to read these survey results.

First of all, survey design is a very complicated process, a craft really, if you want to uncover the nuance of certain phenomena. But surveys designed for marketing purposes are not very rigorous. Part of it has to do with how you select the sample, and part it has to do with how you ask the questions. I havenít examined the methodology of the full report, but itís likely that TDWI members were polled and responses self-selected through a web-based survey tool, so bias is significant. But what is more interesting is the questions themselves. For example:

ďHas your organization suffered problems due to poor master data?Ē 83% said yes. Thatís like asking if youíve failed to plan sufficiently for your retirement. Without some quantification, who would ever answer no to question like that? A better question would have been, "Is your organization impeded in meeting its goals in a measurable way due to poor or lacking master data?"

So from the beginning, the impression is made that 83% of companies feel they have suffered problems due to poor master data. This defines the market. The number will (and already has been) quoted over and over. Now the report goes on to list the categories of problems theyíve had such as Inaccurate Reporting, Arguments Over Data, Data Governance problems, etc. These questions are so open-ended and soft they are like asking people if they ever wished they had more money, were taller or had more time for leisure activities. In fact, itís amazing the response rates werenít higher than 70-80%.

Dissecting the answers on benefits from MDM, there were 741 respondents, but only 54% claimed to realize benefits, or roughly 375 of the total. Yet when the kinds of benefits are charted, the denominator for the response is 375, not 741, inflating and distorting the impact of each benefit. In other words, we are led to believe that 76% of the respondents achieved improvements in data quality when in fact, it was only about 40%. Other soft questions, those that are designed to get a desired response, are:

Improvements in Data Quality: How much of an improvement would it take to answer yes to this question? Is a 1% improvement enough, even if it cost $5,000,000 to implement? A better question would have been, "Can you measure the effect of improvements in data quality as a result of this initiative?"

Better decision making: Who can actually measure this, how much better, and what state was decision making in before the initiative? I wouldn't substitute another question here because I don't this is an objective question.

Easier auditing of informationís origin: Again, is it just fractionally better or a lot better?

But the one question that really got my attention was ďAccurate Reporting.Ē This is tied with Improvements in Data Quality as the most frequently cited benefit (about 30%) but it begs the question, how inaccurate was the reporting before MDM? After all, these companies didnít crawl out from under a rock, they are TDWI members and have presumably been around Data Warehousing for a number of years. They may have won awards for their efforts, presented case studies of their successes at conferences, who knows? Were their reports inaccurate last year or the year before? If we went back in the literature and checked on them, would they have mentioned that their reports were inaccurate and they were just waiting for a solution, such as MDM, for which they didnít have a name yet?

Data Warehousing has been around for 20 years and accurate, integrated data is the heart of the solution. How is it that the leading proponents of Data Warehousing, especially the visible and vocal ones, failed to notice for 18 years that everyoneís reports were inaccurate until MDM came along? I know the answer will be that MDM is broader than Data Warehousing in its reach, it is designed for the enterprise, not just analytics, but this begs one big question Ė isnít reporting part of Data Warehousing??? If reports were inaccurate before MDM, then Data Warehousing didnít do its job and you should take the experts and gurus to task on this one. Unless Iím mistaken, the lionís share of output from data warehouses has always been, and continues to be, reports in one form or another. If theyíve been inaccurate, what does that say about existing data warehouses, data warehouse products and data warehouse methodologies?

On the other hand, for an industry to promote a new solution (MDM) that solves a problem (inaccurate reporting)that was supposedly already solved by prior approaches (data warehousing) is troublesome. In 2003, TDWI awarded special recognition in eight categories to companies that achieved excellence in data warehousing. You can read about it here:
The names of the judges are also listed. Was Inaccurate Reporting overlooked as a criterion? With 81% of today's respondents reporting it as a problem, it seems pretty likely that at least some of these award recipients were experiencing it in 2003. Did everyone miss it?

I think the issues cited with the survey are important, but not dangerous. The sponsors of the study are satisfied that they have some ammunition for their sales force and the analyst has raised the awareness of an important, emerging issue. Few will come away remembering which benefits ranked in which order and for what amounts. Rather, they will have the impression that MDM can solve some of these problems for some people, if not necessarily for them. That's not so bad.

Posted by Neil Raden at January 30, 2007 4:45 PM


Post a comment

Remember Me?