I have as usual, been to a lot of trade shows on Data Governance and MDM. And in the past month or so, I have run into more unemployed Directors of Data Governance than I care to admit. They were all very intelligent, thoughtful, and new quite knowledgeable about data management in general. So why is it that so many of them are looking for jobs. After talking to quite a few of them, I have some thoughts on why is it that the tenure of these folks is short.

Quite of few of them were working in financial services, specifically in companies where the mortgage credit crisis forced quite a few layoffs When asked why they were laid off. Many of them commented that business management didn’t appreciate the importance of data quality and governance. When I asked what were the benefits they were presenting, they commented that the benefits were obvious, but never articulated a specific example where data governance had saved money, prevented a disaster or improved the business in a specific measurable manner.

All of this doesn’t mean that data governance isn’t valuable. However, I think that it is perceived by business managers as just another one of those overhead activities that IT wants to promote. It feels a lot like the general “quality” craze in the 80s that started with Japanese companies and at first, U.S. and European manufacturers didn’t really understand. They would try “quality” programs for a little bit, then fail, and the quality managers would lose their jobs.

Eventually, good product quality became expected by the consumer. You could buy a good quality product for the same as the cost of a bad quality product. So quality manufacturing became an expected part of doing business.

Unfortunately, most companies don't seem ready to think about data quality in the same we think about product quality. Actually, if you turn it around, and thought about product quality the way we think about data quality, I doubt you could drive your car out of the driveway in the morning. The wheels would fall off, the steering wheel wouldn't turn and the car probably wouldn't even start if the quality level of your automobile were as bad as most data environments.

Anyway, so what can anyone do about this? Well here is my short list of thoughts on the topic:

- Stop talking about data governance and data quality as an objective in and of itself.
-Focus on business value: What if we could reduce the number of miss-configured products? What if we could more easily measure the risk of our financial portfolio so we could keep less cash reserves on hand? What if we could more easily cross-sell and up-sell our customers?
- When you create value from good data governance, market that internally. Remind people the savings you are achieving every day as a result. If you don’t market the success, people won’t notice. This is one of the hardest things for an IT professional to do. We aren’t extroverts by nature.
- Don’t call what you are doing data governance. Just do it, measure it, measure the value, then declare your success from the mountain top later. Then go ask for more resources to do more of the good stuff you just did.

What other ideas do you all have about this? I know my list is pretty short, but I am sure that people reading this blog have additional thoughts, please share them.

Perhaps that is what both Symantec and EMC are doing with their recent acquisitions?

Or, maybe it is just as simple as the fact that security cuts across everything, so as a discipline, it is one of the most diverse topics you can possibly think of. You have physical security, network security, identity management etc etc. So it really isn't a surprise that as enterprise data management is still kind of young, with MDM and data lineage deployments just now starting to take off, the overlay of how security of structured data is a little behind.

Ack. It's late and I am rambling. check back in with y'all later.

btw, this week I am in lovely Orlando Florida at the Data Governance Conference. So I will send you all my thoughts while I have them.

Also, a belated Happy Thanksgiving to all of my U.S. readers. And if those of you in the U.K. feel left out, then Happy Guy Hawkes day. :)

This came up an incredible amount. Many companies have built their master or purchased their master. (Note for those of you who are not in the finance world, you can buy a pre-populated MDM system where the data sources are already pre-mapped to each other. This is because there are 100 or so companies in the industry that provide data feeds to financial firms and the financial MDM vendors not only sell the MDM software, but the model with the data feeds pre-integrated. ). Regardless, when it comes to integrating to the hundreds of downstream legacy applications that currently get their data directly from outside data vendors they find that the cost of this integration is too expensive.

This is clearly a case of “If you build it, they will [not] come.” (for your “Field of Dreams” and Kevin Costner fans!]. The problem is multifold:
a. The cost of integrating to the downstream applications is an expensive manual process
b. The IT group which built the master often publishes an interface to integrate with and tells the downstream groups to integrate with it. The problem is that the downstream application groups don’t have the skills or the budget to do the integration so it doesn’t happen.
c. The IT groups don’t want to map their data to the downstream applications themselves because they don’t know the data structure of the downstream applications and SMEs for the downstream apps aren’t available to help. Very Catch-22.

The result is that millions of dollars are spent building the reference master and they currently end up being underutilized.

My talk at FIMA was on this very topic. I went through a real example of a talented data analyst that had to map 6 columns of data to 3 different data sources and what was expected to take 3 weeks ended up taking 7 months. As it turned out, the 6 columns were overloaded columns that contained different codes depending on which of the 50 U.S states a customer might be located. The result was that it was more like 300 columns shoved into those 6 columns and sorting out the mess ended up being very time consuming. This is just an example of why no one wants to step up to doing this work. The second half of my presentation talked about a new approach that automates the discovery of the relationship between the master and the downstream application data and speeds up this process.

My thinking on this is that if financial institutions want to resolve this issue, their master data management IT groups will have to step up to the plate (“Field of Dreams” pun intended) and develop a competency to do this last mile of integration. It just doesn’t make sense to make the downstream app groups do the work. They would only do the work one time each for their application and it doesn’t make sense to staff up for that. But the central group can develop a competency in this and as there is technology now to help, there is really no longer an excuse for not providing full service to their users and developing repeatable processes to solve this problem.

The two topics that came up over and over again during the two day marathon sessions were Data Governance and integration of the master to downstream systems. I will write bout the data governance issue today and leave the downstream integration for next time (my attempt at a cliff hanger as I am sure you are all just waiting with excitement to read the next installment ;) )

Data Governance: This has been a hot topic in the states for the past year with the start of the Data Governance conference but clearly people are realizing that you can’t just have the data management piece of a reference master without also have business people who will provide governance oversight to handle the exceptions.

Lots of discussion about where governance should sit. Is it an IT function or is it a business function. Outside of the financial services world, I think there is little debate about this, that governance and the data stewards sit on the business side as they have to make business decisions about the data. However at FIMA, there didn’t seem to be as much consensus on the topic. Perhaps it is because very often there is a lot of overlap and moving back and forth from business to IT and vice versa in financial institutions. In general, there is a lot more IT savy in financial institutions on the business side and the IT folks tend to take a lot of interest and have a lot of knowledge.

One consultant I spoke to commented that perhaps it is just that no one wants to step up to take “ownership” of the data. This is even more likely. Because “ownership” would imply responsibility and with Basel, MiFID, Sarbox etc. hanging over all these firms, no one at lower levels in the organization want to be responsible for the data because then they might implicitly be accountable in a legal sense.

As for the next topic, integrating with downstream systems, you will have to wait until tomorrow. Time for me to board the plane back to sunny California.

Great talk on data quality by Akady Maydanchic. The top line takeaways were:

- Data quality is not getting better it is getting worse. While there has been tremendous progress specifically in the area of customer data, all other areas of data have actually gotten worse. That is related to the next point, that speed of data movement is bad for
- Speed is antithetical to data quality. What has happened over the years is that while systems have become faster, that has left less time to fix DQ problems. When data used to get moved once a month, there was plenty of time to go back and fix data quality problems before they propagated to downstream systems. Then movement time shrunk to a week, and there was less time to fix problems but they could still get fixed. In fact, lots of DQ problems are self correcting if there is enough time for users to go back and fix their errors. But has data movement has gone real time, data is moved to downstream systems before it can be corrected and if it gets corrected in a down stream system and not an upstream system, then you end up with data inconsistency problems.
- Tools are lacking. The DQ tools are still immature and have not progressed much in the past 5 years. The tools available today are too focused on single column values. The big issue is in tracking complex business rules and data relationships between tables and between systems. The tools don’t do a good job of dealing with complex rules. Tools also need to provide a better infrastructure for doing quality scorecards. There needs to be an environment that enables performing a data quality assessment then allow ongoing scorecarding to see progress

Bob writes:
Isn't MDM just another form of data consolidation? What makes it so radically different that it warrants it's own moniker?

My response:

Bob, great question. I would say the difference between a data consolidation and MDM is that in a data consolidation, you consolidate the original data sources into a single data source and eliminate the original sources. This means that you only have to deal with that consolidation once.

MDM keeps all the original sources and merges the data values according to a set of business rules that you define for survivorship. However, you don't eliminate the original data sources so as new data is added to those sources, those new values get merged, purged and matched into the master data system. The master data, sometimes also refered to as reference data is then used by downstream applications as the "golden master".

The idea of MDM is to create common master data that is used by other applications so you have consistent reference data that is used by all applications. The idea of a data consolidation is usually tied to an application consolidation where you are trying to eliminate excess or redundant applications and the result is that you also have to consolidate the data.

Craig McGill writes:
"but perhaps the reason for including MDM at the start of all the articles was for search engines finding the article (and the analyst) as well as part of the SEO strategy."

My response:

Craig, only the last two blogs have been about MDM and I wrote about it because it is a topic that interests me and I had just come back from the Gartner MDM conference... wow, how cynical some of us have become. :(

Craig McGill writes:
As for the other point of understanding the landscape, that's a fair point, but any MDM company worth the title (disclaimer: I do the PR for VisionWare PLC) is going to have staff trained up to find out where you are, what you want from the MDM and what you don't want (as important as the first two in my opinion).

My response:
The issue I am writing about isn't understanding "what you want and don't want" from MDM. The challenge, especially when you move out of the CDI space and into more complex data environments like financial reference masters, counterparty data masters or just product masters for manufacturing firms, the level of complexity moves into dealing with thousands of data attributes. Furthermore, the level of comlexity of the data relationships that must be rationalized go beyond simple one to one matches and you end up dealing with complext transformations and multinested case statements that are needed to handle the merge, perge and match process.

So while I can't speak for what is going on over on your side of the pond in the UK, I can say that over here in the U.S. the companies I am seeing are really struggling with where to start... hence my recommendation.

In a nutshell, this person talks about the cornerstones for MDM and they are:
- Vision
- Strategy
- Governance - Organization
- Processes
- Technology Infrastructure
- Metrics

the difference between what this person wrote and what I just wrote above was that they prefaced every term with "MDM", so these cornerstoes were actually MDM Vision, MDM Strategy and so on. Wow, thanks for the brilliant insight... not! (as my 12 year old, or Borat for that matter, would say).

Let's see what other prefixes I could put into that list and still have it work:

- Corporate (Corporate vision, corporate strategy, corporate governance ... yup it works!)
- IT
- Business
- Financial
- HR
- Data
- Web 2.0 (not perfect, but not too bad)

yes, they all work, more or less as well. Let's continue:

- X-ray (OK that just works for Vision)

I think you get the point. This advice is so generic as to not be usefull at all. So rather than just complain, I will give you my more practical advice for how to start.

First, start by understanding your existing data landscape. You can't get where you are going if you don't know where you are. Since MDM is all about improving data consistency, your presumption is that you have data inconsistency problems or you wouldn't be considering MDM in the first place. So figure out what your current environment looks like.

More specifically, start by analyzing the data sources that you are planning on using to populate the MDM hub. Understand how attributes (columns) align both within and across your data sources. What are the commonalities, what are the differences and what are the conflicts. Do the same for each row of data, the cells and the data structures.

You can do this by writing SQL and using TOAD or by using any of a number of tools for data analysis available in the market (full disclosure... my company Exeros sells this kind of a tool). Make sure however that you can do analysis both within data sources and between data sources. Remember you are trying to find sources of redundancy and inconsistency across your data sources so cross system analysis is critical. Note that you don't have to analyze all of your data sources, just take a look at the most critical ones, you know what they are, and use that to extrapolate. From there you can establish a baseline from which to work and then put a more realistic plan in place.

Once you have done this, you have the basis for determining the work involved in rationalizing your data sources, you will have raw data to calculate the ROI of an MDM project by quantifying the % of inconsistencies in your data and by estimating the value of reducing or eliminating those inconsistencies. And you will also get a better grip on what else you need to do before you really get started.

For lack of a better term, I would call this an MDM Readiness Assessment.

So go get an MDM health check... start with an MDM Readiness Assessment.

However, I decided that this is still worth a quick entry.

My top takeways from the conference were:

1) End users are just learning how to spell MDM. The most common questions that came up were, "How do I justify the ROI of an MDM investment?" and "How do I get started?" Both questions reflect that the end user market is still in learning mode.

2) Presentations were still very high level. There were not a lot of precriptive presentations with suggestions on what to do and how to do it. There clearly is a need for more precise description of what exactly people need to do to get going. This is reflected as well by the questions that people where asking per my first point. If this market is going to make it, we all have to get off the high level blah blah blah and get specific.

3) Just because a vendor calls what they do MDM doesn't make it so. As a vendor, this one kills me. If you are going to get up on the dais and talk about how you help improve MDM deployments, don't just take the stuff you used to sell as data quality or ETL or EAI or whatever and just give your old pitch and add MDM at the end of every third sentence. There are too many vendors who think that good marketing is just taking their existing stuff, adding the latest buzzword as a suffix or prefix and then trying to sell it. NOTE TO VENDORS - Your prospects are smarter than that!

By the way, there were also vendors who do provide real valued added MDM products as well so I don't want to lump everyone into the same bucket. But as for the charletons selling ice to Eskimos, remember that you can only do that once.

4) There is definitely a need out there for solutions that enable data consistency. Distributed computing has driven distribution of data which has driven inconsistency in data which has driven bad decision making, poor customer service, slow time to market for data intensive projects not to mention the difficulties of data governance in a distributed environment. End users are really suffering and there is definitely a big opportunity for those solution providers who can help improve data consistency for their clients.

SAP decides to enter the acquisition frenzy and purchase Business Objects for $6.8B. For those of you who don't watch this market that much, that is a huge deal for SAP. SAP has traditionally been a not invented here kind of place and has tended to build rather than buy. Most of their acquisitions are of small technology firms... so for them to belly up to the bar and buy BOBJ, that is like buying a dozen pints and downing them all.

Regardless, this continues to be exciting times in data management land with ongoing consolidation. The downside is that while the number of large suppliers is thining, there is little investment in the market from the VC community. It could be that all of the easy ideas have been implemented, but that is usually a good thing for startups since most big innovation in software has come from the start up community in recent years.

First off, Princeton Softech got acquired by IBM. Princeton has software for data archiving and data de-identification. http://www.infoworld.com/article/07/08/03/IBM-to-buy-Princeton-Softech_1.html

Then, EMC acquired acquired Tablus, which is the sensitive content identification and protection business. Their product identifies sensitive data in emails and files. They don't have capabilities in the structured data arena.

So after a few years of quiet in the early stage acquisition market, we are seeing the giants wake up again using the free market and small firms to take the risk in emerging markets, prove out the markets, then snapping them up.

That is good news after what I would consider a dry spell for data management. Although there have been acquisitions in this space, they were either of medium sized public companies, like IBM buying Ascential, or the purchase of also rans in the industry just trying to get some kind of exit.

Anyway, I like both of these companies (full disclosure, my company, Exeros, has partnerships with both of them) as they have great products, provide great value in new and emerging markets and they both have great teams.

Only time will tell if the data management market continues to serve up more acquistions or were these two very closely timed acquisition merely just a coincidence.

http://www.intelligententerprise.com/blog/archives/2007/07/data_discovery.html;jsessionid=RUJMN5NWB54LAQSNDLPCKHSCJUNN2JVN

While he calls this Data Discovery, I would refer to it as Data Relationship Discovery. You have to be able to go beyond just analyzing a single data source for information like mean, median, mode, cardinality, selectivity, frequency distributions etc and even just the simple ability to discovery primary foreign key relationships. You have to be able to discover complex business rules between systems. It is the "between" that really counts here. So what is really interesting in a distributed computing world is to not just do data discovery, but to do "Data Relationship Discovery"

per their website "The HP Neoview platform is an enterprise data warehouse that enables businesses to confidently capitalize on their information at a dramatically lower cost."

So what does this have to do with data governance? On the landing page the Google add sends you to there is not one mention of the term "Data Governance"... not one.

Not to pick on HP, I am a former HP employee and really love the company, but they and just about every other vendor in the data management space seem to have decided that because "Data Governance" is a hot term, they will take their EAI, ETL, SOA, ERP, DW and whatever other two or three letter acronym they sell and call it "Data Governance".

It seems to be the trend is that every vendor took the thing they had and now just labled it "Data Governance". It is just a sign that "Data Governance" is in a big hype cycle. I know that 12 months ago that term was purchased by maybe 3 or 4 vendors, now there is a list as long as my arm.

I sat in on a presentation by Bob Seiner of KIK Consulting and tdan.com. By the way, if you don’t know about him, I recommend that you check him out. He is giving a very practical approach to data governance.

A very gutsy and aggressive presentation. He flat out said that there should be no such position as a data steward. You can’t list a position called “data steward” and bring in someone from outside to take on that role. It won’t work. You have to find people in the organization that are already dealing with data governance issues and “recognize” them as the data steward as part of their job.

That is quite different than what most consultants say but it makes sense. Bob is preaching a very practical approach that companies can actual execute. He is proposing an incremental approach to data governance that is focused on finding specific projects with specific goals that solve specific company problems. Some of his top level recommendations:

Recommendation: Identify/gain a management sponsor and make sure he/she understands the value of data governance

Recommendation: define a scope for your data governance initiative. Pick ONE project, define the scope for that then grow it over time. Don’t boil the ocean.

Recommendation: define a communications plan. Make sure the implementation is understood and the results are communicated as well. Build a data governance home page, make an email address like datagovernance@company.com available. Make sure people know what you are doing and the value it delivers

Recommendation: Define your goals and objectives. Know where you want to go.