« Business Intelligence @ Crossroads | Main | BI Strategy Definition - A "First Principles" Approach »
April 13, 2008
Metadata in the BI World
For as long as I can remember, the definition given for Metadata is "Data about Data".
We have all said this in interviews, heard it from candidates, seen it on presentations and have (almost) always nodded our heads in agreement. In the transaction processing world, where "data-in" is the paradigm, the definition is precise. The databases store the business data in the relational format and the system tables / catalogs describe the structure of that data - the columns, type, size, etc. This data about the structure of business data is "Metadata".
In the Business Intelligence world, that definition of metadata is incomplete. A more precise definition of metadata has two components:
Metadata in BI = "Data about data" plus "Information about information".
The first component "Data about data" is "Technical Metadata" and is similar to the metadata in the OLTP world. Having said that, the technical metadata in BI is arguably more complex, as it not only encompasses the databases but needs to cover the ETL and Reporting tools as well. Each of the tools in the overall BI landscape has its own metadata and this data has to be looked at in a comprehensive fashion to understand data lineage etc.
Even among BI tools, there are different categories - Tools that expose its metadata completely, tools that gives an handle to its metadata through pre-defined APIs and tools that do not allow any access to the metadata. Given the industry direction and the evolution of Common Warehouse Metamodel (CWM) compliance standards, it is only a matter of time before the tool architecture is designed to expose the technical metadata. CWM is a fascinating topic of its own and you can get a feel for it by visiting this website: http://www.omg.org/technology/cwm/
To me, as a BI practitioner, the second piece of the metadata puzzle is the more interesting part. "Information about information" aspect of metadata is "Business Metadata" and its understanding is crucial to implementing the BI vision in any enterprise.
As an analytical information consumer, I have 2 important questions:
1) Need direction to access the required analytical content. Examples are:
a) Where can I get Sales by Product for different geographies over the last 2 years?
b) I am interested in Customer Churn Analytics. Now where do I go? ("system"ically speaking!)
2) Once the content is retrieved, need guidance on how to make sense of it. Examples are:
a) I see the Forecasted Sales for next quarter in the chart. How is this value calculated?
b) Total Inventory value shown in this report ? does it include the Raw material inventory or excludes it?
To the analytics provider, this is a complex problem that cuts across Knowledge Management and context based search disciplines. Having said that, it is important for BI practitioners to understand the true nature of business metadata and provide implementable solutions in their specific organizational context.
I would discuss this fascinating area of Metadata management, encompassing both technical and business metadata, in my future posts.
Please do keep reading and share your thoughts as well.
Posted by Karthikeyan Sankaran at April 13, 2008 6:45 AM
Comments
thanks a lot for ur interesting posting.
i would be very interested to get to know more about metadata and masterdata in generell. but also about the relation between those two data types. especially definitions from a practical point of view would be nice.
greetings
-martin
Posted by: martin at April 18, 2008 7:40 AM
Hi Martin,
Thanks a lot for your comments.
With respect to Metadata and Master data, I had written a blog post on February 9th titled "Zen and the Art of Data Management - The Starting Point". This is an introductory post on identifying the types of data in organizations and I have also tried differentiating between Metadata and Master data.
Please do keep reading and share your thoughts.
Thanks once again.
Karthik
Posted by: Karthikeyan Sankaran at April 19, 2008 10:20 PM
