« Don’t Shoot the Report Writer | Main
January 11, 2010
The Integrated Marriage: Modeling and Profiling
For the road weary integration warrior, the announcement of the union of data modeling and data profiling is no CNN late breaking news update.
Most integration projects understand the need to map all data sources to a data model which later (or concurrently) is mapped to the target data source.
Unfortunately, the data modelers (or ETL designers) are usually working without a net. They either interview system experts for the definitions of the source columns to get 'memory recollection of data content'..or they complete one off queries (read as: time consuming, inefficient task) to check the values of a data source.
Recommendation:
Identify a scope of work on the project,
complete data VALUE profiling for all of the data source objects for that scope of work
sort the data sources at two levels:
= on whether the data source table/file is References Data, Transaction Data, or Summary data
= on the data values found during profiling
Most profiling tools can find Char(3) matches to Char(3) but they can NOT find Char(3) and Integer matches when the values of each are 1 - 999.
It is only at this point when you really understand what you have. NOW the data modeling / data mapping work should begin.
Throwing data modelers and data profilers at an integration project is not wedded bliss unless your data profiler courts the project appropriately and presents VALUES to the betrothed data model. But done right...your integration project can result into the long term relationship you always dreamed about!
Posted by DataGoddess at January 11, 2010 6:15 AM
