« Business Intelligence and Six Sigma | Main | Data Modeling in the BI World »
March 8, 2008
"Right" Time Data Integration - How "Real" can it get?
Data Integration in the BI sense, is all about, extracting data from multiple source systems, transforming them using business rules and loading it back into data repositories built to facilitate analysis, reporting, mining etc.
Given that the raw data has to be converted to a different form (subject-oriented rather than being process oriented) more amenable for analysis & decision-making, there are 2 basic questions to be answered:
1) From a business standpoint, how fast should the "data-information" conversion happen?
2) From a technology standpoint, how fast can the "data-information" conversion happen?
First question is related to the concept of "Right-Time" BI while the second one deals with "Real-Time" data integration. You can get a feel for this topic at the link below: http://www.tdwi.org/research/display.aspx?ID=7095
Traditionally, BI being used more for strategic decision-making, we were happy with the batch mode of data integration with periodicity of a day or later. But increasingly, business demands that the data to information conversion has to happen much faster and that technology has to support it.
Since the answer to the first question above from the business side, is fast becoming "as fast as possible", the focus has shifted to the technology side. Some solutions to the problem are highlighted below:
1) Enterprise Information Integration (EII) - The paradigm here is to "Leave the transaction data where it resides". Business Intelligence reporting/query/analytical tools have to seek data from the OLTP systems through a semantic layer that defines the required analytical relationships. This is probably as real time as you can get!
2) Active Data Warehousing - The most popular proponent of this approach is Teradata. This is the concept of "BI on the Fly". By intelligently combining the hardware and software power, tools like Teradata and other DW appliances can provide analytical outputs from transactional data with terrific performance.
3) BI with EAI Architecture - In the traditional approach to DW construction of integrating multiple sources through ETL tools, one area where I foresee a lot of activity is in the close interaction of EAI tools like IBM Websphere MQ, TIBCO etc. with data integration tools like Informatica etc. At this point in time, though the technology is available, there aren't too many places where messaging is embedded into the BI architectural landscape.
Bottom-line is that there is significant value gained by ensuring that raw business data is transformed to information by the BI infrastructure, as fast as possible, with the limits being prescribed by business imperatives. The best explanation I have come across to explain the value of information latency is the article by Richard Hackathorn (http://www.tdan.com/view-articles/5132).
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at March 8, 2008 4:45 AM
Comments
hi karthikeyan,
Your archive shows that both you and us started blogging at the same time.
better late than never... Keep on blogging!
Posted by: MAIA Blog Admin at March 10, 2008 9:51 AM
