May 31, 2009
Time to get BASHED UP! - BI greets Mashups
Not many applications become killer-apps in their life-time and neither have Data Mashups become one. But of what I have seen of Data Mashups is enough to get a feel of what killer applications are all about. I, for one, strongly believe that data mashups can have profound impact on the way BI is delivered to business users. Let's explore a bit of bashups (BI Mashups) in this blog.
A "mashup" combines applications or data from different sources (sources not originally intended to be used together) into a single site or page. Content used in mashups are typically obtained from a third party source through a public interface API, web services, RSS or screen scraping. A mashup has 3 components:
a) A Web page that creates the mashup by aggregating data from multiple sources
b) Additional content provider
c) Client / Web Browser
Mashups are not compound pages obtained by simply embedding the content from another page nor is it equivalent to a portal. Mashups must obtain real-time data from 3rd party content providers on the fly.
In the context of BI, Enterprise Data Mashups can be a game-changer in the way information has been disseminated to business users. Traditionally, BI has been delivered by aggregating data into data warehouses and marts that are modeled in specific ways to aid end-user reporting and self-service capabilities. That is, traditional BI is one of tightly coupled information chain. Though this model has served the needs of first-generation BI, this model is not going to be sufficient for future needs. With ever increasing data volumes, real-time requirements imposed by Operational BI, increased sophistication for end-user analytics and the clamor for leveraging unstructured data, it is not going to be practically possible to physically aggregate ALL the enterprise data that is required for business decision making. The future of BI is going to be one of loosely coupled information integration.
Business users have (now and in the future) the necessity to combine data from multiple sources before taking a decision. Though ad-hoc query features provided by BI tools are considered a wonderful improvement to canned reporting, the users have been restricted to view data that is available in data warehouses & marts. Data Mashups provide a way to combine this formal data (present in data repositories) with data that is available outside of this domain (informal data). Informal data can be external data provided by RSS feeds or can be any type of unstructured data like documents and mails. The only thing of importance is the fact that the combination of both formal and informal data adds significant value to the business user. In this dimension, Mashups truly empowers the business user by providing "complete" end-user self-service capability.
Here is some friendly advice to BI practitioners to get started on Mashups:
1) Mashups are closely related to webservices and so it is important for BI practitioners to get a good handle on XML, webservices, SOAP, REST etc.
2) Programmableweb.com is an excellent website that showcases exciting mashups and also provides a catalogue of mashup API's
3) Mashup editors such as Microsoft Popfly, Yahoo!Pipes, Google Mashup editor etc. can help BI practitioners get started in creating their own mashups
4) Mashup servers like Presto (JackBe.com), WSO2 etc. once installed in your environment can help combine information from multiple sources.
5) Create prototypes of BI dashboard type mashups. Mashup that I created recently on Presto mashup server had the following pieces of information on a dashboard kind of web-page:
a) Data from a table in a DW exposed as a web-service
b) Source system information exposed as a web-service
c) Information from a RSS Feed
d) Used the Google Map Mashup API
That should give some idea of the possibilities and power of "Loosely Coupled Information Integration" through Mashups.
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 1:30 AM | Comments (0)
May 6, 2009
Ascent of BI - The Five Elements
My inspiration for this post is a book that I recently read titled "Five Equations that Changed the World" by Michael Guillen. This is a fascinating account of the history and people behind what are arguably the most important set of scientific equations that humankind has ever laid its eyes on. The top 5 equations according to the book are:
1) Issac Newton and the Universal Law of Gravity
2) Daniel Bernoulli and the Law of Hydrodynamic pressure
3) Michael Faraday and the Law of Electromagnetic Induction
4) Rudolf Clausius and the Second Law of Thermodynamics
5) Albert Einstein and the Theory of Special Relativity
These 5 fundamental equations have made possible several achievements like electricity, airplanes etc. and more significantly in understanding the nature of life and death.
The world of Business Intelligence has seen rapid strides in the last 10-15 years. In my view, the top 5 reasons for the "Ascent of BI" are:
1) Proliferation of powerful transaction systems (ERP, CRM, SCM etc.)
2) Internet Explosion that created the dissonance between availability & requirement of information and finally solved the problem too.
3) Globalization generated the need to have sophisticated analytical systems for businesses that span multiple geographies
4) Regulatory compliance requirements like SoX, Basel 2, GAAP etc.
5) New Business Models in industries (for example in Financial Services, Telecom etc.) that demands management by metrics
Over and above this, the BI product vendors have shown tremendous visionary zeal in coming up with vast range of BI platforms and tools across the entire BI landscape - be it data integration, databases and reporting tools, that has helped enterprises visualize the power of analytics.
Putting on my predictive hat let me list down the 5 things I think will take BI to the next orbit. They are:
1) Cloud Analytics (Analytics as a service)
2) Analytics that combine structured and unstructured data
3) Deeper Analytical Layer with Predictive capabilities and simulations
4) Real-time analytics (likes of Complex Event Processing (CEP), etc.)
5) Loosely coupled information integration (likes of data mashups etc.)
I will delve into each of these areas in my future posts. Please do share your thoughts. Thanks for reading.
Posted by Karthikeyan Sankaran at 8:45 AM | Comments (0)
April 18, 2009
So, What's Your Business Model?
In my last post, I expressed the view that the world of BI is expanding at a rapid pace. Substantiating this hypothesis is the fact that there are many types of BI solutions possible.
BI systems:
1) Can have many different end points (My view here)
2) Can have different industry flavors (My view here)
3) Can have business process implications (My view here)
4) Are spawning new types of databases and data models (My view here)
5) Are grappling with many unconquered territories (My view here)
The list goes on and on. Ultimately this results in a vexing problem for BI practitioners who's job is to advise organizations on the type of BI system to be built for them. BI consultants know that there is no cookie cutter approach to solving enterprise BI needs across companies but they are not very sure on where to start.
This post tries to address the issue of - "If there are so many different types of BI solutions possible, what is that first question to be clarified so that the BI practitioner is on the right track to provide the best fit BI solutions for that particular organization". In my humble opinion, the question should be, "So, What's your business model?" If this question conjures images of white boards filled with arcane mathematical formulas, that's probably because we don't completely understand what a business model is. Business Model is anything but arcane - it is just a story of how an enterprise works.
Business Model is a set of assumptions about how an organization will perform by creating value for all the players, on whom it depends, including its customers. Like all good stories, a business model relies on the basics of character, motivation and plot. For a business, the plot revolves around how will it make money. Characters are the different stakeholders (internal & external) in that business and each one of them acting in their own self-interest (motivation) makes the plot plausible and meaningful. All successful businesses have a twist in their plot (think of Google, eBay, Dell etc.) that has made them fantastically successful. You can read more about business models in the book by Joan Magretta titled "What Management Is".
From BI perspective, once the practitioner understands the business model of the company, many questions gets answered:
1) What drives the company's success and how BI can help?
2) Who are the stakeholders and what information are they looking for?
3) What needs to be optimized and how it can be done?
4) What is the architectural blueprint and how will it evolve?
5) How fast should information get delivered?
6) How much data needs to be collated and how far into the past should one go?
7) What are the regulatory requirements for the company?
And many more. In essence, clarity on one aspect of the problem (business model) will go a long way in selecting the right kind of BI solution for that particular organization.
So next time, you are on a BI consulting engagement, make this as your first question - "So, What's your business model?". Hopefully this would get you started in the right direction. Please do share your thoughts. Thanks for reading.
Posted by Karthikeyan Sankaran at 1:30 PM | Comments (2)
March 15, 2009
Hubble's Law and Business Intelligence
Edwin Hubble first proposed his now famous Hubble's law in 1929 and that is considered the first observational basis for the expanding space paradigm - in simple words, Hubble proved (using Doppler effects and Red shifts) that the universe is expanding at an accelerated pace. At a more earthy level, it is my contention that the universe of Business Intelligence is also expanding at a fairly rapid rate. If Hubble had been a BI practitioner, he would have probably explained the whole expansion quantitatively but I (and you as a reader) have to be content with some qualitative analysis.
Business Intelligence has come a long-way from the days when Howard Dresner defined it in 1989 as "a set of concepts and methodologies to improve decision making in business through use of facts and fact-based systems". You can read about the changing face of business intelligence at this link
Looking beyond definitions, practitioners would most certainly feel the expansion of BI boundaries in the last 8-10 years. What was initially a technology centric Data Warehousing domain has expanded to subsume areas like Performance Management, Business process analytics, Data and Text Mining, Packaged Industry-specific analytics to name a few.
With the advent of real-time operational BI, Information Services Bus architecture, Complex Event processing (CEP), Business Activity Monitoring (BAM) etc., I will not be surprised if the term Business Intelligence itself gets subsumed under a much more broader & comprehensive concept. With that, I rest my case regarding the relationship between Business Intelligence and Hubble's law.
Now here is the real point of this blog post. Though the boundaries of Business Intelligence are definitely expanding, there is still lot of play left well-within the boundaries. Organizations fundamentally require insights into their business processes so as to the optimize them. Enterprises embarking on BI initiatives should have one eye on the expanding boundaries but should be pragmatic enough to keep the other eye on what gets delivered at the end of the day. Incremental Business Intelligence is still the right step forward in the BI journey for many organizations but awareness of new developments would make them ready to take the leap when their business demands it.
Fundamentally, BI as we know it, encompasses a wide spectrum and is constantly growing. Organizations should be able identify the right kind of BI that is applicable to them given their current business context and be nimble enough to make changes to the contours of BI within their business as they move forward.
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 7:45 AM | Comments (0)
February 21, 2009
Industry Specific BI - What's the common denominator?
My previous post on business process fundamentals concluded with a friendly exhortation to BI practitioners inciting them to view their craft from the point of optimizing business process. So the next time you are involved in any BI endeavor, please ask this question to yourself and the people involved in the project - "So which business process is this BI project supposed to optimize, why and how?" I define 'Optimization' loosely as anything that leads to bottom-line or top-line benefits.
Business processes by its very definition belong to the industry domain. Companies have their own business processes - some of them are standard across firms in that particular domain and many of them are unique to specific companies. Efficiency of business processes is a source of competitive advantage and the fact that ERP vendors like SAP has special configurations for every industry illustrates this point. So by corollary, for BI to be effective in optimizing business processes, it has to be tied to specific industry needs creating what can be called as "Verticalized Business Intelligence". (V-BI in short)
At Hexaware's Business Intelligence & Analytics practice (the company and team that I belong to), we have taken the concept of V-BI pretty seriously and have built solutions aimed at industry verticals. You can view our vertical specific BI offerings at this link and we definitely welcome your comments on that.
Though Verticalized BI is a powerful idea, companies typically need an "analytics anchor point" to establish a BI infrastructure before embarking on their domain specific BI initiatives. The analytics anchor point, mentioned above, should have the following characteristics:
1) All organizations across domains should have the necessity to implement it
2) Business process associated with these analytics needs to be fairly standardized and should be handled by experts
3) Should involve some of the most critical stakeholders within the organization as the success of this first initiative will lay the foundation for future work.
Based on my experience in providing consulting services for organizations in laying down an Enterprise BI roadmap, I feel that "Financial Analytics" has all the right characteristics to become the analytics anchor point for companies. Financial Analytics, the common denominator, typically comprises of:
1) General Ledger Analysis - (also known as Financial Statements Analysis)
2) Profitability Analysis (Customer / Product Profitability etc.)
3) Budgeting, Planning & Forecasting
4) Monitoring & Controlling - The Dashboards & Scorecards
5) General Ledger Consolidation
The above mentioned areas are also classified as Enterprise Performance Management. The convergence of Performance Management and BI is another interesting topic (recent announcements of Microsoft have made this subject doubly interesting!) and I will write about it in my future posts.
In my humble opinion, the prescription for Enterprise BI is:
1) Select one or more areas of Financial Analytics (as mentioned above) as your first target for Enterprise BI.
2) During the process of completing step 1, establish the technology and process infrastructure for BI in the organization
3) Add your industry specific BI initiatives (Verticalized Business Intelligence) as you move up the curve
I, for one, truly believe in the power of Verticalized BI to develop solutions that provide the best fit between business and technology. That business and IT people can sit across the table and look at each other with mutual respect is another important non-trivial benefit.
Thanks for reading. Do you have any other analytics anchor points for organizations to jumpstart their BI initiatives? Please do share your thoughts.
Posted by Karthikeyan Sankaran at 11:45 PM | Comments (0)
January 26, 2009
Business Process for BI Practitioners - A Primer
Business Intelligence has a fairly wide scope but at the fundamental level it is all about "Business Processes". Let me explain a bit here.
BI, without the bells and whistles, is about understanding an organization's business model, its business processes and ultimately find the reason (analytics) and way to optimize the processes. The actions are carried out based on informed judgments (aided by BI), to make the organization better in whatever endeavor it has set itself to accomplish.
Assuming that BI practitioners are convinced that understanding business process is critical to their work, let me delve a bit into the basics of it.
1) What is a business process? (As a side note, one of the best explanation for business models is given by Joan Magretta in her book 'What Management Is')
Business processes are set of activities involved within or outside an organization that work together to produce a business outcome for a customer or to an organization. The fact is that for an organization to function, there are many outcomes that are required to happen on a daily basis.
2) What are BPM Tools?
Business Process Management (BPM) tools are used to create an application that is helpful in designing business process models, process flow models, data flow models, rules and also helpful in simulating, optimizing, monitoring and maintaining various processes that occur within an organization.
3) The Mechanics of Business Modeling
Business Process Modeling is the first step, followed by Process Flow Modeling and Data Flow diagrams. All these 3 diagrams and associated documentation will help in getting the complete picture of an organization's business processes. Brief explanation of these 3 types are given below:
a) In Business Process Modeling, an organization's functions are represented by using boxes and arrows. Boxes represent activities and arrows represent information associated with that activity. Input, Output, Control and Mechanism are the 4 types of arrows. A box and arrows combination that describes one activity is called a context diagram and obviously there would be many context diagrams to explain all the activities within the enterprise.
b) Process Flow Modeling is a model that is a collection of several activities of the business. IDEF3 is the process description capture method and this workflow model explains the activity dependencies, timing, branching and merging of process flows, choice, looping and parallelism in much greater detail.
c) Data Flow Diagrams (DFD) are used to capture the flow of data between various business processes. DFD's describe data sources, destinations, flows, data storage and transformations. DFDs contains five basic constructs namely: activities (processes), data flows, data stores, external references and physical resources.
Just like the data modeler goes thro' conceptual, logical and physical modeling steps, a business process modeler creates the Business Process Models, Process Flow Models and Data Flow Diagrams to get a feel for the business processes that take place within an enterprise.
Thoughts for BI Practitioners:
1. Consider viewing BI from the point of optimizing business processes
2. Might be worthwhile to learn about Business Process Modeling, Process Flow Modeling and Data Flow Diagrams
3. Understand the working of BPM tools and its usage in the enterprise BI landscape
4. Beware of the acronym BPM. BPM is Business Process Management but can also be peddled as Business Performance Management.
5. My view is that Performance Management is at a higher level, in the sense, that it is a collective (synergistic) view of the performance of individual business processes. A strong performance management framework can help you drill-down to specific business processes that can be optimized to increase performance.
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 7:15 AM | Comments (2)
January 3, 2009
What is "Safe to Bet On" in Business Intelligence?
While the phrase "Safe to Bet On" is an oxymoron of sorts, it is that time of the year where we first look at the past, derive some insights and look forward to what the future has in store for us. I have no doubts that 2009 will be doubly interesting for BI practitioners as compared to 2008. Having said that, I decided to do a bit of introspection to figure out what skills (can also be read as competencies) should I be looking at to stay relevant in the Business Intelligence world far into the future, say at 2020. Hopefully that resonates with some of you.
Let me first try and get down to defining the skills required for Business Intelligence and Analytics. The trick here is to stay "high-level" as any BI person will acknowledge the fact that one we get down to look at the trees (rather than the forest), the sheer number of skills required for enterprise level BI can get daunting.
Taking inspiration from the fact that any business can be condensed into 2 basic functions, viz. Making & Selling, I propose that there are 3 key skills that make for successful BI. They are:
Skill 1 - Business Process Understanding: If you are a core industry expert and can still talk about multi-dimensional expressions, that's great! But most BI practitioners have their formative years rooted on the technology side and have implemented solutions across industries. The ability to understand the value-chain of any industry, map out business processes, identify optimization areas, translating IT benefits to business benefits are the key sub-skills in this area.
Skill 2 - Architecting BI Solutions: This skill is all about answering the question of "What is the blue-print" for building the Business Intelligence Landscape in the organization. Traditionally, we have built data warehouses & data marts either top-down or bottom-up, integrated data from multiple sources into physical repositories, modeled them dimensionally, provided adhoc query capability and we are done! - NOT ANYMORE. With ever increasing data volumes, real-time requirements imposed by Operational BI, increased sophistication for end-user analytics, the clamor for leveraging unstructured data on one hand and the advent of On-Demand Analytics, Data Mashups, Data Warehouse appliances, etc., there is no single best way to build a BI infrastructure. So the answer to "What is the blueprint?" is "It depends". It depends on many factors (some of which are known today and many which aren't) and the person / organization who appreciates these factors and finds the best fit to a particular situation is bound to succeed.
Skill 3 - BI Tools Expertise: Once a blue-print is defined and optimization areas identified, we need the tools that can turn those ideas into reality. BI practitioners have many tools at their disposal straddling the entire spectrum with excel spreadsheets at one end to high-end data mining tools at the other extreme. If you bring in the ETL & data modeling tools, the number of industry-strength tools gets into the 50s and beyond. With convergence of web technologies, XML, etc. into mainstream BI, it probably makes sense to simplify and say "Anything you imagine can be done with appropriate BI tools". "Appropriate" is the key word here and it takes good amount of experience (and some luck) to get it right.
In essence, my prescription for BI practitioners to stay relevant in 2020 is to be aware of developments on these 3 major areas, develop specific techniques / sub-skills for each one of them and more importantly respect & collaborate with the BI practitioner in the next cubicle (which translates to anywhere across the globe in this flat world) for he/she would bring in complementary strengths.
Thanks for reading. Wish you all a very happy new year 2009!
Posted by Karthikeyan Sankaran at 11:30 AM | Comments (2)
December 21, 2008
The Esoteric World of Predictive Analytics
Let me start with the defintion of Predictive Analytics as used in literature - "The nontrivial extraction of implicit, previously unknown and potentially useful information from data". If that doesn't sound esoteric enough, you are probably more advanced than what this post gives you credit for!
For a BI practitioner, it is important to get an understanding of Predictive Analytics (also known as Data Mining) as this subject definitely deserves a place in the wide spectrum of Business Intelligence disciplines. BI at a broad level is about optimizing business through "Hindsight, Insight and Foresight". Predictive analytics adds the powerful "Foresight" part to business decision making.
Most BI practitioners tend to equate statistics with predictive analytics and this post explains why such a view is inaccurate. To understand this let's start at the very beginning (a la Alice in Wonderland). Broadly, this world is divided into 2 types of systems:
1) Physical Systems - Has causality and hence can be modeled mathematically with relative ease
2) Human Behavioral Systems - Lacks causality and can be modeled only with specialized techniques
Predictive analytics for business decision making is all about modeling human behavioral systems.
Why Traditional Statistics is insufficient?
Though the entry into predictive analytics requires that we understand the implications of traditional statistical analysis, statistics by itself is insufficient in the business context. Traditional statistical analysis allows us to understand the general group behavior and is primarily concerned with common behavior within the group - the central tendencies.
In business we generally develop models to anticipate human behavior of some type. Human behavior is inconsistent, lacks causality and distributions based on human behavior almost always violate the assumptions of traditional statistical analysis (like normal distribution of data, stability of mean and standard deviation etc). The strength of data mining comes from the ability of the associated techniques to deal with the tails of the distributions, rather than the central tendencies, and from the techniques' ability to deal with the realities of the data in a more precise manner.
In the realm of predictive analytics, we are concerned with modeling human behavior and hence are interested with the tail of our distribution - small percentage of the population that responds to a campaign, commits a fraud, leave our business or purchase the next service.
Though there are specialized techniques used for Predictive Analytics (viz. Non-linear statistics, Induction Algorithms, Cluster Analysis, Neural Networks to name a few), a BI practitioner is only expected to appreciate its usage in different business situations, prepare and model data as required by the tools and interpret the results correctly (a much less daunting task indeed!)
Typically the model development process involves the following steps - a) Define Project, b) Select Data, c) Prepare Data, d) Transform Variables, e) Process Model, f) Validate Model, g) Implement Model. I will explain these steps in more detail in subsequent posts.
Fundamentally, an end-to-end BI view requires the practitioner to learn the concepts around statistics and predictive analytical techniques as available in tools (like say SQL Server Analysis Services) in addition to their technology bag of tricks around data integration, data modeling and OLAP.
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 11:15 AM | Comments (1)
November 28, 2008
Zachman Framework for BI environments
As part of the corporate initiative (I work for a company called Hexaware Technologies based out of India), I co-author the blog called "A Practitioner's View" with a colleague of mine. The blog at beyeblogs titled "A Practitioner's Thoughts" is a re-iteration of my views expressed in the corporate blog with some minor changes.
The latest post on my corporate blog was on Zachman Framework and its adaptation for the BI environment. You can read about it at this link
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 7:15 AM | Comments (0)
November 11, 2008
Valuing your Business Intelligence System - Part 1
Sample these statements:
- Dow Jones Industrial Average jumped 200 points today, a 2% increase from the previous close
- The carbon footprint of an average individual in the world is about 4 tonnes per year which is a 3% increase over last year
- The number of unique URL's as on July 2008 in the World Wide web is 1 trillion. The previous landmark of 1 billion was reached in 2000
- One day 5% VaR (Value at Risk) for the portfolio is $1 Million as compared to the VaR of $1.3 Million a couple of weeks back
Most of us buy into the idea of having a single number that encapsulates complex phenomena. Though the details of the underlying processes are important, the single number (and the trend) does act like a bellwether of sorts helping us quickly get a feel of the current situation.
As a BI practitioner, I feel that it is about time that we formulated a way for valuing the BI infrastructure in organizations. Imagine a scenario where the Director of BI in company X can announce thus: "The value of the BI system in this organization has grown 15% over the past 1 year to touch $50 Million" (substitute your appropriate currencies here).
The core idea of this post is to find a way to "scientifically put a number to your data warehouse". Here are a few level setting points:
1. Valuation of BI systems is different from computing the Return on Investment (ROI) for BI initiatives. ROI calculations are typically done using Discounted Cash Flow techniques and are used in organizations to some extent
2. More than the absolute number, the trends are important which means that the BI system has to be valued using the same norms at different points in time. Scientific / Mathematical rigor helps in bringing the consistency aspect.
My perspective to valuation is based on the "Outside-in" logic where the fundamental premise is that the value of the BI infrastructure is completely determined by its consumption. Or in other words, if there are no consumers for your data warehouse, the value of such a system is zero.
One simple, yet powerful technique in the "Outside-in" category is RFM Analysis. RFM stands for Recency, Frequency and Monetary and is very popular in the direct marketing world. My 2-step hypothesis for BI system valuation using the RFM technique is:
Step 1: Value of BI system = Sum of the values of individual BI consumers
Step 2: Value of each individual consumer = Function (Recency, Frequency, Monetary parameters)
Qualitatively speaking, from the business user standpoint, one who has accessed information from the BI system more recently, has been using data more frequently and uses that information to make decisions that are critical to the organization will be given a higher value.
A calibration chart will provide the specific value associated with RFM parameters based on the categories within them. For example: For the Recency parameter, usage of information within the last 1 day can be fixed at 10 points while access 10 days back will fetch 1 point. I will explain my version of the calibration chart in detail in subsequent posts. (Please note that the conversion of points to dollar values is also an interesting, non-trivial exercise).
Thanks for reading. Please do share your thoughts.
Posted by Karthikeyan Sankaran at 10:15 PM | Comments (0)
