A small note on those 2 "Obvious" books that I read - The first one is a classic called "Obvious Adams: The Story of a Successful Business Man" written by Robert R. Updegraff in 1916 and the second being a more contemporary one by Eliyahu M. Goldratt titled "Isn't it Obvious". Though these books talk about very different business domains (Obvious Adams is on Advertising while Goldratt's book is on Theory of Constraints as applied to the Retail industry), the central theme of these books is the fact that decision makers tend to overlook the basic principles when confronted with problems.

In my humble opinion, a simple, implementable, commonsensical approach based on fundamental principles is the need of the hour in many areas and Business Intelligence is no exception. Based on my experience, given below are some of those basic principles on Business Intelligence that the practitioner would dismiss as being too "obvious" (but hey, isn't that the intent of this post!). Let's roll...

1) Business & Business Stakeholders are the key to successful BI
2) BI should help in making decisions that support the business goals
3) BI systems should provide Hindsight, Insight and Foresight to optimize the business process
4) Quality of BI output and hence the quality of decisioning is directly dependent of the quality of data. Remember "Garbage In, Garbage Out"
5) Ensure that the business units agree on what the business really is. Educate the business about the business, if required.
6) Without strategic focus and executive sponsorship, BI projects are set for failure
7) Organizations are dynamic entities and hence ensure that BI systems can adapt to change
8 ) Enable & Empower the BI users (both operational and strategic)
9) Market the value of DW / BI platforms to the user community
10) Don't boil the ocean - There is no perfect BI system. Try building a successful one instead
11) Always build the BI system with 'detail data'. Summaries & Aggregations can follow the detail
12) Pick the technology based on Business Fitment not on Tool sophistication
13) Prototype and Visualize the end-state before embarking on major BI initiatives
14) Have the right team to sustain and grow the BI infrastructure
15) Be aware of latest developments and trends in BI

And so, here is my quick checklist on the fundamental principles around thinking, building and using BI in organizations. Please do share your thoughts. Thanks for reading!

My biggest learning from consulting assignments during 2009 is the increasing emphasis on the artistic elements of Business Intelligence. The first question asked by senior executives across organizations was, "So, how will this finally look?" before any of the other elements in the BI landscape had taken shape.

"Form" had taken center-stage or at the very least is sharing the same elevated platform as "Content". For BI practitioners like me, who have been using their left brain more than its opposite number, working through architecture, data elements, models, ETL etc. it is time to take a step back and give the right brain its due. Visual Business Intelligence, which deals with data visualization, is rapidly gaining ground and the BI practitioner would do well to pay attention to it.

Stephen Few, one of the foremost experts in this area, writes this way in the wonderful website of his and I quote:

"We are overwhelmed by information, not because there is too much, but because we don't know how to tame it. Information lies stagnant in rapidly expanding pools as our ability to collect and warehouse it increases, but our ability to make sense of and communicate it remains inert, largely without notice.

Computers speed the process of information handling, but they don't tell us what the information means or how to communicate its meaning to decision makers. These skills are not intuitive; they rely largely on analysis and presentation skills that must be learned".

So true! Yet, I do feel that visualization is not carried out with the same rigor as done for other pieces of the BI landscape, in many organizations. For example, I have seen Data Modelers, ETL architects, Reporting specialists in BI teams but haven't come across a role of "Visualization Specialist" or "BI Usability Architect" in the BI space. And I think that day is not far off!

Given below are some resources that I found extremely interesting from the Visual BI standpoint (am sure that there are many more!)

1) Stephen Few's website
2) Infosthetics
3) Gapminder
4) Dashboard Spy
5) Amazing number of BI gadgets & widgets that can be found across the web (Easiest to try out are the Google gadgets ? Load sample data in Google spreadsheets, insert gadgets like motion charts and get going!)

Going back to the title of this blog post, who knows, dashboards developed by BI visualization experts might be auctioned at Sotheby's for multi-million dollars in the future!

Wish you all a very happy new year 2010.

Thanks for reading. Please do share your thoughts.

To illustrate the fact that the world of BI is moving & innovating at a rapid pace, let us play a small "Do you know this?" game in this blog. Given below are some of the new ideas / developments that I have evaluated in the BI space recently and quite a few of them can be viewed as game-changers in this domain.

As a BI practitioner, "Do You Know" that:

1) Large / Medium sized organizations can completely manage its Planning & Budgeting cycle for as low an investment as $25,000 - Check out Adaptive Planning which is a on-demand 'cloud' solution for Planning & Budgeting.

2) The ability to analyze multi-million rows of data in Excel on your laptop is not far-away - Look-out for PowerPivot 2010 (erstwhile codenamed Project 'Gemini') from Microsoft. Fundamentally, In-Memory Analytics, Column-based storage etc. are having a profound impact on large-volume data analysis.

3) Technology to integrate data sources virtually without the need to have hard-wired ETL is available - Enterprise Information Integration (EII) software like Composite and its variations like Cognos Virtual View Manager are taking its rightful place in the realm of information delivery.

4) Creation of awesome visualization like in Gapminder is possible without writing a single piece of code - Take a look at Google gadgets, Mashup API's at Programmableweb and this is sure to add a lot of power to your BI visualization capabilities.

5) Business Process Simulations are making its way into the BI landscape - Simulations based on System Dynamics and its manifestation in tools like Vensim, Powersim etc. are becoming end-points of the BI value chain.

6) Data in Petabyte range (>1000 terabytes) will become commonplace and will be handled effortlessly by BI tools - Massively Parallel Processing architectures (Greenplum, Project Madison from Microsoft to name a few), DW Appliances (Teradata, Netezza, Oracle Exadata, QimBase etc.) are gearing up to handle these challenges.

7) There are comprehensive BI platforms on the cloud - Products like LiveAccess from Birst can combine data uploaded to the cloud with data in on-demand applications like SalesForce.com and also get into on-premise data to provide comprehensive analytical capabilities, in a matter of minutes.

8) Large Data Warehouses can have their development, test, quality and production environment in ONE SINGLE BOX - Virtualization (both Hardware Partitioning & Hypervisor technology) platforms like Microsoft Hyper V, VMware, Xen, Virtuozzo can enable such platform consolidation in the near future

9) Technology exist to analyze all types of logs (call logs, IVR logs, web logs etc.) to provide profound insights - Products like ClickFox operate in the area of Customer Experience Analytics and provides extensive capabilities to model & analyze semi-structured information as present in log files.

10) Along with providing business foresight thro' Predictive Analytics, BI is also helping organizations analyze data streams in real 'real-time' - Complex Event Processing (CEP) and its manifestation in tools like Streambase, Oracle CEP etc. are making its way into mainstream BI.

So, how much did you score? Am sure that you also would have come across some interesting developments in BI recently. Please do share your thoughts. Thanks for reading.

Let me sign-off with a quote which I think is apt for this post:
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man" - George Bernard Shaw, Man and Superman (1903).

The act of creating consolidated physical repositories is a cumbersome task, especially in large enterprises, that such investments need solid financial models, elaborate planning and even then there is no guarantee of success. For many years, the only viable option for enterprises is to perform "physical consolidation", i.e. create the consolidated logical and physical data models, create hard-wired ETL programs to load data into the consolidated database and use this infrastructure for reporting and analytics. Not anymore! - It is my view that Enterprise Information Integration (EII) provides a viable option for enterprise wide BI consolidation and this post it to initiate practitioners to look at EII as a business solution for problems in the BI domain.

EII falls in the realm of data virtualization and refers to technology behind real-time aggregation of corporate data across multiple, widely disparate data sources. EII delivers comprehensive, reusable "views" that is exposed via SQL and/or web services to whole lot of consuming applications (Reports, Dashboards, etc.). Though the concept of data virtualization is very apt for BI consolidation, there are many other class of problems in which EII technology can play a major role.

At a high level, EII tools work in the following way:

1) The designer uses the data connectivity feature of the tool and can potentially connect to a whole lot of data repositories, viz. databases, XML, excel sheets, etc.
2) Data Modelers then can combine the required data sets and model them in the EII software.
3) EII software then creates a virtual data layer and exposes the metadata in the form of views.
4) The views can be optimized by using the features available within the software.
5) Views are exposed to consuming applications and at run-time, data is fetched from the underlying data repositories

A Simple yet Powerful Value Proposition!

It is important to understand that EII does not replace EAI or ETL. All 3 technologies can co-exist and each on its own provides solutions for different kinds of problems.

EAI stands for Enterprise Application Integration and is used in cases where different applications (Sales & Inventory application for example) want to talk to each other. Target for EAI is an application.

ETL stands for Extract, Transform and Load and is used in cases where data aggregation is required to provide an integrated, consistent definition of data for decision support process. Target for ETL is a database.

EII stands for Enterprise Information Integration and is used as a framework for real-time integration of data from multiple sources both inside & outside an enterprise, empowering business users to "pull" any kind of data from anywhere in the enterprise. Target for EII is the business end user.

A classic reference architecture in which all 3 tools can play a part is when, transactional applications are integrated thro' EAI, data from these applications flow into an Enterprise Data Warehouse (EDW) by leveraging ETL capability and then EII tools help to combine data from OLTP applications, EDW, external data repositories and local excel sheets for business decision making.

In terms of technology, there are specialized EII tools like Composite Software and also many of the standard reporting tools have an EII component embedded in them (Data Federator in Business Objects XI, Virtual View Manager in Cognos 8.x etc.). In my view, EII does have a strong business case and BI practitioners would do well to evaluate this option when faced with data consolidation challenges.

Thanks for reading. Please do share your thoughts.

1) Custom-built BI applications from scratch
2) Implement & Customize Packaged BI Apps
3) Implement "Analytical Packs" - Build connectors and Improve on business functionality based on specific needs.

Analytical Packs are developed for a specific functional purpose. This purpose can be completely domain focused (Insurance Analytics) or can be applicable across multiple industries (Human Resource, CRM analytics etc). The Analytical Packs provide the flexibility of a custom built solution and also the benefit of faster turn-around time as provided by packaged BI apps. Also, the analytical packs can be used by organizations to understand their analytical needs better before embarking on bigger BI initiatives.

The following are the pre-built components of an Analytical pack:

1. List of Subject Areas that make up a functional domain
(Example - HR Analytics will cover the subject areas of Staffing, Retention, Workforce,Organization Effectiveness, Compensation & Benefits, Environment etc.)
2. Set of Business Questions for each subject area
3. Data Model for the functional domain / specific subject areas
4. Semantic Layer for ad-hoc analysis
5. Canned Reports
6. Pre-defined Metrics / KPI's
7. Executive Level Dashboards (based on roles)
8. Predictive Analytics Scenarios and Mining models
9. Connectors to source systems (if feasible)

At Hexaware (the company I work for), we have created close to around 25 analytical packs for multiple industry domains and these are constantly being improved upon. Many more packs around Leasing, Credit-risk, Collections, Accounts Receivables etc. are in progress.

How does an Analytical Pack get built? The steps at a high level are given below:

1) Identify the business process associated with the functional domain
2) Identify the data elements / entities generated by the business processes
3) Identify the analytical scope by listing out the scope of analysis / KPI's etc
4) Organize the data generated by business process in a meaningful way (cut out the operational noise and focus on analysis) - Create the data model
5) Identify meaningful ways to analyze the data - Reports, Graphs, Dashboards etc.
6) Identify scenarios for predictive analytics and build mining models

I will explain specific analytical packs in more detail in subsequent blogs. Thanks for reading. Please do share your thoughts.

10 Point Framework looks at the BI technology infrastructure from ten different dimensions and provides a set of recommendations for each of the areas. The coverage is as tabulated below:

1. Reporting and Delivery
a) Determine user groups and reporting requirements
b) Study the challenges involved in delivery mechanism
c) Identify the security requirements for reports
d) Identify the performance and frequency requirements

2. Data Integration
a) Study the current data integration process
b) Analyze the existing tools used for data extraction
c) Understand the Error handling and auditing process

3. Data Sources
a) Understand various sources of data
b) Analyze the source system data
c) Understand source system dependency, feed layout, frequency and mechanism of data transfer
d) Understand the change data capture process

4. Data Quality
a) Audit current databases
b) Study the accuracy of data

5. Enteprise Data Model
a) Study the existing conceptual, logical and physical data models
b) Study the entity relationships

6. Reference Data Strucuture
a) Understand existing system process and data flow
b) Study the existing data architecture

7. Security Framework
a) Study the existing data security

8. Metadata Availability
a) Study the existing metadata
b) Study the metadata capture process

9. Standards & Process
a) Study existing business process and problems
b) Study the current information flow
c) Analyze existing business requirements

10. Infrastructure
a) Study the current IT Infrastructure
b) Study the existing applications
c) Review the existing ETL tools
d) Review current BI Architecture

The following are key highlights of Ten Point Framework:
1) Covers all aspects of application process, technology, requirements and infrastructure
2) Helps in prioritizing and bucketing the requirements
3) Identifies critical positives and negatives
4) Evaluates against industry best practices
5) Provides a roadmap and recommendation

Hexaware delivers the Ten Point Framework as a consulting engagement spanning 6-8 weeks with distinct Understand, Analyze and Recommend phases

Once the Business Profiling and 10 Point assessment is completed, the organization has a pretty good handle on their current BI environment which is the starting point for newer initiatives.

Thanks for reading. Please do share your thoughts

The first step to reinventing your BI program is to calibrate the current state of the BI landscape. Though many organizations do this in many different ways, I use 2 major tools, perfected over a period of time. They are:

1) BI Profiling Questionnaire - This measures the customer propensity and necessity to reinvent.
2) 10 Point Framework - This is a detailed framework that looks at the current landscape from 10 different dimensions (viz. Sources, ETL, Reporting, Security, Metadata etc.) and provides a health sheet on the current state.

These 2 tools put together constitute the Business Intelligence X-Ray for any organization. With this, the organization has a good feel for where they currently are (Point A). Point B, where you want to go, is basically a menu of items straddling the entire spectrum of possibilities (viz. Platform migration, Consolidation, EII, Data mining, Simulations, Complex Event Processing, Customer Experience Analytics, Cloud Analytics, DW Appliances etc.)

Part 2 of this blog topic will cover the 10 Point Framework and Part 3 will categorize the menu of BI re-invention possibilities for organizations.

The focus of this post is on the first part of the Business Intelligence X-Ray which is the Profiling Questionnaire. This questionnaire at this stage of evolution has a set of 30 questions and covers the following areas:

a) Business Process
b) BI Architecture
c) Data
d) Analytics or Knowledge
e) Decisioning

This questionnaire is answered through a focused workshop which involves all the key BI stakeholders and provides the first view of current state BI landscape. This workshop is followed by an education session on those new areas within BI that are relevant for the organization. The other outcome of the workshop is a consulting engagement using the 10 Point Framework that analyzes the technology landscape in lot more detail.

I will discuss the other parts of the Business Intelligence X-Ray in my subsequent posts. Thanks for reading. Please do send in your thoughts.

A "mashup" combines applications or data from different sources (sources not originally intended to be used together) into a single site or page. Content used in mashups are typically obtained from a third party source through a public interface API, web services, RSS or screen scraping. A mashup has 3 components:
a) A Web page that creates the mashup by aggregating data from multiple sources
b) Additional content provider
c) Client / Web Browser

Mashups are not compound pages obtained by simply embedding the content from another page nor is it equivalent to a portal. Mashups must obtain real-time data from 3rd party content providers on the fly.

In the context of BI, Enterprise Data Mashups can be a game-changer in the way information has been disseminated to business users. Traditionally, BI has been delivered by aggregating data into data warehouses and marts that are modeled in specific ways to aid end-user reporting and self-service capabilities. That is, traditional BI is one of tightly coupled information chain. Though this model has served the needs of first-generation BI, this model is not going to be sufficient for future needs. With ever increasing data volumes, real-time requirements imposed by Operational BI, increased sophistication for end-user analytics and the clamor for leveraging unstructured data, it is not going to be practically possible to physically aggregate ALL the enterprise data that is required for business decision making. The future of BI is going to be one of loosely coupled information integration.

Business users have (now and in the future) the necessity to combine data from multiple sources before taking a decision. Though ad-hoc query features provided by BI tools are considered a wonderful improvement to canned reporting, the users have been restricted to view data that is available in data warehouses & marts. Data Mashups provide a way to combine this formal data (present in data repositories) with data that is available outside of this domain (informal data). Informal data can be external data provided by RSS feeds or can be any type of unstructured data like documents and mails. The only thing of importance is the fact that the combination of both formal and informal data adds significant value to the business user. In this dimension, Mashups truly empowers the business user by providing "complete" end-user self-service capability.

Here is some friendly advice to BI practitioners to get started on Mashups:
1) Mashups are closely related to webservices and so it is important for BI practitioners to get a good handle on XML, webservices, SOAP, REST etc.
2) Programmableweb.com is an excellent website that showcases exciting mashups and also provides a catalogue of mashup API's
3) Mashup editors such as Microsoft Popfly, Yahoo!Pipes, Google Mashup editor etc. can help BI practitioners get started in creating their own mashups
4) Mashup servers like Presto (JackBe.com), WSO2 etc. once installed in your environment can help combine information from multiple sources.
5) Create prototypes of BI dashboard type mashups. Mashup that I created recently on Presto mashup server had the following pieces of information on a dashboard kind of web-page:
a) Data from a table in a DW exposed as a web-service
b) Source system information exposed as a web-service
c) Information from a RSS Feed
d) Used the Google Map Mashup API

That should give some idea of the possibilities and power of "Loosely Coupled Information Integration" through Mashups.

Thanks for reading. Please do share your thoughts.

1) Issac Newton and the Universal Law of Gravity
2) Daniel Bernoulli and the Law of Hydrodynamic pressure
3) Michael Faraday and the Law of Electromagnetic Induction
4) Rudolf Clausius and the Second Law of Thermodynamics
5) Albert Einstein and the Theory of Special Relativity

These 5 fundamental equations have made possible several achievements like electricity, airplanes etc. and more significantly in understanding the nature of life and death.

The world of Business Intelligence has seen rapid strides in the last 10-15 years. In my view, the top 5 reasons for the "Ascent of BI" are:

1) Proliferation of powerful transaction systems (ERP, CRM, SCM etc.)
2) Internet Explosion that created the dissonance between availability & requirement of information and finally solved the problem too.
3) Globalization generated the need to have sophisticated analytical systems for businesses that span multiple geographies
4) Regulatory compliance requirements like SoX, Basel 2, GAAP etc.
5) New Business Models in industries (for example in Financial Services, Telecom etc.) that demands management by metrics

Over and above this, the BI product vendors have shown tremendous visionary zeal in coming up with vast range of BI platforms and tools across the entire BI landscape - be it data integration, databases and reporting tools, that has helped enterprises visualize the power of analytics.

Putting on my predictive hat let me list down the 5 things I think will take BI to the next orbit. They are:

1) Cloud Analytics (Analytics as a service)
2) Analytics that combine structured and unstructured data
3) Deeper Analytical Layer with Predictive capabilities and simulations
4) Real-time analytics (likes of Complex Event Processing (CEP), etc.)
5) Loosely coupled information integration (likes of data mashups etc.)

I will delve into each of these areas in my future posts. Please do share your thoughts. Thanks for reading.

BI systems:
1) Can have many different end points (My view here)
2) Can have different industry flavors (My view here)
3) Can have business process implications (My view here)
4) Are spawning new types of databases and data models (My view here)
5) Are grappling with many unconquered territories (My view here)

The list goes on and on. Ultimately this results in a vexing problem for BI practitioners who's job is to advise organizations on the type of BI system to be built for them. BI consultants know that there is no cookie cutter approach to solving enterprise BI needs across companies but they are not very sure on where to start.

This post tries to address the issue of - "If there are so many different types of BI solutions possible, what is that first question to be clarified so that the BI practitioner is on the right track to provide the best fit BI solutions for that particular organization". In my humble opinion, the question should be, "So, What's your business model?" If this question conjures images of white boards filled with arcane mathematical formulas, that's probably because we don't completely understand what a business model is. Business Model is anything but arcane - it is just a story of how an enterprise works.

Business Model is a set of assumptions about how an organization will perform by creating value for all the players, on whom it depends, including its customers. Like all good stories, a business model relies on the basics of character, motivation and plot. For a business, the plot revolves around how will it make money. Characters are the different stakeholders (internal & external) in that business and each one of them acting in their own self-interest (motivation) makes the plot plausible and meaningful. All successful businesses have a twist in their plot (think of Google, eBay, Dell etc.) that has made them fantastically successful. You can read more about business models in the book by Joan Magretta titled "What Management Is".

From BI perspective, once the practitioner understands the business model of the company, many questions gets answered:

1) What drives the company's success and how BI can help?
2) Who are the stakeholders and what information are they looking for?
3) What needs to be optimized and how it can be done?
4) What is the architectural blueprint and how will it evolve?
5) How fast should information get delivered?
6) How much data needs to be collated and how far into the past should one go?
7) What are the regulatory requirements for the company?

And many more. In essence, clarity on one aspect of the problem (business model) will go a long way in selecting the right kind of BI solution for that particular organization.

So next time, you are on a BI consulting engagement, make this as your first question - "So, What's your business model?". Hopefully this would get you started in the right direction. Please do share your thoughts. Thanks for reading.

Business Intelligence has come a long-way from the days when Howard Dresner defined it in 1989 as "a set of concepts and methodologies to improve decision making in business through use of facts and fact-based systems". You can read about the changing face of business intelligence at this link

Looking beyond definitions, practitioners would most certainly feel the expansion of BI boundaries in the last 8-10 years. What was initially a technology centric Data Warehousing domain has expanded to subsume areas like Performance Management, Business process analytics, Data and Text Mining, Packaged Industry-specific analytics to name a few.

With the advent of real-time operational BI, Information Services Bus architecture, Complex Event processing (CEP), Business Activity Monitoring (BAM) etc., I will not be surprised if the term Business Intelligence itself gets subsumed under a much more broader & comprehensive concept. With that, I rest my case regarding the relationship between Business Intelligence and Hubble's law.

Now here is the real point of this blog post. Though the boundaries of Business Intelligence are definitely expanding, there is still lot of play left well-within the boundaries. Organizations fundamentally require insights into their business processes so as to the optimize them. Enterprises embarking on BI initiatives should have one eye on the expanding boundaries but should be pragmatic enough to keep the other eye on what gets delivered at the end of the day. Incremental Business Intelligence is still the right step forward in the BI journey for many organizations but awareness of new developments would make them ready to take the leap when their business demands it.

Fundamentally, BI as we know it, encompasses a wide spectrum and is constantly growing. Organizations should be able identify the right kind of BI that is applicable to them given their current business context and be nimble enough to make changes to the contours of BI within their business as they move forward.

Thanks for reading. Please do share your thoughts.

Business processes by its very definition belong to the industry domain. Companies have their own business processes - some of them are standard across firms in that particular domain and many of them are unique to specific companies. Efficiency of business processes is a source of competitive advantage and the fact that ERP vendors like SAP has special configurations for every industry illustrates this point. So by corollary, for BI to be effective in optimizing business processes, it has to be tied to specific industry needs creating what can be called as "Verticalized Business Intelligence". (V-BI in short)

At Hexaware's Business Intelligence & Analytics practice (the company and team that I belong to), we have taken the concept of V-BI pretty seriously and have built solutions aimed at industry verticals. You can view our vertical specific BI offerings at this link and we definitely welcome your comments on that.

Though Verticalized BI is a powerful idea, companies typically need an "analytics anchor point" to establish a BI infrastructure before embarking on their domain specific BI initiatives. The analytics anchor point, mentioned above, should have the following characteristics:
1) All organizations across domains should have the necessity to implement it
2) Business process associated with these analytics needs to be fairly standardized and should be handled by experts
3) Should involve some of the most critical stakeholders within the organization as the success of this first initiative will lay the foundation for future work.

Based on my experience in providing consulting services for organizations in laying down an Enterprise BI roadmap, I feel that "Financial Analytics" has all the right characteristics to become the analytics anchor point for companies. Financial Analytics, the common denominator, typically comprises of:
1) General Ledger Analysis - (also known as Financial Statements Analysis)
2) Profitability Analysis (Customer / Product Profitability etc.)
3) Budgeting, Planning & Forecasting
4) Monitoring & Controlling - The Dashboards & Scorecards
5) General Ledger Consolidation

The above mentioned areas are also classified as Enterprise Performance Management. The convergence of Performance Management and BI is another interesting topic (recent announcements of Microsoft have made this subject doubly interesting!) and I will write about it in my future posts.

In my humble opinion, the prescription for Enterprise BI is:
1) Select one or more areas of Financial Analytics (as mentioned above) as your first target for Enterprise BI.
2) During the process of completing step 1, establish the technology and process infrastructure for BI in the organization
3) Add your industry specific BI initiatives (Verticalized Business Intelligence) as you move up the curve

I, for one, truly believe in the power of Verticalized BI to develop solutions that provide the best fit between business and technology. That business and IT people can sit across the table and look at each other with mutual respect is another important non-trivial benefit.

Thanks for reading. Do you have any other analytics anchor points for organizations to jumpstart their BI initiatives? Please do share your thoughts.

BI, without the bells and whistles, is about understanding an organization's business model, its business processes and ultimately find the reason (analytics) and way to optimize the processes. The actions are carried out based on informed judgments (aided by BI), to make the organization better in whatever endeavor it has set itself to accomplish.

Assuming that BI practitioners are convinced that understanding business process is critical to their work, let me delve a bit into the basics of it.

1) What is a business process? (As a side note, one of the best explanation for business models is given by Joan Magretta in her book 'What Management Is')

Business processes are set of activities involved within or outside an organization that work together to produce a business outcome for a customer or to an organization. The fact is that for an organization to function, there are many outcomes that are required to happen on a daily basis.

2) What are BPM Tools?
Business Process Management (BPM) tools are used to create an application that is helpful in designing business process models, process flow models, data flow models, rules and also helpful in simulating, optimizing, monitoring and maintaining various processes that occur within an organization.

3) The Mechanics of Business Modeling
Business Process Modeling is the first step, followed by Process Flow Modeling and Data Flow diagrams. All these 3 diagrams and associated documentation will help in getting the complete picture of an organization's business processes. Brief explanation of these 3 types are given below:

a) In Business Process Modeling, an organization's functions are represented by using boxes and arrows. Boxes represent activities and arrows represent information associated with that activity. Input, Output, Control and Mechanism are the 4 types of arrows. A box and arrows combination that describes one activity is called a context diagram and obviously there would be many context diagrams to explain all the activities within the enterprise.

b) Process Flow Modeling is a model that is a collection of several activities of the business. IDEF3 is the process description capture method and this workflow model explains the activity dependencies, timing, branching and merging of process flows, choice, looping and parallelism in much greater detail.

c) Data Flow Diagrams (DFD) are used to capture the flow of data between various business processes. DFD's describe data sources, destinations, flows, data storage and transformations. DFDs contains five basic constructs namely: activities (processes), data flows, data stores, external references and physical resources.

Just like the data modeler goes thro' conceptual, logical and physical modeling steps, a business process modeler creates the Business Process Models, Process Flow Models and Data Flow Diagrams to get a feel for the business processes that take place within an enterprise.

Thoughts for BI Practitioners:
1. Consider viewing BI from the point of optimizing business processes
2. Might be worthwhile to learn about Business Process Modeling, Process Flow Modeling and Data Flow Diagrams
3. Understand the working of BPM tools and its usage in the enterprise BI landscape
4. Beware of the acronym BPM. BPM is Business Process Management but can also be peddled as Business Performance Management.
5. My view is that Performance Management is at a higher level, in the sense, that it is a collective (synergistic) view of the performance of individual business processes. A strong performance management framework can help you drill-down to specific business processes that can be optimized to increase performance.

Thanks for reading. Please do share your thoughts.

Let me first try and get down to defining the skills required for Business Intelligence and Analytics. The trick here is to stay "high-level" as any BI person will acknowledge the fact that one we get down to look at the trees (rather than the forest), the sheer number of skills required for enterprise level BI can get daunting.

Taking inspiration from the fact that any business can be condensed into 2 basic functions, viz. Making & Selling, I propose that there are 3 key skills that make for successful BI. They are:

Skill 1 - Business Process Understanding: If you are a core industry expert and can still talk about multi-dimensional expressions, that's great! But most BI practitioners have their formative years rooted on the technology side and have implemented solutions across industries. The ability to understand the value-chain of any industry, map out business processes, identify optimization areas, translating IT benefits to business benefits are the key sub-skills in this area.

Skill 2 - Architecting BI Solutions: This skill is all about answering the question of "What is the blue-print" for building the Business Intelligence Landscape in the organization. Traditionally, we have built data warehouses & data marts either top-down or bottom-up, integrated data from multiple sources into physical repositories, modeled them dimensionally, provided adhoc query capability and we are done! - NOT ANYMORE. With ever increasing data volumes, real-time requirements imposed by Operational BI, increased sophistication for end-user analytics, the clamor for leveraging unstructured data on one hand and the advent of On-Demand Analytics, Data Mashups, Data Warehouse appliances, etc., there is no single best way to build a BI infrastructure. So the answer to "What is the blueprint?" is "It depends". It depends on many factors (some of which are known today and many which aren't) and the person / organization who appreciates these factors and finds the best fit to a particular situation is bound to succeed.

Skill 3 - BI Tools Expertise: Once a blue-print is defined and optimization areas identified, we need the tools that can turn those ideas into reality. BI practitioners have many tools at their disposal straddling the entire spectrum with excel spreadsheets at one end to high-end data mining tools at the other extreme. If you bring in the ETL & data modeling tools, the number of industry-strength tools gets into the 50s and beyond. With convergence of web technologies, XML, etc. into mainstream BI, it probably makes sense to simplify and say "Anything you imagine can be done with appropriate BI tools". "Appropriate" is the key word here and it takes good amount of experience (and some luck) to get it right.

In essence, my prescription for BI practitioners to stay relevant in 2020 is to be aware of developments on these 3 major areas, develop specific techniques / sub-skills for each one of them and more importantly respect & collaborate with the BI practitioner in the next cubicle (which translates to anywhere across the globe in this flat world) for he/she would bring in complementary strengths.

Thanks for reading. Wish you all a very happy new year 2009!

For a BI practitioner, it is important to get an understanding of Predictive Analytics (also known as Data Mining) as this subject definitely deserves a place in the wide spectrum of Business Intelligence disciplines. BI at a broad level is about optimizing business through "Hindsight, Insight and Foresight". Predictive analytics adds the powerful "Foresight" part to business decision making.

Most BI practitioners tend to equate statistics with predictive analytics and this post explains why such a view is inaccurate. To understand this let's start at the very beginning (a la Alice in Wonderland). Broadly, this world is divided into 2 types of systems:
1) Physical Systems - Has causality and hence can be modeled mathematically with relative ease
2) Human Behavioral Systems - Lacks causality and can be modeled only with specialized techniques

Predictive analytics for business decision making is all about modeling human behavioral systems.

Why Traditional Statistics is insufficient?
Though the entry into predictive analytics requires that we understand the implications of traditional statistical analysis, statistics by itself is insufficient in the business context. Traditional statistical analysis allows us to understand the general group behavior and is primarily concerned with common behavior within the group - the central tendencies.

In business we generally develop models to anticipate human behavior of some type. Human behavior is inconsistent, lacks causality and distributions based on human behavior almost always violate the assumptions of traditional statistical analysis (like normal distribution of data, stability of mean and standard deviation etc). The strength of data mining comes from the ability of the associated techniques to deal with the tails of the distributions, rather than the central tendencies, and from the techniques' ability to deal with the realities of the data in a more precise manner.

In the realm of predictive analytics, we are concerned with modeling human behavior and hence are interested with the tail of our distribution - small percentage of the population that responds to a campaign, commits a fraud, leave our business or purchase the next service.

Though there are specialized techniques used for Predictive Analytics (viz. Non-linear statistics, Induction Algorithms, Cluster Analysis, Neural Networks to name a few), a BI practitioner is only expected to appreciate its usage in different business situations, prepare and model data as required by the tools and interpret the results correctly (a much less daunting task indeed!)

Typically the model development process involves the following steps - a) Define Project, b) Select Data, c) Prepare Data, d) Transform Variables, e) Process Model, f) Validate Model, g) Implement Model. I will explain these steps in more detail in subsequent posts.

Fundamentally, an end-to-end BI view requires the practitioner to learn the concepts around statistics and predictive analytical techniques as available in tools (like say SQL Server Analysis Services) in addition to their technology bag of tricks around data integration, data modeling and OLAP.

Thanks for reading. Please do share your thoughts.