BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

« March 2010 | Main | May 2010 »

April 27, 2010

Answers to a Reporter's Questions About Pervasive BI

After publishing the recent post about Pervasive BI, I have done several interviews covering the topic in more depth. Here is the dialogue I had with Canadian reporter David Hamilton of Web Host Industry Review (WHIR) who asked some interesting questions.

WHIR: Some business people I have spoken to have said that what they’ve seen over the past 18 months or so has had them question a lot of their conventional wisdom on how they operate their organizations. Do you see the rise in business intelligence use corresponding to the declining economic conditions during the financial crisis?

VF: With the recession have come emerging regulatory and transparency requirements, the need to better manage risk, control fraud, and not just increase the ability to cross-sell and up-sell, but to improve the whole customer experience and provide better service as a means to retaining customers and increasing their long-term value.

Prior to the financial crisis, most companies’ BI  focus was on generating a consistent set of accurate financial reports to help them understand and manage the business. The goal was to achieve a single version of the truth of (historical) data in the data warehouse for analysis. But analysis based on history means you’re using yesterday’s results to determine tomorrow’s performance. And many companies were beginning to make their data warehouses and BI systems more operational and predictive.

Some companies kept data mart proliferation in check, but many did not. Where there was data mart consolidation, the approach was often to move the existing data structure (problems and all) to a new platform. In most cases, data quality, data governance, and MDM were left to be dealt with later.

With the onset of the financial crisis and economic uncertainty, priority shifted. IT went into cost-cutting mode, becoming more scrutinizing about acquisition costs and payback periods, and replacing traditional data marts and warehouses with new low-cost appliances. Dollars per terabyte moved to the forefront in evaluating data warehouse platforms. Low-cost and open source BI tools took hold in the market. The need to improve staff productivity put emphasis on increasing the efficiency of costly data integration processes. We have seen increased investments in the past couple of years in data quality, data governance and MDM products and services.

As we begin to see a leveling off of the economic decline, we’re seeing another shift. The need to cut and control costs continues. There is a sentiment that this is not a temporary recession, but a new economic reality. At the same time, competition is keener, driving the need for better customer management. There is a recognition that if you can’t respond to conditions in the moment, you can’t influence their outcome. It is this dilemma of extended cost control while having to increase customer service that is driving a change in companies’ use of business intelligence. There is a strong belief that BI can be tapped to help control business and IT costs, but at the same time, it is imperative that they do a much better job of exploiting data as a valuable asset, to do predictive analysis, make use of data generated by social media and other unstructured content, optimize operational decisions, etc.

In the “Top 10 Trends in Business Intelligence for 2010” whitepaper from HP, we note the likelihood that companies and industries will come out of the recession with a transformation agenda. They will look to IT, and specifically BI, as an important lever for transformation initiatives.

This will call for a different BI infrastructure than what companies have been using, which explains why, in a recent survey by The Data Warehousing Institute (TDWI), 46% of respondents indicated that they anticipate replacing their current primary data warehouse platform by 2012. Here are just a few examples:

In summary, we see both a rise in use of BI as well as a different use, driven by not just the recession itself, but the widespread expectation of a new economic reality as we move forward that will require a fundamental change in the way we make use of business data.

WHIR: Unlike some other business services, BI is not one-size-fits-all when it comes to implementation. When a business comes to HP asking for BI, how involved do businesses have to be in order to ensure very pervasive BI?

VF: The goal may not necessarily be for BI to be pervasive, although wise organizations would like to leverage and maximize their investment in DW/BI. The initial goal is to address a business problem or opportunity. The more business people can be involved in the design, development and implementation of the BI system, the more likely it is that it will meet their needs. In addition to defining requirements, they can act as advocates to the rest of the organization in validating and encouraging use of the DW/BI system and communicating its success.

One of the things that HP helps clients with is overcoming the cultural and political issues in sharing and integrating data. HP guides clients in the establishment of not just a data warehouse, but a trusted data environment that will allow business users to standardize data usage across business teams and develop a thriving data user community.

The use of BI has not become pervasive in organizations. BI usage is only about 24%. In my post about pervasive BI, I discussed many of the obstacles that must be overcome before it can be more pervasive.

In the next post, I will answer more of David Hamilton’s questions, including this very interesting one:

WHIR: What are some of the unforeseen or unexpected benefits of BI for organizations?

Posted by HP Business Intelligence Solutions at 8:31 AM

April 21, 2010

Live BI, CeNSE and BRAIN are HP Lab’s newest projects for BI

A couple of my recent posts have talked about innovation and modernist thinking for business intelligence and data warehousing.  I thought it would be fun for you to hear more about what's going on in HP Lab's. This stuff is true BI innovation.

While HP Labs has a number of innovative projects involving information management, the CeNSE project is especially remarkable. I mentioned it in my previous post on Information in Motion. HP's Central Nervous System for the Earth (CeNSE) is designed to radically change the way we gather, communicate and analyze information.  CeNSE combines nanotechnology, networking and business analytics breakthroughs from the Labs to make the world a better, safer, more efficient place for all us. 

With CeNSE, billions of nano-scale sensors will be placed around the globe and will be used to feel, taste, smell, see and hear the world around us.  Then, using powerful computing networks, the information can be analyzed and actions taken in real-time to benefit society.  For example, with CeNSE, people will be able to prepare for an earthquake before it hits, or anticipate that a building is due to collapse in time to evacuate all tenants. Think of numerous applications where we can monitor what is happening around the world, respond with intelligence, and make the world a safer and more sustainable place.

BRAIN (Behaviorally Robust Aggregation of Information) is a method of more accurately predicting behavior and improving forecasts by taking the bias, manipulation and hierarchy out of prediction.   My good friend, Colin White would call this Decision Science.

BRAIN uses powerful algorithms to more accurately predict behavior and improve forecasting. Imagine being able to use the simplicity of a survey to discover the truth and predict the future based on common market data. It is designed to perform in environments where market mechanisms are either not useful (due to the environment) or non-viable for complexity reasons.

How perfect is this for the uncertainty faced by our post-recession economy? It's a way of addressing the need for "Pattern-based Strategies," a concept recently evangelized by lead analysts and executives at the Gartner Group.

Live BI is a project that takes this new information in motion and information everywhere from sensors and pushes it toward real-time. It is a unified data and analytics platform that allows much more powerful and sophisticated analysis of highly complex data in real time. Imagine being able to take the pulse of the planet at any moment in time. There are countless uses for global data on temperatures, vibrations, pressures, chemical readings and more from all over the planet.

A few days ago, HP published its annual Global Citizenship Report with more details in the Information Explosion section. Check out the report and let me know your thoughts.  What do you think about the interesting new innovations that HP is pursuing to help harness the information explosion?  What projects or initiatives do you find most interesting?

This is why I love my company. HP is committed to using technology in a number of areas that really will make our world better-improving the economy, healthcare, education, the environment and more. 

Post by John Santaferraro
Twitter: santaferraro

 

Posted by HP Business Intelligence Solutions at 4:05 PM

April 15, 2010

Information in Motion: A Key Driver for Business Intelligence Modernization

We have transitioned from a world of static data and transactional batch loads to the world of information in motion. It's a world where the old, static, data warehouse is rapidly being replaced by a world where we stream data in and stream intelligence out. The streaming data sources are on the increase, the need for near real-time data is increasing, and the volume of information in motion is staggering.

The influx of new streams of data is beyond what we ever imagined possible. Consider sensors in the digital oilfield, monitoring of the smart grid, smart meters and home monitors, the digital hospital, the digital healthcare system, location data flowing from mobile devices, and RFID on everything from shipments to assets to people in the field.

For example, on an energy smart grid there are monitors that read electrical phase 60-100 times per second, and thousands of these phase monitors over the entire grid. In addition, millions of energy smart meters on homes and business send reads back to a central repository every 15 minutes, in some cases more frequently, producing terabytes of data every single day.

Along with data pouring in, there is a need for intelligence pouring out. Analyzing near real-time phase readings can drive the optimization of the energy grid and help identify potential outages before they occur. For smart meters, every smart meter can be connected to a web-based portal in each home or business. When smart meter readings are analyzed in near real-time, smart meter intelligence provides insight into customer offers or automated control of home appliances.

On the digital oilfield, there has been a global effort to modernize drilling rigs and production platforms by covering them with devices that monitor pressure, temperature, flow and a host of other measurements. The result is massive amounts of data pouring into data historians and operational systems, all very useful from an analytic perspective.

To add to oilfield information in motion, HP and Shell Oil recently announced a joint project to provide a wireless sensing system to acquire extremely high-resolution seismic data. HP technology combines sensor research originally intended for the printers with memristor technology which combines memory and logic in a technology that's so small and so low power that it enables a pushpin-sized sensor. These sensors are about 1,000 times more sensitive than today's mass produced devices.

There is amazing potential for harnessing billions of drilling and production readings, millions of engineer interactions, and hundreds of thousands of financial transactions every day into digital oilfield intelligence solutions. Streaming out as intelligence, there are opportunities to predict outages or slowdowns in production, automate decision making in the field, and ultimately, to increase well profitability by individual well, basin, or region.

Business intelligence and data warehousing are changing right before our eyes. Without question, the changes mentioned above drive the need for BI modernization. Traditional data warehousing and business intelligence systems were not designed for this kind of environment. When you consider the matrix of information in motion, here are some questions to consider:

And just for the record, HP Neoview Advantage was built for such world as this!

For some fun reading, check out the blog post on CeNSE (Central Nervous System for the Earth), or just watch the video...

 

Post by John Santaferraro
Twitter: santaferraro

Posted by HP Business Intelligence Solutions at 12:48 PM

Why Isn't Business Intelligence More Pervasive?

Pervasive business intelligence (BI) is loosely defined as BI for the masses; empowering everyone in the organization, at all levels, with analytics, alerts and feedback mechanisms to make the right decision at the right time.

Pervasive BI: The Hype

For the past several years there have been widespread projections of pervasive BI: accessible by employees, partners, and customers by the, and I quote, “millions.” Not just for “quants” any more, with today’s tools, “anyone with access to a computer, PDA or smartphone can do analytics.”

Pervasive BI: The Reality

 Fueled by the possibilities described in Thomas Davenport’s 2006 Harvard Business Review article “Competing on Analytics,” many organizations aim to deploy advanced analytics and extend use of the data warehouse (DW) to a wider community of users, supporting operational as well as strategic decisions.

But the reality of pervasive BI has fallen far short of the hype. In a case study published in August 2008, IDC analysts made the point that “Despite the fact that the term Business Intelligence was first coined in 1958 and the first BI software tools emerged in the 1970s, BI is not truly pervasive in any organization.”

And according to BI tools guru Cindi Howson, in 2009 the percentage of employees using BI was 24%, down from 25% in 2007. Even today, BI is used mostly by business analysts and power users (roughly 50% penetration vs. 23% use among field staff).

Pervasive BI: The Obstacles

It’s not for a lack of promise that BI holds, but there are a number of reasons why it has not become more pervasive, despite intent on the part of many.

BI is critical to competitive success, but before organizations can take full advantage, they need to overcome these obstacles.

Technology to the Rescue

Not to worry. In the high tech industry, there is never a shortage of solutions. Here are just a few of the new technologies (and some not so new, but marketed with renewed vigor) standing ready to make BI more pervasive:

Pervasive BI: The First Step

Depending on your infrastructure, one of more of these may help ease the issues of performance, data access, cost, and management overhead. But fundamentally, inconsistent data does create barriers to reliable analytics and strategic use of data. Organizations need to overcome that before BI can be pervasive.

HP consultants worked with client Steven Yon, Division Director, Common Services, Corporate Operations and Information Services at National City, one of the largest financial holding companies in the US, to develop a standard process for enterprise information management and information governance. He talks about how important it was to establish an environment of data governance with consistent metrics and rules, allowing National City to exploit their data as a strategic asset in order to better serve their customers: “If data must go through several conversion steps before it can be used, it makes it difficult to respond to customer needs. We can’t be agile unless data is integrated and coherent. The result is better customer experience. Differentiating ourselves through insight into customer behavior allows us to win relationships even in a commodity business.”

Our research and consulting work at HP indicate an increasing recognition that to truly make the most of BI and analytics, you need to “get the data right.”

Click here for more information about HP’s Information Governance Services to help clients protect, manage and develop data as a valued asset.

Posted by HP Business Intelligence Solutions at 4:47 AM

April 8, 2010

Look Who's Talking About Connected Intelligence

With our recent talk about connected intelligence at HP, I thought it was interesting that Tim Berners-Lee, the inventer of the internet, is talking about the value of connected data. Rather than trying to wax eloquent, I'll let him speak for himself. This is his presentation at the TED Conference in 2009. Check this out...

 Post by John Santaferraro
Twitter: santaferraro

 

Posted by HP Business Intelligence Solutions at 11:06 AM

April 5, 2010

Are You Ready? A New Generation of Data Warehouse and BI Systems is Here

You’re gonna need a bigger disk.” That’s what Amity Police Chief Brody would say if he were chasing down today’s voracious data analysis needs.

A recent InformationWeek Analytics 2010 State of Enterprise Storage survey revealed “an alarming state of affairs.” Nearly half of the IT professionals said they had insufficient storage resources for critical applications, up from 30% the previous year.

I doubt that anyone has gone more than a week or two without reading the latest statistics on the dramatic rate of data growth, so I won’t repeat any here. But that reality, along with regulatory requirements to keep much of the data longer is putting a strain on IT storage budgets and causing IT to search for ways to reduce that strain: SANs, cheaper drives, data compression, multi-temperature storage, etc.

And yet, according to industry analysts, in the data warehousing/business intelligence (DW/BI) world there was a resurgence in independent data marts in 2009 – the same year in which storage budgets were reaching a breaking point. Wouldn’t consolidation, or the reduction of persistent data marts (which are the embodiment of redundant data and therefore wasted storage) make more sense in the belt-tightened reality of the new economy?

A rise in data marts means not only more disk drives, but more processors, memory, ETL scripts to manage, maintenance, floor space, power, and overall system management, with less corporate control over the reliability of the data. One way to help meet today’s user’s appetite for data while maintaining its security and control is to stop duplicating it. Keep it in one place and manage the access, providing all users with the ability to analyze it as they see fit.

What is the reason for this increase in data marts after a data mart consolidation trend that dates back to late 2001? I think there are several.

That last point gets at the current challenge of trying to run and manage multiple types of workloads all on the same system with good performance and the ability to scale to high volumes, all without increasing the administrative overhead. This is known as a “mixed workload” which could easily include a mix of batch data load, continuous real-time data load, a large number of standard reports, pre-determined as well as ad hoc queries, complex queries, data mining and operational analytics.

As industry experts point out, most existing data warehouse implementations are overwhelmed by mixed workload requirements, from both a performance and management perspective. In response to users’ needs, IT departments have resorted to multiple warehouse deployments. To be sure, today’s mixed workload reality is quite different from the world in which most of today’s DW/BI systems were developed. Tuning them to optimize for these conflicting requirements is not always possible. But proliferating specialized data marts is not feasible either.

In fact, in a survey by The Data Warehousing Institute (TDWI) regarding next generation data warehouse platforms, 46% of respondents say they are “planning a data warehouse platform replacement by 2012. Many others anticipate keeping their current platforms, but updating them significantly.” TDWI’s position is that “certain relatively new technologies, techniques, and business practices are driving the majority of data warehouses and their platforms toward a redesign, major retrofit, or even replacement significant enough to be recognized as a new generation.”

At HP, we agree that a new generation is at hand. In the “Top 10 Trends in Business Intelligence for 2010” whitepaper, we describe our view of the new generation of DW/BI systems and note that we are seeing a change in the BI culture. Leading companies are beginning to think not just about “informing the user” but also about “improving the process,” and using analytic results directly in the workflow context to drive operational execution and dynamic process change. We see the coming economic recovery as an impetus for organizations to invest in a new generation decision management system that will help them to transform their business for a new generation of uses and users (while easing the strain on their storage systems and budgets). The whitepaper discusses some of the technologies which hold promise for enabling this new generation of data warehouses and BI systems.

What do you see as the drivers of the new generation, and what form is it taking? I welcome your comments and wish you a happy 35th anniversary of the movie classic, “Jaws.”

Posted by HP Business Intelligence Solutions at 7:42 PM

April 2, 2010

Welcome to the World of Connected Intelligence!

Connect. It's what we do with dots. It's what we do online. It's what networking is all about. It's the last thing someone thinks about when they are building a stand-alone data mart on a single purpose database or analytic appliance. It's not on the mind of the business sponsor who ships his data off to a cloud in order to get a quick reporting need met.

It is the answer to our disconnected world of information. We've seen people build siloed applications and store data in silos for years. Now they are building business intelligence applications in silos. Marketing owns its own set of BI apps; sales has their own reporting systems; finance owns the applications that track profitability; and product data management lives its own world.

In the world of connected intelligence, every element of business intelligence is tied together in a way that drives more business value and allows companies to respond more quickly to changing requirements.

Connected Intelligence starts with getting the data right. Most companies are already thinking of connecting their own data. We look beyond that to identify the value of connecting to outside information or looking at streams of events that would open up the world of instantaneous decisions. Industries like communications, finance, and energy have complex sets of data, complex data stores, complex data flows and complex networks of people need information to do their jobs. This is an area that requires more than technology. It's the combination of people, processes, and technology that helps companies lay the foundation for connected intelligence.

I recently spoke to two veteran data warehouse and business intelligence leaders in the retail and manufacturing industries. Both had been on the job for more than 10 years. Both had built mature information management programs in their companies. And both were faced with the same challenge: "How do I take the still complex analytics I'm producing, simplify the content, and get it to the front lines in the form of a recommended action."

Once you have the data right, the second level of connected intelligence helps companies use that new view of the enterprise in more ways than any other approach. It's all about connecting insight to action. Think of what you could do by embedding analytics in business processes or distributing the right analytic results to millions of front line decision makers every single day. When analytics reach the front lines, companies can finally make decisions at the speed of business and make every decision count.

Once the business is blanketed with analytics, the third level of connected intelligence opens the door to new business possibilities. What happens when you have connections between your business intelligence applications? What happens when you discover 5 or 10 new combinations of analytics that were never before possible in a disconnected environment? Innovation happens. You begin to discover new business models that support breakthrough initiatives. You monetize the value information and transform your business.

In the utility industry, I was surprised that most companies don't know their actual cost of power. As a result, their pricing models tend to require a mix of regulatory guidance, prediction, and intuition. If these companies could connect trading information with regulatory, demand, grid, and customer information, they would be able to create models to automate the pricing function. If they were able to connect this kind of data in near real-time, they could create a dynamic pricing engine that could potentially change the way companies do business in their industry.

That's connected intelligence, an approach to BI that connects analytics in more ways than ever before to drive increasing business value and open the world of intelligent innovation.

And this is why I love working at HP! I love working with a team of innovators that are committed to helping companies use technology in ways they never before dreamed possible.

But I'm curious what you think? What do you think about the idea of connected intelligence? I would love to hear your thoughts...

Post by John Santaferraro
Twitter: santaferraro

Posted by HP Business Intelligence Solutions at 10:53 AM