In many ways the threads of change are already here. Big data. A growing number of data types and sources. The drive to faster decision making. More business users asking more complex questions. The evolution of platforms to address these new demands. And they have the promise to transform analytics within the enterprise.

I will never be considered a statistician super model. I lack a PhD in mathematics. (The bald spot isn’t helping my cause either.) But then for analytics to be truly transformative, won’t business users and decision makers also need to be comfortable wearing statistical models and computational algorithms? Even if they don’t have the flair to design them?

We’ll be exploring the threads of change over the coming months, and Sybase’s unique approach to weaving them for transformative change. Stay tuned.

Enterprises looking to upgrade to the latest version of Oracle will face a dilemma. Purchase shiny new Sun hardware at the same time or migrate to a different database. Both are expensive propositions.

Collateral damage. That’s what customers are when vendors look to grow market share by hurting the very customers they want to win over. There’s irony here.

Some people will probably point to this as a reason to consolidate to one vendor with a complete stack. Personally I think there’s wisdom in picking vendors who don’t have this software/hardware conflict of interest. Besides, having the freedom to choose best-of-breed hardware and software will put savings in your company’s pocket, and probably net you a promotion at the same time.

Challenge is there aren’t too many best of breed data warehousing and analytics vendors that fit the profile. (See latest Gartner MQ for Enterprise Data Warehouse DBMSs and Forrester Wave Enterprise Data Warehouse Platforms). Sybase and SAP being exceptions, of course!

And you won’t be collateral damage either.

I am sure I have your attention by now, but I imagine you are wondering what hang gliding has to do with big data.  Did you wander to the wrong blogging site by accident? Did you search on ”aeronautics”, rather than ”analytics” by mistake?  No, you are in the right place.  Be patient, and let me tell my story.

First, here is a picture of me standing at the top of the training hill near Lookout Mountain, poised with my glider on my shoulders.  My objective is to walk, jog, run, then propel myself into the air for a flight that with luck will keep me in the air for 15 seconds or so, at a maximum height of maybe 20 feet.  I am processing data (as well as a lot of emotion) – keep my wings level, maintain the glider’s nose at a particular angle, keep my eye on my target, don’t grip the control bars too tight, don’t look at the ground, make slight adjustments as I fly.  It is a lot to deal with, and my mind is racing:

 

This is a relatively small amount of data that I am processing, and somehow I manage to pull it together and fly through the air:

 

I spent three days on the training hill, gaining confidence and some skills.  I found myself looking at the sky every morning, checking the wind sock in the landing zone, sensing the movement of the air, and observing the clouds.  I learned that birds circling overhead were not tracking prey, but lifting and soaring through thermals – the quest of all hang glider pilots.

I realized how much of hang gliding is an understanding of the weather, and how ground topography and temperature affect air currents.  A pilot will spend more time assessing the conditions in which he will fly, than flying itself.  Hang gliding is as much about observation, judgment, and patience:

 

as launching, flying, and landing:

 

The vast collection of data points that characterize atmospheric conditions of concern to pilots constitute ”big data”.  This data is sensor generated, streaming and dynamic.  The National Oceanic and Atmospheric Administration (NOAA) is one of the most cited examples of an organization that amasses huge amounts of data to aid in climate, ecosystem, and weather research.  It uses ”Big Data” technologies to load, store and analyze petabytes of data.

On March 23, 2011, while I was musing on big data at Lookout Mountain, Sybase’s CTO, Irfan Khan, was talking big data at the GigaOm Structure Big Data 2011 conference in  New York City (http://event.gigaom.com/bigdata/).   His presentation was titled ”Streamlining the Information Pipeline:  How to Drive Towards Zero Latency”.  He discussed the increasing importance of real time data and analytics, the race to zero latency and how enterprises can implement and take advantage of next generation information platforms in these times of massive data volumes.  Check out this videocast to watch the presentation:

http://www.livestream.com/gigaombigdata/video?clipId=pla_792874d1-8473-43be-91ab-2b78ab2a40c0&utm_source=lslibrary&utm_medium=ui-thumb

I have just begun my journey as a fledgling pilot, and have much to learn.  I look forward to the day when I can not only maneuver my glider with confidence and joy, but interpret the environmental cues that will allow me to assess the conditions for soaring and safe flight.  Maybe someday, big data technology will allow me to visualize those thermals in real time from instruments mounted on my glider.

Happy flying,
Courtney Claussen

Authored by Trevor Moore, the book draws upon his considerable hands-on experience with Sybase IQ. Trevor has spent 10 years working on Sybase IQ at Lehman Brothers, Credit Suisse, the UK government and Sybase Professional Services, so the book definitely offers up advice from the perspective of someone who has been there and done that.

I had the opportunity to shoot a couple of questions Trevor’s way regarding his book.

Q: Why did you write this book?

I wanted a quick guide though the world of IQ that gave me practical examples of how things worked. Unfortunately, there did not seem to be one in the market, so I set about writing one myself. The book took about 18 months on and off to complete and has been received quite well, selling over 100 copies in the first couple of months. I am currently taking a break from writing until Version 15.3, when I plan to update the book.

Q: Who will benefit from reading your book?

As the world of IQ is smaller than ASE, I set about writing a book that could be used with versions 12.7 upwards (although a lot of information is still relevant to earlier versions). I have included all useful features that a beginner would want to know, such as the different types of indexes and tables, through to more advanced features such as using Java procedures and XML. I have also included a tips and tricks section that covers some of the more common issues/quirks I have found over the years that may help DBA’s and Developers to resolve issues quickly.

It looks like there’s something for everyone, novices or experienced users, in this book, so if you are interested, the book is currently available from Lulu and Amazon . Get one for yourself or your Sybase IQ user friends!

http://blogs.sybase.com/sybaseiq/2010/09/improved-data-%e2%80%93-shown-to-promote-innovation/

The study has been completed, and the third part focuses on the relationship between better data, and a company’s operational efficiency.  Operational efficiency is characterized by the following:

• Asset utilization: the ability of a firm to efficiently utilize inventories, accounts receivables and hard assets for the production of goods and services
• Planning and forecasting accuracy: the ability of a firm to accurately predict demand for its products and services
• On-time delivery: the ability of a firm to deliver products or services to its customers in a timely manner

The following diagram illustrates the relationship of the variables examined in the study – data and control processes driving operational efficiency:

The key findings of the study are these:

• An increase of 10% in the mobility of the sales team (better availability of data) leads to a 7.28% increase in asset utilization
• An increase of 10% in business intelligence (better intelligence from data) leads to an 18.5% increase in planning and forecasting accuracy
• An increase of 10% in data quality (better accuracy of data) leads to a 2.8% increase in the timeliness of product delivery

It is not surprising to see that there are relationships between effective data and the ability of a company to execute its business.  What is unique about this study is the attempt to measure the magnitude of the impact.  The results demonstrate the substantial benefits from small improvements in data quality and usability.  You can find the complete published report here:

http://www.sybase.com/detail?id=1082805

Courtney Claussen

Sybase Redefines Massively Parallel Processing (MPP) With Increased Analytic Performance, Scalability and Architectural Flexibility

Sybase IQ 15.3 Customers Benefit From Extreme Query Performance With PlexQ Distributed Query Platform 

ORLANDO, Fla., Nov 09, 2010 (BUSINESS WIRE) — Sybase, Inc., an SAP(R) an industry leader in enterprise and mobile software, today announced the release of Sybase IQ 15.3 to beta customers. Sybase IQ 15.3 introduces the PlexQ(TM) Distributed Query Platform, a Massively Parallel Processing (MPP) architecture that accelerates highly complex queries by distributing work to many computers in a grid configuration. Unlike shared-nothing MPP architectures, PlexQ utilizes a shared-everything approach that dynamically manages and balances query workloads across all the compute nodes in a Sybase IQ MPP configuration. The PlexQ’s automatic workload re-balancer aggressively works to avoid contention among users for system resources, thereby providing predictable high performance for a spectrum of concurrent workloads.

What we’re talking about here is MPP done right. Leveraging the Sybase IQ column-based arhcitecture, the  PlexQ Distributed Query platform tops standard shared-nothing MPP architectures by providing better concurrency, enabling higher-performing self-service ad-hoc queries, and supporting independent scale-out of compute and storage resources. Extreme performance and architectural flexibility have always been key parts of the Sybase IQ story. As you’ll see in the weeks and months to come, that story is about to become a lot more interesting.

Meanwhile, if you happen to be attending TDWI this week, be sure and stop by the SAP Booth (#300) and look for Sybase Analytics co-blogger Kevin Leong. Kevin is on hand to tell you all about how Sybase IQ is redefining MPP.

Sybase Redefines Massively Parallel Processing (MPP) With Increased Analytic Performance, Scalability and Architectural Flexibility

Sybase IQ 15.3 Customers Benefit From Extreme Query Performance With PlexQ Distributed Query Platform 

ORLANDO, Fla., Nov 09, 2010 (BUSINESS WIRE) — Sybase, Inc., an SAP(R) an industry leader in enterprise and mobile software, today announced the release of Sybase IQ 15.3 to beta customers. Sybase IQ 15.3 introduces the PlexQ(TM) Distributed Query Platform, a Massively Parallel Processing (MPP) architecture that accelerates highly complex queries by distributing work to many computers in a grid configuration. Unlike shared-nothing MPP architectures, PlexQ utilizes a shared-everything approach that dynamically manages and balances query workloads across all the compute nodes in a Sybase IQ MPP configuration. The PlexQ’s automatic workload re-balancer aggressively works to avoid contention among users for system resources, thereby providing predictable high performance for a spectrum of concurrent workloads.

What we’re talking about here is MPP done right. Leveraging the Sybase IQ column-based arhcitecture, the  PlexQ Distributed Query platform tops standard shared-nothing MPP architectures by providing better concurrency, enabling higher-performing self-service ad-hoc queries, and supporting independent scale-out of compute and storage resources. Extreme performance and architectural flexibility have always been key parts of the Sybase IQ story. As you’ll see in the weeks and months to come, that story is about to become a lot more interesting.

Meanwhile, if you happen to be attending TDWI this week, be sure and stop by the SAP Booth (#300) and look for Sybase Analytics co-blogger Kevin Leong. Kevin is on hand to tell you all about how Sybase IQ is redefining MPP.

This comes as no big surprise to us, seeing as our own research shows that only about 15% of our analytics customers have a formal analytics plan in place.  That’s right, 85% of those currently doing analytics don’t have an analytics plan in place. What’s going on? Are these businesses living in the Wild West – enjoying some of the trappings of civilization (analytics), but not really there yet?

Source: Wikimedia Commons

Jon Lucker, a principal at Deloitte Consulting, explains  the lack of analytics capability and awareness by identifying two core business issues that limit the adoption of analytics software.  The first of these is that users tend to miss the value offered by analytics software because they have not been shown what these types of tools are capable of doing.  The second issue comes from the technical side of the house. Lucker says that IT organizations tend to be overly ”hung up” on data quality.

I would tend to agree with Lucker’s first point, but – working for a database software company – I have to be a little wary of the second. Data quality is a hugely important issue and, while there are inevitably tradeoffs  between varying goals within any IT environment, it should never come down to having to choose between data quality and analytics capability. Those should both be doable and, as we’ll see below, will in fact work together to bring increased value to the business.

Despite the fact that businesses are investing large sums of money into Information Technology, and specifically into analytics, Deloitte’s  survey testifies to the fact that there is still a gap where quantifying and understanding the benefits  are concerned. So what is it about data analytics that’s so important to businesses and how does having effective data impact the business?

A recent Sybase-sponsored study (conducted by the University of Texas at Austin, in conjunction with the Indian School of Business) examined  150 Fortune 1000 firms from every major industry. The study found that five attributes of data – quality, usability, intelligence, remote accessibility and sales mobility have a dramatically positive effect on key financial measures. Relatively small improvements in these key attributes can pay off with large financial returns. As we have noted before, here’s how the numbers broke down for  median Fortune 1000 business (36,000 employees and $388,000 in sales per employee) as an example.

1. Productivity of Employees: The study found that employee productivity could be considerably affected by increasing the usability of data within the organization. Increasing  the usability of data by 10% would translate to a $2.01 billion increase in total revenue every year. 
2. Return on Equity: If the average Fortune 1000 company increased its data quality and the ability of sales people to access data by 10%, the company could increase ROE by 16%.
3. Return on Invested Capital:  Increasing mobility of the sales organization’s data by 10% would  increase ROIC by 1.4% as a result of net income increasing by $5.4 million for the average Fortune 1000 business.
4. Return on Assets: A 10% increase in both data intelligence and accessibility of data resulted in a 0.7% increase in ROA for the average Fortune 1000 company. This is equivalent to squeezing an additional $2.87 million of income out of the average business’s assets (assuming they remain fixed).

So it would seem that investing in IT does pay off. (And maybe being ”hung up” on data quality isn’t such a bad thing!)

Like the Deloitte study referenced above, these results imply a world in which the value of analytics is not yet fully realized.  Going forward, data intelligence and accessibility should be pushing the needle a lot harder for employee and overall organizational productivity than they do for return on assets. Equipping decision-makers to make better decisions faster is a part of what analytics is all about. (Actually automating those decisions where possible is another part.)

When the railroads finally made their way to the small towns out west, true civilization inevitably followed. Likewise, a business that makes better decisions faster can’t help but be more productive and, ultimately, profitable. Companies beginning to recognize these kinds of benefits from analytics solutions – or looking to do so, having seen their partners or competitors achieving those benefits – will begin to put more planning and a deeper organizational understanding of analytics in place.

Any of the surveys referenced above, if repeated five years, or three years, or even one year from now, are likely to show  a significant increase in how well organization understand, use, and reap the business benefits of analytics solutions.

The U.S. government can save more than $1 trillion over the next 10 years by consolidating its IT infrastructure, reducing its energy use and moving to more Web-based citizen services, a group of tech CEOs said in a report released Wednesday.

The Technology CEO Council’s report, delivered to President Barack Obama’s National Commission on Fiscal Responsibility and Reform, also recommends that the U.S. government streamline its supply chains and move agencies to shared services for mission-support activities.

“America’s growing national debt is undermining our global competitiveness,” said the council, chaired by IBM CEO Samuel Palmisano. “How we choose to confront and address this challenge will determine our future environment for growth and innovation.”

So far, very interesting. But why am I posting about it on the Sybase Analytics blog? Well, here’s why:

The federal government could also reduce IT energy consumption by 25%, and it could save $200 billion over 10 years by using advanced analytics to stop improper payments, the report said.

So out of a trillion dollars in savings that this group has outlined for the  US federal government, 20% is coming from implementing advanced analytics.

This is a strong validation of what we’ve been saying all along about the real business benefits that come from implementing an effective analytics environment. If Uncle Sam decides to take the CEOs up on their ideas, let’s hope the government puts a solution in place that can generate true insight into the problem area and deliver the kind of scalability and performance that will actually deliver the $200 billion savings.

If anyone from the government wants to talk about the database platform and BI tools that would best deliver all that, I’d be happy to take their call.

http://blogs.sybase.com/sybaseiq/2010/09/the-benefits-of-improved-data-%e2%80%93-quantified-at-last/

The second part of the study has just been released, and focuses on the relationship between better data, and a company’s ability to innovate and grow revenue from new products and services.  This is important, because new business – an increasing customer base and variety of products – is a key indicator of a firm’s competitiveness and ability to thrive.

Part two of the documented study includes some thoughts from Mark Deck, lead director of product development practice at PRTM, a management consulting firm in Waltham, MA, who describes why there is a relationship between better data and innovation.  Mr. Deck says ”...increased data accessibility enables different groups such as product development in an organization to obtain data from functions like sales and services, which provide important information on customer preferences and demand patterns”.  Accurate data and correct analysis is critical for the creation and marketing of new products and services.

The study focuses on two aspects of data effectiveness – accessibility (availability by remote users) and intelligence (trends and patterns) – and how a 10% improvement in these attributes leads to increased revenue from both new products and services, and new customers.  These results were generated based on a detailed survey of over 150 Fortune 1000 companies.  Here are the key findings:

• An increase of 10% in data accessibility and intelligence leads to a .81% increase in revenue due to new products: $17 million additional revenue for the median firm in the study’s sample
• An increase of 10% in data accessibility and intelligence leads to a .7% increase in revenue due to new customers: $14.7 million additional revenue for the median firm in the study’s sample

The results demonstrate the often dramatic impacts from even marginal improvements in data quality and usability.  The third and final part of the report will explore customer-focused and operational impacts.  You can find the published report here:

http://www.sybase.com/detail?id=1082805

Sybase recently revealed the results of the first part of a ground-breaking three part study that attempts to measure the direct correlation between a company’s IT investments and business performance.  The study was commissioned by Sybase and conducted by researchers from the University of Texas in conjunction with the Indian School of Business.  The first part of the study looks at five distinct attributes of data: quality, usability, intelligence, accessibility and mobility, and examines how a 10 percent improvement in any one or two of these attributes affects business financial performance. 

Here is a depiction of the conceptual model that formed the basis of the study:

The methodology of the study involved these steps:

1. Develop a questionnaire that would elicit measures of the data attributes and financial impacts
2. Refine the survey with feedback from employees of Fortune 1000 companies across various industries
3. Conduct the survey
4. Analyze the results
   1. Use factor analysis to derive the specific variables that impact operational performance
   2. Apply multiple regression analysis statistical techniques to quantify the relationship between independent and dependent variables

The study focuses on the impacts of effective data on productivity of employees, and key financial performance indicators for a company.  Here are the highlights for a Fortune 1000 business (median  36,000 employees and $388,000 in sales per employee):

• A 10% increase in data conciseness and consistency results in a 15% average increase in sales per employee
• A 10% increase in data accuracy and accessibility results in a 16% average increase in return on equity (ROE)
• A 10% increase in mobility of data for the sales force results in a 1.4% average increase in return on invested capital (ROIC)
• A 10% increase in intelligence and remote accessibility of data results in a .7% average increase in return on assets (ROA)

The findings demonstrate the often dramatic impacts from even marginal improvements in data quality and usability.  Two subsequent reports will explore customer-focused and operational impacts.

The first part of the full report can be found here:

http://www.sybase.com/detail?id=1082805

Enjoy!

Courtney Claussen

Some of the findings are pretty much in line with what you would expect from such a survey, while others are more surprising. Not surprisingly, more than 2/3s of those surveyed cited rampant data growth as a major driver behind their implementation of analytics. More surprisingly, fewer than 15% of these customers (all analytics customers, remember) have a formalized analytics plan in place.

Let’s look at some of the results in a little more detail.

Drivers

Rapid data growth continues to be a major driver for the adoption of analytics. As companies are faced with the challenge of managing the growing influx of data, they are turning to analytics to make sense of it all.

• 72% of those surveyed noted that data growth and the need to better manage and understand data was the biggest driver for implementing analytics
• Increasing and maintaining competitive advantage was ranked as the highest benefit sought with an analytics solution
• The highest priority for analytics solutions is time to results, followed by the total cost of ownership and initial cost of the implementation
• The number one impact companies report seeing from analytics is better planning, forecasting and decision-making

 

Analytics as a Priority

Although businesses feel that analytics is something their organizations must have, most companies still do not have a formalized analytics strategy or even include it as part of their annual IT planning process:

• A majority of those surveyed describe analytics as a ”must-have” for their business
• 86% of the survey participants do not have a formalized analytics strategy
• While half of the respondents do not include analytics as part of their annual IT planning process, 93% see the growth of business data and the distribution of that data increasingly impacting IT planning
• Many cited the change from needing data monthly, weekly, or daily, to the need to have answers continuously and in real time
• Half of those surveyed have expanded the use of analytics throughout their business since the original deployment

The Road Ahead

These results indicate that, while organizations are making progress with analytics, there is still a lot to be done. Although organizations are increasingly using analytics to drive business results with faster and more accurate answers, the discipline of developing, implementing, and managing an overall analytics strategy is emerging slowly. Meanwhile, organizations at an earlier stage are looking for how best to make the initial move in that direction. Greater recognition of the kinds of success reported by our users can only help in making such a call.

Do you have an analytics strategy in place? What are some of the challenges and what kind of successes have you seen from analytics? How are you dealing with the complexities of incorporating BI solutions to drive business results? Let’s discuss.

The Sybase booth was hot!  200 individuals passed through, and 55 completed surveys.  Those surveyed came from a variety of industries:

We had 50 individuals complete at least 3 of the 8 classes, and we even had a few graduates of the entire curriculum.  Today’s BI professionals know that knowledge is power, and TDWI attendees availed themselves of the opportunity to learn and grow.  Thanks to all those who participated.

Check out this blog to learn details about the Performance Series at TDWI:

http://blogs.sybase.com/sybaseiq/2010/08/the-performance-series/

Until later,
Courtney Claussen

Since the TDWI conferences are all about education, we decided that we would go beyond sharing information about Sybase IQ and our tremendous success in the analytics market. So this time out, we are sponsoring an eight-part course outlining the key business and technological considerations for achieving high-performance business intelligence/data warehousing.We’re calling it The Performance Series.

The eight 10-minute mini-classes are as follows:

BI/DW Performance: The Warning Signs
Identifying the business and technical factors that foreshadow BI/DW performance issues.

High-Performance Architecture Part 1: Columnar Databases
Why has the column-based approach garnered so much recentattention for BI/DW? A straightforward overview of the approach and the benefits it brings.

High-Performance Architecture Part 2: SMP or MPP?
Symmetric or massively parallel? Shared disk? Shared nothing? Shared everything? How to choose an architecture that best fits the needs of your organization.

Performance BI/DW Part 1: In-Database Analytics
A look at the performance and security advantages that resultfrom moving analytics processing into the database.

Performance BI/DW Part 2: Text Search and Analysis
It’s estimated that 80% or more of most organizations’ data is trapped in unstructured formats. What if you could leverage those valuable data assets within your BI/DW environment?

Performance BI/DW Part 3: Web Enabled Analytics
Ensuring that your BI/DW environment is Web 2.0 app-friendly,and that it stays on the cutting edge of Web development.

Managing Information Lifecycle in a Performance Environment
Having data where and when it is needed is a major consideration;making the best use of available resources is another. Sound Information Lifecycle Management Policies help to ensure thatyou can have both.

The Four-Phase Approach to BI/DW Performance
Beyond the lifecycle of the data is the lifecycle of the BI/DW environment. How to ensure that your environment is providing the answers your organization needs.

If you’re planning to attend the TDWI conference, we look forward to having you join us for this course. The classes will be presented at booth #102 in the main exhibit hall during show floor hours.

If you can’t make it to TDWI, we are also sponsors of an in-depth certification course from BEYE University. Called The Analytics Performance Series, this online course covers the same topics listed above in somewhat greater depth. It is presented by industry luminaries Claudia Imhoff, Seth Grimes, and David Loshin.

]]> http://www.beyeblogs.com/sybaseiq/archive/2010/08/theperformances.php http://www.beyeblogs.com/sybaseiq/archive/2010/08/theperformances.php Wed, 11 Aug 2010 18:55:11 -0700 Our friend Jim Ericson at Information Management has some thoughts on a recent DM Radio segment with James Kobelius of Forrester research and Neil McGovern from Sybase. The topic of the program was real-time data warehousing. Jim notes that one of the key buzzwords to emerge in the discussion was Big Data, and that some pretty interesting stuff starts to happen when big data meets CEP (complex event processing):

McGovern was talking about complex event processing or CEP (another pregnant term) and the idea of bringing the data to the query, rather than the other way around, which is something we’re seeing more of now with in-memory analytic processing and the advent of the petabyte-scale data warehouse.

That’s what CEP tends to be about, a kind of forensic analysis (like a series of corpses to pick over moving by on a conveyor belt) of something that’s actually just an invisible blur. McGovern says “one of the faster feeds” he knows about in financial services uses an engine that monitors one million transactions per second.

Ahem. I stopped to confirm he’d said “one million transactions per second.”

Indeed he did.

Not that long ago, the only attribute of data bigness that got much attention was the size of the data itself. And it’s still a huge consideration — if you’ll pardon the expression. Managing exponential data growth is important; we do live in an age of petabyte-sized data warehouses (and of taking the data to the query.) But increasingly we are realizing that it isn’t just about size, that there are a number of different factors that contribute to the bigness of Big Data.

To name just a few…

• The number of users accessing the data
• The volume of queries / analyses that data is subject to
• The complexity of the queries / analyses performed
• The speed with which results must be returned

Even a modest amount of data becomes Big Data as such requirements become more and more demanding. In the end, a Big Data solution has to be one that accommodates more than one factor at a time. As organizations experience growing pains with data “bigness” emerging along multiple,  often unexpected dimensions, the need for an analytics environment that scales flexibly, and in line with what the business demands, becomes increasinglyclear.