BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

Main | February 2009 »

December 31, 2008

Discovering Big Data

We keep hearing industry analysts and companies making claims regarding data volumes doubling every year and exploding data and I have always turned around and questioned from where? If a company is growing well, they could expect 30-50% more order data volumes, perhaps 20-30% more customer care tickets. I do not think that financial systems data would grow too significantly. So where does all of this exploding data come that we have to worry so much about?

What I described above is a very basic brick and mortar form of running a business where operational data came from business applications and functions. In todays world, several other factors provide new data which is necessary to be a successful competitive business.

Recently, I was fortunate to meet and spend some time with Andrew Cardno, Chief Technology Officer at BIS2, Business Intelligence Systems Solutions. Andrew is an extremely bright, energetic and charismatic person who is driving one clear message in his company; that data volumes are becoming so large that the only way to interact with data in the future will be through data visualization techniques. Later that evening, he made an interesting comment that really stuck with me. He said that less than 0.1% of all data captured is entered by humans , the rest comes from instrumentation and that we have generated more data since 2005 then we have in the past 40,000 years!

The word that stuck with me was instrumentation. Maybe this was because my engineering degree specialized in instrumentation and controls or maybe because that is where I spent most of my time in building data warehouses and decision support systems. But it did rattle in my brain and got me looking around; computers, scanners, digital pictures, text messages, etc are all silently being captured and stored (or not) somewhere. Surfing the internet was not really surfing, it was hiking and exploring and we are leaving digital footprints every where.

Businesses today should not only count the record that represents the order that you place with them. They should be more interested in all the many steps and processes you followed before and after placing that order. And all of the orders that should have been placed and were not! All the records that represent my digital footprints all over the company website and how many other people have taken similar hiking paths yet finished with a much smaller shopping cart. Did they miss something along the way? All the records associated with other customers who placed the same order yet also had several other products in their shopping cart. Amazon does this well with their others who bought this also bought... and their your last products viewed column. Similarly, iTunes has recently launched their Genius service which uploads your playlists, compares them with others and then makes suggestions for what else I might be interested in purchasing because others did. This is genius and it is going on all around us.

Now that we know that all of this exploding data is coming from our ability to instrument everything around us and store it somewhere. Next, the business value and competitive edge will come from a companys ability to access and manage all of this data and to recognize the patterns which are valuable to our customers. And let us not forget the user experience...

Posted by John O'Brien at 11:00 AM | Comments (7)

December 15, 2008


I tend to organize things according to where they fit into the big picture. What I find fascinating is exploring the real world drivers behind actions and evaluating the linkage of a given set of decisions. In many ways, this is similar to Zen practices which encourage reflection and understanding of the world in which we live. It is with this principle of reflection and analysis that I approach data turning it into information and subsequently knowledge. It is also the way I think about business intelligence and emerging technologies. I believe that systems, like people, are part of something larger, a dynamic system comprised of many parts. Only by studying all of the parts can you understand the system itself.

So how did I get here? On my resume you will read that I received my bachelors degree in mechanical engineering from California State University and specialized in automated control systems. Later, I received my Executive MBA from University of Colorado, where I currently live. I have worked for Bank of America, Southern California Edison, Sprint PCS and Level 3 Communications before becoming an independent consulting and later co-founding Dataupia in 2005 taking on the role as Chief Technology Officer. I also regularly present, write and contribute to The Data Warehousing Institute conferences around the world.

As the years have gone by I have come to realize that my career of working in business intelligence is much more than just a job. It is a lifelong quest to understand how businesses, organizations and individuals make sense of all the data they are presented. And that still holds true today. The only difference is that we have exponentially more data to deal with than we did two decades ago and a business environment that has become far more hyper competitive.

I hope to use this forum as a way of providing thought-provoking commentary on how data affects our working and personal lives. And I hope you will take the journey with me, sharing ideas, interesting articles and real life stories from within your organizations as we desperately seek a Zen balance in our work places, in our data and in our lives.

Posted by John O'Brien at 12:15 PM | Comments (0)