BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

« Inspiration | Main | Predictions for BI in 2009 »

December 31, 2008

Discovering Big Data

We keep hearing industry analysts and companies making claims regarding data volumes doubling every year and exploding data and I have always turned around and questioned from where? If a company is growing well, they could expect 30-50% more order data volumes, perhaps 20-30% more customer care tickets. I do not think that financial systems data would grow too significantly. So where does all of this exploding data come that we have to worry so much about?

What I described above is a very basic brick and mortar form of running a business where operational data came from business applications and functions. In todays world, several other factors provide new data which is necessary to be a successful competitive business.

Recently, I was fortunate to meet and spend some time with Andrew Cardno, Chief Technology Officer at BIS2, Business Intelligence Systems Solutions. Andrew is an extremely bright, energetic and charismatic person who is driving one clear message in his company; that data volumes are becoming so large that the only way to interact with data in the future will be through data visualization techniques. Later that evening, he made an interesting comment that really stuck with me. He said that less than 0.1% of all data captured is entered by humans , the rest comes from instrumentation and that we have generated more data since 2005 then we have in the past 40,000 years!

The word that stuck with me was instrumentation. Maybe this was because my engineering degree specialized in instrumentation and controls or maybe because that is where I spent most of my time in building data warehouses and decision support systems. But it did rattle in my brain and got me looking around; computers, scanners, digital pictures, text messages, etc are all silently being captured and stored (or not) somewhere. Surfing the internet was not really surfing, it was hiking and exploring and we are leaving digital footprints every where.

Businesses today should not only count the record that represents the order that you place with them. They should be more interested in all the many steps and processes you followed before and after placing that order. And all of the orders that should have been placed and were not! All the records that represent my digital footprints all over the company website and how many other people have taken similar hiking paths yet finished with a much smaller shopping cart. Did they miss something along the way? All the records associated with other customers who placed the same order yet also had several other products in their shopping cart. Amazon does this well with their others who bought this also bought... and their your last products viewed column. Similarly, iTunes has recently launched their Genius service which uploads your playlists, compares them with others and then makes suggestions for what else I might be interested in purchasing because others did. This is genius and it is going on all around us.

Now that we know that all of this exploding data is coming from our ability to instrument everything around us and store it somewhere. Next, the business value and competitive edge will come from a companys ability to access and manage all of this data and to recognize the patterns which are valuable to our customers. And let us not forget the user experience...

Posted by John O'Brien at December 31, 2008 11:00 AM


John, interesting thoughts for sure. I wonder what Andrew would say about the task of visualizing the 99.9% of data that is machine created. The process side is where the value is and anyone that can give insight into this fast paced and dynamic data will provide a value to the end user that is very important. I like the "instrumentation" concept. Glad to have you blogging on the Network.

Posted by: Shawn Rogers at December 31, 2008 1:54 PM

Very good point. With so much instrumented data available (in real-time and extremely micro-sampled) at your fingertips, the true business value will come from the business users who can leverage intuitive BI technologies followed by the users who can configure the predictive analytic systems. The technology is arriving, now it's time for the human! thanks, John

Posted by: John at January 2, 2009 4:47 PM

To see Big Data visualization, take a look at the BIS2 website at where Andrew Cardno is CTO.

Posted by: John at January 2, 2009 4:50 PM

I enjoy your tale, ok, i'll save this blog and also come back here in next couple of days.

Posted by: swarovskiemu at May 16, 2011 9:33 AM

Heya i am for the first time here. I found this board and I find It truly useful & it helped me out a lot. I hope to give something back and help others like you aided me.

Posted by: escort service London at May 16, 2011 1:33 PM

I’ve been exploring for a little for any high-quality articles or blog posts on this sort of area . Exploring in Yahoo I at last stumbled upon this web site. Reading this info So i am happy to convey that I've an incredibly good uncanny feeling I discovered just what I needed. I most certainly will make certain to do not forget this web site and give it a glance on a constant basis.

Posted by: London escort girls at May 17, 2011 12:44 AM

very nice post! thx for that - keep writing

Posted by: at May 17, 2011 12:56 AM

Post a comment

Remember Me?