BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

April 14, 2009

Splash at Enterprise Data World

I was fortunate to attend the Enterprise Data World conference in Tampa, Florida. It always gets my energy levels up when I visit with a passionate group of people pioneering the data management world.

During the conference I presented a topic that is near and dear to my heart, Physical Architectures for Data Warehouse Workload Optimization. The presentation brings together concepts found is recent articles that I have had published; Degrees of MPP in Information Management and Building a Best-Fit Data Warehouse in TDWIs BI Journal(requires TDWI registration).

The heart of the presentation reminds data architects to clearly understand user behavior and workload before starting the data design process. Where we used to focus on data modeling techniques to match user workloads, e.g. Dimensional modeling is great for metrics based analytics; in todays world you also need to consider how the underlying physical infrastructure contributes to the user experience. This is not what we traditionally call physical data modeling where we adapt the models for the specifics of a given database vendor. But rather should this data model benefit from a shared-everything or shared-nothing database architecture. To download the presentation please visit

It is always great when you present and have an interactive audience, great questions, discussions and a few a-ha! moments. Here are some of my favorites:
What is big data? Greater than single digit terabytes? Double digit terabytes? How about the point where the old rules for accessing data have to change.
The Cloud is simply a SMP server on demand for as long as you need it and not about scalability or GRID.
Major rule changes are needed as we transition from transactions to working with data sets and also when we go from small to big data.

I can hear them announcing the boarding of my flight home. As I sign off, I am struck with the simplicity but importance of my key take away from the conference; the old rules for data management do not apply when dealing with big data. How do you define Big Data?

Posted by BeyeBLOGS at 12:30 PM | Comments (0)

March 10, 2009

History In the Making

A historic event occurred recently when President Obama named a Chief Information Officer, Vivek Kundra. For many of us, it seems almost impossible that it took until 2009 for the Executive Administration to assign what has become such an integral part of the strategy for a successful business. Is running the government so different from running a business? Well yes, it is in many ways. But when it comes to making smart IT decisions I think there are more similarities than differences.

Meeting the needs of many different user communities requires careful planning, governance, standard platforms and a flexible infrastructure that can support a wide and diverse set of tools.

A few years ago it was almost impossible to imagine a corporation which had data volumes that could come close to those of the federal government. That is no longer true. Today, businesses struggle to build scalable solutions that accommodate exploding data volumes, a broad range of applications and growing user communities.

The convergence of extreme data volumes and near real-time availability has brought data quality challenges to center stage.

It goes without saying that in both government and the private sector security & privacy protection are paramount.

A systems overall cost of ownership is often elusive and difficult to predict, but even an organization who can print its own money has to be concerned with TCO especially since it is the taxpayer's money they are spending.

Mr. Kundra, like most all CIOs, has a lot of challenges ahead and like the rest of us he will need to wrestle through many options and set priorities. If he is smart he will surround himself with seasoned data professionals and build a strong and diverse community.

And as they say timing is everything. I just returned from the most recent TDWI conference in Las Vegas. It was an intense week of teaching classes, meeting with BI & DW professionals and enjoy the infrequent opportunities to catch up with long time industry friends. In times like these, we find ourselves asking more questions to better understand how companies are looking to the future. So during the conference Dataupia asked conference attendees how they would set priorities for the Obama administration. Surprisingly, or hopefully not too surprisingly, data quality tops the charts. If you are interested in reading the complete survey results check them out at:

Posted by BeyeBLOGS at 4:15 PM | Comments (0)

February 2, 2009

Predictions for BI in 2009

It is that time of year again when we get to read industry thought leaders sharing their predictions for BI. As I read and thought about what I would predict for 2009, it became clear to me that I was not really making any predictions; but rather I was urging companies, colleagues, and friends to take clear and certain actions throughout 2009 for their companies and careers.

I will not make a prediction for what is happening with the global economy and its impact on all of us. This is a once in a lifetime magnitude event. I read the headlines and send my thoughts and prayers to all who are affected by this crisis. I also remind as many people as I can to stay positive, work through this and we will get through to better days again. As my friend Maureen Clarry reminded me this morning, negativism begets negativism and keep nudging people in a positive direction.

So my predictions for 2009 are what I believe many proactive, positive attitude, best in class companies are going to be doing this year. Take a look at the list and let me know what resonates with your own action plan for 2009.

1. The need for more information, business intelligence and better decision making will drive higher data demands with fewer resources. This is not about competition, it is about survival.

2. Companies will refocus BI projects from strategic planning towards projects that drive operational efficiencies.

3. Companies will consolidate the number of BI tools and applications used within their companies to save money and gain economies of scale.

4. Adoption of data warehouse appliances will be accelerated and will move from supporting a single application to supporting multiple applications and more users within the business.

5. Green IT will be also become a standard which BI programs must adopt as companies become strict about minimizing operating costs.

Since this blogging is more recent an activity for me, we will have to pull up this blog next year and see how we did. In the meantime, keep nudging everyone around you, work smart and we will get through this.

Posted by BeyeBLOGS at 9:45 AM | Comments (4)

December 31, 2008

Discovering Big Data

We keep hearing industry analysts and companies making claims regarding data volumes doubling every year and exploding data and I have always turned around and questioned from where? If a company is growing well, they could expect 30-50% more order data volumes, perhaps 20-30% more customer care tickets. I do not think that financial systems data would grow too significantly. So where does all of this exploding data come that we have to worry so much about?

What I described above is a very basic brick and mortar form of running a business where operational data came from business applications and functions. In todays world, several other factors provide new data which is necessary to be a successful competitive business.

Recently, I was fortunate to meet and spend some time with Andrew Cardno, Chief Technology Officer at BIS2, Business Intelligence Systems Solutions. Andrew is an extremely bright, energetic and charismatic person who is driving one clear message in his company; that data volumes are becoming so large that the only way to interact with data in the future will be through data visualization techniques. Later that evening, he made an interesting comment that really stuck with me. He said that less than 0.1% of all data captured is entered by humans , the rest comes from instrumentation and that we have generated more data since 2005 then we have in the past 40,000 years!

The word that stuck with me was instrumentation. Maybe this was because my engineering degree specialized in instrumentation and controls or maybe because that is where I spent most of my time in building data warehouses and decision support systems. But it did rattle in my brain and got me looking around; computers, scanners, digital pictures, text messages, etc are all silently being captured and stored (or not) somewhere. Surfing the internet was not really surfing, it was hiking and exploring and we are leaving digital footprints every where.

Businesses today should not only count the record that represents the order that you place with them. They should be more interested in all the many steps and processes you followed before and after placing that order. And all of the orders that should have been placed and were not! All the records that represent my digital footprints all over the company website and how many other people have taken similar hiking paths yet finished with a much smaller shopping cart. Did they miss something along the way? All the records associated with other customers who placed the same order yet also had several other products in their shopping cart. Amazon does this well with their others who bought this also bought... and their your last products viewed column. Similarly, iTunes has recently launched their Genius service which uploads your playlists, compares them with others and then makes suggestions for what else I might be interested in purchasing because others did. This is genius and it is going on all around us.

Now that we know that all of this exploding data is coming from our ability to instrument everything around us and store it somewhere. Next, the business value and competitive edge will come from a companys ability to access and manage all of this data and to recognize the patterns which are valuable to our customers. And let us not forget the user experience...

Posted by John O'Brien at 11:00 AM | Comments (7)

December 15, 2008


I tend to organize things according to where they fit into the big picture. What I find fascinating is exploring the real world drivers behind actions and evaluating the linkage of a given set of decisions. In many ways, this is similar to Zen practices which encourage reflection and understanding of the world in which we live. It is with this principle of reflection and analysis that I approach data turning it into information and subsequently knowledge. It is also the way I think about business intelligence and emerging technologies. I believe that systems, like people, are part of something larger, a dynamic system comprised of many parts. Only by studying all of the parts can you understand the system itself.

So how did I get here? On my resume you will read that I received my bachelors degree in mechanical engineering from California State University and specialized in automated control systems. Later, I received my Executive MBA from University of Colorado, where I currently live. I have worked for Bank of America, Southern California Edison, Sprint PCS and Level 3 Communications before becoming an independent consulting and later co-founding Dataupia in 2005 taking on the role as Chief Technology Officer. I also regularly present, write and contribute to The Data Warehousing Institute conferences around the world.

As the years have gone by I have come to realize that my career of working in business intelligence is much more than just a job. It is a lifelong quest to understand how businesses, organizations and individuals make sense of all the data they are presented. And that still holds true today. The only difference is that we have exponentially more data to deal with than we did two decades ago and a business environment that has become far more hyper competitive.

I hope to use this forum as a way of providing thought-provoking commentary on how data affects our working and personal lives. And I hope you will take the journey with me, sharing ideas, interesting articles and real life stories from within your organizations as we desperately seek a Zen balance in our work places, in our data and in our lives.

Posted by John O'Brien at 12:15 PM | Comments (0)