DataSense http://www.beyeblogs.com/DataSense/ I am often asked usually by programmers - What is Data Warehousing & how do I learn it? I explain to them we use all the same tools that you do but differently. That’s when I coined the term Data Sense. It describes the essence of Data Warehousing and separates Data Warehousing from rest of Programming. Every aspect of IT from Hardware / Software infrastructure to Design, Development and QA is done with massive data flows and need for data precession, accuracy and meaning. en Copyright 2008 Sat, 03 May 2008 19:30:00 -0700 http://www.movabletype.org/?v=3.33 http://blogs.law.harvard.edu/tech/rss How many Dimensions should a data warehouse have? Of course that will depend on the subject and requirements. However I refuse to believe that a business is / can be analyzed by 100 or even 25 dimensions. There are not simply that many dimensions.

Then why do I see models with 150 dimensions (e.g. in a big stock exchange). What is the mistake that the data warehouse architect made?

Analyzing the dimensions I find they are not dimensions and lookups fitted into dimensional model. A lookup is a code that has a description attached to it. Basically the origin is the transactional systems where there is a need to abbreviate and minimize the transaction length. I would prefer to convert them entirely from codes to proper names. The codes and descriptions are not dimensions.

This got the stock exchange model to less than 15 dimensions easily manageable and understandable. However it was too late to make the change 2 years and 10 million into the project.

The minute you hear the number of dimensions to be a large number - question your data warehouse architect and get a second opinion.

]]> http://www.beyeblogs.com/DataSense/archive/2008/05/how_many_dimensions_should_a_d.php http://www.beyeblogs.com/DataSense/archive/2008/05/how_many_dimensions_should_a_d.php Sat, 03 May 2008 19:30:00 -0700 Top 5 Reasons Why Data Warehouses Fail Lack of Executive sponsor
A data warehouse touches so many areas deeply, requires work from so many people and opens so many cans of worms in so many miscoded applications that it requires strong executive leadership to workthrough or workaround the turf battles.
Confusion between Data Warehousing tools and Data Warehouse
Although vendors will tell otherwise Data Warehousing tools are neither necessary nor sufficient to implement a data warehouse. They are in very nice to have category, making the technical details of developing a data warehouse easier and making it less likely to fail due to bad programming for an OTHERWISE sucessful effort. Tools don't turn around a badly understood / designed data warehouse by themselves.
Confusion between OLTP Application and Data Warehouse Development
Although the tools used are same - databases, programming languages, Hardware software and network infrastructure - they are used differently. Every aspect of IT from Hardware / Software infrastructure to Design, Development and QA is done with massive data flows and need for data precision, accuracy and meaning.
Work within Corporate IT Procedures
Big Corporations are setup with inflexable IT procedures, built over years of managing IT design, development and Operations within the company. It is very difficult to get them to think through and realize that some of these don't apply to Data Warehousing. Working around their procedures creates additional cost and risk to the project.
Violate KeepItSimpleS***** Principle
Once the project gets off the ground in a company starved off information for years, there is a tendancy to make the project comprehensive, even working in future requirements and source applications still in design phase. A sucessful data warehouse must have a vision and the ability to reject all requirements that don't fall within that vision. Else the project becomes too vague and too big to succeed.

ZEROTH law of Data warehousing:
Hire the right people. They will take you through ...

]]> http://www.beyeblogs.com/DataSense/archive/2008/03/top_5_reasons_why_data_warehou.php http://www.beyeblogs.com/DataSense/archive/2008/03/top_5_reasons_why_data_warehou.php Sun, 23 Mar 2008 17:45:00 -0700 Data Warehousing and Corporate IT Practices Big Corporations have streamlined IT practices to enable them to support applications in cost effective manner. many of these make sense for OLTP applications and are most likely designed so. However they will make BI development difficult.
e.g.:
BI tasks are more data intensive per transaction compared to OLTP tasks. A BI developer will need the ability to create tables on the fly, to take subset of data to analyze trends or even issues with data. An OLTP developer will likely not need this ability. Corporate policies that prevent such ability impede on the ability of BI developer to diagnose and fix problems. There are work arounds - but it is never a simple matter to backup a terabyte of data, move it across network, find a machine to handle it, and work off it. No one has such powerful machines laying around just in case....

]]> http://www.beyeblogs.com/DataSense/archive/2008/03/data_warehousing_and_corporate.php http://www.beyeblogs.com/DataSense/archive/2008/03/data_warehousing_and_corporate.php Fri, 14 Mar 2008 16:30:00 -0700