BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

May 30, 2009

Performance Benchmarking 101

I have participated in many performance benchmarks and it still amazes me how sometimes they are thrown together at the last minute just to meet a milestone in the project plan. In BI development we are working with astronomical amounts of data in limited batch windows and in some cases near real time so this step should not be taken lightly. Following are some basic guidelines to ensure a successful performance benchmark.

1) Clear and Concise Benchmark Goals.
Don't use fuzzy requirements like I want it to run really, really fast or to execute as fast as a fat man eats a cinnamon crunch bagel but instead use clearly defined benchmarks such as "Response Time for Function X must complete in 15 milliseconds."

2) Establish a Clean and Controlled Test Environment
This environment should also mirror Production as close as possible in structure, workload and configuration. This sometimes can be a costly proposition so compromises may need to be made but be careful that any compromises made do not invalidate the performance benchmarks performed. One example is to make sure your DBMS' SQL optimizer works identical in both environments.

3) Introduce Changes One at Time and Measure Performance After Each Change
If you don't do this carefully and instead rush to make a batch of changes it could actually take you longer to untangle the mess you have made and revert back to your last good run. Automate your runs and results as much as possible so that you can easily determine at an atomic level what was a productive versus detrimental change.

4) Start Early in the Lifecycle Not the Night Before Implementation
One of the values of agile development is to iterate through development and testing and build upon each release. Introduce performance testing early in the development lifecycle so whenever a new release of code is ready (after functional and possibly integration test), promote it to the performance test environment and compare against earlier benchmarks to ensure it is still within acceptable limits. You can do this many times before moving one line of code to Production. In fact, I like a performance environment set up as soon as possible after the development and integration test environments are set up.


5) Learn From Your Mistakes and Broadcast Them

This is the time when you don't want to make a process repeatable. How many times have you picked up a reusable code snippet or maplet that someone else developed and it hadn't been tested to ensure it would perform adequately. Multiply that across an application and you could have a serious bottleneck. Test any reusable component thoroughly for any performance shortcomings.

6) Plan Performance Benchmarks for Each Layer and Then Across the Whole Application
In a typical BI environment you have multiple layers each with different performance goals, staging versus atomic versus derived, etc. Plan benchmark goals for each layer and manage them carefully. As components within each layer are proven, broaden your tests to inter-layer and then across the entire application.

This is just some high level guidelines that will hopefully ensure a high performance, widely-used BI application.

Posted by Dan Sutherland at 8:00 PM | Comments (0)

January 25, 2009

Exciting Times in 2009

It is an exciting time to be in the Business Intelligence industry. New technologies are being introduced all the time, some areas of the industry are converging and many small, medium and even large companies are being swallowed by even larger behemoths, e.g., SAP/Business Objects, Microsoft/DATAllegro and IBM/Cognos. At the same time newer companies are starting to make inroads into the BI market and are competing with innovative solutions, including companies such as Aster Data Systems and ParAccel. Open source BI is gaining strength and wider acceptance especially in the small and medium business arena. The total market for BI is expected to continue to grow even in the midst of an economic downturn because it is becoming an integral tool to finding and exploiting efficiencies for any firm.

As practitioners we have a chance to learn, influence and grow in an exciting industry. To learn we can read, study, try new things with new tools and expand out of our comfortable niche. We can influence by suggesting new ways to do things, trying a proof of concept on a small scale instead of just writing about it and by actively contributing to the BI community through whatever method works for you; blogging, authoring articles and/or submitting intellectual capital. Growth will be a natural outgrowth from learning and influencing but the key concept is donít sit on the sidelines and let opportunity pass you by. Grab hold of the next challenge and do not be surprised if you like the adrenaline flow. Whether you succeed or fail, you will learn and can apply those lessons the next time.

Good luck with all your challenges through 2009 and may you not only survive but thrive!

Posted by Dan Sutherland at 4:45 PM | Comments (3)