BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

« January 2009 | Main

May 30, 2009

Performance Benchmarking 101

I have participated in many performance benchmarks and it still amazes me how sometimes they are thrown together at the last minute just to meet a milestone in the project plan. In BI development we are working with astronomical amounts of data in limited batch windows and in some cases near real time so this step should not be taken lightly. Following are some basic guidelines to ensure a successful performance benchmark.

1) Clear and Concise Benchmark Goals.
Don't use fuzzy requirements like I want it to run really, really fast or to execute as fast as a fat man eats a cinnamon crunch bagel but instead use clearly defined benchmarks such as "Response Time for Function X must complete in 15 milliseconds."

2) Establish a Clean and Controlled Test Environment
This environment should also mirror Production as close as possible in structure, workload and configuration. This sometimes can be a costly proposition so compromises may need to be made but be careful that any compromises made do not invalidate the performance benchmarks performed. One example is to make sure your DBMS' SQL optimizer works identical in both environments.

3) Introduce Changes One at Time and Measure Performance After Each Change
If you don't do this carefully and instead rush to make a batch of changes it could actually take you longer to untangle the mess you have made and revert back to your last good run. Automate your runs and results as much as possible so that you can easily determine at an atomic level what was a productive versus detrimental change.

4) Start Early in the Lifecycle Not the Night Before Implementation
One of the values of agile development is to iterate through development and testing and build upon each release. Introduce performance testing early in the development lifecycle so whenever a new release of code is ready (after functional and possibly integration test), promote it to the performance test environment and compare against earlier benchmarks to ensure it is still within acceptable limits. You can do this many times before moving one line of code to Production. In fact, I like a performance environment set up as soon as possible after the development and integration test environments are set up.

5) Learn From Your Mistakes and Broadcast Them

This is the time when you don't want to make a process repeatable. How many times have you picked up a reusable code snippet or maplet that someone else developed and it hadn't been tested to ensure it would perform adequately. Multiply that across an application and you could have a serious bottleneck. Test any reusable component thoroughly for any performance shortcomings.

6) Plan Performance Benchmarks for Each Layer and Then Across the Whole Application
In a typical BI environment you have multiple layers each with different performance goals, staging versus atomic versus derived, etc. Plan benchmark goals for each layer and manage them carefully. As components within each layer are proven, broaden your tests to inter-layer and then across the entire application.

This is just some high level guidelines that will hopefully ensure a high performance, widely-used BI application.

Posted by Dan Sutherland at 8:00 PM | Comments (0)