BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

January 8, 2010

Watch out for Data Integration Silver Bullets

The market for data integration tools has never been either as strong or rich as it is today. 10 years ago today's giants in the marketplace offered the classic Extract, Transform and Load capability, and have been evolving their platforms ever since to provide the current batch of feature rich, high performance packages that now extend to cover data migration, data quality (profile, monitor and cleanse), web services and enterprise metadata management.

A recent Gartner paper (available courtesy of Syncsort here), reveals the market for Data Management and Integration tools is nearly $1.7 billion, and is expected to grow to $2.7 billion by 2013. This growth is driven by an awareness of the high cost of delivering data centric projects by using the manually intensive programming techniques of the 3GL languages, and businesses are increasingly attracted to the productivity gains offered by Data Integration tools. At the same time businesses listen intently to the marketing machines and accept at face value that the data integration tool they end up buying really will be the silver bullet to solve all their data problems.

Wrong, wrong, wrong.

Choosing a data integration technology is just the first step on a journey to improving productivity and responsiveness within a business, making it work over the long term is a little more difficult. Having worked on countless data integration projects over the last 12 years, my biggest source of frustration is when the customer has been set an unrealistic expectation about how easy it is to work with the technology.

Yes, DI tools are certainly an order of magnitude easier than hand cranking code, but the architecture will not take care of itself, and the out of the box settings almost always last little more than a few months before progress falters - it may even halt until things are fixed.

Why does this happen?

Most DI tools find their way into a business following a Proof of Concept project. Proof of concepts are just that, they prove something. And the intent is prove the concept as quickly as possible. They don't usually provide production ready code, and the environment isn't usually set up at this stage to support wide scale usage. It is usually impressive in terms of results but can also be very fast and dirty.

It also helps to understand the business model of the vendor, and therefore their ultimate motivation. Some vendors don't have professional services, and rely on licenses and maintenance as their main income stream. This leads to the possibility that the immediate sale becomes the focus - at the expense of making it last

Credit to the vendor presales who perform the POCs. They are highly skilled and deliver at tremendous speeds. The problem however manifests itself in the fact that they make it all seem very easy.........too easy. Remember, because the vendor does it with consummate ease and the tool is shiny, it doesn't mean that even your brightest technical people can achieve the same feat the day after returning from the training course. Your team will need support to get to the apex of that learning curve as quickly as possible - particularly around implementing a stable and scalable architecture.

So ask your vendor what their business model is. If they're not geared up to do professional services, you probably need to find a partner to help you with the transition of your delivery team from newbies to experts - and the sooner the partner gets involved, the fewer long term problems you're likely to see.

Any integration partner worth his or her salt will start with a discovery phase during which they audit the environment and map the business needs to a technology strategy and plan. Many times I've been here, and in very short time I often find that the software has been poorly configured so it won't scale, in one or more of performance, complexity or enterprise growth. What follows is a difficult conversation with the business sponsor who still can't believe that the wonder tool has failed to deliver.

The moral of the tale is this: when you start evaluating a data integration tool, begin to evaluate an integration partner that you can trust and work closely with. Get them in early, preferably when you're doing the proof of concept with the vendor, so they can make ensure a smooth transition between vendor and internal team. They can help with your evaluation criteria, your architecture and governance and help you avoid pain later in your delivery programme.

It's often said that there's no substitute for experience, and that's never truer than when applied to data integration projects; a few days consulting early in the project lifecycle can save tens or even hundreds of thousands of pounds down the line.

This article was first posted as a guest entry to Rick Barton's BeyeNETWORK blog on September 7th 2009.

Posted by Phil Watt at 10:30 AM | Comments (5)