BeyeBLOGS | BeyeBLOGS Home | Get Your Own Blog

February 17, 2011

Managers : New Data Migration? Some advice.

I wanted to communicate some high level advice targeted at managers starting up new projects for Data Migration. I have leveraged an online presentation service to communicate this message.

Link :



Wade Walker

View Wade Walker's profile on LinkedIn

Share: Digg Furl ma.gnolia Netscape Newsvine reddit StumbleUpon Yahoo MyWeb  

Posted by Wade Walker at 10:30 AM | Comments (3)

October 9, 2009

Balance of Power : Consultant <--> Client

As a consultant, and now as the owner of a consulting company I have worked at many client sites over my years. One thing that I often see, which is disturbing to say the least, is the balance of power in the relationship between clients and the consultants they hire.

What do I mean? This is a bit of a sensitive subject, so I will proceed with caution...

Well, just the nature of the reason to hire a consultant - a job needs to be done, and there is likely not in-house expertise (or more dangerously, knowledge) to do the job. Thus, an external resource must be resourced.

There's the crux of the problem. In the absence of in-house expertise and knowledge, there is an opportunity created for a consultant to take advantage of their client.

For example, I was recently engaged by a well known multinational to perform some data migration. The project was ongoing and they had had some very serious interpersonal problems within the team of externals, so they cleaned house and brought me in. Over the months, I was told the detailed story. There was one consultant in particular, whose work I constantly had to correct or rebuild due to errors and performance issues. No mentality of quality focussed work or adherence to standards. And a lot of mistakes.

The client was amazed at how fast I was able to deliver my work. Where, for an equivalent piece of work, I was able to deliver (a stable, high-quality standards-based product) in a day, the other consultant would take two weeks! Further, at one point, he apparently told his other external colleagues "Don't work so fast, you guys"! In his mind, the longer it took to deliver something, the more he could bill!

This guy was clearly an extreme case, but there is a lesson to be learned here. It is normal on the market that a client should bring in an external resource to deliver work that they cannot perhaps do internally. However, a small investment in preparation will not only protect you from this kind of exploitation, but also position your development environment and culture for the future.

Here are my recommendations:

- Define Standards of Development (eg. Database, Software, ETL, etc.)
- Define Naming Conventions (DB Tables, Applets, ETL jobs, objects, etc)
- Define or adopt a methodology for development.
- Define acceptance processes that enforce adherence to these
- Once work has been rejected once or twice, developers will change their mindset
and the development culture will evolve towards delivering at a higher standard
of quality and instances of rejected work should diminish.
- Establish Code Libraries; Templates; Shared objects
- Benchmark expected development time for a "standard component"

This last point serves as the tool for protection against exploitation such as I described above. My client now has the templates that I delivered, and they know the expected development time for a given "standard" piece of work.

This helps the PM with estimating the level of effort for delivery, and also gives them a tool to protect themselves against unscrupulous consultants. They are out there.

Wade C. Walker View Wade Walker's profile on LinkedIn

Methodata Sàrl

(0)78 708.36.34

Share: Digg Furl ma.gnolia Netscape Newsvine reddit StumbleUpon Yahoo MyWeb  

Posted by Wade Walker at 1:46 AM | Comments (0)

May 29, 2008

Did I get data quality in my data migration??

What IS data quality ?

Your project is complete. You have carefully spent your budget and countless man-days on moving all that data from across your organization into your new database. You had a team of Data Migration specialists cleaning that data as it went across your network to its new home. You can relax now, right?

Well, maybe. In some ways, your organizational data repositories (and that can mean databases, documents, spreadsheets, flat files, web pages, external data providers and so on) are like Switzerland itself - a miraculous coming together over many years of a very diverse population of different perspectives, beliefs, and languages that create a harmonious whole. This population lives under a common identity, but they have retained their traditions, foods, languages and perspectives. That is what diversity is all about, and is one of the things that makes Switzerland, and perhaps your company, such a strong, unified whole. However, with diversity, in this case, data diversity, comes complexity

So what's my point? Good question. I find that during large (or small) migration projects, it is an assumption that Data Quality is part of what is delivered. However, in an environment such as the above - which is all but inevitable in your organization over time - your data in all those different repositories contains redundancies and...let's just that is lacking in the accuracy that one should expect.

In general, no matter how good or well-intentioned your hard-working Data Migration specialists are, they can only do what they can with the tools they have. In a 'vanilla' data migration project, they probably don't have the bandwidth (i.e. time or budget)...or possibly even the technology to really DO data quality within a reasonable time period. 'Vanilla' Data Migration itself IS by its very nature, extremely complex...and the out-of-the box ETL (or ELT, EAI ... Exx) tools are not generally equipped to do geocoding, intelligent duplicate elimination and other hardcore data quality processes without substantial innovative customization.

I am a data migration architect as one of my specialized disciplines, and I focus on process quality as my first deliverable. However, in all of the projects I have worked on, there has just not been a budget for true data quality transformation. It is likely that in most cases, the clients believe that data quality is implicitly delivered in the project. Unfortunately, despite our best efforts, apart from the standard data quality processes we data migration developers apply, this is not the case in most projects. Data Quality is often a project in itself.

The (simplistic) CUSTOMER example

Certain data quality issues can be resolved using vanilla tools – some simple redundancies caused by slight variations in a customer's name, for example:

Wade Walker
wade walker

can be resolved. However, add in some typos, phonetic spelling, linguistic phonetic differences, abbreviations things become a bit more complex:

Waid Walker
Wadi Walker
W. Walker
Wade Wlker
Weed Wacker (I always hated that one!) get the idea.

...and yes - I have seen all of these bastardizations of my name - and more - appearing on envelopes addressed to me over the years...

Standard data migration tools don't normally have the built-in processes, or intelligence to deal with such variations without extensive customization. The address information for a given customer can work wonders for recognizing duplicates…but of course, there are variations, obsolete data, typos, etc which also further complicate things. But at least, we can say that we probably have a substantial improvement!

This is why the leading migration tools have formed partnerships with data quality vendors - these actually become plug-ins to the vanilla migration product. Now we are talking about the ability to "clean" / "scrub" our data. Some of the vendors offer separate complimentary products that audit data to determine content and assess data quality. Unfortunately, more often than not, these are either eliminated due to cost constraints, or the client hasn't been offered the option!

Think I've said enough on the subject for now. I would be interested in your comments.

Wade Walker

View Wade Walker's profile on LinkedIn

Share: Digg Furl ma.gnolia Netscape Newsvine reddit StumbleUpon Yahoo MyWeb  

Posted by Wade Walker at 9:34 AM | Comments (2)