Even though I have a license for the Ab Initio GDE, I haven't had a chance to use it. Nor has the promised availability of installing Metadata Manager for Informatica 8.

Informatica has been giving us problems again. The new version, even though its been out for a while, has some problems; mostly with the transition from older versions to new. But that has precluded me from installing Metadata Manager yet.

In the meantime, I was able to attend the 3rd Annual Semantics Conference in San Jose last week. It was really helpful. I learned that Ontology and Semantics is useful to me as the next level of abstraction above Metadata Modeling; so I will be able to use it to find the common points between Ab Initio's EME and Informatica's MM and my own metadata model. More will be coming soon on the details of the conference. In the meantime, here is a link.

http://www.semantic-conference.com/default.html

Next week, I will try setting up the Metadata Manager again.

Its hard to type with crossed fingers... ;-)

Oh! And I almost forgot! I have a temporary license for Ab Initio's GDE now!

Ab Initio's Enterprise Meta Environment is so different from Informatica. The EME is set up on a corporate server (we are in a subsidiary); and after only a small amount of tweaking, it can show a complete Data Lineage diagram from our ultimate source system, through Informatica, to Ab Initio, to the ultimate target and into a Business Objects Universe!

(I heard that overuse of exclamation points is the sign of a diseased mind!)

It was very exciting.

Apparently, it not only collects statistics, but does analysis across and within data sets for dependencies and correlation; as well as being able to generate transformation code to use for validation.

Looking through the docs, the only thing I can think of that its missing is trending analysis; and it may just be because I missed it.

-------------
Added Feb. 1, 2007:

We had the demo, see the next entry. I realized that I did't say much about the Profiler in that entry, and it seems more relevant here.

The Ab Initio Data Profiler is better than others I have seen. You can profile an entire dataset, or a sample thereof (with parameters), or you can profile a single run of a graph against that data; allowing you to schedule the heavy load of a full profile, and then keep it up to date piece by piece.

We did run into problems with our initial profile runs on some data, but that is because our server we can run the profiler from is in a different location from where the data is, and we were filling up the pipe between. We were able to throttle the process and keep from hogging the pipe, but it makes things run quite slowly.

What was really impressive was that we got a pre-view of the demo about 3 or 4 weeks prior, and my manager asked about analyzing trends between runs of the incremental profile, and using that trending to determine, in the map, wheter we wanted to load the data to the target. They weren't able to do it at the time, but when they brought the tool in for the demo this week, they had a way to accomplish that task.

I know, it was probably something they were already working on, but it was impressive none the less.

It was a great demo. I have worked with Ab Initio before, and all the great things I remember liking about it were still there, plus a few new features.

After working with both Informatica and Ab Initio, I have to say that I prefer Ab Initio for several reasons.
Ab Initio is easier to work with.
Informatica has the PowerCenter Designer; where you put together the mappings of the data from source to target, and enter the business rules for transformation. But to define a source there is another "tab" that you have to switch to, and the targets are defined on another tab; and the reusable pieces of maps (called mapplets) are on yet another tab. Then the connection between a logical source definition in the tool, and the actual table/file in the computer is in a completely separate tool; called the Workflow Manager. Then, when you run the thing; there is yet another application that you use to monitor the execution. If you want to see the data you are operating upon, you have to use some other tool to get to it (I use TOAD to see the data we have on Oracle, Teradata SQL assistant, and QMF for our DB2 source). If you want to see what each individual component is up to? You are out of luck. *If* you can get the debugger to run, you might be able to track what the components are up to, but beware, of you have too many components on the map, the debugger won't even load. In Informatica, parallelism is left to the physical hardware implementation at the network level. (That is, to get parallelism in Informatica, you require more than one server to run it on).

With Ab Initio, it is all in one place. You drag and drop the components in the window of the Graphical Development Environment (GDE); there are database table components and file components and myriad transformation components. Then, to define the input columns, you can import DDL, or double click on the component and use a text edit mode to enter it. Same thing for the output definitions. Plus, you can also put a URL or the database connection information into the component and actually browse the data you are defining the DDL for. The tool gives you a visual indication when the information in the component is not complete enough for the graph to execute. Then, when you are ready to run it, click on a button and it starts to execute; no window switching. Plus, you can see the record counts as each component processes, so you can see which components are working as expected; or if there are bottlenecks in your process. And thats not even mentioning the debugger, which is a quantum leap beyond the execution data. Another huge advantage of Ab Initio is that parallelism is built in from the graph level. With Ab Initio, there are components to "Partition" and "Departition" a data flow; which allows the programmer to insert parallelism in the process at the graph level. And between graphs or between checkpoints within graphs you can land your data flows to "multifile" data sets on disk.

There is even more, especially when it comes to metadata and the tools related to it, but I'm afraid my posting would stray too far toward a rant at that point.

We will not be going whole hog to Ab Initio, because we have significant sunk cost in Informatica. But we will be using Ab Initio as much as we can (sharing an implementation at the Corporate office); especially for metadata.

Its organized along the lines of the Zachman Architecture Framework, with a few tunings of his own; and goes along building data models; the first one is a data model of a data model (hmm, maybe thats why Aristotle is on the cover).

It sounds esoteric, but I found it invaluable in refining my own metamodel and catching some nuances I had missed.

For a flavor of what the book is like, check out Mr. Hay's articles in TDAN, especially the one about Modeling Baseball Cards.

Sorry its been such a long time since posting; here's the catch-up.

Things are still moving along on the Metadata front; we now have SuperGlue from Informatica; and in between deliverables on other projects I try to get it running.

Its been installed twice; the first time I had it to the point where I had a couple Xconnects defined and running, although not on a scheduled basis. The Xconnects are the canned set of mapping that ship with SuperGlue and import the data from various sources; the most important of which is the Informatica Repository itself.

Unfortunately, not long after, we lost a disk in the array where our databases were kept; and the DBA's decided not to recover the SuperGlue repository.

So I started from scratch.

I am now to the point where its time to start defining the sources and then configure the Xconnects again; but other projects are eating up my time again.

On the Ab Initio front; one of my co-workers went to a seminar recently on Ab Initio; and also spoke to a manager in another part of our company who is receptive to the idea of sharing his instance with us.

We will see which avenue bears the most fruit.

Things are moving along finally.

After an energy infusion from the DebTech conference, and an enforced lull from metadata concerns (enforced that is by other work deadlines rearing up)

DebTech put on a nice conference.

I was able to hear Claudia Imhoff speak on "Sarbanes-Oxley - A Legislative Mandate for Business Intelligence!";
Tom Haughey gave a great tutorial on Advanced Modeling;
Bonnie O'Neil gave us examples and tools for building a Data Dictionary with "whatever is lying around".

Dave Hay waxed philosophical on "Data Model Patterns" and "Advanced Data Model Patterns";
and Steve Hoberman brought us back to concreteness.

Malcolm Chisholm gave a couple of good talks on Repositories (Build or Buy) and "Master Data Management vs. Reference Data Management".

David Schlesinger gave us some great ideas of how to facilitate a sane regulatory compliance and information security strategy with some simple metadata techniques.

I found myself wishing it would last a little longer.

Bonnie O'Neil infected me with her energy and enthusiasm.

Dave Hay took us back to Aristotle's "Posterior Analytics" and modeled semantics and made me want his new book "Data Model Patterns - A Metadata Map" (not to be confused with his 1995 title "Data Model Patterns: Conventions of Thought"). He also humbled me with his dazzling display of Origami expertise.

Thank you all for an edifying and fun experience.

Company A had a data warehouse, and company B had a large Operational Data Store (ODS); Company C has a very large data warehouse that wants data from A and B. I support the collection of data from companies A+B to provide reporting, as well as trying to supply the data requirements of C.

We use Informatica to transport data between DB2, Oracle and Teradata, and its working pretty well for us. Our data volume from the source system (yes, there is only 1) is getting to the point where we are going to have to do something to keep up.

Metadata wise, we are still in the infancy stage. We recognize (well, maybe not *all* of us) that metadata is important, but maybe not how important it is. I have a metadata model that I would like to implement, but without tools its going to be too much work for 1 person.

I guess thats deep enough for an introduction. I will expand on what we are doing, what challenges we are facing, and some of the solutions we are persuing in later posts.