1. Implement and enforce strong data governance across your organization so that the definitions, standards, ownership, location and history (audit trail) of key enterprise data is understood.


2. Implement a data quality program as an integral part of your data governance. This is not a once-off initiative.


3. Select data integration tools that fully support the governance organization and provide tracking, audit trail, alerts and other similar key performance parameters to allow effective management of the process.


4. Start small and grow. Don't be tempted to undertake a data integration or migration project using a 'big bang' approach.


5. Choose a data integration tool that is designed for purpose and avoid tools that are composed of multiple (bolted together) and not necessarily well-integrated components or modules.


6. Select tools that support a strong business-oriented focus for data integration. Many of today's tools are complex, sophisticated and difficult to learn, requiring specialists.


7. Select a tool that will offer the scalability needed to encompass your future business needs.


8. Seek vendors that can offer role-based implementation where (as described above) tasks currently undertaken by developers can be shared more effectively among your current experienced business staff.


9. Steer clear of products that require you to hard code or embed business logic into the internals of the tool.


10. Remember that total cost of ownership is what really matters. Not just the negotiation of license costs, but all cost aspects both now and in the future of your integration strategy.

http://www.expressor-software.com/deep-dive.htm

Register now!

This training event was very well organized and there were at least 300 attendees at the venue during the day. We had a great reception at our exhibitor booth and were able to interact with over 100 attendees at our booth.

Many of the attendees we talked to are already using or planning to use SSIS, which confirms that the SQL Server community is increasingly embracing ETL technology. SSIS folks generally like the product but are very much aware of the limitations it has. We handed out a survey at this event were we asked booth visitors to answer a few questions aimed at finding out common pain points SSIS users have. Here is what they had to say:

• migrating SSIS packages from development, through testing, into production is very painful


• achieving good SSIS performance can be a difficult task and often requires serious performance tuning


• it is difficult to handle complex transformations such as multi-step business rules, complex multi-joins, multi-level aggregations, etc


• it is difficult to integrate non-Microsoft data sources (e.g. Oracle)

These survey results are very much in line with what we are hearing from our SSIS customers and sales prospects.

During lunch time, Michael Ruland, our director of field engineering, gave another great 45 minute technical overview presentation, which was well attended and received by the audience. Michael's presentation can be viewed by clicking here.

In summary, this was another great SQLSaturday event for us - and it is good to be amidst a vibrant SQL Server community that is very upbeat about the SQL Server product. As it regards expressor, we have a great opportunity to complement SSIS in SQL Server ETL projects where customers need enterprise-class performance, lifecycle management, and heterogeneous data integration capabilities.

On to the next SQLSaturday event in mid March.

Michael Waclawiczek, VP Marketing

In my view, Rick is completely right that this expansion is taking place. I am witnessing it every day as we market and sell our highly affordable, enterprise-class ETL tool to a growing number of SMBs who are increasingly facing similar data management and integration challenges that you find in G2000 companies.

In addition to expressor's focus on the higher-end of the SMB market, SMBs are also increasingly using free database such as SSIS and open source tools to tackle basic ETL tasks instead of continuing to hand code these types of smaller applications.

Michael Waclawiczek, VP Marketing

expressor illustrator -- Microsoft Visio based application used by ETL developers to build expressor drawings

Only individuals assigned to an ETL developer role can open an illustrator project. Developers use illustrator to design a data integration application as one or more drawings. expressor illustrator stores its artifacts in the expressor repositor version control system so individual developers have access to all the illustrator drawings and other artifacts comprising the project.

To create a drawing in illustrator, you highlight the project in which to include the drawing and, if necessary, check the project out of the repositor version control system. When you bring up a blank drawing, you will see a number of shapes representing the processor motors and operators.

expressor shapes are classified in functional groups.

• collate: operators for collating, grouping and identifying a specific record in a collection of records


• data-in: motors that retrieve data records from external sources and submits these records to the application


• data-out: motors that send records from the application to external sources


• database: motors that retrieve and submit data records to external relational database management systems


• partition: operators that merge, replicate or partition data records


• sort: operators that sort and merge data records


• transform: operators that manipulate (modify, delete, extend) the content of data records


• utility: operators that provide various support functions, such as generating test data

You create a drawing by simply dragging and dropping shapes onto the Visio drawing, then using the Connector Tool to establish the data flow in the drawing. For each shape, the expressor message panel at the bottom of drawing panel lists warnings, errors and tasks that must be addressed before the drawing is valid.

Shape color is a visual clue to the correctness of its configuration. When it is first added to a drawing, a shape is yellow, indicating that its configuration options have not been set. White indicates that a shape has been properly configured.

Colors other than white can indicate a correctly configured shape and provide additional information about how the shape's processing will be performed. For example, turquoise indicates that the processing will use memory to store processing intermediates, red indicates that the processing will discard data, and orange indicates that data is being filtered.

To save a drawing, select File > Save. The drawing is saved into the project as a Microsoft Visio XML Drawing (.vdx extension). You do not need to supply the extension when you name the file.

John Lifter, expressor training and documentation

It was a terrific day with over 400 registrants and nearly 300 attendees. While the attendees were interested in a wide variety of topics, we received very positive feedback, and quite a few sales opportunities with the message that expressor can "turbocharge" SQL Server SSIS applications. Attendees were crowded around our exhibitor station all day learning more about how expressor can complement SSIS implementations that have hit the wall when encountering higher data volumes, complex data, complex transformations or heterogeneous environments. As one of our visitors noted, "It's great to have an upgrade path for our ETL platform that won't cost us six figures."

Thanks to the organizers of this grass roots event. We look forward to participating in our next SQLSaturday event this coming weekend in Waltham, MA.

Frank Fallon, VP of Sales and Business Dev

Last week we had a briefing with a well-known and highly respected Microsoft SQL Server MVP, whom I won't mention by name as I haven't asked him for permission to do so.

The main goal of our meeting was to discuss our product in detail and to find out if our value proposition will resonate with SQL Server Integration Services (SSIS) customers and users.

Fortunately for us, this particular MVP felt extremely positive about expressor and said that expressor in his view is very much complementary to SSIS. By the way, the MVP entered the conversation thinking that expressor might compete with SSIS, which in his opinion would not go down well with the existing SSIS community, which generally likes SSIS and is satisfied with using it for basic ETL tasks.

So how does expressor complement SSIS? The answer is quite simple. If you or your team encounters one or more of these types of SSIS issues, expressor might very well be the solution for you:

1. You require support for team-based application development, something SSIS doesn't offer today.

2. You run into performance issues that you find hard to overcome even though you spent time optimizing your SSIS packages.

3. You find it too painful to map heterogeneous data sources, e.g. Oracle or DB2, to a SQL Server. Or you even find it hard to map data between two SQL Server databases.

4. You experience that writing and troubleshooting complex scripts in SSIS is just too hard.

These are some of the reasons why expressor can truly be a complementary product for SSIS customers. We designed and developed our product to address these types of usage, performance and complexity issues, and offer these capabilities at an affordable price, which is key to all SSIS customers.

- Michael Waclawiczek, VP marketing

expressor just announced it closed 2009 with a strong Q4 - which included four new customers, the introduction of its new 'expressor expressway' methodology and aggressive expansion into the small-to-medium business (SMB) market served by Microsoft SQL Server.

expressor signed new licensing transactions in Q4 with Baseline Management, Integrated DNA, STi Prepaid and a major US health benefits company.

In addition to exhibiting at the PASS Summit - covered previously in this blog -- expressor is building awareness with Microsoft SSIS users by sponsoring several SQL Saturday user group training events scheduled in Q1 2010 as well as a webinar to be conducted with SQL Server Magazine in March.

The company's new expressor expressway methodology will become available in Q1 2010 and be free for expressor users.

expressor also added another data integration expert: Hugo Sheng, who joined the company as a director of field engineering and was previously worldwide technical sales operations manager for IBM's InfoSphere group.

Click here to access the full press release.

- Steve Casey, marketing

This is the second in a series of mini-tutorials designed to provide insight into expressor's features and capabilities. In this particular blog, I'll explain some of the basics of an expressor project, which is the foundation for building an expressor data integration application.

Each project is composed of:

- image files - describing the data records being processed

- network files - describing where the data for each step of the application is located

- configuration files - describing runtime environment settings, parameters and values

- drawing files - describing data flows and transformations

Projects also include relevant data on role assignments, server names and security credentials and policies.
The responsibilities for developing, testing, and deploying an expressor project are distributed across multiple roles established during project definition. To develop a new project, an architect, working with expressor administrator, creates the necessary networks and channels, images and configuration files and assigns the appropriate analysts, stewards and ETL developers.

Once defined, the project can be checked out by the individuals assigned to the project, who use expressor's administrator, constructor, initiator and illustrator tools to create additional project artifacts - as shown in the following diagram.

expressor's unique approach to project design, creation and management makes it easier to optimize your resources and improve application quality.

John Lifter, senior manager of education, expressor

Check out our expressor educational videos including our expressor initiator demo. Additional videos are available at http://www.expressor-software.com/expressor-educational-videos.htm.

Michael Waclawiczek - VP, Marketing]]> http://www.beyeblogs.com/expressor/archive/2010/01/expressor_initiator_demo.php http://www.beyeblogs.com/expressor/archive/2010/01/expressor_initiator_demo.php Wed, 13 Jan 2010 09:30:00 -0700

This has been a very exciting year for all of us here at expressor and we are looking forward to a great 2010 with lots of exciting things to write and talk about. I will end this year with my top ten premium content links which you�ll hopefully enjoy as well:

1. top 10 for expressor
2. turbocharge your ETL with expressor
3. Bloor Research on expressor 2.0
4. expressor vs. Talend pricing model, transparency debate
5. expressor eval edition educational videos
6. top 10 video series
7. semantic rationalization blog series
8. rapid ETL migration to expressor (archived WEBEX webinar)
9. operational DI in the real world (DM Radio interview with Bob Potter)
10. expressor complex data online demo

Happy Holidays Michael Waclawiczek, VP Marketing]]> http://www.beyeblogs.com/expressor/archive/2010/01/expressor_2009_top_10_premium.php http://www.beyeblogs.com/expressor/archive/2010/01/expressor_2009_top_10_premium.php Wed, 13 Jan 2010 09:00:00 -0700
This is the first in a series of mini-tutorials designed to provide additional insight into specific technical features and capabilities of the expressor semantic data integration system. We're starting with an examination of our implementation of parallel processing.

expressor uses three compatible and combinable techniques for parallel processing to provide high scalability while maintaining high performance:

• pipeline parallelism


• depth parallelism


• partition parallelism

With pipeline parallelism, an arbitrary set of contiguous records is processed sequentially from component to component. Working like a process pipe on a command line, each expressor processor operator in a drawing (data flow diagram) is simultaneously processing different data records in the same record stream.

In the following diagram, while the furthest downstream operator is processing record 1, its upstream neighbor is processing record 2, the next upstream operator is processing record 3, etc. This means that as many records as the application has operators can be processed in parallel.

Depth parallelism

With depth parallelism, multiple input and output processor motors run simultaneously. One motor does not need to finish its processing before another motor can begin. Depth parallelism allows a high-speed CPU, or multiple CPUs or cores, to simultaneously service multiple slower I/O connections.

The following figure illustrates depth parallelism where multiple input and output motors run simultaneously. In the drawing, two input motors feed data records into a transform operator (e.g. a joiner or joiner-multi operator). The last operator splits the records into multiple channels (see partition parallelism), which are delivered to multiple output motors. The operators also use pipeline parallelism and all the motors and operators run simultaneously in separate processes.

An alternative depth parallelism design is to use multiple data processing pathways in a single drawing. Again, all the input and output motors and operators can run simultaneously in separate processes.

Partition parallelism

With partition parallelism (also called data parallelism), multiple I/O channels are set up between sequential data processing operators. As with depth parallelism, a high speed CPU, or multiple CPUs or cores, can simultaneously service multiple slower I/O channels; each tool in the deployed application is serviced by multiple channels and can process multiple data records concurrently.

There is a significant difference between partition and depth parallelism. Partition parallelism uses multiple channels in a single instance of a motor or operator while depth parallelism uses multiple distinct instances of identical or different motors. In the above drawing, each operator and motor is running in a single, separate process with four data processing threads. In the depth parallelism diagrams (see previous section), multiple instances of the input and output motors are running in multiple processes with, in this example, only a single data processing thread.

This concludes my brief description of expressor's parallel processing techniques.

Bill Kehoe, lead engineer, expressor

In my last blog, I discussed the benefits of our abstraction layer. In this final blog in the series, I'll wrap things up with a discussion of reusability and ease of use.

As you might imagine, our new paradigm for viewing data involves a number of new concepts that bear careful explanation. Issues such as different categories of the same metatype: inches and millimeters are really subclasses of length with a static relationship; Dollars and Euros are subclasses of currency with a dynamic relationship. As we have learned with object-oriented implementations, there are powerful benefits to hierarchical structures, but they can also render the system very complex and do not support our goal of creating an intuitive, easy to use system for a non-technical audience. We can certainly learn from object-oriented approaches how to balance between functionality and ease of use - a non-trivial endeavor but not an impossible one.

Another aspect of reusability and ease of use which our abstraction layer is architected to provide is the ability to create a relationship of a rule with a definition. The intent is for these rules to be applied to data in an automated fashion to align or cleanse it as the data is processed, without relying on the developer to repeatedly specify a set of rules. By removing these rules from the inline processing we can also optionally allow the rules to be managed independently of the data flow specification. In fact, we have demonstrated an approach similar to this with some indirect programming projects (sometimes referred to as meta-programming) completed for a number of our customers and prospects.

As we extend our abstraction layer, our goal is to provide an environment that is significantly easier to work in while at the same time providing a very rich set of functionality which can be used to process the data. We have introduced the concept of rules libraries in a recent release of the system, which allows complex algorithms to be abstracted in a way that they appear to be first-class functions within the environment (as if they were distributed as part of the system). This approach allows organizations to extend the capabilities of the system to provide increasingly complex functionality to less sophisticated users while retaining a consistent, easy to use interface.

Finally, we continually solicit feedback from our customers, partners, prospects, and industry analysts regarding both the data integration requirements of the business and how business users could become more closely involved in the delivery of solutions for their business problems. These discussions provide insights that drive our plans to introduce new functionality and ease of use. We welcome your comments.

expressor is excited about the possibilities for combining the ease of use of this semantic abstraction layer with the revolutionary cost of ownership model we have pioneered. Our approach is also attracting attention from the analyst community - Gartner Group recognized expressor with the 'Cool Vendor' award for our innovative approach to data integration.

I hope you enjoyed this series - let us know what you think -- and I look forward to exploring additional topics here in the future.

- Michael Ruland, field engineering

We'll be talking a lot more about our new Q4 customers in early 2010. If you want to stay abreast of the rapid progress at which we are expanding our customer base, go to www.expressor-software.com/customers.htm. Be assured that this page will be updated frequently.

Happy Holidays,
Dr. Michael Waclawiczek, VP Marketing