cooper to logan - expressor software's semantic data integration blog: January 2010 Archives - BeyeBLOGS.com

« December 2009 | Main | February 2010 »

January 31, 2010

expressor expands into smb market

I just had a chance to read Rick Sherman's latest blog on important trends in data integration for 2010 and what stuck out to me was trend 5 which asserts that ETL is expanding into the SMB market.

In my view, Rick is completely right that this expansion is taking place. I am witnessing it every day as we market and sell our highly affordable, enterprise-class ETL tool to a growing number of SMBs who are increasingly facing similar data management and integration challenges that you find in G2000 companies.

In addition to expressor's focus on the higher-end of the SMB market, SMBs are also increasingly using free database such as SSIS and open source tools to tackle basic ETL tasks instead of continuing to hand code these types of smaller applications.

Michael Waclawiczek, VP Marketing

Posted by expressor software at 5:45 PM | Comments (0)

January 29, 2010

insight series: creating drawings with expressor illustrator

expressor illustrator is a graphical interface based on Microsoft Visio that application developers use to creating data-flow diagrams, called "drawings" in expressor. expressor illustrator drawings are comprised of expressor processor image, network and channel descriptions, and connectors (called "motors" in expressor) and operators.

expressor illustrator

expressor illustrator -- Microsoft Visio based application used by ETL developers to build expressor drawings

Only individuals assigned to an ETL developer role can open an illustrator project. Developers use illustrator to design a data integration application as one or more drawings. expressor illustrator stores its artifacts in the expressor repositor version control system so individual developers have access to all the illustrator drawings and other artifacts comprising the project.

To create a drawing in illustrator, you highlight the project in which to include the drawing and, if necessary, check the project out of the repositor version control system. When you bring up a blank drawing, you will see a number of shapes representing the processor motors and operators.

expressor shapes are classified in functional groups.

collate: operators for collating, grouping and identifying a specific record in a collection of records

data-in: motors that retrieve data records from external sources and submits these records to the application

data-out: motors that send records from the application to external sources

database: motors that retrieve and submit data records to external relational database management systems

partition: operators that merge, replicate or partition data records

sort: operators that sort and merge data records

transform: operators that manipulate (modify, delete, extend) the content of data records

utility: operators that provide various support functions, such as generating test data

You create a drawing by simply dragging and dropping shapes onto the Visio drawing, then using the Connector Tool to establish the data flow in the drawing. For each shape, the expressor message panel at the bottom of drawing panel lists warnings, errors and tasks that must be addressed before the drawing is valid.

Shape color is a visual clue to the correctness of its configuration. When it is first added to a drawing, a shape is yellow, indicating that its configuration options have not been set. White indicates that a shape has been properly configured.

Colors other than white can indicate a correctly configured shape and provide additional information about how the shape's processing will be performed. For example, turquoise indicates that the processing will use memory to store processing intermediates, red indicates that the processing will discard data, and orange indicates that data is being filtered.

To save a drawing, select File > Save. The drawing is saved into the project as a Microsoft Visio XML Drawing (.vdx extension). You do not need to supply the extension when you name the file.

John Lifter, expressor training and documentation

Posted by expressor software at 11:45 AM | Comments (1)

January 25, 2010

expressor at SQLSaturday Tampa

There is a growing phenomenon within the Microsoft user community with events known as SQLSaturdays. These are all day meetings held on Saturdays (hence the name!) hosted by local SQL Server user groups. expressor just returned from the Tampa SQLSaturday event, where we were a Platinum sponsor.

It was a terrific day with over 400 registrants and nearly 300 attendees. While the attendees were interested in a wide variety of topics, we received very positive feedback, and quite a few sales opportunities with the message that expressor can "turbocharge" SQL Server SSIS applications. Attendees were crowded around our exhibitor station all day learning more about how expressor can complement SSIS implementations that have hit the wall when encountering higher data volumes, complex data, complex transformations or heterogeneous environments. As one of our visitors noted, "It's great to have an upgrade path for our ETL platform that won't cost us six figures."

Thanks to the organizers of this grass roots event. We look forward to participating in our next SQLSaturday event this coming weekend in Waltham, MA.

Frank Fallon, VP of Sales and Business Dev

Posted by expressor software at 11:30 AM | Comments (0)

January 20, 2010

is expressor complementary to SSIS?

Last week we had a briefing with a well-known and highly respected Microsoft SQL Server MVP, whom I won't mention by name as I haven't asked him for permission to do so.

The main goal of our meeting was to discuss our product in detail and to find out if our value proposition will resonate with SQL Server Integration Services (SSIS) customers and users.

Fortunately for us, this particular MVP felt extremely positive about expressor and said that expressor in his view is very much complementary to SSIS. By the way, the MVP entered the conversation thinking that expressor might compete with SSIS, which in his opinion would not go down well with the existing SSIS community, which generally likes SSIS and is satisfied with using it for basic ETL tasks.

So how does expressor complement SSIS? The answer is quite simple. If you or your team encounters one or more of these types of SSIS issues, expressor might very well be the solution for you:

1. You require support for team-based application development, something SSIS doesn't offer today.

2. You run into performance issues that you find hard to overcome even though you spent time optimizing your SSIS packages.

3. You find it too painful to map heterogeneous data sources, e.g. Oracle or DB2, to a SQL Server. Or you even find it hard to map data between two SQL Server databases.

4. You experience that writing and troubleshooting complex scripts in SSIS is just too hard.

These are some of the reasons why expressor can truly be a complementary product for SSIS customers. We designed and developed our product to address these types of usage, performance and complexity issues, and offer these capabilities at an affordable price, which is key to all SSIS customers.

- Michael Waclawiczek, VP marketing

Posted by expressor software at 10:45 AM | Comments (0)

January 14, 2010

expressor posts strong Q4

expressor just announced it closed 2009 with a strong Q4 - which included four new customers, the introduction of its new 'expressor expressway' methodology and aggressive expansion into the small-to-medium business (SMB) market served by Microsoft SQL Server.

expressor signed new licensing transactions in Q4 with Baseline Management, Integrated DNA, STi Prepaid and a major US health benefits company.

In addition to exhibiting at the PASS Summit - covered previously in this blog -- expressor is building awareness with Microsoft SSIS users by sponsoring several SQL Saturday user group training events scheduled in Q1 2010 as well as a webinar to be conducted with SQL Server Magazine in March.

The company's new expressor expressway methodology will become available in Q1 2010 and be free for expressor users.

expressor also added another data integration expert: Hugo Sheng, who joined the company as a director of field engineering and was previously worldwide technical sales operations manager for IBM's InfoSphere group.

Click here to access the full press release.

- Steve Casey, marketing

Posted by expressor software at 12:15 PM | Comments (0)

Insight series: expressor projects

This is the second in a series of mini-tutorials designed to provide insight into expressor's features and capabilities. In this particular blog, I'll explain some of the basics of an expressor project, which is the foundation for building an expressor data integration application.

Each project is composed of:

- image files - describing the data records being processed

- network files - describing where the data for each step of the application is located

- configuration files - describing runtime environment settings, parameters and values

- drawing files - describing data flows and transformations

Projects also include relevant data on role assignments, server names and security credentials and policies.
The responsibilities for developing, testing, and deploying an expressor project are distributed across multiple roles established during project definition. To develop a new project, an architect, working with expressor administrator, creates the necessary networks and channels, images and configuration files and assigns the appropriate analysts, stewards and ETL developers.

Once defined, the project can be checked out by the individuals assigned to the project, who use expressor's administrator, constructor, initiator and illustrator tools to create additional project artifacts - as shown in the following diagram.

expressor's unique approach to project design, creation and management makes it easier to optimize your resources and improve application quality.

John Lifter, senior manager of education, expressor

Posted by expressor software at 12:15 PM | Comments (0)

January 13, 2010

expressor initiator demo

Check out our expressor educational videos including our expressor initiator demo. Additional videos are available at http://www.expressor-software.com/expressor-educational-videos.htm.

Michael Waclawiczek - VP, Marketing

Posted by expressor software at 9:30 AM | Comments (0)

expressor: 2009 top 10 premium content links

This has been a very exciting year for all of us here at expressor and we are looking forward to a great 2010 with lots of exciting things to write and talk about. I will end this year with my top ten premium content links which you�ll hopefully enjoy as well:

top 10 for expressor
turbocharge your ETL with expressor
Bloor Research on expressor 2.0
expressor vs. Talend pricing model, transparency debate
expressor eval edition educational videos
top 10 video series
semantic rationalization blog series
rapid ETL migration to expressor (archived WEBEX webinar)
operational DI in the real world (DM Radio interview with Bob Potter)
expressor complex data online demo

Happy Holidays Michael Waclawiczek, VP Marketing

Posted by expressor software at 9:00 AM | Comments (0)

Insight series: parallel processing

This is the first in a series of mini-tutorials designed to provide additional insight into specific technical features and capabilities of the expressor semantic data integration system. We're starting with an examination of our implementation of parallel processing.

expressor uses three compatible and combinable techniques for parallel processing to provide high scalability while maintaining high performance:

pipeline parallelism

depth parallelism

partition parallelism

Pipeline parallelism

With pipeline parallelism, an arbitrary set of contiguous records is processed sequentially from component to component. Working like a process pipe on a command line, each expressor processor operator in a drawing (data flow diagram) is simultaneously processing different data records in the same record stream.

In the following diagram, while the furthest downstream operator is processing record 1, its upstream neighbor is processing record 2, the next upstream operator is processing record 3, etc. This means that as many records as the application has operators can be processed in parallel.

expressor_pipeline_parallelism_graphic

Depth parallelism

With depth parallelism, multiple input and output processor motors run simultaneously. One motor does not need to finish its processing before another motor can begin. Depth parallelism allows a high-speed CPU, or multiple CPUs or cores, to simultaneously service multiple slower I/O connections.

The following figure illustrates depth parallelism where multiple input and output motors run simultaneously. In the drawing, two input motors feed data records into a transform operator (e.g. a joiner or joiner-multi operator). The last operator splits the records into multiple channels (see partition parallelism), which are delivered to multiple output motors. The operators also use pipeline parallelism and all the motors and operators run simultaneously in separate processes.

expressor_depth_parallelism_graphic_1

An alternative depth parallelism design is to use multiple data processing pathways in a single drawing. Again, all the input and output motors and operators can run simultaneously in separate processes.

expressor_depth_parallelism_graphic_2

Partition parallelism

With partition parallelism (also called data parallelism), multiple I/O channels are set up between sequential data processing operators. As with depth parallelism, a high speed CPU, or multiple CPUs or cores, can simultaneously service multiple slower I/O channels; each tool in the deployed application is serviced by multiple channels and can process multiple data records concurrently.

expressor_partition_parallelism_graphic

There is a significant difference between partition and depth parallelism. Partition parallelism uses multiple channels in a single instance of a motor or operator while depth parallelism uses multiple distinct instances of identical or different motors. In the above drawing, each operator and motor is running in a single, separate process with four data processing threads. In the depth parallelism diagrams (see previous section), multiple instances of the input and output motors are running in multiple processes with, in this example, only a single data processing thread.

This concludes my brief description of expressor's parallel processing techniques.

Bill Kehoe, lead engineer, expressor

Posted by expressor software at 8:30 AM | Comments (0)

semantic rationalization blog series: part 5 - reusability and ease of use

In my last blog, I discussed the benefits of our abstraction layer. In this final blog in the series, I'll wrap things up with a discussion of reusability and ease of use.

As you might imagine, our new paradigm for viewing data involves a number of new concepts that bear careful explanation. Issues such as different categories of the same metatype: inches and millimeters are really subclasses of length with a static relationship; Dollars and Euros are subclasses of currency with a dynamic relationship. As we have learned with object-oriented implementations, there are powerful benefits to hierarchical structures, but they can also render the system very complex and do not support our goal of creating an intuitive, easy to use system for a non-technical audience. We can certainly learn from object-oriented approaches how to balance between functionality and ease of use - a non-trivial endeavor but not an impossible one.

Another aspect of reusability and ease of use which our abstraction layer is architected to provide is the ability to create a relationship of a rule with a definition. The intent is for these rules to be applied to data in an automated fashion to align or cleanse it as the data is processed, without relying on the developer to repeatedly specify a set of rules. By removing these rules from the inline processing we can also optionally allow the rules to be managed independently of the data flow specification. In fact, we have demonstrated an approach similar to this with some indirect programming projects (sometimes referred to as meta-programming) completed for a number of our customers and prospects.

As we extend our abstraction layer, our goal is to provide an environment that is significantly easier to work in while at the same time providing a very rich set of functionality which can be used to process the data. We have introduced the concept of rules libraries in a recent release of the system, which allows complex algorithms to be abstracted in a way that they appear to be first-class functions within the environment (as if they were distributed as part of the system). This approach allows organizations to extend the capabilities of the system to provide increasingly complex functionality to less sophisticated users while retaining a consistent, easy to use interface.

Finally, we continually solicit feedback from our customers, partners, prospects, and industry analysts regarding both the data integration requirements of the business and how business users could become more closely involved in the delivery of solutions for their business problems. These discussions provide insights that drive our plans to introduce new functionality and ease of use. We welcome your comments.

expressor is excited about the possibilities for combining the ease of use of this semantic abstraction layer with the revolutionary cost of ownership model we have pioneered. Our approach is also attracting attention from the analyst community - Gartner Group recognized expressor with the 'Cool Vendor' award for our innovative approach to data integration.

I hope you enjoyed this series - let us know what you think -- and I look forward to exploring additional topics here in the future.

- Michael Ruland, field engineering

Posted by expressor software at 8:00 AM | Comments (0)

January 12, 2010

q4 is turning out to be a huge quarter for expressor

In Q3 of this year we began to publicly announce new customer acquisitions. In this tradition, we cited American Tower, Sybron Dental, and Silverlink among those new customers willing to allow us to refer to them by name. Although Q4 hasn't yet closed, we are proud to add three more names to this list: Integrated DNA Technologies, Baseline Management, and STi Prepaid.

We'll be talking a lot more about our new Q4 customers in early 2010. If you want to stay abreast of the rapid progress at which we are expanding our customer base, go to www.expressor-software.com/customers.htm. Be assured that this page will be updated frequently.

Happy Holidays,
Dr. Michael Waclawiczek, VP Marketing

Posted by expressor software at 8:15 AM | Comments (0)

January 1, 2010

semantic rationalization blog series: part 4 - the benefits of abstraction

In my previous blog, I discussed our approach to semantic rationalization, and introduced the functionality necessary to implement our semantic metadata abstraction layer. I’d like to continue this discussion by talking about the benefits of our abstraction layer.

The key advantage is that all integration and cleansing rules created in our system are built on business definitions, not external names. So regardless of how the data is named or stored externally, once a rule is built for a particular piece of business data, it is automatically available to be used with another datum with the same business definition – regardless of what it is called externally.

This is radically different than other ETL implementations, where even if a business glossary is provided, it is generally built on top of the ETL component. This approach is understandable for prior-generation ETL tools, since it would be cost-prohibitive to re-architect those tools to incorporate this capability at even a rudimentary level. Placing this functionality on top of the ETL component allows for reporting and analysis, but does not accommodate automating the reuse process at the developer level, where it can provide the most influence on productivity.

By automating reuse at the developer level, the expressor semantic data integration system can suggest a list of rules to a developer when he or she picks a column with which to work, for example. It also provides the ability for an analyst or developer to see all the rules associated with a logical name regardless of the interface with which they are working.

expressor’s use of metatyping provides an even deeper form of reusability and productivity. Metatypes are the vehicle used to define the way data is used in the business and can provide guidelines for how the data should look and what operations should be supported against it. From a purist viewpoint, this is similar to the concept of associating methods with objects in an object-oriented paradigm.

This is a very exciting approach that has the promise of fundamentally changing how we view integration. At the core of the approach is the ability to convert data from a complex external environment to a harmonized environment within the abstraction layer. The properties of the metatype assigned to the business definition define the internal representation of the data and the operators that read and write the data (which we call “motors”) are responsible for marshaling the data from the external complexities to the internally aligned formats.

The promise of this approach is that we can isolate the complexities of data type conversion from the developer/analyst, allowing him or her to focus more intently on the business problem at hand rather than the details of how the data is persisted externally. As we begin to deemphasize the physical complexities and accentuate the business complexion of the data, we converge on providing an environment in which the technical business users can participate in more readily.

In the next and final blog in this series, I’ll wrap things up with a discussion of reusability and ease of use.